tahoe-lafs/trac-2024-07-25
2
0
Issues 1.1k Wiki Activity

'DeprecationWarning: the sets module is deprecated' from foolscap causes failure in test_client_no_noise #1329

New issue
Closed
opened 2011-01-16 05:52:24 +00:00 by davidsarah · 8 comments
davidsarah commented 2011-01-16 05:52:24 +00:00
Owner
Copy link

(http://tahoe-lafs.org/buildbot/builders/Brian%20ubuntu-i386%20linode/builds/133/steps/test/logs/stdio)

[FAIL]: allmydata.test.test_runner.RunNode.test_client_no_noise

Traceback (most recent call last):
  File "/home/buildbot/bb-tahoe/brian-linode/build/src/allmydata/test/test_runner.py", line 442, in _cb2
    and "from pkg_resources import load_entry_point" not in line)], errstr)
twisted.trial.unittest.FailTest: cc=0, OUT: '', ERR: '/home/buildbot/bb-tahoe/brian-linode/build/support/lib/python2.6/site-packages/foolscap-0.6.0-py2.6.egg/foolscap/banana.py:2: DeprecationWarning: the sets module is deprecated
  import struct, sets, time
'
(http://tahoe-lafs.org/buildbot/builders/Brian%20ubuntu-i386%20linode/builds/133/steps/test/logs/stdio) ``` [FAIL]: allmydata.test.test_runner.RunNode.test_client_no_noise Traceback (most recent call last): File "/home/buildbot/bb-tahoe/brian-linode/build/src/allmydata/test/test_runner.py", line 442, in _cb2 and "from pkg_resources import load_entry_point" not in line)], errstr) twisted.trial.unittest.FailTest: cc=0, OUT: '', ERR: '/home/buildbot/bb-tahoe/brian-linode/build/support/lib/python2.6/site-packages/foolscap-0.6.0-py2.6.egg/foolscap/banana.py:2: DeprecationWarning: the sets module is deprecated import struct, sets, time ' ```
tahoe-lafs added the
code
major
defect
1.8.1
 labels 2011-01-16 05:52:24 +00:00
tahoe-lafs added this to the soon milestone 2011-01-16 05:52:24 +00:00
david-sarah@jacaranda.org commented 2011-01-16 20:32:12 +00:00
Author
Owner
Copy link

In [4965/ticket1306]:

src/allmydata/_auto_deps.py: Fix 'DeprecationWarning: the sets module is deprecated'. refs #1329
In [4965/ticket1306]: ``` src/allmydata/_auto_deps.py: Fix 'DeprecationWarning: the sets module is deprecated'. refs #1329 ```
tahoe-lafs modified the milestone from soon to 1.8.2 2011-01-16 21:23:57 +00:00
warner commented 2011-01-17 05:41:12 +00:00
Author
Owner
Copy link

I just released foolscap-0.6.1, which removes the offending import sets, so this ticket can be resolved by increasing Tahoe's requirement to foolscap-0.6.1 .

I just released foolscap-0.6.1, which removes the offending `import sets`, so this ticket can be resolved by increasing Tahoe's requirement to foolscap-0.6.1 .
zooko commented 2011-01-17 06:45:25 +00:00
Author
Owner
Copy link

-0 from me -- I would rather suppress the warning the way [20110116200112-93fa1-0397f85493a9fd0ae82ce58d7291d598a4f6e2d6]/ticket1306 attempts to do or change the test to tolerate the noise than raise the dependency requirement. But, it is just a -0. If you prefer it, go ahead.

-0 from me -- I would rather suppress the warning the way [20110116200112-93fa1-0397f85493a9fd0ae82ce58d7291d598a4f6e2d6]/ticket1306 attempts to do or change the test to tolerate the noise than raise the dependency requirement. But, it is just a -0. If you prefer it, go ahead.
warner commented 2011-01-17 06:51:15 +00:00
Author
Owner
Copy link

my main problem with [20110116200112-93fa1-0397f85493a9fd0ae82ce58d7291d598a4f6e2d6]/ticket1306 is that it involved what looked like a signficant refactoring of _auto_deps.py, so I couldn't apply just a single patch (there were more like six). But if there's a clean way to use that approach, I'm happy to go with it.

The clock is ticking, though :)

my main problem with [20110116200112-93fa1-0397f85493a9fd0ae82ce58d7291d598a4f6e2d6]/ticket1306 is that it involved what looked like a signficant refactoring of `_auto_deps.py`, so I couldn't apply just a single patch (there were more like six). But if there's a clean way to use that approach, I'm happy to go with it. The clock is ticking, though :)
zooko commented 2011-01-18 07:25:52 +00:00
Author
Owner
Copy link

Replying to warner:

my main problem with [20110116200112-93fa1-0397f85493a9fd0ae82ce58d7291d598a4f6e2d6]/ticket1306 is that it involved what looked like a signficant refactoring of _auto_deps.py, so I couldn't apply just a single patch (there were more like six). But if there's a clean way to use that approach, I'm happy to go with it.

For the record, here are the patches that (from darcs's perspective) are required for [20110116200112-93fa1-0397f85493a9fd0ae82ce58d7291d598a4f6e2d6]/ticket1306:

HACL Zooko-Ofsimplegeos-MacBook-Pro:~/playground/tahoe-lafs/ticket1329$ time darcs push zooko@dev.allmydata.org:/home/darcs/tahoe-lafs/trunk
Pushing to "zooko@dev.allmydata.org:/home/darcs/tahoe-lafs/trunk"...
The remote repository has 2 patches to pull.
Sat Jan  1 04:01:41 MST 2011  david-sarah@jacaranda.org
  * allmydata/__init__.py: don't use pkg_resources.require to get versions and paths. Also change pyOpenSSL to pyopenssl since that is the package name. fixes #1287
Shall I push this patch? (1/7)  [ynWsfvplxdaqjk], or ? for help: w
Sat Jan  1 13:05:02 MST 2011  david-sarah@jacaranda.org
  * allmydata/_auto_deps.py: add a comment reminding to change __init__.py when dependencies are added or removed. Clarify a comment in __init__.py. refs #1287
Shall I push this patch? (2/7)  [ynWvplxdaqjk], or ? for help: w
Sat Jan  1 12:25:59 MST 2011  david-sarah@jacaranda.org
  * test_runner: add test_import_in_repl, which uses 'bin/tahoe debug repl' to import the allmydata module and checks that it comes from the right path. Also, fix a latent bug that caused test_the_right_code to incorrectly conclude that a path mismatch was due to a Unicode path (causing a skip rather than a failure). This version of the patch avoids a confusing shadowing of 'srcfile'. refs #1258
Shall I push this patch? (3/7)  [ynWsfvplxdaqjk], or ? for help: w
Wed Jan 12 21:24:57 MST 2011  david-sarah@jacaranda.org
  * Make bb-freeze work. This also implements more robust checking of dependency requirements, as discussed in ticket:585#comment:10 . refs #585
Shall I push this patch? (4/7)  [ynWvplxdaqjk], or ? for help: w
Wed Jan 12 21:30:40 MST 2011  david-sarah@jacaranda.org
  * Move get_package_versions_and_locations from __init__.py to _auto_deps.py (because it needs to be updated when dependencies change). refs #585
Shall I push this patch? (5/7)  [ynWvplxdaqjk], or ? for help: w
Fri Jan 14 22:02:45 MST 2011  david-sarah@jacaranda.org
  * Add mock to get_package_versions_and_locations.
Shall I push this patch? (6/7)  [ynWsfvplxdaqjk], or ? for help: w
Sun Jan 16 13:01:12 MST 2011  david-sarah@jacaranda.org
  * src/allmydata/_auto_deps.py: Fix 'DeprecationWarning: the sets module is deprecated'. refs #1329
Shall I push this patch? (7/7)  [ynWsfvplxdaqjk], or ? for help: w

And below and attached is the unified diff of all seven of them coalesced.

I propose that we start by applying the first one -- see #1258.

diff -rN -u old-ticket1329/Makefile new-ticket1329/Makefile
--- old-ticket1329/Makefile	2011-01-17 23:54:16.000000000 -0700
+++ new-ticket1329/Makefile	2011-01-17 23:54:16.000000000 -0700
@@ -179,7 +179,7 @@
 
 
 pyflakes:
-	$(PYTHON) -OOu `which pyflakes` src/allmydata |sort |uniq
+	$(PYTHON) -OOu `which pyflakes` src/allmydata static |sort |uniq
 check-umids:
 	$(PYTHON) misc/coding_tools/check-umids.py `find src/allmydata -name '*.py'`
 
diff -rN -u old-ticket1329/src/allmydata/__init__.py new-ticket1329/src/allmydata/__init__.py
--- old-ticket1329/src/allmydata/__init__.py	2011-01-17 23:54:16.000000000 -0700
+++ new-ticket1329/src/allmydata/__init__.py	2011-01-17 23:54:16.000000000 -0700
@@ -4,57 +4,6 @@
 community web site: U{http://tahoe-lafs.org/}
 """
 
-# We want to call require_auto_deps() before other imports, because the setuptools
-# docs claim that if a distribution is installed with --multi-version, it might not
-# be importable until after pkg_resources.require() has been called for it. We don't
-# have an example of this happening at this time. It is possible that require() isn't
-# actually needed because we set __requires__ in the generated startup script, but
-# that would be an undocumented property of the setuptools implementation.
-
-from allmydata import _auto_deps
-_auto_deps.require_auto_deps()
-
-# This is just to suppress DeprecationWarnings from nevow and twisted.
-# See http://allmydata.org/trac/tahoe/ticket/859 and
-# http://divmod.org/trac/ticket/2994 .
-import warnings
-warnings.filterwarnings("ignore", category=DeprecationWarning,
-    message="the sha module is deprecated; use the hashlib module instead",
-    append=True)
-warnings.filterwarnings("ignore", category=DeprecationWarning,
-    message="object.__new__\(\) takes no parameters",
-    append=True)
-warnings.filterwarnings("ignore", category=DeprecationWarning,
-    message="The popen2 module is deprecated.  Use the subprocess module.",
-    append=True)
-warnings.filterwarnings("ignore", category=DeprecationWarning,
-    message="the md5 module is deprecated; use hashlib instead",
-    append=True)
-warnings.filterwarnings("ignore", category=DeprecationWarning,
-    message="twisted.web.error.NoResource is deprecated since Twisted 9.0.  See twisted.web.resource.NoResource.",
-    append=True)
-try:
-    import nevow
-    from twisted.persisted import sob
-    from twisted.python import filepath
-    hush_pyflakes = (nevow, sob, filepath)
-    del hush_pyflakes
-finally:
-    warnings.filters.pop()
-    warnings.filters.pop()
-    warnings.filters.pop()
-    warnings.filters.pop()
-    # Don't pop the filter for the sha module warning because it is also generated
-    # by pycrypto (which we don't want to import unless needed).
-    # warnings.filters.pop()
-
-# This warning is generated by twisted, PyRex, and possibly other packages,
-# but can happen at any time, not only when they are imported. See
-# http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1129 .
-warnings.filterwarnings("ignore", category=DeprecationWarning,
-    message="BaseException.message has been deprecated as of Python 2.6",
-    append=True)
-
 __version__ = "unknown"
 try:
     from allmydata._version import __version__
@@ -175,88 +124,93 @@
     else:
         return platform.platform()
 
-def get_package_versions_from_setuptools():
-    import pkg_resources
-    return dict([(p.project_name, (p.version, p.location)) for p in pkg_resources.require(__appname__)])
-
-def package_dir(srcfile):
-    return os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile))))
-
-def get_package_versions_and_locations():
-    # because there are a few dependencies that are outside setuptools's ken
-    # (Python and platform, and sqlite3 if you are on Python >= 2.5), and
-    # because setuptools might fail to find something even though import
-    # finds it:
-    import OpenSSL, allmydata, foolscap.api, nevow, platform, pycryptopp, setuptools, simplejson, twisted, zfec, zope.interface
-    pysqlitever = None
-    pysqlitefile = None
-    sqlitever = None
-    try:
-        import sqlite3
-    except ImportError:
-        try:
-            from pysqlite2 import dbapi2
-        except ImportError:
-            pass
-        else:
-            pysqlitever = dbapi2.version
-            pysqlitefile = package_dir(dbapi2.__file__)
-            sqlitever = dbapi2.sqlite_version
-    else:
-        pysqlitever = sqlite3.version
-        pysqlitefile = package_dir(sqlite3.__file__)
-        sqlitever = sqlite3.sqlite_version
-
-    d1 = {
-        'pyOpenSSL': (OpenSSL.__version__, package_dir(OpenSSL.__file__)),
-        __appname__: (allmydata.__version__, package_dir(allmydata.__file__)),
-        'foolscap': (foolscap.api.__version__, package_dir(foolscap.__file__)),
-        'Nevow': (nevow.__version__, package_dir(nevow.__file__)),
-        'pycryptopp': (pycryptopp.__version__, package_dir(pycryptopp.__file__)),
-        'setuptools': (setuptools.__version__, package_dir(setuptools.__file__)),
-        'simplejson': (simplejson.__version__, package_dir(simplejson.__file__)),
-        'pysqlite': (pysqlitever, pysqlitefile),
-        'sqlite': (sqlitever, 'unknown'),
-        'zope.interface': ('unknown', package_dir(zope.interface.__file__)),
-        'Twisted': (twisted.__version__, package_dir(twisted.__file__)),
-        'zfec': (zfec.__version__, package_dir(zfec.__file__)),
-        'python': (platform.python_version(), sys.executable),
-        'platform': (get_platform(), None),
-        }
 
-    # But we prefer to get all the dependencies as known by setuptools:
-    import pkg_resources
-    try:
-        d2 = get_package_versions_from_setuptools()
-    except pkg_resources.DistributionNotFound:
-        # See docstring in _auto_deps.require_auto_deps() to explain why it makes sense to ignore this exception.
-        pass
+from allmydata._auto_deps import get_package_versions_and_locations
+
+_vers_and_locs = get_package_versions_and_locations()
+_vers_and_locs['platform'] = (get_platform(), None)
+
+
+def check_all_requirements():
+    import platform
+    from distutils.version import LooseVersion
+    from allmydata._auto_deps import install_requires
+
+    # we require 2.4.4 on non-UCS-2, non-Redhat builds to avoid <http://www.python.org/news/security/PSF-2006-001/>
+    # we require 2.4.3 on non-UCS-2 Redhat, because 2.4.3 is common on Redhat-based distros and will have patched the above bug
+    # we require at least 2.4.2 in any case to avoid a bug in the base64 module: <http://bugs.python.org/issue1171487>
+    if sys.maxunicode == 65535:
+        if sys.version_info < (2, 4, 2) or sys.version_info[0] > 2:
+            raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.2 or greater "
+                                      "for a UCS-2 build (but less than v3), not %r" %
+                                      (sys.version_info,))
+    elif platform.platform().lower().find('redhat') >= 0:
+        if sys.version_info < (2, 4, 3) or sys.version_info[0] > 2:
+            raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.3 or greater "
+                                      "on Redhat-based distributions (but less than v3), not %r" %
+                                      (sys.version_info,))
     else:
-        d1.update(d2)
+        if sys.version_info < (2, 4, 4) or sys.version_info[0] > 2:
+            raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.4 or greater "
+                                      "for a non-UCS-2 build (but less than v3), not %r" %
+                                      (sys.version_info,))
+
+    def check_requirement(req):
+        # We only support a single >= or ==.
+        def _check(s, ok, relation):
+            name = s[0].strip(' ').partition('[')[0]
+            required = s[1].strip(' ')
+            if name not in _vers_and_locs:
+                raise AssertionError("no version info for %s" % (name,))
+            (actual, location) = _vers_and_locs[name]
+            actual = str(actual)
+            if actual != 'unknown' and not ok(LooseVersion(actual), LooseVersion(required)):
+                msg = ("We require %s version %s of %s, but could only find version %s.\n"
+                      % (relation, required, name, actual))
+                if location and location != 'unknown':
+                    msg += "The version we found is from %r.\n" % (location,)
+                msg += ("To resolve this problem, uninstall that version, either using your\n"
+                        "operating system's package manager or by moving aside the directory.")
+                raise AssertionError(msg)
+
+        s = req.split('>=')
+        if len(s) == 2:
+            _check(s, lambda x, y: x >= y, "at least")
+            return
+        s = req.split('==')
+        if len(s) == 2:
+            _check(s, lambda x, y: x == y, "exactly")
+            return
+        if req not in _vers_and_locs:
+            raise AssertionError("no version info or could not understand requirement for %r" % (req))
+
+    for requirement in install_requires:
+        check_requirement(requirement)
+
+check_all_requirements()
 
-    return d1
 
 def get_package_versions():
-    return dict([(k, v) for k, (v, l) in get_package_versions_and_locations().iteritems()])
+    return dict([(k, v) for k, (v, l) in _vers_and_locs.iteritems()])
 
 def get_package_locations():
-    return dict([(k, l) for k, (v, l) in get_package_versions_and_locations().iteritems()])
+    return dict([(k, l) for k, (v, l) in _vers_and_locs.iteritems()])
 
 def get_package_versions_string(show_paths=False):
-    vers_and_locs = get_package_versions_and_locations()
     res = []
-    for p in [__appname__, "foolscap", "pycryptopp", "zfec", "Twisted", "Nevow", "zope.interface", "python", "platform"]:
-        (ver, loc) = vers_and_locs.get(p, ('UNKNOWN', 'UNKNOWN'))
+    first = [__appname__, "foolscap", "pycryptopp", "zfec", "Twisted", "Nevow", "zope.interface", "python", "platform"]
+    for p in first:
+        (ver, loc) = _vers_and_locs.get(p, ('UNKNOWN', 'UNKNOWN'))
         info = str(p) + ": " + str(ver)
         if show_paths:
             info = info + " (%s)" % str(loc)
         res.append(info)
-        if vers_and_locs.has_key(p):
-            del vers_and_locs[p]
 
-    for p, (v, loc) in vers_and_locs.iteritems():
-        info = str(p) + ": " + str(v)
-        if show_paths:
-            info = info + " (%s)" % str(loc)
-        res.append(info)
+    for p, (v, loc) in _vers_and_locs.iteritems():
+        if p not in first:
+            info = str(p) + ": " + str(v)
+            if show_paths:
+                info = info + " (%s)" % str(loc)
+            res.append(info)
+
     return ', '.join(res)
diff -rN -u old-ticket1329/src/allmydata/_auto_deps.py new-ticket1329/src/allmydata/_auto_deps.py
--- old-ticket1329/src/allmydata/_auto_deps.py	2011-01-17 23:54:16.000000000 -0700
+++ new-ticket1329/src/allmydata/_auto_deps.py	2011-01-17 23:54:16.000000000 -0700
@@ -1,9 +1,10 @@
 # Note: do not import any module from Tahoe-LAFS itself in this
 # file. Also please avoid importing modules from other packages than
-# the Python Standard Library if at all possible (exception: we rely
-# on importing pkg_resources, which is provided by setuptools,
-# zetuptoolz, distribute, and perhaps in the future distutils2, for
-# the require_auto_deps() function.)
+# the Python Standard Library if at all possible. That includes
+# setuptools and pkg_resources.
+
+# If you add or remove dependencies, please remember to update
+# get_package_versions_and_locations() below.
 
 install_requires=[
                   # we require newer versions of setuptools (actually
@@ -27,7 +28,7 @@
 
                   # Needed for SFTP. pyasn1 is needed by twisted.conch in Twisted >= 9.0.
                   # pycrypto 2.2 doesn't work due to https://bugs.launchpad.net/pycrypto/+bug/620253
-                  "pycrypto == 2.0.1, == 2.1, >= 2.3",
+                  "pycrypto >= 2.3",
                   "pyasn1 >= 0.0.8a",
 
                   # Will be needed to test web apps, but not yet. See #1001.
@@ -74,49 +75,105 @@
     install_requires=[]
 del sys # clean up namespace
 
-def require_python_version():
-    import sys, platform
 
-    # we require 2.4.4 on non-UCS-2, non-Redhat builds to avoid <http://www.python.org/news/security/PSF-2006-001/>
-    # we require 2.4.3 on non-UCS-2 Redhat, because 2.4.3 is common on Redhat-based distros and will have patched the above bug
-    # we require at least 2.4.2 in any case to avoid a bug in the base64 module: <http://bugs.python.org/issue1171487>
-    if sys.maxunicode == 65535:
-        if sys.version_info < (2, 4, 2) or sys.version_info[0] > 2:
-            raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.2 or greater "
-                                      "for a UCS-2 build (but less than v3), not %r" %
-                                      (sys.version_info,))
-    elif platform.platform().lower().find('redhat') >= 0:
-        if sys.version_info < (2, 4, 3) or sys.version_info[0] > 2:
-            raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.3 or greater "
-                                      "on Redhat-based distributions (but less than v3), not %r" %
-                                      (sys.version_info,))
+def get_package_versions_and_locations():
+    import warnings, os, sys
+    def package_dir(srcfile):
+        return os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile))))
+
+    # pkg_resources.require returns the distribution that pkg_resources attempted to put
+    # on sys.path, which can differ from the one that we actually import due to #1258,
+    # or any other bug that causes sys.path to be set up incorrectly. Therefore we
+    # must import the packages in order to check their versions and paths. Note that
+    # this needs to be updated if dependencies are added or removed.
+
+    # This warning is generated by twisted, PyRex, and possibly other packages,
+    # but can happen at any time, not only when they are imported. See
+    # http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1129 .
+    warnings.filterwarnings("ignore", category=DeprecationWarning,
+        message="BaseException.message has been deprecated as of Python 2.6",
+        append=True)
+
+    # This is to suppress DeprecationWarnings from nevow, twisted, and pycrypto.
+    # See http://allmydata.org/trac/tahoe/ticket/859 and
+    # http://divmod.org/trac/ticket/2994 .
+    warnings.filterwarnings("ignore", category=DeprecationWarning,
+        message="the sha module is deprecated; use the hashlib module instead",
+        append=True)
+    warnings.filterwarnings("ignore", category=DeprecationWarning,
+        message="object.__new__\(\) takes no parameters",
+        append=True)
+    warnings.filterwarnings("ignore", category=DeprecationWarning,
+        message="The popen2 module is deprecated.  Use the subprocess module.",
+        append=True)
+    warnings.filterwarnings("ignore", category=DeprecationWarning,
+        message="the md5 module is deprecated; use hashlib instead",
+        append=True)
+    warnings.filterwarnings("ignore", category=DeprecationWarning,
+        message="twisted.web.error.NoResource is deprecated since Twisted 9.0.  See twisted.web.resource.NoResource.",
+        append=True)
+    warnings.filterwarnings("ignore", category=DeprecationWarning,
+        message="the sets module is deprecated",
+        append=True)
+    try:
+        import nevow
+        from twisted.persisted import sob
+        from twisted.python import filepath
+        import Crypto
+        import foolscap.api
+        [sob, filepath]  # hush pyflakes
+    finally:
+        for n in range(6):
+            warnings.filters.pop()
+
+    import OpenSSL, allmydata, platform, pycryptopp, simplejson, twisted, zfec, zope.interface, pyasn1, mock
+    pysqlitever = None
+    pysqlitefile = None
+    sqlitever = None
+    try:
+        import sqlite3
+    except ImportError:
+        try:
+            from pysqlite2 import dbapi2
+        except ImportError:
+            pass
+        else:
+            pysqlitever = dbapi2.version
+            pysqlitefile = package_dir(dbapi2.__file__)
+            sqlitever = dbapi2.sqlite_version
     else:
-        if sys.version_info < (2, 4, 4) or sys.version_info[0] > 2:
-            raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.4 or greater "
-                                      "for a non-UCS-2 build (but less than v3), not %r" %
-                                      (sys.version_info,))
-
-def require_auto_deps():
-    """
-    The purpose of this function is to raise a pkg_resources exception if any of the
-    requirements can't be imported.  This is just to give earlier and more explicit error
-    messages, as opposed to waiting until the source code tries to import some module from one
-    of these packages and gets an ImportError.  This function gets called from
-    src/allmydata/__init__.py .
-    """
-    require_python_version()
+        pysqlitever = sqlite3.version
+        pysqlitefile = package_dir(sqlite3.__file__)
+        sqlitever = sqlite3.sqlite_version
+
+    packages = {
+        'pyopenssl': (OpenSSL.__version__, package_dir(OpenSSL.__file__)),
+        allmydata.__appname__: (allmydata.__version__, package_dir(allmydata.__file__)),
+        'foolscap': (foolscap.api.__version__, package_dir(foolscap.__file__)),
+        'Nevow': (nevow.__version__, package_dir(nevow.__file__)),
+        'pycryptopp': (pycryptopp.__version__, package_dir(pycryptopp.__file__)),
+        'simplejson': (simplejson.__version__, package_dir(simplejson.__file__)),
+        'pysqlite': (pysqlitever, pysqlitefile),
+        'sqlite': (sqlitever, 'unknown'),
+        'zope.interface': ('unknown', package_dir(zope.interface.__file__)),
+        'Twisted': (twisted.__version__, package_dir(twisted.__file__)),
+        'zfec': (zfec.__version__, package_dir(zfec.__file__)),
+        'pycrypto': (Crypto.__version__, package_dir(Crypto.__file__)),
+        'pyasn1': ('unknown', package_dir(pyasn1.__file__)),
+        'mock': (mock.__version__, package_dir(mock.__file__)),
+        'python': (platform.python_version(), sys.executable),
+        }
 
-    import pkg_resources
-    for requirement in install_requires:
+    if sys.platform == 'win32':
         try:
-            pkg_resources.require(requirement)
-        except pkg_resources.DistributionNotFound:
-            # there is no .egg-info present for this requirement, which
-            # either means that it isn't installed, or it is installed in a
-            # way that pkg_resources can't find it (but regular python
-            # might).  There are several older Linux distributions which
-            # provide our dependencies just fine, but they don't ship
-            # .egg-info files. Note that if there *is* an .egg-info file,
-            # but it shows a too-old version, then we'll get a
-            # VersionConflict error instead of DistributionNotFound.
+            import win32api
+            packages['pywin32'] = ('unknown', package_dir(win32api.__file__))
+        except ImportError:
             pass
+
+    if not hasattr(sys, 'frozen'):
+        # Don't try to get the setuptools version if using bbfreeze, py2exe etc. See #585.
+        import setuptools
+        packages['setuptools'] = (setuptools.__version__, package_dir(setuptools.__file__))
+
+    return packages
diff -rN -u old-ticket1329/src/allmydata/test/test_runner.py new-ticket1329/src/allmydata/test/test_runner.py
--- old-ticket1329/src/allmydata/test/test_runner.py	2011-01-17 23:54:16.000000000 -0700
+++ new-ticket1329/src/allmydata/test/test_runner.py	2011-01-17 23:54:16.000000000 -0700
@@ -3,7 +3,7 @@
 
 from twisted.python import usage, runtime
 from twisted.internet import utils
-import os.path, re, sys
+import os.path, re, sys, subprocess
 from cStringIO import StringIO
 from allmydata.util import fileutil, pollmixin
 from allmydata.util.encodingutil import unicode_to_argv, unicode_to_output, get_filesystem_encoding
@@ -14,24 +14,36 @@
 
 timeout = 240
 
+def get_root_from_file(src):
+    srcdir = os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(src))))
+
+    root = os.path.dirname(srcdir)
+    if os.path.basename(srcdir) == 'site-packages':
+        if re.search(r'python.+\..+', os.path.basename(root)):
+            root = os.path.dirname(root)
+        root = os.path.dirname(root)
+    elif os.path.basename(root) == 'src':
+        root = os.path.dirname(root)
+
+    return root
+
 srcfile = allmydata.__file__
-srcdir = os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile))))
+rootdir = get_root_from_file(srcfile)
 
-rootdir = os.path.dirname(srcdir)
-if os.path.basename(srcdir) == 'site-packages':
-    if re.search(r'python.+\..+', os.path.basename(rootdir)):
-        rootdir = os.path.dirname(rootdir)
-    rootdir = os.path.dirname(rootdir)
-elif os.path.basename(rootdir) == 'src':
-    rootdir = os.path.dirname(rootdir)
-
-bintahoe = os.path.join(rootdir, 'bin', 'tahoe')
-if sys.platform == "win32":
-    bintahoe += ".pyscript"
-    if not os.path.exists(bintahoe):
-       alt_bintahoe = os.path.join(rootdir, 'Scripts', 'tahoe.pyscript')
-       if os.path.exists(alt_bintahoe):
-           bintahoe = alt_bintahoe
+
+if hasattr(sys, 'frozen'):
+    bintahoe = os.path.join(rootdir, 'tahoe')
+
+    if sys.platform == "win32" and os.path.exists(bintahoe + '.exe'):
+        bintahoe += '.exe'
+else:
+    bintahoe = os.path.join(rootdir, 'bin', 'tahoe')
+    if sys.platform == "win32":
+        bintahoe += '.pyscript'
+        if not os.path.exists(bintahoe):
+            alt_bintahoe = os.path.join(rootdir, 'Scripts', 'tahoe.pyscript')
+            if os.path.exists(alt_bintahoe):
+                bintahoe = alt_bintahoe
 
 
 class SkipMixin:
@@ -49,16 +61,20 @@
 
 
 class BinTahoe(common_util.SignalMixin, unittest.TestCase, SkipMixin):
-    def test_the_right_code(self):
+    def _check_right_code(self, file_to_check):
+        root_to_check = get_root_from_file(file_to_check)
+        if os.path.basename(root_to_check) == 'dist':
+            root_to_check = os.path.dirname(root_to_check)
+
         cwd = os.path.normcase(os.path.realpath("."))
         root_from_cwd = os.path.dirname(cwd)
         if os.path.basename(root_from_cwd) == 'src':
             root_from_cwd = os.path.dirname(root_from_cwd)
 
-        same = (root_from_cwd == rootdir)
+        same = (root_from_cwd == root_to_check)
         if not same:
             try:
-                same = os.path.samefile(root_from_cwd, rootdir)
+                same = os.path.samefile(root_from_cwd, root_to_check)
             except AttributeError, e:
                 e  # hush pyflakes
 
@@ -66,9 +82,9 @@
             msg = ("We seem to be testing the code at %r,\n"
                    "(according to the source filename %r),\n"
                    "but expected to be testing the code at %r.\n"
-                   % (rootdir, srcfile, root_from_cwd))
+                   % (root_to_check, file_to_check, root_from_cwd))
 
-            root_from_cwdu = os.path.normcase(os.path.normpath(os.getcwdu()))
+            root_from_cwdu = os.path.dirname(os.path.normcase(os.path.normpath(os.getcwdu())))
             if os.path.basename(root_from_cwdu) == u'src':
                 root_from_cwdu = os.path.dirname(root_from_cwdu)
 
@@ -81,6 +97,24 @@
                 msg += "Please run the tests from the root of the Tahoe-LAFS distribution."
                 self.fail(msg)
 
+    def test_the_right_code(self):
+        self._check_right_code(srcfile)
+
+    def test_import_in_repl(self):
+        self.skip_if_cannot_run_bintahoe()
+
+        args = [bintahoe, "debug", "repl"]
+        if not hasattr(sys, 'frozen'):
+            args = [sys.executable] + args
+
+        p = subprocess.Popen(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        (out, err) = p.communicate("import allmydata; print allmydata.__file__")
+
+        self.failUnlessEqual(p.returncode, 0)
+        first = out.splitlines()[0]
+        self.failUnless(first.startswith('>>> '))
+        self._check_right_code(first[4:])
+
     def test_path(self):
         self.skip_if_cannot_run_bintahoe()
         d = utils.getProcessOutputAndValue(bintahoe, args=["--version-and-path"], env=os.environ)
@@ -112,6 +146,7 @@
                 else:
                     altverstr = verstr
 
+            srcdir = os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile))))
             required_ver_and_path = "%s: %s (%s)" % (allmydata.__appname__, verstr, srcdir)
             alt_required_ver_and_path = "%s: %s (%s)" % (allmydata.__appname__, altverstr, srcdir)
 
@@ -121,6 +156,8 @@
 
     def test_unicode_arguments_and_output(self):
         self.skip_if_cannot_run_bintahoe()
+        if hasattr(sys, 'frozen') and sys.platform == "win32":
+            raise unittest.SkipTest("This test can't currently be made to work for frozen executables on Windows,\ndue to lack of support for Unicode in twisted.internet.utils.getProcessOutputAndValue.")
 
         tricky = u"\u2621"
         try:
@@ -139,6 +176,8 @@
 
     def test_run_with_python_options(self):
         self.skip_if_cannot_run_bintahoe()
+        if hasattr(sys, 'frozen'):
+            raise unittest.SkipTest("This test doesn't apply to frozen executables.")
 
         # -t is a harmless option that warns about tabs.
         d = utils.getProcessOutputAndValue(sys.executable, args=['-t', bintahoe, '--version'],
diff -rN -u old-ticket1329/src/allmydata/test/test_system.py new-ticket1329/src/allmydata/test/test_system.py
--- old-ticket1329/src/allmydata/test/test_system.py	2011-01-17 23:54:16.000000000 -0700
+++ new-ticket1329/src/allmydata/test/test_system.py	2011-01-17 23:54:16.000000000 -0700
@@ -1769,8 +1769,7 @@
 
         if env is None:
             env = os.environ
-        d = utils.getProcessOutputAndValue(sys.executable, args=[bintahoe] + argv,
-                                           env=env)
+        d = utils.getProcessOutputAndValue(bintahoe, argv, env=env)
         return d
 
     def _test_checker(self, res):
Replying to [warner](/tahoe-lafs/trac-2024-07-25/issues/1329#issuecomment-123830): > my main problem with [20110116200112-93fa1-0397f85493a9fd0ae82ce58d7291d598a4f6e2d6]/ticket1306 is that it involved what looked like a signficant refactoring of `_auto_deps.py`, so I couldn't apply just a single patch (there were more like six). But if there's a clean way to use that approach, I'm happy to go with it. For the record, here are the patches that (from darcs's perspective) are required for [20110116200112-93fa1-0397f85493a9fd0ae82ce58d7291d598a4f6e2d6]/ticket1306: ``` HACL Zooko-Ofsimplegeos-MacBook-Pro:~/playground/tahoe-lafs/ticket1329$ time darcs push zooko@dev.allmydata.org:/home/darcs/tahoe-lafs/trunk Pushing to "zooko@dev.allmydata.org:/home/darcs/tahoe-lafs/trunk"... The remote repository has 2 patches to pull. Sat Jan 1 04:01:41 MST 2011 david-sarah@jacaranda.org * allmydata/__init__.py: don't use pkg_resources.require to get versions and paths. Also change pyOpenSSL to pyopenssl since that is the package name. fixes #1287 Shall I push this patch? (1/7) [ynWsfvplxdaqjk], or ? for help: w Sat Jan 1 13:05:02 MST 2011 david-sarah@jacaranda.org * allmydata/_auto_deps.py: add a comment reminding to change __init__.py when dependencies are added or removed. Clarify a comment in __init__.py. refs #1287 Shall I push this patch? (2/7) [ynWvplxdaqjk], or ? for help: w Sat Jan 1 12:25:59 MST 2011 david-sarah@jacaranda.org * test_runner: add test_import_in_repl, which uses 'bin/tahoe debug repl' to import the allmydata module and checks that it comes from the right path. Also, fix a latent bug that caused test_the_right_code to incorrectly conclude that a path mismatch was due to a Unicode path (causing a skip rather than a failure). This version of the patch avoids a confusing shadowing of 'srcfile'. refs #1258 Shall I push this patch? (3/7) [ynWsfvplxdaqjk], or ? for help: w Wed Jan 12 21:24:57 MST 2011 david-sarah@jacaranda.org * Make bb-freeze work. This also implements more robust checking of dependency requirements, as discussed in ticket:585#comment:10 . refs #585 Shall I push this patch? (4/7) [ynWvplxdaqjk], or ? for help: w Wed Jan 12 21:30:40 MST 2011 david-sarah@jacaranda.org * Move get_package_versions_and_locations from __init__.py to _auto_deps.py (because it needs to be updated when dependencies change). refs #585 Shall I push this patch? (5/7) [ynWvplxdaqjk], or ? for help: w Fri Jan 14 22:02:45 MST 2011 david-sarah@jacaranda.org * Add mock to get_package_versions_and_locations. Shall I push this patch? (6/7) [ynWsfvplxdaqjk], or ? for help: w Sun Jan 16 13:01:12 MST 2011 david-sarah@jacaranda.org * src/allmydata/_auto_deps.py: Fix 'DeprecationWarning: the sets module is deprecated'. refs #1329 Shall I push this patch? (7/7) [ynWsfvplxdaqjk], or ? for help: w ``` And below and attached is the unified diff of all seven of them coalesced. I propose that we start by applying the first one -- see #1258. ``` diff -rN -u old-ticket1329/Makefile new-ticket1329/Makefile --- old-ticket1329/Makefile 2011-01-17 23:54:16.000000000 -0700 +++ new-ticket1329/Makefile 2011-01-17 23:54:16.000000000 -0700 @@ -179,7 +179,7 @@ pyflakes: - $(PYTHON) -OOu `which pyflakes` src/allmydata |sort |uniq + $(PYTHON) -OOu `which pyflakes` src/allmydata static |sort |uniq check-umids: $(PYTHON) misc/coding_tools/check-umids.py `find src/allmydata -name '*.py'` diff -rN -u old-ticket1329/src/allmydata/__init__.py new-ticket1329/src/allmydata/__init__.py --- old-ticket1329/src/allmydata/__init__.py 2011-01-17 23:54:16.000000000 -0700 +++ new-ticket1329/src/allmydata/__init__.py 2011-01-17 23:54:16.000000000 -0700 @@ -4,57 +4,6 @@ community web site: U{http://tahoe-lafs.org/} """ -# We want to call require_auto_deps() before other imports, because the setuptools -# docs claim that if a distribution is installed with --multi-version, it might not -# be importable until after pkg_resources.require() has been called for it. We don't -# have an example of this happening at this time. It is possible that require() isn't -# actually needed because we set __requires__ in the generated startup script, but -# that would be an undocumented property of the setuptools implementation. - -from allmydata import _auto_deps -_auto_deps.require_auto_deps() - -# This is just to suppress DeprecationWarnings from nevow and twisted. -# See http://allmydata.org/trac/tahoe/ticket/859 and -# http://divmod.org/trac/ticket/2994 . -import warnings -warnings.filterwarnings("ignore", category=DeprecationWarning, - message="the sha module is deprecated; use the hashlib module instead", - append=True) -warnings.filterwarnings("ignore", category=DeprecationWarning, - message="object.__new__\(\) takes no parameters", - append=True) -warnings.filterwarnings("ignore", category=DeprecationWarning, - message="The popen2 module is deprecated. Use the subprocess module.", - append=True) -warnings.filterwarnings("ignore", category=DeprecationWarning, - message="the md5 module is deprecated; use hashlib instead", - append=True) -warnings.filterwarnings("ignore", category=DeprecationWarning, - message="twisted.web.error.NoResource is deprecated since Twisted 9.0. See twisted.web.resource.NoResource.", - append=True) -try: - import nevow - from twisted.persisted import sob - from twisted.python import filepath - hush_pyflakes = (nevow, sob, filepath) - del hush_pyflakes -finally: - warnings.filters.pop() - warnings.filters.pop() - warnings.filters.pop() - warnings.filters.pop() - # Don't pop the filter for the sha module warning because it is also generated - # by pycrypto (which we don't want to import unless needed). - # warnings.filters.pop() - -# This warning is generated by twisted, PyRex, and possibly other packages, -# but can happen at any time, not only when they are imported. See -# http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1129 . -warnings.filterwarnings("ignore", category=DeprecationWarning, - message="BaseException.message has been deprecated as of Python 2.6", - append=True) - __version__ = "unknown" try: from allmydata._version import __version__ @@ -175,88 +124,93 @@ else: return platform.platform() -def get_package_versions_from_setuptools(): - import pkg_resources - return dict([(p.project_name, (p.version, p.location)) for p in pkg_resources.require(__appname__)]) - -def package_dir(srcfile): - return os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile)))) - -def get_package_versions_and_locations(): - # because there are a few dependencies that are outside setuptools's ken - # (Python and platform, and sqlite3 if you are on Python >= 2.5), and - # because setuptools might fail to find something even though import - # finds it: - import OpenSSL, allmydata, foolscap.api, nevow, platform, pycryptopp, setuptools, simplejson, twisted, zfec, zope.interface - pysqlitever = None - pysqlitefile = None - sqlitever = None - try: - import sqlite3 - except ImportError: - try: - from pysqlite2 import dbapi2 - except ImportError: - pass - else: - pysqlitever = dbapi2.version - pysqlitefile = package_dir(dbapi2.__file__) - sqlitever = dbapi2.sqlite_version - else: - pysqlitever = sqlite3.version - pysqlitefile = package_dir(sqlite3.__file__) - sqlitever = sqlite3.sqlite_version - - d1 = { - 'pyOpenSSL': (OpenSSL.__version__, package_dir(OpenSSL.__file__)), - __appname__: (allmydata.__version__, package_dir(allmydata.__file__)), - 'foolscap': (foolscap.api.__version__, package_dir(foolscap.__file__)), - 'Nevow': (nevow.__version__, package_dir(nevow.__file__)), - 'pycryptopp': (pycryptopp.__version__, package_dir(pycryptopp.__file__)), - 'setuptools': (setuptools.__version__, package_dir(setuptools.__file__)), - 'simplejson': (simplejson.__version__, package_dir(simplejson.__file__)), - 'pysqlite': (pysqlitever, pysqlitefile), - 'sqlite': (sqlitever, 'unknown'), - 'zope.interface': ('unknown', package_dir(zope.interface.__file__)), - 'Twisted': (twisted.__version__, package_dir(twisted.__file__)), - 'zfec': (zfec.__version__, package_dir(zfec.__file__)), - 'python': (platform.python_version(), sys.executable), - 'platform': (get_platform(), None), - } - # But we prefer to get all the dependencies as known by setuptools: - import pkg_resources - try: - d2 = get_package_versions_from_setuptools() - except pkg_resources.DistributionNotFound: - # See docstring in _auto_deps.require_auto_deps() to explain why it makes sense to ignore this exception. - pass +from allmydata._auto_deps import get_package_versions_and_locations + +_vers_and_locs = get_package_versions_and_locations() +_vers_and_locs['platform'] = (get_platform(), None) + + +def check_all_requirements(): + import platform + from distutils.version import LooseVersion + from allmydata._auto_deps import install_requires + + # we require 2.4.4 on non-UCS-2, non-Redhat builds to avoid <http://www.python.org/news/security/PSF-2006-001/> + # we require 2.4.3 on non-UCS-2 Redhat, because 2.4.3 is common on Redhat-based distros and will have patched the above bug + # we require at least 2.4.2 in any case to avoid a bug in the base64 module: <http://bugs.python.org/issue1171487> + if sys.maxunicode == 65535: + if sys.version_info < (2, 4, 2) or sys.version_info[0] > 2: + raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.2 or greater " + "for a UCS-2 build (but less than v3), not %r" % + (sys.version_info,)) + elif platform.platform().lower().find('redhat') >= 0: + if sys.version_info < (2, 4, 3) or sys.version_info[0] > 2: + raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.3 or greater " + "on Redhat-based distributions (but less than v3), not %r" % + (sys.version_info,)) else: - d1.update(d2) + if sys.version_info < (2, 4, 4) or sys.version_info[0] > 2: + raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.4 or greater " + "for a non-UCS-2 build (but less than v3), not %r" % + (sys.version_info,)) + + def check_requirement(req): + # We only support a single >= or ==. + def _check(s, ok, relation): + name = s[0].strip(' ').partition('[')[0] + required = s[1].strip(' ') + if name not in _vers_and_locs: + raise AssertionError("no version info for %s" % (name,)) + (actual, location) = _vers_and_locs[name] + actual = str(actual) + if actual != 'unknown' and not ok(LooseVersion(actual), LooseVersion(required)): + msg = ("We require %s version %s of %s, but could only find version %s.\n" + % (relation, required, name, actual)) + if location and location != 'unknown': + msg += "The version we found is from %r.\n" % (location,) + msg += ("To resolve this problem, uninstall that version, either using your\n" + "operating system's package manager or by moving aside the directory.") + raise AssertionError(msg) + + s = req.split('>=') + if len(s) == 2: + _check(s, lambda x, y: x >= y, "at least") + return + s = req.split('==') + if len(s) == 2: + _check(s, lambda x, y: x == y, "exactly") + return + if req not in _vers_and_locs: + raise AssertionError("no version info or could not understand requirement for %r" % (req)) + + for requirement in install_requires: + check_requirement(requirement) + +check_all_requirements() - return d1 def get_package_versions(): - return dict([(k, v) for k, (v, l) in get_package_versions_and_locations().iteritems()]) + return dict([(k, v) for k, (v, l) in _vers_and_locs.iteritems()]) def get_package_locations(): - return dict([(k, l) for k, (v, l) in get_package_versions_and_locations().iteritems()]) + return dict([(k, l) for k, (v, l) in _vers_and_locs.iteritems()]) def get_package_versions_string(show_paths=False): - vers_and_locs = get_package_versions_and_locations() res = [] - for p in [__appname__, "foolscap", "pycryptopp", "zfec", "Twisted", "Nevow", "zope.interface", "python", "platform"]: - (ver, loc) = vers_and_locs.get(p, ('UNKNOWN', 'UNKNOWN')) + first = [__appname__, "foolscap", "pycryptopp", "zfec", "Twisted", "Nevow", "zope.interface", "python", "platform"] + for p in first: + (ver, loc) = _vers_and_locs.get(p, ('UNKNOWN', 'UNKNOWN')) info = str(p) + ": " + str(ver) if show_paths: info = info + " (%s)" % str(loc) res.append(info) - if vers_and_locs.has_key(p): - del vers_and_locs[p] - for p, (v, loc) in vers_and_locs.iteritems(): - info = str(p) + ": " + str(v) - if show_paths: - info = info + " (%s)" % str(loc) - res.append(info) + for p, (v, loc) in _vers_and_locs.iteritems(): + if p not in first: + info = str(p) + ": " + str(v) + if show_paths: + info = info + " (%s)" % str(loc) + res.append(info) + return ', '.join(res) diff -rN -u old-ticket1329/src/allmydata/_auto_deps.py new-ticket1329/src/allmydata/_auto_deps.py --- old-ticket1329/src/allmydata/_auto_deps.py 2011-01-17 23:54:16.000000000 -0700 +++ new-ticket1329/src/allmydata/_auto_deps.py 2011-01-17 23:54:16.000000000 -0700 @@ -1,9 +1,10 @@ # Note: do not import any module from Tahoe-LAFS itself in this # file. Also please avoid importing modules from other packages than -# the Python Standard Library if at all possible (exception: we rely -# on importing pkg_resources, which is provided by setuptools, -# zetuptoolz, distribute, and perhaps in the future distutils2, for -# the require_auto_deps() function.) +# the Python Standard Library if at all possible. That includes +# setuptools and pkg_resources. + +# If you add or remove dependencies, please remember to update +# get_package_versions_and_locations() below. install_requires=[ # we require newer versions of setuptools (actually @@ -27,7 +28,7 @@ # Needed for SFTP. pyasn1 is needed by twisted.conch in Twisted >= 9.0. # pycrypto 2.2 doesn't work due to https://bugs.launchpad.net/pycrypto/+bug/620253 - "pycrypto == 2.0.1, == 2.1, >= 2.3", + "pycrypto >= 2.3", "pyasn1 >= 0.0.8a", # Will be needed to test web apps, but not yet. See #1001. @@ -74,49 +75,105 @@ install_requires=[] del sys # clean up namespace -def require_python_version(): - import sys, platform - # we require 2.4.4 on non-UCS-2, non-Redhat builds to avoid <http://www.python.org/news/security/PSF-2006-001/> - # we require 2.4.3 on non-UCS-2 Redhat, because 2.4.3 is common on Redhat-based distros and will have patched the above bug - # we require at least 2.4.2 in any case to avoid a bug in the base64 module: <http://bugs.python.org/issue1171487> - if sys.maxunicode == 65535: - if sys.version_info < (2, 4, 2) or sys.version_info[0] > 2: - raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.2 or greater " - "for a UCS-2 build (but less than v3), not %r" % - (sys.version_info,)) - elif platform.platform().lower().find('redhat') >= 0: - if sys.version_info < (2, 4, 3) or sys.version_info[0] > 2: - raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.3 or greater " - "on Redhat-based distributions (but less than v3), not %r" % - (sys.version_info,)) +def get_package_versions_and_locations(): + import warnings, os, sys + def package_dir(srcfile): + return os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile)))) + + # pkg_resources.require returns the distribution that pkg_resources attempted to put + # on sys.path, which can differ from the one that we actually import due to #1258, + # or any other bug that causes sys.path to be set up incorrectly. Therefore we + # must import the packages in order to check their versions and paths. Note that + # this needs to be updated if dependencies are added or removed. + + # This warning is generated by twisted, PyRex, and possibly other packages, + # but can happen at any time, not only when they are imported. See + # http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1129 . + warnings.filterwarnings("ignore", category=DeprecationWarning, + message="BaseException.message has been deprecated as of Python 2.6", + append=True) + + # This is to suppress DeprecationWarnings from nevow, twisted, and pycrypto. + # See http://allmydata.org/trac/tahoe/ticket/859 and + # http://divmod.org/trac/ticket/2994 . + warnings.filterwarnings("ignore", category=DeprecationWarning, + message="the sha module is deprecated; use the hashlib module instead", + append=True) + warnings.filterwarnings("ignore", category=DeprecationWarning, + message="object.__new__\(\) takes no parameters", + append=True) + warnings.filterwarnings("ignore", category=DeprecationWarning, + message="The popen2 module is deprecated. Use the subprocess module.", + append=True) + warnings.filterwarnings("ignore", category=DeprecationWarning, + message="the md5 module is deprecated; use hashlib instead", + append=True) + warnings.filterwarnings("ignore", category=DeprecationWarning, + message="twisted.web.error.NoResource is deprecated since Twisted 9.0. See twisted.web.resource.NoResource.", + append=True) + warnings.filterwarnings("ignore", category=DeprecationWarning, + message="the sets module is deprecated", + append=True) + try: + import nevow + from twisted.persisted import sob + from twisted.python import filepath + import Crypto + import foolscap.api + [sob, filepath] # hush pyflakes + finally: + for n in range(6): + warnings.filters.pop() + + import OpenSSL, allmydata, platform, pycryptopp, simplejson, twisted, zfec, zope.interface, pyasn1, mock + pysqlitever = None + pysqlitefile = None + sqlitever = None + try: + import sqlite3 + except ImportError: + try: + from pysqlite2 import dbapi2 + except ImportError: + pass + else: + pysqlitever = dbapi2.version + pysqlitefile = package_dir(dbapi2.__file__) + sqlitever = dbapi2.sqlite_version else: - if sys.version_info < (2, 4, 4) or sys.version_info[0] > 2: - raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.4 or greater " - "for a non-UCS-2 build (but less than v3), not %r" % - (sys.version_info,)) - -def require_auto_deps(): - """ - The purpose of this function is to raise a pkg_resources exception if any of the - requirements can't be imported. This is just to give earlier and more explicit error - messages, as opposed to waiting until the source code tries to import some module from one - of these packages and gets an ImportError. This function gets called from - src/allmydata/__init__.py . - """ - require_python_version() + pysqlitever = sqlite3.version + pysqlitefile = package_dir(sqlite3.__file__) + sqlitever = sqlite3.sqlite_version + + packages = { + 'pyopenssl': (OpenSSL.__version__, package_dir(OpenSSL.__file__)), + allmydata.__appname__: (allmydata.__version__, package_dir(allmydata.__file__)), + 'foolscap': (foolscap.api.__version__, package_dir(foolscap.__file__)), + 'Nevow': (nevow.__version__, package_dir(nevow.__file__)), + 'pycryptopp': (pycryptopp.__version__, package_dir(pycryptopp.__file__)), + 'simplejson': (simplejson.__version__, package_dir(simplejson.__file__)), + 'pysqlite': (pysqlitever, pysqlitefile), + 'sqlite': (sqlitever, 'unknown'), + 'zope.interface': ('unknown', package_dir(zope.interface.__file__)), + 'Twisted': (twisted.__version__, package_dir(twisted.__file__)), + 'zfec': (zfec.__version__, package_dir(zfec.__file__)), + 'pycrypto': (Crypto.__version__, package_dir(Crypto.__file__)), + 'pyasn1': ('unknown', package_dir(pyasn1.__file__)), + 'mock': (mock.__version__, package_dir(mock.__file__)), + 'python': (platform.python_version(), sys.executable), + } - import pkg_resources - for requirement in install_requires: + if sys.platform == 'win32': try: - pkg_resources.require(requirement) - except pkg_resources.DistributionNotFound: - # there is no .egg-info present for this requirement, which - # either means that it isn't installed, or it is installed in a - # way that pkg_resources can't find it (but regular python - # might). There are several older Linux distributions which - # provide our dependencies just fine, but they don't ship - # .egg-info files. Note that if there *is* an .egg-info file, - # but it shows a too-old version, then we'll get a - # VersionConflict error instead of DistributionNotFound. + import win32api + packages['pywin32'] = ('unknown', package_dir(win32api.__file__)) + except ImportError: pass + + if not hasattr(sys, 'frozen'): + # Don't try to get the setuptools version if using bbfreeze, py2exe etc. See #585. + import setuptools + packages['setuptools'] = (setuptools.__version__, package_dir(setuptools.__file__)) + + return packages diff -rN -u old-ticket1329/src/allmydata/test/test_runner.py new-ticket1329/src/allmydata/test/test_runner.py --- old-ticket1329/src/allmydata/test/test_runner.py 2011-01-17 23:54:16.000000000 -0700 +++ new-ticket1329/src/allmydata/test/test_runner.py 2011-01-17 23:54:16.000000000 -0700 @@ -3,7 +3,7 @@ from twisted.python import usage, runtime from twisted.internet import utils -import os.path, re, sys +import os.path, re, sys, subprocess from cStringIO import StringIO from allmydata.util import fileutil, pollmixin from allmydata.util.encodingutil import unicode_to_argv, unicode_to_output, get_filesystem_encoding @@ -14,24 +14,36 @@ timeout = 240 +def get_root_from_file(src): + srcdir = os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(src)))) + + root = os.path.dirname(srcdir) + if os.path.basename(srcdir) == 'site-packages': + if re.search(r'python.+\..+', os.path.basename(root)): + root = os.path.dirname(root) + root = os.path.dirname(root) + elif os.path.basename(root) == 'src': + root = os.path.dirname(root) + + return root + srcfile = allmydata.__file__ -srcdir = os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile)))) +rootdir = get_root_from_file(srcfile) -rootdir = os.path.dirname(srcdir) -if os.path.basename(srcdir) == 'site-packages': - if re.search(r'python.+\..+', os.path.basename(rootdir)): - rootdir = os.path.dirname(rootdir) - rootdir = os.path.dirname(rootdir) -elif os.path.basename(rootdir) == 'src': - rootdir = os.path.dirname(rootdir) - -bintahoe = os.path.join(rootdir, 'bin', 'tahoe') -if sys.platform == "win32": - bintahoe += ".pyscript" - if not os.path.exists(bintahoe): - alt_bintahoe = os.path.join(rootdir, 'Scripts', 'tahoe.pyscript') - if os.path.exists(alt_bintahoe): - bintahoe = alt_bintahoe + +if hasattr(sys, 'frozen'): + bintahoe = os.path.join(rootdir, 'tahoe') + + if sys.platform == "win32" and os.path.exists(bintahoe + '.exe'): + bintahoe += '.exe' +else: + bintahoe = os.path.join(rootdir, 'bin', 'tahoe') + if sys.platform == "win32": + bintahoe += '.pyscript' + if not os.path.exists(bintahoe): + alt_bintahoe = os.path.join(rootdir, 'Scripts', 'tahoe.pyscript') + if os.path.exists(alt_bintahoe): + bintahoe = alt_bintahoe class SkipMixin: @@ -49,16 +61,20 @@ class BinTahoe(common_util.SignalMixin, unittest.TestCase, SkipMixin): - def test_the_right_code(self): + def _check_right_code(self, file_to_check): + root_to_check = get_root_from_file(file_to_check) + if os.path.basename(root_to_check) == 'dist': + root_to_check = os.path.dirname(root_to_check) + cwd = os.path.normcase(os.path.realpath(".")) root_from_cwd = os.path.dirname(cwd) if os.path.basename(root_from_cwd) == 'src': root_from_cwd = os.path.dirname(root_from_cwd) - same = (root_from_cwd == rootdir) + same = (root_from_cwd == root_to_check) if not same: try: - same = os.path.samefile(root_from_cwd, rootdir) + same = os.path.samefile(root_from_cwd, root_to_check) except AttributeError, e: e # hush pyflakes @@ -66,9 +82,9 @@ msg = ("We seem to be testing the code at %r,\n" "(according to the source filename %r),\n" "but expected to be testing the code at %r.\n" - % (rootdir, srcfile, root_from_cwd)) + % (root_to_check, file_to_check, root_from_cwd)) - root_from_cwdu = os.path.normcase(os.path.normpath(os.getcwdu())) + root_from_cwdu = os.path.dirname(os.path.normcase(os.path.normpath(os.getcwdu()))) if os.path.basename(root_from_cwdu) == u'src': root_from_cwdu = os.path.dirname(root_from_cwdu) @@ -81,6 +97,24 @@ msg += "Please run the tests from the root of the Tahoe-LAFS distribution." self.fail(msg) + def test_the_right_code(self): + self._check_right_code(srcfile) + + def test_import_in_repl(self): + self.skip_if_cannot_run_bintahoe() + + args = [bintahoe, "debug", "repl"] + if not hasattr(sys, 'frozen'): + args = [sys.executable] + args + + p = subprocess.Popen(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + (out, err) = p.communicate("import allmydata; print allmydata.__file__") + + self.failUnlessEqual(p.returncode, 0) + first = out.splitlines()[0] + self.failUnless(first.startswith('>>> ')) + self._check_right_code(first[4:]) + def test_path(self): self.skip_if_cannot_run_bintahoe() d = utils.getProcessOutputAndValue(bintahoe, args=["--version-and-path"], env=os.environ) @@ -112,6 +146,7 @@ else: altverstr = verstr + srcdir = os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile)))) required_ver_and_path = "%s: %s (%s)" % (allmydata.__appname__, verstr, srcdir) alt_required_ver_and_path = "%s: %s (%s)" % (allmydata.__appname__, altverstr, srcdir) @@ -121,6 +156,8 @@ def test_unicode_arguments_and_output(self): self.skip_if_cannot_run_bintahoe() + if hasattr(sys, 'frozen') and sys.platform == "win32": + raise unittest.SkipTest("This test can't currently be made to work for frozen executables on Windows,\ndue to lack of support for Unicode in twisted.internet.utils.getProcessOutputAndValue.") tricky = u"\u2621" try: @@ -139,6 +176,8 @@ def test_run_with_python_options(self): self.skip_if_cannot_run_bintahoe() + if hasattr(sys, 'frozen'): + raise unittest.SkipTest("This test doesn't apply to frozen executables.") # -t is a harmless option that warns about tabs. d = utils.getProcessOutputAndValue(sys.executable, args=['-t', bintahoe, '--version'], diff -rN -u old-ticket1329/src/allmydata/test/test_system.py new-ticket1329/src/allmydata/test/test_system.py --- old-ticket1329/src/allmydata/test/test_system.py 2011-01-17 23:54:16.000000000 -0700 +++ new-ticket1329/src/allmydata/test/test_system.py 2011-01-17 23:54:16.000000000 -0700 @@ -1769,8 +1769,7 @@ if env is None: env = os.environ - d = utils.getProcessOutputAndValue(sys.executable, args=[bintahoe] + argv, - env=env) + d = utils.getProcessOutputAndValue(bintahoe, argv, env=env) return d def _test_checker(self, res): ```
zooko commented 2011-01-18 07:26:18 +00:00
Author
Owner
Copy link

Attachment 1329.diff (26465 bytes) added

**Attachment** 1329.diff (26465 bytes) added
1329.diff
26 KiB
davidsarah commented 2011-01-18 07:31:54 +00:00
Author
Owner
Copy link

Yes, yes, I know these patches need refactoring! It will be easy to apply the #1329 fix after I've done that; it makes no sense to apply it first (because the relevant code has been moved to a different file).

Yes, yes, I know these patches need refactoring! It will be easy to apply the #1329 fix after I've done that; it makes no sense to apply it first (because the relevant code has been moved to a different file).
david-sarah@jacaranda.org commented 2011-01-19 07:04:37 +00:00
Author
Owner
Copy link

In changeset:8d2983c49348e4e7:

Update foolscap requirement to >= 0.6.1. fixes #1329
In changeset:8d2983c49348e4e7: ``` Update foolscap requirement to >= 0.6.1. fixes #1329 ```
tahoe-lafs added the
fixed
 label 2011-01-19 07:04:37 +00:00
david-sarah@jacaranda.org closed this issue 2011-01-19 07:04:37 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
No results found.
No results found.
Labels
Clear labels   
0.2.0

   
0.3.0

   
0.4.0

   
0.5.0

   
0.5.1

   
0.6.0

   
0.6.1

   
0.7.0

   
0.8.0

   
0.9.0

   
1.0.0

   
1.1.0

   
1.10.0

   
1.10.1

   
1.10.2

   
1.10a2

   
1.11.0

   
1.12.0

   
1.12.1

   
1.13.0

   
1.14.0

   
1.15.0

   
1.15.1

   
1.2.0

   
1.3.0

   
1.4.1

   
1.5.0

   
1.6.0

   
1.6.1

   
1.7.0

   
1.7.1

   
1.7β

   
1.8.0

   
1.8.1

   
1.8.2

   
1.8.3

   
1.8β

   
1.9.0

   
1.9.0-s3branch

   
1.9.0a1

   
1.9.0a2

   
1.9.0b1

   
1.9.1

   
1.9.2

   
1.9.2a1

   
LeastAuthority.com automation

   
blocker

   
cannot reproduce

   
cloud-branch

   
code

catch-all for anything that can be fixed by writing code but isn't classified better

  
code-dirnodes

anything involving tahoe directories

  
code-encoding

immutable-file encoding format/layout, cap format, encoding/decoding algorithms

  
code-frontend

general frontend things: new user-facing interfaces, FTP/SFTP servers, FUSE-like interfaces

  
code-frontend-cli

CLI tools

  
code-frontend-ftp-sftp

   
code-frontend-magic-folder

   
code-frontend-web

webapi interfaces, user-facing "WUI" interfaces

  
code-mutable

anything involving mutable files: encoding/decoding, share format

  
code-network

Introducer, locating storage servers, establishing/maintaining connections to them

  
code-nodeadmin

node startup/shutdown, tools to create nodes, logging tools, management/monitoring tools, manhole, disk-space-limiting tools

  
code-peerselection

server selection/sorting algorithm, rebalancing code, allocation policy, swarming-download

  
code-storage

storage-server implementation, share-storage directory layout, client-to-server share transfer protocol

  
contrib

problems with non-core things that live in the source:contrib/ directory

  
critical

   
defect

   
dev-infrastructure

things that help us build Tahoe: buildbot, automated speed/memory/code-coverage tests and results-displayers, IRC channels, mailing lists, this Trac instance

  
documentation

docs items. Also use 'docs' keyword on tickets in more specific components.

  
duplicate

   
enhancement

   
fixed

   
invalid

   
major

   
minor

   
n/a

   
normal

   
operational

things involved in running a tahoe grid, external tools to monitor/maintain a grid (like "nodeadmin" but larger-scale). Was also used for issues with the allmydata.com grid.

  
packaging

setup.py script, library dependencies, binary packages, licensing, installation instructions

  
somebody else's problem

   
supercritical

   
task

   
trivial

   
unknown

Uncategorized. All tickets default to this component until someone updates it.

  
was already fixed

   
website

things about the tahoe-lafs.org website (and sometimes this Trac instance, shared with 'dev-infrastructure')

  
wontfix

   
worksforme
No labels
0.2.0
0.3.0
0.4.0
0.5.0
0.5.1
0.6.0
0.6.1
0.7.0
0.8.0
0.9.0
1.0.0
1.1.0
1.10.0
1.10.1
1.10.2
1.10a2
1.11.0
1.12.0
1.12.1
1.13.0
1.14.0
1.15.0
1.15.1
1.2.0
1.3.0
1.4.1
1.5.0
1.6.0
1.6.1
1.7.0
1.7.1
1.7β
1.8.0
1.8.1
1.8.2
1.8.3
1.8β
1.9.0
1.9.0-s3branch
1.9.0a1
1.9.0a2
1.9.0b1
1.9.1
1.9.2
1.9.2a1
LeastAuthority.com automation
blocker
cannot reproduce
cloud-branch
code
code-dirnodes
code-encoding
code-frontend
code-frontend-cli
code-frontend-ftp-sftp
code-frontend-magic-folder
code-frontend-web
code-mutable
code-network
code-nodeadmin
code-peerselection
code-storage
contrib
critical
defect
dev-infrastructure
documentation
duplicate
enhancement
fixed
invalid
major
minor
n/a
normal
operational
packaging
somebody else's problem
supercritical
task
trivial
unknown
was already fixed
website
wontfix
worksforme
Milestone
Clear milestone
No items
No milestone
1.8.2
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: tahoe-lafs/trac-2024-07-25#1329
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?