tahoe-lafs/trac

'DeprecationWarning: the sets module is deprecated' from foolscap causes failure in test_client_no_noise #1329

New issue

Closed

opened 2011-01-16 05:52:24 +00:00 by daira · 8 comments

daira commented

2011-01-16 05:52:24 +00:00

(http://tahoe-lafs.org/buildbot/builders/Brian%20ubuntu-i386%20linode/builds/133/steps/test/logs/stdio)

[FAIL]: allmydata.test.test_runner.RunNode.test_client_no_noise

Traceback (most recent call last):
  File "/home/buildbot/bb-tahoe/brian-linode/build/src/allmydata/test/test_runner.py", line 442, in _cb2
    and "from pkg_resources import load_entry_point" not in line)], errstr)
twisted.trial.unittest.FailTest: cc=0, OUT: '', ERR: '/home/buildbot/bb-tahoe/brian-linode/build/support/lib/python2.6/site-packages/foolscap-0.6.0-py2.6.egg/foolscap/banana.py:2: DeprecationWarning: the sets module is deprecated
  import struct, sets, time
'

(http://tahoe-lafs.org/buildbot/builders/Brian%20ubuntu-i386%20linode/builds/133/steps/test/logs/stdio) ``` [FAIL]: allmydata.test.test_runner.RunNode.test_client_no_noise Traceback (most recent call last): File "/home/buildbot/bb-tahoe/brian-linode/build/src/allmydata/test/test_runner.py", line 442, in _cb2 and "from pkg_resources import load_entry_point" not in line)], errstr) twisted.trial.unittest.FailTest: cc=0, OUT: '', ERR: '/home/buildbot/bb-tahoe/brian-linode/build/support/lib/python2.6/site-packages/foolscap-0.6.0-py2.6.egg/foolscap/banana.py:2: DeprecationWarning: the sets module is deprecated import struct, sets, time ' ```

daira added the

labels 2011-01-16 05:52:24 +00:00

daira added this to the soon milestone 2011-01-16 05:52:24 +00:00

daira commented

2011-01-16 20:32:12 +00:00

Author

In [4965/ticket1306]:

src/allmydata/_auto_deps.py: Fix 'DeprecationWarning: the sets module is deprecated'. refs #1329

In [4965/ticket1306]: ``` src/allmydata/_auto_deps.py: Fix 'DeprecationWarning: the sets module is deprecated'. refs #1329 ```

daira modified the milestone from soon to 1.8.2

2011-01-16 21:23:57 +00:00

warner commented

2011-01-17 05:41:12 +00:00

I just released foolscap-0.6.1, which removes the offending import sets, so this ticket can be resolved by increasing Tahoe's requirement to foolscap-0.6.1 .

I just released foolscap-0.6.1, which removes the offending `import sets`, so this ticket can be resolved by increasing Tahoe's requirement to foolscap-0.6.1 .

zooko commented

2011-01-17 06:45:25 +00:00

-0 from me -- I would rather suppress the warning the way [20110116200112-93fa1-0397f85493a9fd0ae82ce58d7291d598a4f6e2d6]/ticket1306 attempts to do or change the test to tolerate the noise than raise the dependency requirement. But, it is just a -0. If you prefer it, go ahead.

warner commented

2011-01-17 06:51:15 +00:00

my main problem with [20110116200112-93fa1-0397f85493a9fd0ae82ce58d7291d598a4f6e2d6]/ticket1306 is that it involved what looked like a signficant refactoring of _auto_deps.py, so I couldn't apply just a single patch (there were more like six). But if there's a clean way to use that approach, I'm happy to go with it.

The clock is ticking, though :)

my main problem with [20110116200112-93fa1-0397f85493a9fd0ae82ce58d7291d598a4f6e2d6]/ticket1306 is that it involved what looked like a signficant refactoring of `_auto_deps.py`, so I couldn't apply just a single patch (there were more like six). But if there's a clean way to use that approach, I'm happy to go with it. The clock is ticking, though :)

zooko commented

2011-01-18 07:25:52 +00:00

Replying to warner:

my main problem with [20110116200112-93fa1-0397f85493a9fd0ae82ce58d7291d598a4f6e2d6]/ticket1306 is that it involved what looked like a signficant refactoring of _auto_deps.py, so I couldn't apply just a single patch (there were more like six). But if there's a clean way to use that approach, I'm happy to go with it.

For the record, here are the patches that (from darcs's perspective) are required for [20110116200112-93fa1-0397f85493a9fd0ae82ce58d7291d598a4f6e2d6]/ticket1306:

HACL Zooko-Ofsimplegeos-MacBook-Pro:~/playground/tahoe-lafs/ticket1329$ time darcs push zooko@dev.allmydata.org:/home/darcs/tahoe-lafs/trunk
Pushing to "zooko@dev.allmydata.org:/home/darcs/tahoe-lafs/trunk"...
The remote repository has 2 patches to pull.
Sat Jan  1 04:01:41 MST 2011  david-sarah@jacaranda.org
  * allmydata/__init__.py: don't use pkg_resources.require to get versions and paths. Also change pyOpenSSL to pyopenssl since that is the package name. fixes #1287
Shall I push this patch? (1/7)  [ynWsfvplxdaqjk], or ? for help: w
Sat Jan  1 13:05:02 MST 2011  david-sarah@jacaranda.org
  * allmydata/_auto_deps.py: add a comment reminding to change __init__.py when dependencies are added or removed. Clarify a comment in __init__.py. refs #1287
Shall I push this patch? (2/7)  [ynWvplxdaqjk], or ? for help: w
Sat Jan  1 12:25:59 MST 2011  david-sarah@jacaranda.org
  * test_runner: add test_import_in_repl, which uses 'bin/tahoe debug repl' to import the allmydata module and checks that it comes from the right path. Also, fix a latent bug that caused test_the_right_code to incorrectly conclude that a path mismatch was due to a Unicode path (causing a skip rather than a failure). This version of the patch avoids a confusing shadowing of 'srcfile'. refs #1258
Shall I push this patch? (3/7)  [ynWsfvplxdaqjk], or ? for help: w
Wed Jan 12 21:24:57 MST 2011  david-sarah@jacaranda.org
  * Make bb-freeze work. This also implements more robust checking of dependency requirements, as discussed in ticket:585#comment:10 . refs #585
Shall I push this patch? (4/7)  [ynWvplxdaqjk], or ? for help: w
Wed Jan 12 21:30:40 MST 2011  david-sarah@jacaranda.org
  * Move get_package_versions_and_locations from __init__.py to _auto_deps.py (because it needs to be updated when dependencies change). refs #585
Shall I push this patch? (5/7)  [ynWvplxdaqjk], or ? for help: w
Fri Jan 14 22:02:45 MST 2011  david-sarah@jacaranda.org
  * Add mock to get_package_versions_and_locations.
Shall I push this patch? (6/7)  [ynWsfvplxdaqjk], or ? for help: w
Sun Jan 16 13:01:12 MST 2011  david-sarah@jacaranda.org
  * src/allmydata/_auto_deps.py: Fix 'DeprecationWarning: the sets module is deprecated'. refs #1329
Shall I push this patch? (7/7)  [ynWsfvplxdaqjk], or ? for help: w

And below and attached is the unified diff of all seven of them coalesced.

I propose that we start by applying the first one -- see #1258.

diff -rN -u old-ticket1329/Makefile new-ticket1329/Makefile
--- old-ticket1329/Makefile	2011-01-17 23:54:16.000000000 -0700
+++ new-ticket1329/Makefile	2011-01-17 23:54:16.000000000 -0700
@@ -179,7 +179,7 @@
 
 
 pyflakes:
-	$(PYTHON) -OOu `which pyflakes` src/allmydata |sort |uniq
+	$(PYTHON) -OOu `which pyflakes` src/allmydata static |sort |uniq
 check-umids:
 	$(PYTHON) misc/coding_tools/check-umids.py `find src/allmydata -name '*.py'`
 
diff -rN -u old-ticket1329/src/allmydata/__init__.py new-ticket1329/src/allmydata/__init__.py
--- old-ticket1329/src/allmydata/__init__.py	2011-01-17 23:54:16.000000000 -0700
+++ new-ticket1329/src/allmydata/__init__.py	2011-01-17 23:54:16.000000000 -0700
@@ -4,57 +4,6 @@
 community web site: U{http://tahoe-lafs.org/}
 """
 
-# We want to call require_auto_deps() before other imports, because the setuptools
-# docs claim that if a distribution is installed with --multi-version, it might not
-# be importable until after pkg_resources.require() has been called for it. We don't
-# have an example of this happening at this time. It is possible that require() isn't
-# actually needed because we set __requires__ in the generated startup script, but
-# that would be an undocumented property of the setuptools implementation.
-
-from allmydata import _auto_deps
-_auto_deps.require_auto_deps()
-
-# This is just to suppress DeprecationWarnings from nevow and twisted.
-# See http://allmydata.org/trac/tahoe/ticket/859 and
-# http://divmod.org/trac/ticket/2994 .
-import warnings
-warnings.filterwarnings("ignore", category=DeprecationWarning,
-    message="the sha module is deprecated; use the hashlib module instead",
-    append=True)
-warnings.filterwarnings("ignore", category=DeprecationWarning,
-    message="object.__new__\(\) takes no parameters",
-    append=True)
-warnings.filterwarnings("ignore", category=DeprecationWarning,
-    message="The popen2 module is deprecated.  Use the subprocess module.",
-    append=True)
-warnings.filterwarnings("ignore", category=DeprecationWarning,
-    message="the md5 module is deprecated; use hashlib instead",
-    append=True)
-warnings.filterwarnings("ignore", category=DeprecationWarning,
-    message="twisted.web.error.NoResource is deprecated since Twisted 9.0.  See twisted.web.resource.NoResource.",
-    append=True)
-try:
-    import nevow
-    from twisted.persisted import sob
-    from twisted.python import filepath
-    hush_pyflakes = (nevow, sob, filepath)
-    del hush_pyflakes
-finally:
-    warnings.filters.pop()
-    warnings.filters.pop()
-    warnings.filters.pop()
-    warnings.filters.pop()
-    # Don't pop the filter for the sha module warning because it is also generated
-    # by pycrypto (which we don't want to import unless needed).
-    # warnings.filters.pop()
-
-# This warning is generated by twisted, PyRex, and possibly other packages,
-# but can happen at any time, not only when they are imported. See
-# http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1129 .
-warnings.filterwarnings("ignore", category=DeprecationWarning,
-    message="BaseException.message has been deprecated as of Python 2.6",
-    append=True)
-
 __version__ = "unknown"
 try:
     from allmydata._version import __version__
@@ -175,88 +124,93 @@
     else:
         return platform.platform()
 
-def get_package_versions_from_setuptools():
-    import pkg_resources
-    return dict([(p.project_name, (p.version, p.location)) for p in pkg_resources.require(__appname__)])
-
-def package_dir(srcfile):
-    return os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile))))
-
-def get_package_versions_and_locations():
-    # because there are a few dependencies that are outside setuptools's ken
-    # (Python and platform, and sqlite3 if you are on Python >= 2.5), and
-    # because setuptools might fail to find something even though import
-    # finds it:
-    import OpenSSL, allmydata, foolscap.api, nevow, platform, pycryptopp, setuptools, simplejson, twisted, zfec, zope.interface
-    pysqlitever = None
-    pysqlitefile = None
-    sqlitever = None
-    try:
-        import sqlite3
-    except ImportError:
-        try:
-            from pysqlite2 import dbapi2
-        except ImportError:
-            pass
-        else:
-            pysqlitever = dbapi2.version
-            pysqlitefile = package_dir(dbapi2.__file__)
-            sqlitever = dbapi2.sqlite_version
-    else:
-        pysqlitever = sqlite3.version
-        pysqlitefile = package_dir(sqlite3.__file__)
-        sqlitever = sqlite3.sqlite_version
-
-    d1 = {
-        'pyOpenSSL': (OpenSSL.__version__, package_dir(OpenSSL.__file__)),
-        __appname__: (allmydata.__version__, package_dir(allmydata.__file__)),
-        'foolscap': (foolscap.api.__version__, package_dir(foolscap.__file__)),
-        'Nevow': (nevow.__version__, package_dir(nevow.__file__)),
-        'pycryptopp': (pycryptopp.__version__, package_dir(pycryptopp.__file__)),
-        'setuptools': (setuptools.__version__, package_dir(setuptools.__file__)),
-        'simplejson': (simplejson.__version__, package_dir(simplejson.__file__)),
-        'pysqlite': (pysqlitever, pysqlitefile),
-        'sqlite': (sqlitever, 'unknown'),
-        'zope.interface': ('unknown', package_dir(zope.interface.__file__)),
-        'Twisted': (twisted.__version__, package_dir(twisted.__file__)),
-        'zfec': (zfec.__version__, package_dir(zfec.__file__)),
-        'python': (platform.python_version(), sys.executable),
-        'platform': (get_platform(), None),
-        }
 
-    # But we prefer to get all the dependencies as known by setuptools:
-    import pkg_resources
-    try:
-        d2 = get_package_versions_from_setuptools()
-    except pkg_resources.DistributionNotFound:
-        # See docstring in _auto_deps.require_auto_deps() to explain why it makes sense to ignore this exception.
-        pass
+from allmydata._auto_deps import get_package_versions_and_locations
+
+_vers_and_locs = get_package_versions_and_locations()
+_vers_and_locs['platform'] = (get_platform(), None)
+
+
+def check_all_requirements():
+    import platform
+    from distutils.version import LooseVersion
+    from allmydata._auto_deps import install_requires
+
+    # we require 2.4.4 on non-UCS-2, non-Redhat builds to avoid <http://www.python.org/news/security/PSF-2006-001/>
+    # we require 2.4.3 on non-UCS-2 Redhat, because 2.4.3 is common on Redhat-based distros and will have patched the above bug
+    # we require at least 2.4.2 in any case to avoid a bug in the base64 module: <http://bugs.python.org/issue1171487>
+    if sys.maxunicode == 65535:
+        if sys.version_info < (2, 4, 2) or sys.version_info[0] > 2:
+            raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.2 or greater "
+                                      "for a UCS-2 build (but less than v3), not %r" %
+                                      (sys.version_info,))
+    elif platform.platform().lower().find('redhat') >= 0:
+        if sys.version_info < (2, 4, 3) or sys.version_info[0] > 2:
+            raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.3 or greater "
+                                      "on Redhat-based distributions (but less than v3), not %r" %
+                                      (sys.version_info,))
     else:
-        d1.update(d2)
+        if sys.version_info < (2, 4, 4) or sys.version_info[0] > 2:
+            raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.4 or greater "
+                                      "for a non-UCS-2 build (but less than v3), not %r" %
+                                      (sys.version_info,))
+
+    def check_requirement(req):
+        # We only support a single >= or ==.
+        def _check(s, ok, relation):
+            name = s[0].strip(' ').partition('[')[0]
+            required = s[1].strip(' ')
+            if name not in _vers_and_locs:
+                raise AssertionError("no version info for %s" % (name,))
+            (actual, location) = _vers_and_locs[name]
+            actual = str(actual)
+            if actual != 'unknown' and not ok(LooseVersion(actual), LooseVersion(required)):
+                msg = ("We require %s version %s of %s, but could only find version %s.\n"
+                      % (relation, required, name, actual))
+                if location and location != 'unknown':
+                    msg += "The version we found is from %r.\n" % (location,)
+                msg += ("To resolve this problem, uninstall that version, either using your\n"
+                        "operating system's package manager or by moving aside the directory.")
+                raise AssertionError(msg)
+
+        s = req.split('>=')
+        if len(s) == 2:
+            _check(s, lambda x, y: x >= y, "at least")
+            return
+        s = req.split('==')
+        if len(s) == 2:
+            _check(s, lambda x, y: x == y, "exactly")
+            return
+        if req not in _vers_and_locs:
+            raise AssertionError("no version info or could not understand requirement for %r" % (req))
+
+    for requirement in install_requires:
+        check_requirement(requirement)
+
+check_all_requirements()
 
-    return d1
 
 def get_package_versions():
-    return dict([(k, v) for k, (v, l) in get_package_versions_and_locations().iteritems()])
+    return dict([(k, v) for k, (v, l) in _vers_and_locs.iteritems()])
 
 def get_package_locations():
-    return dict([(k, l) for k, (v, l) in get_package_versions_and_locations().iteritems()])
+    return dict([(k, l) for k, (v, l) in _vers_and_locs.iteritems()])
 
 def get_package_versions_string(show_paths=False):
-    vers_and_locs = get_package_versions_and_locations()
     res = []
-    for p in [__appname__, "foolscap", "pycryptopp", "zfec", "Twisted", "Nevow", "zope.interface", "python", "platform"]:
-        (ver, loc) = vers_and_locs.get(p, ('UNKNOWN', 'UNKNOWN'))
+    first = [__appname__, "foolscap", "pycryptopp", "zfec", "Twisted", "Nevow", "zope.interface", "python", "platform"]
+    for p in first:
+        (ver, loc) = _vers_and_locs.get(p, ('UNKNOWN', 'UNKNOWN'))
         info = str(p) + ": " + str(ver)
         if show_paths:
             info = info + " (%s)" % str(loc)
         res.append(info)
-        if vers_and_locs.has_key(p):
-            del vers_and_locs[p]
 
-    for p, (v, loc) in vers_and_locs.iteritems():
-        info = str(p) + ": " + str(v)
-        if show_paths:
-            info = info + " (%s)" % str(loc)
-        res.append(info)
+    for p, (v, loc) in _vers_and_locs.iteritems():
+        if p not in first:
+            info = str(p) + ": " + str(v)
+            if show_paths:
+                info = info + " (%s)" % str(loc)
+            res.append(info)
+
     return ', '.join(res)
diff -rN -u old-ticket1329/src/allmydata/_auto_deps.py new-ticket1329/src/allmydata/_auto_deps.py
--- old-ticket1329/src/allmydata/_auto_deps.py	2011-01-17 23:54:16.000000000 -0700
+++ new-ticket1329/src/allmydata/_auto_deps.py	2011-01-17 23:54:16.000000000 -0700
@@ -1,9 +1,10 @@
 # Note: do not import any module from Tahoe-LAFS itself in this
 # file. Also please avoid importing modules from other packages than
-# the Python Standard Library if at all possible (exception: we rely
-# on importing pkg_resources, which is provided by setuptools,
-# zetuptoolz, distribute, and perhaps in the future distutils2, for
-# the require_auto_deps() function.)
+# the Python Standard Library if at all possible. That includes
+# setuptools and pkg_resources.
+
+# If you add or remove dependencies, please remember to update
+# get_package_versions_and_locations() below.
 
 install_requires=[
                   # we require newer versions of setuptools (actually
@@ -27,7 +28,7 @@
 
                   # Needed for SFTP. pyasn1 is needed by twisted.conch in Twisted >= 9.0.
                   # pycrypto 2.2 doesn't work due to https://bugs.launchpad.net/pycrypto/+bug/620253
-                  "pycrypto == 2.0.1, == 2.1, >= 2.3",
+                  "pycrypto >= 2.3",
                   "pyasn1 >= 0.0.8a",
 
                   # Will be needed to test web apps, but not yet. See #1001.
@@ -74,49 +75,105 @@
     install_requires=[]
 del sys # clean up namespace
 
-def require_python_version():
-    import sys, platform
 
-    # we require 2.4.4 on non-UCS-2, non-Redhat builds to avoid <http://www.python.org/news/security/PSF-2006-001/>
-    # we require 2.4.3 on non-UCS-2 Redhat, because 2.4.3 is common on Redhat-based distros and will have patched the above bug
-    # we require at least 2.4.2 in any case to avoid a bug in the base64 module: <http://bugs.python.org/issue1171487>
-    if sys.maxunicode == 65535:
-        if sys.version_info < (2, 4, 2) or sys.version_info[0] > 2:
-            raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.2 or greater "
-                                      "for a UCS-2 build (but less than v3), not %r" %
-                                      (sys.version_info,))
-    elif platform.platform().lower().find('redhat') >= 0:
-        if sys.version_info < (2, 4, 3) or sys.version_info[0] > 2:
-            raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.3 or greater "
-                                      "on Redhat-based distributions (but less than v3), not %r" %
-                                      (sys.version_info,))
+def get_package_versions_and_locations():
+    import warnings, os, sys
+    def package_dir(srcfile):
+        return os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile))))
+
+    # pkg_resources.require returns the distribution that pkg_resources attempted to put
+    # on sys.path, which can differ from the one that we actually import due to #1258,
+    # or any other bug that causes sys.path to be set up incorrectly. Therefore we
+    # must import the packages in order to check their versions and paths. Note that
+    # this needs to be updated if dependencies are added or removed.
+
+    # This warning is generated by twisted, PyRex, and possibly other packages,
+    # but can happen at any time, not only when they are imported. See
+    # http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1129 .
+    warnings.filterwarnings("ignore", category=DeprecationWarning,
+        message="BaseException.message has been deprecated as of Python 2.6",
+        append=True)
+
+    # This is to suppress DeprecationWarnings from nevow, twisted, and pycrypto.
+    # See http://allmydata.org/trac/tahoe/ticket/859 and
+    # http://divmod.org/trac/ticket/2994 .
+    warnings.filterwarnings("ignore", category=DeprecationWarning,
+        message="the sha module is deprecated; use the hashlib module instead",
+        append=True)
+    warnings.filterwarnings("ignore", category=DeprecationWarning,
+        message="object.__new__\(\) takes no parameters",
+        append=True)
+    warnings.filterwarnings("ignore", category=DeprecationWarning,
+        message="The popen2 module is deprecated.  Use the subprocess module.",
+        append=True)
+    warnings.filterwarnings("ignore", category=DeprecationWarning,
+        message="the md5 module is deprecated; use hashlib instead",
+        append=True)
+    warnings.filterwarnings("ignore", category=DeprecationWarning,
+        message="twisted.web.error.NoResource is deprecated since Twisted 9.0.  See twisted.web.resource.NoResource.",
+        append=True)
+    warnings.filterwarnings("ignore", category=DeprecationWarning,
+        message="the sets module is deprecated",
+        append=True)
+    try:
+        import nevow
+        from twisted.persisted import sob
+        from twisted.python import filepath
+        import Crypto
+        import foolscap.api
+        [sob, filepath]  # hush pyflakes
+    finally:
+        for n in range(6):
+            warnings.filters.pop()
+
+    import OpenSSL, allmydata, platform, pycryptopp, simplejson, twisted, zfec, zope.interface, pyasn1, mock
+    pysqlitever = None
+    pysqlitefile = None
+    sqlitever = None
+    try:
+        import sqlite3
+    except ImportError:
+        try:
+            from pysqlite2 import dbapi2
+        except ImportError:
+            pass
+        else:
+            pysqlitever = dbapi2.version
+            pysqlitefile = package_dir(dbapi2.__file__)
+            sqlitever = dbapi2.sqlite_version
     else:
-        if sys.version_info < (2, 4, 4) or sys.version_info[0] > 2:
-            raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.4 or greater "
-                                      "for a non-UCS-2 build (but less than v3), not %r" %
-                                      (sys.version_info,))
-
-def require_auto_deps():
-    """
-    The purpose of this function is to raise a pkg_resources exception if any of the
-    requirements can't be imported.  This is just to give earlier and more explicit error
-    messages, as opposed to waiting until the source code tries to import some module from one
-    of these packages and gets an ImportError.  This function gets called from
-    src/allmydata/__init__.py .
-    """
-    require_python_version()
+        pysqlitever = sqlite3.version
+        pysqlitefile = package_dir(sqlite3.__file__)
+        sqlitever = sqlite3.sqlite_version
+
+    packages = {
+        'pyopenssl': (OpenSSL.__version__, package_dir(OpenSSL.__file__)),
+        allmydata.__appname__: (allmydata.__version__, package_dir(allmydata.__file__)),
+        'foolscap': (foolscap.api.__version__, package_dir(foolscap.__file__)),
+        'Nevow': (nevow.__version__, package_dir(nevow.__file__)),
+        'pycryptopp': (pycryptopp.__version__, package_dir(pycryptopp.__file__)),
+        'simplejson': (simplejson.__version__, package_dir(simplejson.__file__)),
+        'pysqlite': (pysqlitever, pysqlitefile),
+        'sqlite': (sqlitever, 'unknown'),
+        'zope.interface': ('unknown', package_dir(zope.interface.__file__)),
+        'Twisted': (twisted.__version__, package_dir(twisted.__file__)),
+        'zfec': (zfec.__version__, package_dir(zfec.__file__)),
+        'pycrypto': (Crypto.__version__, package_dir(Crypto.__file__)),
+        'pyasn1': ('unknown', package_dir(pyasn1.__file__)),
+        'mock': (mock.__version__, package_dir(mock.__file__)),
+        'python': (platform.python_version(), sys.executable),
+        }
 
-    import pkg_resources
-    for requirement in install_requires:
+    if sys.platform == 'win32':
         try:
-            pkg_resources.require(requirement)
-        except pkg_resources.DistributionNotFound:
-            # there is no .egg-info present for this requirement, which
-            # either means that it isn't installed, or it is installed in a
-            # way that pkg_resources can't find it (but regular python
-            # might).  There are several older Linux distributions which
-            # provide our dependencies just fine, but they don't ship
-            # .egg-info files. Note that if there *is* an .egg-info file,
-            # but it shows a too-old version, then we'll get a
-            # VersionConflict error instead of DistributionNotFound.
+            import win32api
+            packages['pywin32'] = ('unknown', package_dir(win32api.__file__))
+        except ImportError:
             pass
+
+    if not hasattr(sys, 'frozen'):
+        # Don't try to get the setuptools version if using bbfreeze, py2exe etc. See #585.
+        import setuptools
+        packages['setuptools'] = (setuptools.__version__, package_dir(setuptools.__file__))
+
+    return packages
diff -rN -u old-ticket1329/src/allmydata/test/test_runner.py new-ticket1329/src/allmydata/test/test_runner.py
--- old-ticket1329/src/allmydata/test/test_runner.py	2011-01-17 23:54:16.000000000 -0700
+++ new-ticket1329/src/allmydata/test/test_runner.py	2011-01-17 23:54:16.000000000 -0700
@@ -3,7 +3,7 @@
 
 from twisted.python import usage, runtime
 from twisted.internet import utils
-import os.path, re, sys
+import os.path, re, sys, subprocess
 from cStringIO import StringIO
 from allmydata.util import fileutil, pollmixin
 from allmydata.util.encodingutil import unicode_to_argv, unicode_to_output, get_filesystem_encoding
@@ -14,24 +14,36 @@
 
 timeout = 240
 
+def get_root_from_file(src):
+    srcdir = os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(src))))
+
+    root = os.path.dirname(srcdir)
+    if os.path.basename(srcdir) == 'site-packages':
+        if re.search(r'python.+\..+', os.path.basename(root)):
+            root = os.path.dirname(root)
+        root = os.path.dirname(root)
+    elif os.path.basename(root) == 'src':
+        root = os.path.dirname(root)
+
+    return root
+
 srcfile = allmydata.__file__
-srcdir = os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile))))
+rootdir = get_root_from_file(srcfile)
 
-rootdir = os.path.dirname(srcdir)
-if os.path.basename(srcdir) == 'site-packages':
-    if re.search(r'python.+\..+', os.path.basename(rootdir)):
-        rootdir = os.path.dirname(rootdir)
-    rootdir = os.path.dirname(rootdir)
-elif os.path.basename(rootdir) == 'src':
-    rootdir = os.path.dirname(rootdir)
-
-bintahoe = os.path.join(rootdir, 'bin', 'tahoe')
-if sys.platform == "win32":
-    bintahoe += ".pyscript"
-    if not os.path.exists(bintahoe):
-       alt_bintahoe = os.path.join(rootdir, 'Scripts', 'tahoe.pyscript')
-       if os.path.exists(alt_bintahoe):
-           bintahoe = alt_bintahoe
+
+if hasattr(sys, 'frozen'):
+    bintahoe = os.path.join(rootdir, 'tahoe')
+
+    if sys.platform == "win32" and os.path.exists(bintahoe + '.exe'):
+        bintahoe += '.exe'
+else:
+    bintahoe = os.path.join(rootdir, 'bin', 'tahoe')
+    if sys.platform == "win32":
+        bintahoe += '.pyscript'
+        if not os.path.exists(bintahoe):
+            alt_bintahoe = os.path.join(rootdir, 'Scripts', 'tahoe.pyscript')
+            if os.path.exists(alt_bintahoe):
+                bintahoe = alt_bintahoe
 
 
 class SkipMixin:
@@ -49,16 +61,20 @@
 
 
 class BinTahoe(common_util.SignalMixin, unittest.TestCase, SkipMixin):
-    def test_the_right_code(self):
+    def _check_right_code(self, file_to_check):
+        root_to_check = get_root_from_file(file_to_check)
+        if os.path.basename(root_to_check) == 'dist':
+            root_to_check = os.path.dirname(root_to_check)
+
         cwd = os.path.normcase(os.path.realpath("."))
         root_from_cwd = os.path.dirname(cwd)
         if os.path.basename(root_from_cwd) == 'src':
             root_from_cwd = os.path.dirname(root_from_cwd)
 
-        same = (root_from_cwd == rootdir)
+        same = (root_from_cwd == root_to_check)
         if not same:
             try:
-                same = os.path.samefile(root_from_cwd, rootdir)
+                same = os.path.samefile(root_from_cwd, root_to_check)
             except AttributeError, e:
                 e  # hush pyflakes
 
@@ -66,9 +82,9 @@
             msg = ("We seem to be testing the code at %r,\n"
                    "(according to the source filename %r),\n"
                    "but expected to be testing the code at %r.\n"
-                   % (rootdir, srcfile, root_from_cwd))
+                   % (root_to_check, file_to_check, root_from_cwd))
 
-            root_from_cwdu = os.path.normcase(os.path.normpath(os.getcwdu()))
+            root_from_cwdu = os.path.dirname(os.path.normcase(os.path.normpath(os.getcwdu())))
             if os.path.basename(root_from_cwdu) == u'src':
                 root_from_cwdu = os.path.dirname(root_from_cwdu)
 
@@ -81,6 +97,24 @@
                 msg += "Please run the tests from the root of the Tahoe-LAFS distribution."
                 self.fail(msg)
 
+    def test_the_right_code(self):
+        self._check_right_code(srcfile)
+
+    def test_import_in_repl(self):
+        self.skip_if_cannot_run_bintahoe()
+
+        args = [bintahoe, "debug", "repl"]
+        if not hasattr(sys, 'frozen'):
+            args = [sys.executable] + args
+
+        p = subprocess.Popen(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        (out, err) = p.communicate("import allmydata; print allmydata.__file__")
+
+        self.failUnlessEqual(p.returncode, 0)
+        first = out.splitlines()[0]
+        self.failUnless(first.startswith('>>> '))
+        self._check_right_code(first[4:])
+
     def test_path(self):
         self.skip_if_cannot_run_bintahoe()
         d = utils.getProcessOutputAndValue(bintahoe, args=["--version-and-path"], env=os.environ)
@@ -112,6 +146,7 @@
                 else:
                     altverstr = verstr
 
+            srcdir = os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile))))
             required_ver_and_path = "%s: %s (%s)" % (allmydata.__appname__, verstr, srcdir)
             alt_required_ver_and_path = "%s: %s (%s)" % (allmydata.__appname__, altverstr, srcdir)
 
@@ -121,6 +156,8 @@
 
     def test_unicode_arguments_and_output(self):
         self.skip_if_cannot_run_bintahoe()
+        if hasattr(sys, 'frozen') and sys.platform == "win32":
+            raise unittest.SkipTest("This test can't currently be made to work for frozen executables on Windows,\ndue to lack of support for Unicode in twisted.internet.utils.getProcessOutputAndValue.")
 
         tricky = u"\u2621"
         try:
@@ -139,6 +176,8 @@
 
     def test_run_with_python_options(self):
         self.skip_if_cannot_run_bintahoe()
+        if hasattr(sys, 'frozen'):
+            raise unittest.SkipTest("This test doesn't apply to frozen executables.")
 
         # -t is a harmless option that warns about tabs.
         d = utils.getProcessOutputAndValue(sys.executable, args=['-t', bintahoe, '--version'],
diff -rN -u old-ticket1329/src/allmydata/test/test_system.py new-ticket1329/src/allmydata/test/test_system.py
--- old-ticket1329/src/allmydata/test/test_system.py	2011-01-17 23:54:16.000000000 -0700
+++ new-ticket1329/src/allmydata/test/test_system.py	2011-01-17 23:54:16.000000000 -0700
@@ -1769,8 +1769,7 @@
 
         if env is None:
             env = os.environ
-        d = utils.getProcessOutputAndValue(sys.executable, args=[bintahoe] + argv,
-                                           env=env)
+        d = utils.getProcessOutputAndValue(bintahoe, argv, env=env)
         return d
 
     def _test_checker(self, res):

Replying to [warner](/tahoe-lafs/trac/issues/1329#issuecomment-383286): > my main problem with [20110116200112-93fa1-0397f85493a9fd0ae82ce58d7291d598a4f6e2d6]/ticket1306 is that it involved what looked like a signficant refactoring of `_auto_deps.py`, so I couldn't apply just a single patch (there were more like six). But if there's a clean way to use that approach, I'm happy to go with it. For the record, here are the patches that (from darcs's perspective) are required for [20110116200112-93fa1-0397f85493a9fd0ae82ce58d7291d598a4f6e2d6]/ticket1306: ``` HACL Zooko-Ofsimplegeos-MacBook-Pro:~/playground/tahoe-lafs/ticket1329$ time darcs push zooko@dev.allmydata.org:/home/darcs/tahoe-lafs/trunk Pushing to "zooko@dev.allmydata.org:/home/darcs/tahoe-lafs/trunk"... The remote repository has 2 patches to pull. Sat Jan 1 04:01:41 MST 2011 david-sarah@jacaranda.org * allmydata/__init__.py: don't use pkg_resources.require to get versions and paths. Also change pyOpenSSL to pyopenssl since that is the package name. fixes #1287 Shall I push this patch? (1/7) [ynWsfvplxdaqjk], or ? for help: w Sat Jan 1 13:05:02 MST 2011 david-sarah@jacaranda.org * allmydata/_auto_deps.py: add a comment reminding to change __init__.py when dependencies are added or removed. Clarify a comment in __init__.py. refs #1287 Shall I push this patch? (2/7) [ynWvplxdaqjk], or ? for help: w Sat Jan 1 12:25:59 MST 2011 david-sarah@jacaranda.org * test_runner: add test_import_in_repl, which uses 'bin/tahoe debug repl' to import the allmydata module and checks that it comes from the right path. Also, fix a latent bug that caused test_the_right_code to incorrectly conclude that a path mismatch was due to a Unicode path (causing a skip rather than a failure). This version of the patch avoids a confusing shadowing of 'srcfile'. refs #1258 Shall I push this patch? (3/7) [ynWsfvplxdaqjk], or ? for help: w Wed Jan 12 21:24:57 MST 2011 david-sarah@jacaranda.org * Make bb-freeze work. This also implements more robust checking of dependency requirements, as discussed in ticket:585#comment:10 . refs #585 Shall I push this patch? (4/7) [ynWvplxdaqjk], or ? for help: w Wed Jan 12 21:30:40 MST 2011 david-sarah@jacaranda.org * Move get_package_versions_and_locations from __init__.py to _auto_deps.py (because it needs to be updated when dependencies change). refs #585 Shall I push this patch? (5/7) [ynWvplxdaqjk], or ? for help: w Fri Jan 14 22:02:45 MST 2011 david-sarah@jacaranda.org * Add mock to get_package_versions_and_locations. Shall I push this patch? (6/7) [ynWsfvplxdaqjk], or ? for help: w Sun Jan 16 13:01:12 MST 2011 david-sarah@jacaranda.org * src/allmydata/_auto_deps.py: Fix 'DeprecationWarning: the sets module is deprecated'. refs #1329 Shall I push this patch? (7/7) [ynWsfvplxdaqjk], or ? for help: w ``` And below and attached is the unified diff of all seven of them coalesced. I propose that we start by applying the first one -- see #1258. ``` diff -rN -u old-ticket1329/Makefile new-ticket1329/Makefile --- old-ticket1329/Makefile 2011-01-17 23:54:16.000000000 -0700 +++ new-ticket1329/Makefile 2011-01-17 23:54:16.000000000 -0700 @@ -179,7 +179,7 @@ pyflakes: - $(PYTHON) -OOu `which pyflakes` src/allmydata |sort |uniq + $(PYTHON) -OOu `which pyflakes` src/allmydata static |sort |uniq check-umids: $(PYTHON) misc/coding_tools/check-umids.py `find src/allmydata -name '*.py'` diff -rN -u old-ticket1329/src/allmydata/__init__.py new-ticket1329/src/allmydata/__init__.py --- old-ticket1329/src/allmydata/__init__.py 2011-01-17 23:54:16.000000000 -0700 +++ new-ticket1329/src/allmydata/__init__.py 2011-01-17 23:54:16.000000000 -0700 @@ -4,57 +4,6 @@ community web site: U{http://tahoe-lafs.org/} """ -# We want to call require_auto_deps() before other imports, because the setuptools -# docs claim that if a distribution is installed with --multi-version, it might not -# be importable until after pkg_resources.require() has been called for it. We don't -# have an example of this happening at this time. It is possible that require() isn't -# actually needed because we set __requires__ in the generated startup script, but -# that would be an undocumented property of the setuptools implementation. - -from allmydata import _auto_deps -_auto_deps.require_auto_deps() - -# This is just to suppress DeprecationWarnings from nevow and twisted. -# See http://allmydata.org/trac/tahoe/ticket/859 and -# http://divmod.org/trac/ticket/2994 . -import warnings -warnings.filterwarnings("ignore", category=DeprecationWarning, - message="the sha module is deprecated; use the hashlib module instead", - append=True) -warnings.filterwarnings("ignore", category=DeprecationWarning, - message="object.__new__ takes no parameters", - append=True) -warnings.filterwarnings("ignore", category=DeprecationWarning, - message="The popen2 module is deprecated. Use the subprocess module.", - append=True) -warnings.filterwarnings("ignore", category=DeprecationWarning, - message="the md5 module is deprecated; use hashlib instead", - append=True) -warnings.filterwarnings("ignore", category=DeprecationWarning, - message="twisted.web.error.NoResource is deprecated since Twisted 9.0. See twisted.web.resource.NoResource.", - append=True) -try: - import nevow - from twisted.persisted import sob - from twisted.python import filepath - hush_pyflakes = (nevow, sob, filepath) - del hush_pyflakes -finally: - warnings.filters.pop() - warnings.filters.pop() - warnings.filters.pop() - warnings.filters.pop() - # Don't pop the filter for the sha module warning because it is also generated - # by pycrypto (which we don't want to import unless needed). - # warnings.filters.pop() - -# This warning is generated by twisted, PyRex, and possibly other packages, -# but can happen at any time, not only when they are imported. See -# http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1129 . -warnings.filterwarnings("ignore", category=DeprecationWarning, - message="BaseException.message has been deprecated as of Python 2.6", - append=True) - __version__ = "unknown" try: from allmydata._version import __version__ @@ -175,88 +124,93 @@ else: return platform.platform() -def get_package_versions_from_setuptools(): - import pkg_resources - return dict([(p.project_name, (p.version, p.location)) for p in pkg_resources.require(__appname__)]) - -def package_dir(srcfile): - return os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile)))) - -def get_package_versions_and_locations(): - # because there are a few dependencies that are outside setuptools's ken - # (Python and platform, and sqlite3 if you are on Python >= 2.5), and - # because setuptools might fail to find something even though import - # finds it: - import OpenSSL, allmydata, foolscap.api, nevow, platform, pycryptopp, setuptools, simplejson, twisted, zfec, zope.interface - pysqlitever = None - pysqlitefile = None - sqlitever = None - try: - import sqlite3 - except ImportError: - try: - from pysqlite2 import dbapi2 - except ImportError: - pass - else: - pysqlitever = dbapi2.version - pysqlitefile = package_dir(dbapi2.__file__) - sqlitever = dbapi2.sqlite_version - else: - pysqlitever = sqlite3.version - pysqlitefile = package_dir(sqlite3.__file__) - sqlitever = sqlite3.sqlite_version - - d1 = { - 'pyOpenSSL': (OpenSSL.__version__, package_dir(OpenSSL.__file__)), - __appname__: (allmydata.__version__, package_dir(allmydata.__file__)), - 'foolscap': (foolscap.api.__version__, package_dir(foolscap.__file__)), - 'Nevow': (nevow.__version__, package_dir(nevow.__file__)), - 'pycryptopp': (pycryptopp.__version__, package_dir(pycryptopp.__file__)), - 'setuptools': (setuptools.__version__, package_dir(setuptools.__file__)), - 'simplejson': (simplejson.__version__, package_dir(simplejson.__file__)), - 'pysqlite': (pysqlitever, pysqlitefile), - 'sqlite': (sqlitever, 'unknown'), - 'zope.interface': ('unknown', package_dir(zope.interface.__file__)), - 'Twisted': (twisted.__version__, package_dir(twisted.__file__)), - 'zfec': (zfec.__version__, package_dir(zfec.__file__)), - 'python': (platform.python_version(), sys.executable), - 'platform': (get_platform(), None), - } - # But we prefer to get all the dependencies as known by setuptools: - import pkg_resources - try: - d2 = get_package_versions_from_setuptools() - except pkg_resources.DistributionNotFound: - # See docstring in _auto_deps.require_auto_deps() to explain why it makes sense to ignore this exception. - pass +from allmydata._auto_deps import get_package_versions_and_locations + +_vers_and_locs = get_package_versions_and_locations() +_vers_and_locs['platform'] = (get_platform(), None) + + +def check_all_requirements(): + import platform + from distutils.version import LooseVersion + from allmydata._auto_deps import install_requires + + # we require 2.4.4 on non-UCS-2, non-Redhat builds to avoid <http://www.python.org/news/security/PSF-2006-001/> + # we require 2.4.3 on non-UCS-2 Redhat, because 2.4.3 is common on Redhat-based distros and will have patched the above bug + # we require at least 2.4.2 in any case to avoid a bug in the base64 module: <http://bugs.python.org/issue1171487> + if sys.maxunicode == 65535: + if sys.version_info < (2, 4, 2) or sys.version_info[0] > 2: + raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.2 or greater " + "for a UCS-2 build (but less than v3), not %r" % + (sys.version_info,)) + elif platform.platform().lower().find('redhat') >= 0: + if sys.version_info < (2, 4, 3) or sys.version_info[0] > 2: + raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.3 or greater " + "on Redhat-based distributions (but less than v3), not %r" % + (sys.version_info,)) else: - d1.update(d2) + if sys.version_info < (2, 4, 4) or sys.version_info[0] > 2: + raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.4 or greater " + "for a non-UCS-2 build (but less than v3), not %r" % + (sys.version_info,)) + + def check_requirement(req): + # We only support a single >= or ==. + def _check(s, ok, relation): + name = s[0].strip(' ').partition('[')[0] + required = s[1].strip(' ') + if name not in _vers_and_locs: + raise AssertionError("no version info for %s" % (name,)) + (actual, location) = _vers_and_locs[name] + actual = str(actual) + if actual != 'unknown' and not ok(LooseVersion(actual), LooseVersion(required)): + msg = ("We require %s version %s of %s, but could only find version %s.\n" + % (relation, required, name, actual)) + if location and location != 'unknown': + msg += "The version we found is from %r.\n" % (location,) + msg += ("To resolve this problem, uninstall that version, either using your\n" + "operating system's package manager or by moving aside the directory.") + raise AssertionError(msg) + + s = req.split('>=') + if len(s) == 2: + _check(s, lambda x, y: x >= y, "at least") + return + s = req.split('==') + if len(s) == 2: + _check(s, lambda x, y: x == y, "exactly") + return + if req not in _vers_and_locs: + raise AssertionError("no version info or could not understand requirement for %r" % (req)) + + for requirement in install_requires: + check_requirement(requirement) + +check_all_requirements() - return d1 def get_package_versions(): - return dict([(k, v) for k, (v, l) in get_package_versions_and_locations().iteritems()]) + return dict([(k, v) for k, (v, l) in _vers_and_locs.iteritems()]) def get_package_locations(): - return dict([(k, l) for k, (v, l) in get_package_versions_and_locations().iteritems()]) + return dict([(k, l) for k, (v, l) in _vers_and_locs.iteritems()]) def get_package_versions_string(show_paths=False): - vers_and_locs = get_package_versions_and_locations() res = [] - for p in [__appname__, "foolscap", "pycryptopp", "zfec", "Twisted", "Nevow", "zope.interface", "python", "platform"]: - (ver, loc) = vers_and_locs.get(p, ('UNKNOWN', 'UNKNOWN')) + first = [__appname__, "foolscap", "pycryptopp", "zfec", "Twisted", "Nevow", "zope.interface", "python", "platform"] + for p in first: + (ver, loc) = _vers_and_locs.get(p, ('UNKNOWN', 'UNKNOWN')) info = str(p) + ": " + str(ver) if show_paths: info = info + " (%s)" % str(loc) res.append(info) - if vers_and_locs.has_key(p): - del vers_and_locs[p] - for p, (v, loc) in vers_and_locs.iteritems(): - info = str(p) + ": " + str(v) - if show_paths: - info = info + " (%s)" % str(loc) - res.append(info) + for p, (v, loc) in _vers_and_locs.iteritems(): + if p not in first: + info = str(p) + ": " + str(v) + if show_paths: + info = info + " (%s)" % str(loc) + res.append(info) + return ', '.join(res) diff -rN -u old-ticket1329/src/allmydata/_auto_deps.py new-ticket1329/src/allmydata/_auto_deps.py --- old-ticket1329/src/allmydata/_auto_deps.py 2011-01-17 23:54:16.000000000 -0700 +++ new-ticket1329/src/allmydata/_auto_deps.py 2011-01-17 23:54:16.000000000 -0700 @@ -1,9 +1,10 @@ # Note: do not import any module from Tahoe-LAFS itself in this # file. Also please avoid importing modules from other packages than -# the Python Standard Library if at all possible (exception: we rely -# on importing pkg_resources, which is provided by setuptools, -# zetuptoolz, distribute, and perhaps in the future distutils2, for -# the require_auto_deps() function.) +# the Python Standard Library if at all possible. That includes +# setuptools and pkg_resources. + +# If you add or remove dependencies, please remember to update +# get_package_versions_and_locations() below. install_requires=[ # we require newer versions of setuptools (actually @@ -27,7 +28,7 @@ # Needed for SFTP. pyasn1 is needed by twisted.conch in Twisted >= 9.0. # pycrypto 2.2 doesn't work due to https://bugs.launchpad.net/pycrypto/+bug/620253 - "pycrypto == 2.0.1, == 2.1, >= 2.3", + "pycrypto >= 2.3", "pyasn1 >= 0.0.8a", # Will be needed to test web apps, but not yet. See #1001. @@ -74,49 +75,105 @@ install_requires=[] del sys # clean up namespace -def require_python_version(): - import sys, platform - # we require 2.4.4 on non-UCS-2, non-Redhat builds to avoid <http://www.python.org/news/security/PSF-2006-001/> - # we require 2.4.3 on non-UCS-2 Redhat, because 2.4.3 is common on Redhat-based distros and will have patched the above bug - # we require at least 2.4.2 in any case to avoid a bug in the base64 module: <http://bugs.python.org/issue1171487> - if sys.maxunicode == 65535: - if sys.version_info < (2, 4, 2) or sys.version_info[0] > 2: - raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.2 or greater " - "for a UCS-2 build (but less than v3), not %r" % - (sys.version_info,)) - elif platform.platform().lower().find('redhat') >= 0: - if sys.version_info < (2, 4, 3) or sys.version_info[0] > 2: - raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.3 or greater " - "on Redhat-based distributions (but less than v3), not %r" % - (sys.version_info,)) +def get_package_versions_and_locations(): + import warnings, os, sys + def package_dir(srcfile): + return os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile)))) + + # pkg_resources.require returns the distribution that pkg_resources attempted to put + # on sys.path, which can differ from the one that we actually import due to #1258, + # or any other bug that causes sys.path to be set up incorrectly. Therefore we + # must import the packages in order to check their versions and paths. Note that + # this needs to be updated if dependencies are added or removed. + + # This warning is generated by twisted, PyRex, and possibly other packages, + # but can happen at any time, not only when they are imported. See + # http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1129 . + warnings.filterwarnings("ignore", category=DeprecationWarning, + message="BaseException.message has been deprecated as of Python 2.6", + append=True) + + # This is to suppress DeprecationWarnings from nevow, twisted, and pycrypto. + # See http://allmydata.org/trac/tahoe/ticket/859 and + # http://divmod.org/trac/ticket/2994 . + warnings.filterwarnings("ignore", category=DeprecationWarning, + message="the sha module is deprecated; use the hashlib module instead", + append=True) + warnings.filterwarnings("ignore", category=DeprecationWarning, + message="object.__new__ takes no parameters", + append=True) + warnings.filterwarnings("ignore", category=DeprecationWarning, + message="The popen2 module is deprecated. Use the subprocess module.", + append=True) + warnings.filterwarnings("ignore", category=DeprecationWarning, + message="the md5 module is deprecated; use hashlib instead", + append=True) + warnings.filterwarnings("ignore", category=DeprecationWarning, + message="twisted.web.error.NoResource is deprecated since Twisted 9.0. See twisted.web.resource.NoResource.", + append=True) + warnings.filterwarnings("ignore", category=DeprecationWarning, + message="the sets module is deprecated", + append=True) + try: + import nevow + from twisted.persisted import sob + from twisted.python import filepath + import Crypto + import foolscap.api + [sob, filepath] # hush pyflakes + finally: + for n in range(6): + warnings.filters.pop() + + import OpenSSL, allmydata, platform, pycryptopp, simplejson, twisted, zfec, zope.interface, pyasn1, mock + pysqlitever = None + pysqlitefile = None + sqlitever = None + try: + import sqlite3 + except ImportError: + try: + from pysqlite2 import dbapi2 + except ImportError: + pass + else: + pysqlitever = dbapi2.version + pysqlitefile = package_dir(dbapi2.__file__) + sqlitever = dbapi2.sqlite_version else: - if sys.version_info < (2, 4, 4) or sys.version_info[0] > 2: - raise NotImplementedError("Tahoe-LAFS current requires Python v2.4.4 or greater " - "for a non-UCS-2 build (but less than v3), not %r" % - (sys.version_info,)) - -def require_auto_deps(): - """ - The purpose of this function is to raise a pkg_resources exception if any of the - requirements can't be imported. This is just to give earlier and more explicit error - messages, as opposed to waiting until the source code tries to import some module from one - of these packages and gets an ImportError. This function gets called from - src/allmydata/__init__.py . - """ - require_python_version() + pysqlitever = sqlite3.version + pysqlitefile = package_dir(sqlite3.__file__) + sqlitever = sqlite3.sqlite_version + + packages = { + 'pyopenssl': (OpenSSL.__version__, package_dir(OpenSSL.__file__)), + allmydata.__appname__: (allmydata.__version__, package_dir(allmydata.__file__)), + 'foolscap': (foolscap.api.__version__, package_dir(foolscap.__file__)), + 'Nevow': (nevow.__version__, package_dir(nevow.__file__)), + 'pycryptopp': (pycryptopp.__version__, package_dir(pycryptopp.__file__)), + 'simplejson': (simplejson.__version__, package_dir(simplejson.__file__)), + 'pysqlite': (pysqlitever, pysqlitefile), + 'sqlite': (sqlitever, 'unknown'), + 'zope.interface': ('unknown', package_dir(zope.interface.__file__)), + 'Twisted': (twisted.__version__, package_dir(twisted.__file__)), + 'zfec': (zfec.__version__, package_dir(zfec.__file__)), + 'pycrypto': (Crypto.__version__, package_dir(Crypto.__file__)), + 'pyasn1': ('unknown', package_dir(pyasn1.__file__)), + 'mock': (mock.__version__, package_dir(mock.__file__)), + 'python': (platform.python_version(), sys.executable), + } - import pkg_resources - for requirement in install_requires: + if sys.platform == 'win32': try: - pkg_resources.require(requirement) - except pkg_resources.DistributionNotFound: - # there is no .egg-info present for this requirement, which - # either means that it isn't installed, or it is installed in a - # way that pkg_resources can't find it (but regular python - # might). There are several older Linux distributions which - # provide our dependencies just fine, but they don't ship - # .egg-info files. Note that if there *is* an .egg-info file, - # but it shows a too-old version, then we'll get a - # VersionConflict error instead of DistributionNotFound. + import win32api + packages['pywin32'] = ('unknown', package_dir(win32api.__file__)) + except ImportError: pass + + if not hasattr(sys, 'frozen'): + # Don't try to get the setuptools version if using bbfreeze, py2exe etc. See #585. + import setuptools + packages['setuptools'] = (setuptools.__version__, package_dir(setuptools.__file__)) + + return packages diff -rN -u old-ticket1329/src/allmydata/test/test_runner.py new-ticket1329/src/allmydata/test/test_runner.py --- old-ticket1329/src/allmydata/test/test_runner.py 2011-01-17 23:54:16.000000000 -0700 +++ new-ticket1329/src/allmydata/test/test_runner.py 2011-01-17 23:54:16.000000000 -0700 @@ -3,7 +3,7 @@ from twisted.python import usage, runtime from twisted.internet import utils -import os.path, re, sys +import os.path, re, sys, subprocess from cStringIO import StringIO from allmydata.util import fileutil, pollmixin from allmydata.util.encodingutil import unicode_to_argv, unicode_to_output, get_filesystem_encoding @@ -14,24 +14,36 @@ timeout = 240 +def get_root_from_file(src): + srcdir = os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(src)))) + + root = os.path.dirname(srcdir) + if os.path.basename(srcdir) == 'site-packages': + if re.search(r'python.+\..+', os.path.basename(root)): + root = os.path.dirname(root) + root = os.path.dirname(root) + elif os.path.basename(root) == 'src': + root = os.path.dirname(root) + + return root + srcfile = allmydata.__file__ -srcdir = os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile)))) +rootdir = get_root_from_file(srcfile) -rootdir = os.path.dirname(srcdir) -if os.path.basename(srcdir) == 'site-packages': - if re.search(r'python.+\..+', os.path.basename(rootdir)): - rootdir = os.path.dirname(rootdir) - rootdir = os.path.dirname(rootdir) -elif os.path.basename(rootdir) == 'src': - rootdir = os.path.dirname(rootdir) - -bintahoe = os.path.join(rootdir, 'bin', 'tahoe') -if sys.platform == "win32": - bintahoe += ".pyscript" - if not os.path.exists(bintahoe): - alt_bintahoe = os.path.join(rootdir, 'Scripts', 'tahoe.pyscript') - if os.path.exists(alt_bintahoe): - bintahoe = alt_bintahoe + +if hasattr(sys, 'frozen'): + bintahoe = os.path.join(rootdir, 'tahoe') + + if sys.platform == "win32" and os.path.exists(bintahoe + '.exe'): + bintahoe += '.exe' +else: + bintahoe = os.path.join(rootdir, 'bin', 'tahoe') + if sys.platform == "win32": + bintahoe += '.pyscript' + if not os.path.exists(bintahoe): + alt_bintahoe = os.path.join(rootdir, 'Scripts', 'tahoe.pyscript') + if os.path.exists(alt_bintahoe): + bintahoe = alt_bintahoe class SkipMixin: @@ -49,16 +61,20 @@ class BinTahoe(common_util.SignalMixin, unittest.TestCase, SkipMixin): - def test_the_right_code(self): + def _check_right_code(self, file_to_check): + root_to_check = get_root_from_file(file_to_check) + if os.path.basename(root_to_check) == 'dist': + root_to_check = os.path.dirname(root_to_check) + cwd = os.path.normcase(os.path.realpath(".")) root_from_cwd = os.path.dirname(cwd) if os.path.basename(root_from_cwd) == 'src': root_from_cwd = os.path.dirname(root_from_cwd) - same = (root_from_cwd == rootdir) + same = (root_from_cwd == root_to_check) if not same: try: - same = os.path.samefile(root_from_cwd, rootdir) + same = os.path.samefile(root_from_cwd, root_to_check) except AttributeError, e: e # hush pyflakes @@ -66,9 +82,9 @@ msg = ("We seem to be testing the code at %r,\n" "(according to the source filename %r),\n" "but expected to be testing the code at %r.\n" - % (rootdir, srcfile, root_from_cwd)) + % (root_to_check, file_to_check, root_from_cwd)) - root_from_cwdu = os.path.normcase(os.path.normpath(os.getcwdu())) + root_from_cwdu = os.path.dirname(os.path.normcase(os.path.normpath(os.getcwdu()))) if os.path.basename(root_from_cwdu) == u'src': root_from_cwdu = os.path.dirname(root_from_cwdu) @@ -81,6 +97,24 @@ msg += "Please run the tests from the root of the Tahoe-LAFS distribution." self.fail(msg) + def test_the_right_code(self): + self._check_right_code(srcfile) + + def test_import_in_repl(self): + self.skip_if_cannot_run_bintahoe() + + args = [bintahoe, "debug", "repl"] + if not hasattr(sys, 'frozen'): + args = [sys.executable] + args + + p = subprocess.Popen(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + (out, err) = p.communicate("import allmydata; print allmydata.__file__") + + self.failUnlessEqual(p.returncode, 0) + first = out.splitlines()[0] + self.failUnless(first.startswith('>>> ')) + self._check_right_code(first[4:]) + def test_path(self): self.skip_if_cannot_run_bintahoe() d = utils.getProcessOutputAndValue(bintahoe, args=["--version-and-path"], env=os.environ) @@ -112,6 +146,7 @@ else: altverstr = verstr + srcdir = os.path.dirname(os.path.dirname(os.path.normcase(os.path.realpath(srcfile)))) required_ver_and_path = "%s: %s (%s)" % (allmydata.__appname__, verstr, srcdir) alt_required_ver_and_path = "%s: %s (%s)" % (allmydata.__appname__, altverstr, srcdir) @@ -121,6 +156,8 @@ def test_unicode_arguments_and_output(self): self.skip_if_cannot_run_bintahoe() + if hasattr(sys, 'frozen') and sys.platform == "win32": + raise unittest.SkipTest("This test can't currently be made to work for frozen executables on Windows,\ndue to lack of support for Unicode in twisted.internet.utils.getProcessOutputAndValue.") tricky = u"\u2621" try: @@ -139,6 +176,8 @@ def test_run_with_python_options(self): self.skip_if_cannot_run_bintahoe() + if hasattr(sys, 'frozen'): + raise unittest.SkipTest("This test doesn't apply to frozen executables.") # -t is a harmless option that warns about tabs. d = utils.getProcessOutputAndValue(sys.executable, args=['-t', bintahoe, '--version'], diff -rN -u old-ticket1329/src/allmydata/test/test_system.py new-ticket1329/src/allmydata/test/test_system.py --- old-ticket1329/src/allmydata/test/test_system.py 2011-01-17 23:54:16.000000000 -0700 +++ new-ticket1329/src/allmydata/test/test_system.py 2011-01-17 23:54:16.000000000 -0700 @@ -1769,8 +1769,7 @@ if env is None: env = os.environ - d = utils.getProcessOutputAndValue(sys.executable, args=[bintahoe] + argv, - env=env) + d = utils.getProcessOutputAndValue(bintahoe, argv, env=env) return d def _test_checker(self, res): ```

zooko commented

2011-01-18 07:26:18 +00:00

Attachment 1329.diff (26465 bytes) added

**Attachment** 1329.diff (26465 bytes) added

1329.diff

26 KiB

daira commented

2011-01-18 07:31:54 +00:00

Author

Yes, yes, I know these patches need refactoring! It will be easy to apply the #1329 fix after I've done that; it makes no sense to apply it first (because the relevant code has been moved to a different file).