tahoe-lafs/trac-2024-07-25

Twisted 16.0.0 causes test failure when importing cryptography.exceptions #2740

New issue

Closed

opened 2016-03-15 16:02:05 +00:00 by daira · 6 comments

daira commented

2016-03-15 16:02:05 +00:00

Owner

E.g. https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/116153170#L1350 :

allmydata.test.test_client ... Traceback (most recent call last):
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-16.0.0-py2.7-linux-x86_64.egg/twisted/trial/runner.py", line 511, in loadPackage
    module = modinfo.load()
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-16.0.0-py2.7-linux-x86_64.egg/twisted/python/modules.py", line 389, in load
    return self.pathEntry.pythonPath.moduleLoader(self.name)
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-16.0.0-py2.7-linux-x86_64.egg/twisted/python/reflect.py", line 303, in namedAny
    topLevelPackage = _importAndCheckStack(trialname)
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-16.0.0-py2.7-linux-x86_64.egg/twisted/python/reflect.py", line 250, in _importAndCheckStack
    reraise(excValue, excTraceback)
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/test/test_client.py", line 11, in <module>
    from allmydata.frontends.auth import NeedRootcapLookupScheme
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/frontends/auth.py", line 8, in <module>
    from twisted.conch.ssh import keys
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-16.0.0-py2.7-linux-x86_64.egg/twisted/conch/ssh/keys.py", line 16, in <module>
    from cryptography.exceptions import InvalidSignature
exceptions.ImportError: No module named cryptography.exceptions

E.g. <https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/116153170#L1350> : ``` allmydata.test.test_client ... Traceback (most recent call last): File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-16.0.0-py2.7-linux-x86_64.egg/twisted/trial/runner.py", line 511, in loadPackage module = modinfo.load() File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-16.0.0-py2.7-linux-x86_64.egg/twisted/python/modules.py", line 389, in load return self.pathEntry.pythonPath.moduleLoader(self.name) File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-16.0.0-py2.7-linux-x86_64.egg/twisted/python/reflect.py", line 303, in namedAny topLevelPackage = _importAndCheckStack(trialname) File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-16.0.0-py2.7-linux-x86_64.egg/twisted/python/reflect.py", line 250, in _importAndCheckStack reraise(excValue, excTraceback) File "/home/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/test/test_client.py", line 11, in <module> from allmydata.frontends.auth import NeedRootcapLookupScheme File "/home/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/frontends/auth.py", line 8, in <module> from twisted.conch.ssh import keys File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-16.0.0-py2.7-linux-x86_64.egg/twisted/conch/ssh/keys.py", line 16, in <module> from cryptography.exceptions import InvalidSignature exceptions.ImportError: No module named cryptography.exceptions ```

tahoe-lafs added the

labels 2016-03-15 16:02:05 +00:00

tahoe-lafs added this to the undecided milestone 2016-03-15 16:02:05 +00:00

warner commented

2016-03-15 16:51:47 +00:00

Author

Owner

It looks like Twisted only declares a dependency on "cryptography" if you ask for the "conch" extra, as in pip install twistedconch, or install_requires = [ "Twistedconch" ]. The previous twisted-15.5.0 had a conch dependency of "gmpy, pyasn1, pycrypto", which overlapped enough with what Tahoe declares dependencies on that things still worked. Twisted-16.0.0 changed conch to "gmpy, pyasn1, cryptography".

Tahoe has been in the practice of declaring a transitive closure of dependencies, and I think it's time we ended that and let pip have responsibility for figuring out the transitive set. (We can always add a requires.txt if we want to behave like an application, as opposed to a library, and also provide peep-style hashes of upstream tarballs/wheels). We should declare a dependency on Twistedconch. Foolscap should declare Twistedtls (added as foolscap#249).

And I think it might be appropriate for tahoe to have a windows-conditional dependency on Twistedwindows, since that's what triggers twisted to ask for pypiwin32.

It looks like Twisted only declares a dependency on "cryptography" if you ask for the "conch" extra, as in `pip install twistedconch`, or `install_requires = [ "Twistedconch" ]`. The previous twisted-15.5.0 had a `conch` dependency of "gmpy, pyasn1, pycrypto", which overlapped enough with what Tahoe declares dependencies on that things still worked. Twisted-16.0.0 changed conch to "gmpy, pyasn1, cryptography". Tahoe has been in the practice of declaring a transitive closure of dependencies, and I think it's time we ended that and let pip have responsibility for figuring out the transitive set. (We can always add a `requires.txt` if we want to behave like an application, as opposed to a library, and also provide peep-style hashes of upstream tarballs/wheels). We should declare a dependency on `Twistedconch`. Foolscap should declare `Twistedtls` (added as [foolscap#249](https://foolscap.lothar.com/trac/ticket/249)). And I think it might be appropriate for tahoe to have a windows-conditional dependency on `Twistedwindows`, since that's what triggers twisted to ask for `pypiwin32`.

daira commented

2016-03-15 18:15:21 +00:00

Author

Owner

warner wrote:

Tahoe has been in the practice of declaring a transitive closure of dependencies, and I think it's time we ended that and let pip have responsibility for figuring out the transitive set.

warner wrote: > Tahoe has been in the practice of declaring a transitive closure of dependencies, and I think it's time we ended that and let pip have responsibility for figuring out the transitive set. +1

warner commented

2016-03-22 05:03:36 +00:00

Author

Owner

Milestone renamed

tahoe-lafs added this to the 1.11.0 milestone 2016-03-22 05:03:36 +00:00

warner commented

2016-03-25 18:34:29 +00:00

Author

Owner

We'll bump the dependency to twistedtls,conch to cover both this issue, and to get Twisted's TLS stuff installed. (Ideally we'd leave that responsibility to Foolscap, but a setuptools bug means that "extras" only get honored in the first appearance of a package).

We'll bump the dependency to `twistedtls,conch` to cover both this issue, and to get Twisted's TLS stuff installed. (Ideally we'd leave that responsibility to Foolscap, but a setuptools bug means that "extras" only get honored in the first appearance of a package).

warner commented

2016-03-25 18:45:07 +00:00

Author

Owner

Oops, the twistedconch triggered an inclusion of gmpy, which fails to build on most systems (because they lack the libgmp headers). It turns out that gmpy is pretty optional: the twisted code (specifically twisted/conch/ssh/common.py) uses it to monkey-patch the built-in pow() function with a faster version. (Maybe this is old? I thought python used libgmp for pow() already). If conch cannot import gmpy, it silently continues with the default pow().

So we're reducing the dependency to just twistedtls, which currently gets us everything that twistedconch would get except for gmpy. And since gmpy is really optional, SFTP should still just work.

Oops, the `twistedconch` triggered an inclusion of `gmpy`, which fails to build on most systems (because they lack the `libgmp` headers). It turns out that `gmpy` is pretty optional: the twisted code (specifically `twisted/conch/ssh/common.py`) uses it to monkey-patch the built-in `pow()` function with a faster version. (Maybe this is old? I thought python used libgmp for `pow()` already). If conch cannot import `gmpy`, it silently continues with the default `pow()`. So we're reducing the dependency to just `twistedtls`, which currently gets us everything that `twistedconch` would get *except* for `gmpy`. And since `gmpy` is really optional, SFTP should still just work.

Daira Hopwood <daira@jacaranda.org> commented

2016-03-25 18:45:36 +00:00

Author

Owner

In c3d54e0/trunk:

Back out the dependency on the conch extra for Twisted. refs ticket:2740

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

In [c3d54e0/trunk](/tahoe-lafs/trac-2024-07-25/commit/c3d54e0221d36033a0f3611ef0cde247242bfaab): ``` Back out the dependency on the conch extra for Twisted. refs ticket:2740 Signed-off-by: Daira Hopwood <daira@jacaranda.org> ```

tahoe-lafs added the

fixed

label 2016-03-25 19:42:24 +00:00

warner closed this issue

2016-03-25 19:42:24 +00:00