tahoe-lafs/trac

provide binary eggs of PyCrypto for our supported platforms #1065

New issue

Closed

opened 2010-06-04 03:35:27 +00:00 by zooko · 10 comments

zooko commented

2010-06-04 03:35:27 +00:00

http://tahoe-lafs.org/buildbot/builders/Soultcer%20Win7-64%20py2.6

For example, a failing build is:

http://tahoe-lafs.org/buildbot/builders/Soultcer%20Win7-64%20py2.6/builds/47/steps/build/logs/stdio

which ends with

Searching for pycrypto>=2.0.1
Reading http://pypi.python.org/simple/pycrypto/
Reading http://pycrypto.sourceforge.net
Reading http://www.amk.ca/python/code/crypto
Best match: pycrypto 2.1.0
Downloading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/pycrypto-2.1.0.tar.bz2
Processing pycrypto-2.1.0.tar.bz2
Running pycrypto-2.1.0\setup.py -q bdist_egg --dist-dir c:\users\buildbot\appdata\local\temp\easy_install-ynvpn2\pycrypto-2.1.0\egg-dist-tmp-k5u7tq
Failure to get the list of managed files from darcs -- if you are building a package with 'setup.py sdist', 'setup.py bdist_egg', or other package-building commands, then the resulting package might be missing some files.  If you are not building a package then you can ignore this warning.
warning: GMP library not found; Not building Crypto.PublicKey._fastmath.
error: Setup script exited with error: Unable to find vcvarsall.bat

To close this ticket, make it so that that buildbot succeeds at installing Tahoe-LAFS even if its owner (soultcer) does not install any new tools such as a C compiler nor packages such as PyCrypto.

The way to do that is to build or acquire a binary egg of PyCrypto for that platform (Windows 7 64 bit) and host it here: http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/ (or you can host it on the PyPI site for PyCrypto, which will also be examined to find candidate eggs).

Now that we require PyCrypto (see #953), we need to provide binary eggs of it so that people can install Tahoe-LAFS on platforms that don't have a working C compiler. Here is a buildbot that is currently demonstrating how this fails: <http://tahoe-lafs.org/buildbot/builders/Soultcer%20Win7-64%20py2.6> For example, a failing build is: <http://tahoe-lafs.org/buildbot/builders/Soultcer%20Win7-64%20py2.6/builds/47/steps/build/logs/stdio> which ends with ``` Searching for pycrypto>=2.0.1 Reading http://pypi.python.org/simple/pycrypto/ Reading http://pycrypto.sourceforge.net Reading http://www.amk.ca/python/code/crypto Best match: pycrypto 2.1.0 Downloading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/pycrypto-2.1.0.tar.bz2 Processing pycrypto-2.1.0.tar.bz2 Running pycrypto-2.1.0\setup.py -q bdist_egg --dist-dir c:\users\buildbot\appdata\local\temp\easy_install-ynvpn2\pycrypto-2.1.0\egg-dist-tmp-k5u7tq Failure to get the list of managed files from darcs -- if you are building a package with 'setup.py sdist', 'setup.py bdist_egg', or other package-building commands, then the resulting package might be missing some files. If you are not building a package then you can ignore this warning. warning: GMP library not found; Not building Crypto.PublicKey._fastmath. error: Setup script exited with error: Unable to find vcvarsall.bat ``` To close this ticket, make it so that that buildbot succeeds at installing Tahoe-LAFS even if its owner (soultcer) does not install any new tools such as a C compiler nor packages such as PyCrypto. The way to do *that* is to build or acquire a binary egg of PyCrypto for that platform (Windows 7 64 bit) and host it here: <http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/> (or you can host it on the PyPI site for PyCrypto, which will also be examined to find candidate eggs).

zooko added the

labels 2010-06-04 03:35:27 +00:00

zooko added this to the soon (release n/a) milestone 2010-06-04 03:35:27 +00:00

zooko added

c/packaging

and removed

c/unknown

labels 2010-06-04 03:37:03 +00:00

zooko changed title from ~~host binary eggs of PyCrypto for our supported platforms~~ to provide binary eggs of PyCrypto for our supported platforms

2010-06-04 03:37:03 +00:00

zooko commented

2010-06-04 03:43:42 +00:00

Author

Here is the launchpad ticket where I request that the maintainers of PyCrypto provide binary eggs: https://bugs.launchpad.net/tahoe-lafs/+bug/589511

Here is the launchpad ticket where I request that the maintainers of [PyCrypto](wiki/PyCrypto) provide binary eggs: <https://bugs.launchpad.net/tahoe-lafs/+bug/589511>

zooko commented

2010-08-10 04:50:46 +00:00

Author

The maintainer of PyCrypto, Dwayne Litzenberger, replied with "Won't Fix":
I have no intention of distributing PyCrypto binaries any time soon. Sorry.

The maintainer of PyCrypto, Dwayne Litzenberger, replied with "Won't Fix": I have no intention of distributing [PyCrypto](wiki/PyCrypto) binaries any time soon. Sorry.

daira commented

2010-08-27 20:18:46 +00:00

This problem has been exacerbated by [this bug in PyCrypto 2.2](https://bugs.launchpad.net/pycrypto/+bug/620253) (fixed in 2.3). If we were to bump the PyCrypto version requirement in source:_auto_deps.py to 2.3, systems with no compiler would fail to upgrade, even if they already had a working PyCrypto egg (say 2.0.1 or 2.1, which are downloadable from http://www.voidspace.org.uk/python/modules.shtml#pycrypto ).

Even if we don't bump the version requirement, a user whose system has no working compiler may run into http://bugs.python.org/setuptools/issue83 (this happened to bj0). #717 and #1015 are also contributing factors here.

This problem has been exacerbated by [this bug in [PyCrypto](wiki/PyCrypto) 2.2](https://bugs.launchpad.net/pycrypto/+bug/620253) (fixed in 2.3). If we were to bump the PyCrypto version requirement in source:_auto_deps.py to 2.3, systems with no compiler would fail to upgrade, even if they already had a working PyCrypto egg (say 2.0.1 or 2.1, which are downloadable from <http://www.voidspace.org.uk/python/modules.shtml#pycrypto> ). Even if we don't bump the version requirement, a user whose system has no working compiler may run into <http://bugs.python.org/setuptools/issue83> (this happened to bj0). #717 and #1015 are also contributing factors here.

daira commented

2010-08-27 20:39:16 +00:00

According to setuptools docs, we can write the PyCrypto dependency as "pycrypto == 2.0.1, == 2.1, >= 2.3".

According to [setuptools docs](http://peak.telecommunity.com/DevCenter/setuptools#declaring-dependencies), we can write the PyCrypto dependency as `"pycrypto == 2.0.1, == 2.1, >= 2.3"`.

zooko commented

2010-08-29 22:40:29 +00:00

Author

Well, I've never used disjunctive version requirements like that before, but let's try it. I'll commit it to source:ticket798 and have the buildbots build it.

daira commented

2010-08-30 03:52:51 +00:00

This was actually committed to trunk, in changeset:57b96c5e2f851cca. It doesn't seem to have caused any new buildbot failures.

daira commented

2010-08-30 20:11:52 +00:00

Replying to davidsarah:

This was actually committed to trunk, in changeset:57b96c5e2f851cca. It doesn't seem to have caused any new buildbot failures.

Sigh, spoke too soon (log for the test step on 'Ruben Fedora'):

Searching for pycrypto==2.1.0
Best match: pycrypto 2.1.0
Processing pycrypto-2.1.0-py2.7-linux-x86_64.egg
pycrypto 2.1.0 is already the active version in easy-install.pth

Using /home/buildbot/tahoe/Ruben Fedora/build/support/lib/python2.7/
site-packages/pycrypto-2.1.0-py2.7-linux-x86_64.egg
[...]
running trial
[...]
pkg_resources.VersionConflict: (pycrypto 2.2 (/usr/lib64/python2.7/
site-packages), Requirement.parse('pycrypto==2.0.1,==2.1,>=2.3'))

I believe this is due to the same underlying issue as #1137, that is, the handling of sys.path by the setuptools-installed site.py and .pth files being broken. If the path handling were working correctly, it should not have been possible that "pycrypto 2.1.0 is already the active version in easy-install.pth", but we end up importing pycrypto 2.2 when running trial.

Note that we have the same problem with foolscap on 'Shawn jaunty':

Searching for foolscap==0.5.1
Best match: foolscap 0.5.1
Processing foolscap-0.5.1-py2.6.egg
foolscap 0.5.1 is already the active version in easy-install.pth
Installing flappclient script to support/bin
Installing flogtool script to support/bin
Installing flappserver script to support/bin

Using /var/lib/buildbot/tahoe/mordsith/build/support/lib/python2.6/
site-packages/foolscap-0.5.1-py2.6.egg
[...]
running trial
[...]
pkg_resources.VersionConflict: (foolscap 0.4.2 (/usr/local/lib/python2.6/
dist-packages/foolscap-0.4.2-py2.6.egg),
Requirement.parse('foolscap[secure_connections]>=0.5.1'))

So this is not specific to pycrypto or due to the use of a disjunctive requirement. I think that we should keep the pycrypto requirement as "pycrypto == 2.0.1, == 2.1, >= 2.3" for 1.8.0c3, since it at least fails with a message that could give the user a hint that they need to delete the pycrypto 2.2 installation.

Replying to [davidsarah](/tahoe-lafs/trac/issues/1065#issuecomment-378619): > This was actually committed to trunk, in changeset:57b96c5e2f851cca. It doesn't seem to have caused any new buildbot failures. Sigh, spoke too soon ([log for the test step on 'Ruben Fedora'](http://tahoe-lafs.org/buildbot/builders/Ruben%20Fedora/builds/610/steps/test/logs/stdio)): ``` Searching for pycrypto==2.1.0 Best match: pycrypto 2.1.0 Processing pycrypto-2.1.0-py2.7-linux-x86_64.egg pycrypto 2.1.0 is already the active version in easy-install.pth Using /home/buildbot/tahoe/Ruben Fedora/build/support/lib/python2.7/ site-packages/pycrypto-2.1.0-py2.7-linux-x86_64.egg [...] running trial [...] pkg_resources.VersionConflict: (pycrypto 2.2 (/usr/lib64/python2.7/ site-packages), Requirement.parse('pycrypto==2.0.1,==2.1,>=2.3')) ``` I believe this is due to the same underlying issue as #1137, that is, the handling of `sys.path` by the setuptools-installed `site.py` and `.pth` files being broken. If the path handling were working correctly, it should not have been possible that "pycrypto 2.1.0 is already the active version in easy-install.pth", but we end up importing pycrypto 2.2 when running trial. Note that we have the same problem with foolscap [on 'Shawn jaunty'](http://tahoe-lafs.org/buildbot/builders/Shawn%20jaunty%20amd64/builds/563/steps/test/logs/stdio): ``` Searching for foolscap==0.5.1 Best match: foolscap 0.5.1 Processing foolscap-0.5.1-py2.6.egg foolscap 0.5.1 is already the active version in easy-install.pth Installing flappclient script to support/bin Installing flogtool script to support/bin Installing flappserver script to support/bin Using /var/lib/buildbot/tahoe/mordsith/build/support/lib/python2.6/ site-packages/foolscap-0.5.1-py2.6.egg [...] running trial [...] pkg_resources.VersionConflict: (foolscap 0.4.2 (/usr/local/lib/python2.6/ dist-packages/foolscap-0.4.2-py2.6.egg), Requirement.parse('foolscap[secure_connections]>=0.5.1')) ``` So this is not specific to pycrypto or due to the use of a disjunctive requirement. I think that we should keep the pycrypto requirement as "pycrypto == 2.0.1, == 2.1, >= 2.3" for 1.8.0c3, since it at least fails with a message that could give the user a hint that they need to delete the pycrypto 2.2 installation.

daira commented

2010-08-30 20:22:10 +00:00

The using-wrong-version-of-dependency problem is #1190.

zooko commented

2010-10-31 06:08:52 +00:00

Author

Since the newest version of PyCrypto listed on http://pypi.python.org/pypi/pycrypto at the moment is 2.3, then we need to build bdist_egg's of PyCrypto 2.3 for all platforms which we intend for people to be able to install Tahoe-LAFS v1.8.1 without having a C compiler and Python headers installed. (Note that the upstream maintainer of PyCrypto will not provide binaries: https://bugs.launchpad.net/pycrypto/+bug/589511 .)

Sadly, if we do all the work to build and host such binary eggs, and then a new version of PyCrypto gets uploaded to !PyPI in source form, then our build scripts will immediately start refusing to use the PyCrypto 2.3 binaries that we built. Fixing this so that our build scripts would continue to use the PyCrypto 2.3 binaries even if a newer version of PyCrypto is known to them is the subject of #1233.

Since the newest version of PyCrypto listed on <http://pypi.python.org/pypi/pycrypto> at the moment is 2.3, then we need to build bdist_egg's of PyCrypto 2.3 for all platforms which we intend for people to be able to install Tahoe-LAFS v1.8.1 without having a C compiler and Python headers installed. (Note that the upstream maintainer of PyCrypto will not provide binaries: <https://bugs.launchpad.net/pycrypto/+bug/589511> .) Sadly, if we do all the work to build and host such binary eggs, and then a new version of PyCrypto gets uploaded to !PyPI in source form, then our build scripts will immediately start refusing to use the PyCrypto 2.3 binaries that we built. Fixing this so that our build scripts would continue to use the PyCrypto 2.3 binaries even if a newer version of PyCrypto is known to them is the subject of #1233.

zooko modified the milestone from soon (release n/a) to 1.8.1

2010-10-31 06:08:52 +00:00

warner modified the milestone from 1.8.1 to soon (release n/a)

2011-01-06 21:22:01 +00:00

zooko commented

2011-01-17 07:43:00 +00:00

Author

We now host binary eggs of PyCrypto for many fine platforms. See http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/README.html for details.

We now host binary eggs of `PyCrypto` for many fine platforms. See <http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/README.html> for details.

Rows
Columns