prefer binary eggs which satisfy dependencies over source eggs which are even newer

zooko commented

2010-10-28 05:40:17 +00:00

Owner

Currently if the zetuptoolz-based build system finds both a binary egg (for the current platform that it is building on) and a source egg, and both of them satisfy the dependency requirement, it will choose the one with the highest version number.

This is a serious problem in practice because, for example, if we have built and hosted dozens of binary eggs of zfec for many different combinations of platform and Python version, and then a newer source-code distribution of zfec is uploaded to PyPI, then the build system will no longer try to use any of those binary eggs and instead use the newer source distribution exclusively, which will fail on platforms that don't have a C compiler and Python header files present.

To fix this, change the logic in zetuptoolz to prefer a binary egg, provided that it is for your platform and Python version and satisfies the dependency requirement, over a source distribution.

Currently if the zetuptoolz-based build system finds both a binary egg (for the current platform that it is building on) and a source egg, and both of them satisfy the dependency requirement, it will choose the one with the highest version number. This is a serious problem in practice because, for example, if we have built and hosted dozens of binary eggs of zfec for many different combinations of platform and Python version, and then a newer source-code distribution of zfec is uploaded to PyPI, then the build system will no longer try to use any of those binary eggs and instead use the newer source distribution exclusively, which will fail on platforms that don't have a C compiler and Python header files present. To fix this, change the logic in zetuptoolz to prefer a binary egg, provided that it is for your platform and Python version and satisfies the dependency requirement, over a source distribution.

tahoe-lafs added the

labels 2010-10-28 05:40:17 +00:00

tahoe-lafs added this to the 1.8.1 milestone 2010-10-28 05:40:17 +00:00

zooko commented

2010-10-28 05:41:18 +00:00

Author

Owner

comment on a different ticket about one of the (very numerous) times that this has caused trouble for a user:

(@@http://tahoe-lafs.org/trac/tahoe-lafs/ticket/773#comment:-1@@)

bug report against setuptools:

http://bugs.python.org/setuptools/issue83

comment on a different ticket about one of the (very numerous) times that this has caused trouble for a user: (@@http://tahoe-lafs.org/trac/tahoe-lafs/ticket/773#[comment:-1](/tahoe-lafs/trac-2024-07-25/issues/1233#issuecomment--1)@@) bug report against setuptools: <http://bugs.python.org/setuptools/issue83>

zooko commented

2010-10-31 05:55:29 +00:00

Author

Owner

David-Sarah says we shouldn't try changing setuptools/zetuptoolz at the last minute, so instead of doing this ticket for v1.8.1, we're just going to do the time-consuming but predictable work of building and hosting binary eggs for all platforms that we want to support.

tahoe-lafs modified the milestone from 1.8.1 to undecided

2010-10-31 05:55:29 +00:00

davidsarah commented

2010-11-03 04:02:39 +00:00

Author

Owner

Replying to zooko:

David-Sarah says we shouldn't try changing setuptools/zetuptoolz at the last minute, so instead of doing this ticket for v1.8.1, we're just going to do the time-consuming but predictable work of building and hosting binary eggs for all platforms that we want to support.

Having seen the scale of the problem in generating all those binary eggs, zooko and I have had second thoughts about that. I have a patch to zetuptoolz, and zooko has a possible test.

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/1233#issuecomment-122336): > David-Sarah says we shouldn't try changing setuptools/zetuptoolz at the last minute, so instead of doing this ticket for v1.8.1, we're just going to do the time-consuming but predictable work of building and hosting binary eggs for all platforms that we want to support. Having seen [the scale of the problem](http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/README.html) in generating all those binary eggs, zooko and I have had second thoughts about that. I have a patch to zetuptoolz, and zooko has a possible test.

davidsarah commented

2010-11-03 04:13:36 +00:00

Author

Owner

Attachment zetuptoolz-prefer-platform-specific-dists.darcs.patch (15949 bytes) added

bundled zetuptools: prefer platform-specific dists to platform-independent ones. refs #1233

**Attachment** zetuptoolz-prefer-platform-specific-dists.darcs.patch (15949 bytes) added bundled zetuptools: prefer platform-specific dists to platform-independent ones. refs #1233

zetuptoolz-prefer-platform-specific-dists.darcs.patch

16 KiB

zooko commented

2010-11-03 20:46:38 +00:00

Author

Owner

Attachment test-prefer-bdists.darcs.patch (17233 bytes) added

**Attachment** test-prefer-bdists.darcs.patch (17233 bytes) added

test-prefer-bdists.darcs.patch

17 KiB

zooko commented

2010-11-03 20:48:26 +00:00

Author

Owner

Attachment zetuptoolz-debugprints.diff.patch (4733 bytes) added

**Attachment** zetuptoolz-debugprints.diff.patch (4733 bytes) added

zetuptoolz-debugprints.diff.patch

4.6 KiB

tahoe-lafs modified the milestone from undecided to 1.8.1

2010-11-04 07:32:54 +00:00

david-sarah@jacaranda.org commented

2010-11-06 00:25:44 +00:00

Author

Owner

In changeset:306ec02c1d2946b2:

setup.py: fix a bug in the check for whether we are running 'trial' or 'test', that affected zooko's test for #1233. refs #1233

In changeset:306ec02c1d2946b2: ``` setup.py: fix a bug in the check for whether we are running 'trial' or 'test', that affected zooko's test for #1233. refs #1233 ```

zooko commented

2010-11-14 10:04:11 +00:00

Author

Owner

There is now a deterministic test of this behavior, added in changeset:59dcd5014dcb440a. You can run it locally with python misc/build_helpers/test-with-fake-dists.py. Unfortunately it just skips the test if a satisfying version of pycryptopp is already installed. I couldn't figure out how to test it in that case.

If a satisfying version of pycryptopp is not already installed, then it will test this behavior and either raise an exception saying that the build system attempted to build a fake "pycryptopp v9.9.99" from source package (so it failed the test) or it will pass saying that it used a fake "pycryptopp 0.5.24" from binary package.

Looking at our buildslaves, all of ones working well enough to run the test already have pycryptopp installed:

I just manually removed pycryptopp from zomp and rebuilt build #213 to yield build #214:

http://tahoe-lafs.org/buildbot/builders/Zooko%20zomp%20Mac-amd64%2010.6%20py2.6/builds/214/steps/test-with-fake-dists/logs/stdio

Hey! Same behavior. Oh, silly me! pycryptopp is being built and stored in ./support by the build step if it isn't already present, so when the later test-with-fake-dists step happens it is always there! I will change the buildmaster config to do with-with-fake-dists first and see if that shows us something useful.

There is now a deterministic test of this behavior, added in changeset:59dcd5014dcb440a. You can run it locally with `python misc/build_helpers/test-with-fake-dists.py`. Unfortunately it just skips the test if a satisfying version of `pycryptopp` is already installed. I couldn't figure out how to test it in that case. If a satisfying version of `pycryptopp` is not already installed, then it will test this behavior and either raise an exception saying that the build system attempted to build a fake "pycryptopp v9.9.99" from source package (so it failed the test) or it will pass saying that it used a fake "pycryptopp 0.5.24" from binary package. Looking at our buildslaves, all of ones working well enough to run the test already have pycryptopp installed: * <http://tahoe-lafs.org/buildbot/builders/David%20A.%20OpenSolaris%20i386/builds/464/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/MM%20netbsd5%20i386%20warp/builds/159/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/lucid-amd64/builds/72/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/Brian%20ubuntu-i386%20linode/builds/63/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/Ruben%20Fedora/builds/691/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/FreeStorm%20WinXP-x86%20py2.6/builds/356/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/Zooko%20zomp%20Mac-amd64%2010.6%20py2.6/builds/213/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/FreeStorm%20WinXP-x86%20py2.6/builds/356/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/FreeStorm%20Win7-amd64-mingw%20py2.6/builds/168/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/Eugen%20lenny-amd64/builds/671/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/Randy%20FreeBSD-amd64/builds/195/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/Arthur%20lenny%20c7%2032bit/builds/553/steps/test-with-fake-dists/logs/stdio> I just manually removed pycryptopp from zomp and rebuilt build #213 to yield build #214: * <http://tahoe-lafs.org/buildbot/builders/Zooko%20zomp%20Mac-amd64%2010.6%20py2.6/builds/214/steps/test-with-fake-dists/logs/stdio> Hey! Same behavior. Oh, silly me! pycryptopp is being built and stored in `./support` by the `build` step if it isn't already present, so when the later `test-with-fake-dists` step happens it is always there! I will change the buildmaster config to do `with-with-fake-dists` first and see if that shows us something useful.

zooko commented

2010-11-14 10:31:47 +00:00

Author

Owner

Great! Some of the buildslaves don't have pycryptopp installed and so we get a demonstration of issue #1233 on them. If we commit a patch that fixes #1233 then all of these should go from red to green:

These ones have a sufficient version of pycryptopp and so are not really testing #1233:

These ones I went to bed before they finished so at the time of this writing I don't knowL

Great! Some of the buildslaves don't have pycryptopp installed and so we get a demonstration of issue #1233 on them. If we commit a patch that fixes #1233 then all of these should go from red to green: * <http://tahoe-lafs.org/buildbot/builders/lucid-amd64/builds/74/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/Brian%20ubuntu-i386%20linode/builds/65/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/Randy%20FreeBSD-amd64/builds/197/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/FreeStorm%20CentOS5-i386/builds/232/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/FreeStorm%20WinXP-x86%20py2.6/builds/358/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/Zooko%20zomp%20Mac-amd64%2010.6%20py2.6/builds/216/steps/test-with-fake-dists/logs/stdio> These ones have a sufficient version of pycryptopp and so are not really testing #1233: * <http://tahoe-lafs.org/buildbot/builders/Ruben%20Fedora/builds/693/steps/test-with-fake-dists/logs/stdio> * <http://tahoe-lafs.org/buildbot/builders/MM%20netbsd5%20i386%20warp/builds/161/steps/test-with-fake-dists/logs/stdio> These ones I went to bed before they finished so at the time of this writing I don't knowL * <http://tahoe-lafs.org/buildbot/builders/David%20A.%20OpenSolaris%20i386/builds/467> * <http://tahoe-lafs.org/buildbot/builders/Eugen%20lenny-amd64/builds/673> * <http://tahoe-lafs.org/buildbot/builders/Arthur%20lenny%20c7%2032bit/builds/555>

zooko commented

2010-11-16 07:36:44 +00:00

Author

Owner

All of the red steps test-with-fake-dists on our Supported Builders or all builders should go green when we fix this.

All of the red steps `test-with-fake-dists` on our [Supported Builders](http://tahoe-lafs.org/buildbot/waterfall?show_events=false&builder=Kyle+OpenBSD+amd64&builder=MM+netbsd5+i386+warp&builder=lucid-amd64&builder=Eugen+lenny-amd64&builder=Arthur+lenny+c7+32bit&builder=FreeStorm+CentOS5-i386&builder=Ruben+Fedora&builder=David+A.+OpenSolaris+i386&builder=Zooko+zomp+Mac-amd64+10.6+py2.6&builder=FreeStorm+WinXP-x86+py2.6&builder=Dcoder+Win7-64+py2.6&builder=tarballs) or [all builders](http://tahoe-lafs.org/buildbot/waterfall?show_events=false) should go green when we fix this.

zooko commented

2010-11-16 07:38:00 +00:00

Author

Owner

It would be great if someone else would try to hack zetuptoolz to fix this. If they don't, I will try.

zooko@zooko.com commented

2010-11-17 09:06:16 +00:00

Author

Owner

In [4832/ticket1233]:

bundled zetuptoolz: prefer locally-available distributions over remotely-downloaded distributions above all
This fixes #1233. Actually the previous patches—[changeset:a1cef915fd689308] and [changeset:a44330dfeef23757]—fixed it, but with them zetuptoolz would download a higher-numbered distribution from the net instead of using the locally-available (fake) pycryptopp-0.5.24, thus preventing the tests from passing. This patch changes that behavior (which is an improvement in its own right) and also fixes a bug in the tests.

In [4832/ticket1233]: ``` bundled zetuptoolz: prefer locally-available distributions over remotely-downloaded distributions above all This fixes #1233. Actually the previous patches—[changeset:a1cef915fd689308] and [changeset:a44330dfeef23757]—fixed it, but with them zetuptoolz would download a higher-numbered distribution from the net instead of using the locally-available (fake) pycryptopp-0.5.24, thus preventing the tests from passing. This patch changes that behavior (which is an improvement in its own right) and also fixes a bug in the tests. ```

tahoe-lafs added the

fixed

label 2010-11-17 09:06:16 +00:00

zooko@zooko.com closed this issue

2010-11-17 09:06:16 +00:00

zooko commented

2010-11-17 09:54:32 +00:00

Author

Owner

This is fixed by changeset:a1cef915fd689308, changeset:a44330dfeef23757, changeset:b4c14421f7c2f25a, changeset:8fef75851181ce79 on the source:ticket1233 branch! Compare before and after those four patches.

These patches also make a related improvement to behavior: do not download a distribution from a remote server if a distribution already present locally would suffice.

Please review! :-)

This is fixed by changeset:a1cef915fd689308, changeset:a44330dfeef23757, changeset:b4c14421f7c2f25a, changeset:8fef75851181ce79 on the source:ticket1233 branch! Compare [before](http://tahoe-lafs.org/buildbot/builders/FreeStorm%20CentOS5-i386/builds/244/steps/test-with-fake-dists/logs/stdio) and [after](http://tahoe-lafs.org/buildbot/builders/FreeStorm%20CentOS5-i386/builds/251/steps/test-with-fake-dists/logs/stdio) those four patches. These patches also make a related improvement to behavior: do not download a distribution from a remote server if a distribution already present locally would suffice. Please review! :-)

zooko commented

2010-11-17 09:54:45 +00:00

Author

Owner

By the way, here is the issue ticket in the Distribute project:

http://bitbucket.org/tarek/distribute/issue/127/prefer-v141-in-binary-egg-format-over-v145

By the way, here is the issue ticket in the Distribute project: <http://bitbucket.org/tarek/distribute/issue/127/prefer-v141-in-binary-egg-format-over-v145>

tahoe-lafs removed the

fixed

label 2010-11-17 10:02:55 +00:00

zooko reopened this issue

2010-11-17 10:02:55 +00:00

davidsarah commented

2010-11-18 02:51:49 +00:00

Author

Owner

Looks like there are still some unexplained buildslave failures on this step, so not ready for review yet.

davidsarah commented

2010-11-18 03:02:18 +00:00

Author

Owner

Replying to zooko:

These patches also make a related improvement to behavior: do not download a distribution from a remote server if a distribution already present locally would suffice.

However, they always look on remote servers to see what distributions are available, before choosing the local one. It shouldn't be necessary even to look remotely in that case. But fixing that would be part of #1220 (or the longer-term project of replacing our use of setuptools).

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/1233#issuecomment-122348): > These patches also make a related improvement to behavior: do not download a distribution from a remote server if a distribution already present locally would suffice. However, they always look on remote servers to see what distributions are available, before choosing the local one. It shouldn't be necessary even to look remotely in that case. But fixing that would be part of #1220 (or the longer-term project of replacing our use of setuptools).

zooko commented

2010-11-18 06:00:22 +00:00

Author

Owner

Replying to davidsarah:

Looks like there are still some unexplained buildslave failures on this step, so not ready for review yet.

Let's see...

Both of them exhibit the same error message:

pkg_resources.VersionConflict: (foolscap 0.4.2 (/usr/lib/pymodules/python2.6), Requirement.parse('foolscap[secure_connections]>=0.5.1'))

and

pkg_resources.VersionConflict: (foolscap 0.4.2 (/usr/local/lib/python2.6/dist-packages/foolscap-0.4.2-py2.6.egg), Requirement.parse('foolscap[secure_connections]>=0.5.1'))

Now what do these two have in common that the rest of the buildslaves, including their brother lucid-amd64 don't?

The answer seems to be that they have foolscap-0.4.2 installed, as shown by flappclient --version in their show-tool-versions steps. The other buildslaves, which don't hit this error, either have no foolscap installed at all or have a foolscap installed which is >= 0.5.1.

Still investigating...

Replying to [davidsarah](/tahoe-lafs/trac-2024-07-25/issues/1233#issuecomment-122352): > Looks like there are still some unexplained buildslave failures on this step, so not ready for review yet. Let's see... * <http://tahoe-lafs.org/buildbot/builders/Brian%20ubuntu-i386%20linode/builds/78> * <http://tahoe-lafs.org/buildbot/builders/Shawn%20ubuntu-amd64/builds/592/steps/test-with-fake-dists/logs/stdio> Both of them exhibit the same error message: ``` pkg_resources.VersionConflict: (foolscap 0.4.2 (/usr/lib/pymodules/python2.6), Requirement.parse('foolscap[secure_connections]>=0.5.1')) ``` and ``` pkg_resources.VersionConflict: (foolscap 0.4.2 (/usr/local/lib/python2.6/dist-packages/foolscap-0.4.2-py2.6.egg), Requirement.parse('foolscap[secure_connections]>=0.5.1')) ``` Now what do these two have in common that the rest of the buildslaves, including their brother [lucid-amd64](http://tahoe-lafs.org/buildbot/builders/lucid-amd64/builds/84) don't? The answer seems to be that they have foolscap-0.4.2 installed, as shown by `flappclient --version` in their show-tool-versions steps. The other buildslaves, which don't hit this error, either have no foolscap installed at all or have a foolscap installed which is >= 0.5.1. Still investigating...

zooko commented

2010-11-18 08:00:29 +00:00

Author

Owner

Hm, that VersionConflict looks kind of like the problem in #1190. Could that have regressed somehow but only with regard to foolscap? Could it have something to do with the "extra requirement" secure_connections that aren't used in our other dependencies? That fix was suppoesd to be verfied by these tests: changeset:390c40cd8ce1e579, [0101030034303-93fa1-0267f4ed3c9014099eb2c9b4d8e43724d7ff85e1], changeset:c3b1bca95450ae40.

Huh-oh, if I change the test suite that source:trunk/misc/build_helpers/test-with-fake-pkg.py@4801 uses to test from allmydata.test.test_base62 (chosen because it does little) to allmydata.test.test_backupdb (which imports pycryptopp) then test-with-fake-pkg.py fails! On my Mac OS 10.6. This is a regression, because it worked back when we committed it to trunk!

I'll attach a darcs patch which changes test-with-fake-pkg to use test_backupdb...

Hm, that `VersionConflict` looks kind of like the problem in #1190. Could that have regressed somehow but only with regard to foolscap? Could it have something to do with the "extra requirement" `secure_connections` that aren't used in our other dependencies? That fix was suppoesd to be verfied by these tests: changeset:390c40cd8ce1e579, [0101030034303-93fa1-0267f4ed3c9014099eb2c9b4d8e43724d7ff85e1], changeset:c3b1bca95450ae40. Huh-oh, if I change the test suite that source:trunk/misc/build_helpers/test-with-fake-pkg.py@4801 uses to test from `allmydata.test.test_base62` (chosen because it does little) to `allmydata.test.test_backupdb` (which imports pycryptopp) then test-with-fake-pkg.py fails! On my Mac OS 10.6. This is a regression, because it worked back when we committed it to trunk! I'll attach a darcs patch which changes `test-with-fake-pkg` to use `test_backupdb`...

zooko commented

2010-11-18 08:01:07 +00:00

Author

Owner

Attachment test-with-fake-pkg-improvement.darcs.patch (20976 bytes) added

**Attachment** test-with-fake-pkg-improvement.darcs.patch (20976 bytes) added

test-with-fake-pkg-improvement.darcs.patch

20 KiB

zooko commented

2010-11-18 08:12:16 +00:00

Author

Owner

Okay let's move further discussion of this VersionConflict over to http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1190#comment:38 .

Okay let's move further discussion of this `VersionConflict` over to <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1190#comment:38> .

zooko commented

2010-11-18 08:18:13 +00:00

Author

Owner

I kind of suspect that the problems with the VersionConflict aren't actually caused by the patches in this ticket, but are somehow being triggered by the test-with-fake-dists step. Perhaps we don't see this VersionConflict on buildbot of current trunk merely because the test-with-fake-dists step ends early on trunk when it gets an exception from trying to build the fake pycryptopp-9.9.99. If we, for example, hacked the fake pycryptopp-9.9.99 so that it stopped raising an exception, thus allowing test-with-fake-dists to run to the end on trunk, would we get the same VersionConflict on "Brian ubuntu-i386 linode" and "Shawn ubuntu-amd64"?

I kind of suspect that the problems with the `VersionConflict` aren't actually caused by the patches in this ticket, but are somehow being triggered by the `test-with-fake-dists` step. Perhaps we don't see this `VersionConflict` on buildbot of current trunk merely because the `test-with-fake-dists` step ends early on trunk when it gets an exception from trying to build the fake `pycryptopp-9.9.99`. If we, for example, hacked the fake `pycryptopp-9.9.99` so that it stopped raising an exception, thus allowing `test-with-fake-dists` to run to the end on trunk, would we get the same `VersionConflict` on "Brian ubuntu-i386 linode" and "Shawn ubuntu-amd64"?

zooko commented

2010-11-19 14:28:52 +00:00

Author

Owner

The issue with the VersionConflict with foolscap is fixed by changeset:50f8c37a2b0049a5 (on source:ticket1233).

The buildbot is nicely green now with &branch=ticket1233:

http://tahoe-lafs.org/buildbot/waterfall?branch=ticket1233

The issue with the `VersionConflict` with foolscap is fixed by changeset:50f8c37a2b0049a5 (on source:ticket1233). The buildbot is nicely green now with `&branch=ticket1233`: <http://tahoe-lafs.org/buildbot/waterfall?branch=ticket1233>

zooko commented

2010-11-19 14:29:18 +00:00

Author

Owner

Please review!

zooko commented

2010-11-19 22:46:51 +00:00

Author

Owner

On IRC, David-Sarah said: "I think ticket1233 can be committed to trunk"

david-sarah@jacaranda.org commented

2010-11-20 06:05:53 +00:00

Author

Owner

In changeset:a1cef915fd689308:

bundled zetuptools: prefer platform-specific dists to platform-independent ones. refs #1233

In changeset:a1cef915fd689308: ``` bundled zetuptools: prefer platform-specific dists to platform-independent ones. refs #1233 ```

zooko@zooko.com commented

2010-11-20 06:05:53 +00:00

Author

Owner

In changeset:b4c14421f7c2f25a:

bundled zetuptoolz: prefer locally-available distributions over remotely-downloaded distributions above all
This fixes #1233. Actually the previous patches—[changeset:a1cef915fd689308] and [changeset:a44330dfeef23757]—fixed it, but with them zetuptoolz would download a higher-numbered distribution from the net instead of using the locally-available (fake) pycryptopp-0.5.24, thus preventing the tests from passing. This patch changes that behavior (which is an improvement in its own right) and also fixes a bug in the tests.

In changeset:b4c14421f7c2f25a: ``` bundled zetuptoolz: prefer locally-available distributions over remotely-downloaded distributions above all This fixes #1233. Actually the previous patches—[changeset:a1cef915fd689308] and [changeset:a44330dfeef23757]—fixed it, but with them zetuptoolz would download a higher-numbered distribution from the net instead of using the locally-available (fake) pycryptopp-0.5.24, thus preventing the tests from passing. This patch changes that behavior (which is an improvement in its own right) and also fixes a bug in the tests. ```

tahoe-lafs added the

fixed

label 2010-11-20 06:05:53 +00:00

zooko@zooko.com closed this issue

2010-11-20 06:05:53 +00:00

zooko commented

2010-12-04 07:27:41 +00:00

Author

Owner

This appears to have failed for David-Sarah just now. They reported:

$ python setup.py build
running darcsver
setup.py darcsver: wrote '1.8.1-r4867' into src/allmydata/_version.py
running develop
Not found: ../tahoe-deps
running egg_info
writing requirements to src\allmydata_tahoe.egg-info\requires.txt
writing src\allmydata_tahoe.egg-info\PKG-INFO
writing top-level names to src\allmydata_tahoe.egg-info\top_level.txt
writing dependency_links to src\allmydata_tahoe.egg-info\dependency_links.txt
writing entry points to src\allmydata_tahoe.egg-info\entry_points.txt
writing manifest file 'src\allmydata_tahoe.egg-info\SOURCES.txt'
running build_ext
Processing darcsver-1.7.1.egg
removing 'd:\tahoe\tahoe-clean\support\lib\site-packages\darcsver-1.7.1.egg' (and everything under it)
Copying darcsver-1.7.1.egg to d:\tahoe\tahoe-clean\support\lib\site-packages
darcsver 1.7.1 is already the active version in easy-install.pth
Deleting support/Scripts\darcsver
Deleting support/Scripts\darcsver.pyscript
Installing darcsver.pyscript script to support/Scripts
Installing darcsver script to support/Scripts

Installed d:\tahoe\tahoe-clean\support\lib\site-packages\darcsver-1.7.1.egg
Creating d:\tahoe\tahoe-clean\support\lib\site-packages\allmydata-tahoe.egg-link (link to src)
allmydata-tahoe 1.8.1-r4867 is already the active version in easy-install.pth
Deleting support/Scripts\tahoe
Deleting support/Scripts\tahoe.pyscript
Installing tahoe.pyscript script to support/Scripts
Installing tahoe script to support/Scripts

Installed d:\tahoe\tahoe-clean\src
Processing dependencies for allmydata-tahoe==1.8.1-r4867
Searching for pycryptopp>=0.5.20
Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/
Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/
Best match: pycryptopp 0.5.25
Processing pycryptopp-0.5.25.tar.gz
Running pycryptopp-0.5.25\setup.py -q bdist_egg --dist-dir c:\users\david\appdata\local\temp\easy_install-ytavvv\pycrypt
opp-0.5.25\egg-dist-tmp-_ox2l8

Setting up environment to run scripts for the current user...
Already done: associate the filetype .pyscript with Python.File for the current user.
Already done: add .pyscript to the user environment variable PATHEXT.
Already done: add .pyw to the user environment variable PATHEXT.
/usr/lib/gcc/i686-pc-mingw32/3.4.4/../../../../i686-pc-mingw32/bin/ld: cannot find -lmsvcr90
collect2: ld returned 1 exit status
error: Setup script exited with error: command 'g++' failed with exit status 1

I don't understand why it didn't choose http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/pycryptopp-0.5.25-py2.6-win32.egg over pycryptopp-0.5.25.tar.gz . Hm, and from where did it get that pycryptopp-0.5.25.tar.gz , anyway? It doesn't say that it downloaded it. Oh, maybe the logic to "prefer locally available over remote as the highest priority" chose pycryptopp-0.5.25.tar.gz because it was present on the local filesystem.

In that case, perhaps this is the expected behavior and we can reclose this ticket? :-/

Maybe we should patch zetuptoolz to verbosely explain why it chose what it did and where it acquired it?

This appears to have failed for David-Sarah just now. They reported: ``` $ python setup.py build running darcsver setup.py darcsver: wrote '1.8.1-r4867' into src/allmydata/_version.py running develop Not found: ../tahoe-deps running egg_info writing requirements to src\allmydata_tahoe.egg-info\requires.txt writing src\allmydata_tahoe.egg-info\PKG-INFO writing top-level names to src\allmydata_tahoe.egg-info\top_level.txt writing dependency_links to src\allmydata_tahoe.egg-info\dependency_links.txt writing entry points to src\allmydata_tahoe.egg-info\entry_points.txt writing manifest file 'src\allmydata_tahoe.egg-info\SOURCES.txt' running build_ext Processing darcsver-1.7.1.egg removing 'd:\tahoe\tahoe-clean\support\lib\site-packages\darcsver-1.7.1.egg' (and everything under it) Copying darcsver-1.7.1.egg to d:\tahoe\tahoe-clean\support\lib\site-packages darcsver 1.7.1 is already the active version in easy-install.pth Deleting support/Scripts\darcsver Deleting support/Scripts\darcsver.pyscript Installing darcsver.pyscript script to support/Scripts Installing darcsver script to support/Scripts Installed d:\tahoe\tahoe-clean\support\lib\site-packages\darcsver-1.7.1.egg Creating d:\tahoe\tahoe-clean\support\lib\site-packages\allmydata-tahoe.egg-link (link to src) allmydata-tahoe 1.8.1-r4867 is already the active version in easy-install.pth Deleting support/Scripts\tahoe Deleting support/Scripts\tahoe.pyscript Installing tahoe.pyscript script to support/Scripts Installing tahoe script to support/Scripts Installed d:\tahoe\tahoe-clean\src Processing dependencies for allmydata-tahoe==1.8.1-r4867 Searching for pycryptopp>=0.5.20 Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/ Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/ Best match: pycryptopp 0.5.25 Processing pycryptopp-0.5.25.tar.gz Running pycryptopp-0.5.25\setup.py -q bdist_egg --dist-dir c:\users\david\appdata\local\temp\easy_install-ytavvv\pycrypt opp-0.5.25\egg-dist-tmp-_ox2l8 Setting up environment to run scripts for the current user... Already done: associate the filetype .pyscript with Python.File for the current user. Already done: add .pyscript to the user environment variable PATHEXT. Already done: add .pyw to the user environment variable PATHEXT. /usr/lib/gcc/i686-pc-mingw32/3.4.4/../../../../i686-pc-mingw32/bin/ld: cannot find -lmsvcr90 collect2: ld returned 1 exit status error: Setup script exited with error: command 'g++' failed with exit status 1 ``` I don't understand why it didn't choose <http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/pycryptopp-0.5.25-py2.6-win32.egg> over pycryptopp-0.5.25.tar.gz . Hm, and from where did it get that pycryptopp-0.5.25.tar.gz , anyway? It doesn't say that it downloaded it. Oh, maybe the logic to "prefer locally available over remote as the highest priority" chose pycryptopp-0.5.25.tar.gz because it was present on the local filesystem. In that case, perhaps this is the expected behavior and we can reclose this ticket? :-/ Maybe we should patch zetuptoolz to verbosely explain why it chose what it did and where it acquired it?

davidsarah commented

2010-12-04 08:46:22 +00:00

Author

Owner

Replying to zooko:
[...]

I don't understand why it didn't choose http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/pycryptopp-0.5.25-py2.6-win32.egg over pycryptopp-0.5.25.tar.gz . Hm, and from where did it get that pycryptopp-0.5.25.tar.gz , anyway? It doesn't say that it downloaded it.

Oh, Problem Between Keyboard and Chair. I had that in my tahoe-deps directory.

Oh, maybe the logic to "prefer locally available over remote as the highest priority" chose pycryptopp-0.5.25.tar.gz because it was present on the local filesystem.

In that case, perhaps this is the expected behavior and we can reclose this ticket? :-/

Yes. (It wasn't reopened.)

Maybe we should patch zetuptoolz to verbosely explain why it chose what it did and where it acquired it?

That would be useful, as a new ticket.

Replying to [zooko](/tahoe-lafs/trac-2024-07-25/issues/1233#issuecomment-122365): [...] > I don't understand why it didn't choose <http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/pycryptopp-0.5.25-py2.6-win32.egg> over pycryptopp-0.5.25.tar.gz . Hm, and from where did it get that pycryptopp-0.5.25.tar.gz , anyway? It doesn't say that it downloaded it. Oh, Problem Between Keyboard and Chair. I had that in my tahoe-deps directory. > Oh, maybe the logic to "prefer locally available over remote as the highest priority" chose pycryptopp-0.5.25.tar.gz because it was present on the local filesystem. > > In that case, perhaps this is the expected behavior and we can reclose this ticket? :-/ Yes. (It wasn't reopened.) > Maybe we should patch zetuptoolz to verbosely explain why it chose what it did and where it acquired it? That would be useful, as a new ticket.

prefer binary eggs which satisfy dependencies over source eggs which are even newer #1233