don't require newer foolscap if you have older Twisted #1326

New issue

Closed

opened 2011-01-16 00:57:51 +00:00 by zooko · 8 comments

zooko commented

2011-01-16 00:57:51 +00:00

The current stable release—v1.8.1—requires Twisted >= 2.4.0 and foolscap >= 0.5.1. The current trunk requires Twisted >= 2.4.0 and foolscap >= 0.6.0. Brian bemoaned this increased requirement on the mailing list, since there is actually nothing wrong with foolscap 0.5.1 unless you also happen to be using Twisted >= 10.2.

Brian gave as example that people on Ubuntu 10.04 Long-Term-Support "Lucid" have Twisted-10.0 and foolscap-0.5.1.

This patch makes it so that if you already have Twisted installed and the version is < 10.2 then the required version of foolscap is >= 0.5.1, else it is >= 0.6.0.

David-Sarah proposed a different approach, in which python setup.py build would build a local private copy of foolscap >= 0.6.0 even if the system already have foolscap-0.5.1 installed. I'm not sure users would be okay with that behavior -- it runs counter to the desiderata that Tahoe-LAFS will use the packages that are provided in the system if they are of an acceptable version number.

In any case, I would like to apply my patch right away for Tahoe-LAFS v1.8.2, so review-needed!

The current stable release—v1.8.1—[requires Twisted >= 2.4.0 and foolscap >= 0.5.1](http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/src/allmydata/_auto_deps.py?annotate=blame&rev=4799#L19). The current trunk [requires Twisted >= 2.4.0 and foolscap >= 0.6.0](http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/src/allmydata/_auto_deps.py?annotate=blame&rev=4893#L19). Brian [bemoaned](http://tahoe-lafs.org/pipermail/tahoe-dev/2011-January/005842.html) this increased requirement on the mailing list, since there is actually nothing wrong with `foolscap 0.5.1` *unless* you also happen to be using `Twisted >= 10.2`. Brian gave as example that people on Ubuntu 10.04 Long-Term-Support "Lucid" have `Twisted-10.0` and `foolscap-0.5.1`. This patch makes it so that if you already have `Twisted` installed and the version is `< 10.2` then the required version of `foolscap` is `>= 0.5.1`, else it is `>= 0.6.0`. David-Sarah [proposed](http://tahoe-lafs.org/pipermail/tahoe-dev/2011-January/005867.html) a different approach, in which `python setup.py build` would build a local private copy of `foolscap >= 0.6.0` even if the system already have `foolscap-0.5.1` installed. I'm not sure users would be okay with that behavior -- it runs counter to the desiderata that Tahoe-LAFS will use the packages that are provided in the system if they are of an acceptable version number. In any case, I would like to apply my patch right away for Tahoe-LAFS v1.8.2, so `review-needed`!

zooko added the

labels 2011-01-16 00:57:51 +00:00

zooko added this to the 1.8.2 milestone 2011-01-16 00:57:51 +00:00

zooko commented

2011-01-16 00:58:13 +00:00

Author

Attachment foolscap051060.darcs.patch (13690 bytes) added

**Attachment** foolscap051060.darcs.patch (13690 bytes) added

foolscap051060.darcs.patch

13 KiB

zooko commented

2011-01-16 08:57:26 +00:00

Author

The desert island test is currently red because of this:

http://tahoe-lafs.org/buildbot/builders/clean/builds/2667/steps/test-desert-island/logs/stdio

Downloading http://foolscap.lothar.com/releases/foolscap-0.6.0.tar.gz
Failed: build tried to download files
gmake: *** [test-desert-island] Error 1

I guess we need to either update the foolscap version in tahoe-deps or else apply the patch from this ticket. :-)

The desert island test is currently red because of this: <http://tahoe-lafs.org/buildbot/builders/clean/builds/2667/steps/test-desert-island/logs/stdio> ``` Downloading http://foolscap.lothar.com/releases/foolscap-0.6.0.tar.gz Failed: build tried to download files gmake: *** [test-desert-island] Error 1 ``` I guess we need to either update the foolscap version in [tahoe-deps](http://tahoe-lafs.org/source/tahoe-lafs/deps/) or else apply the patch from this ticket. :-)

daira commented

2011-01-16 20:19:02 +00:00

Replying to zooko:

David-Sarah proposed a different approach, in which python setup.py build would build a local private copy of foolscap >= 0.6.0 even if the system already have foolscap-0.5.1 installed.

There must be some confusion -- this is what trunk already does.

If having an installed foolscap-0.5.1 breaks the ability to build a private foolscap 0.6.0 in support/.../site-packages, that's a bug. It works fine for me, and on the buildbots, many of which have foolscap 0.5.1 installed.

(Potentially #1258 could cause such breakage, but only if the installed foolscap is in a directory shared with other packages that are Tahoe dependencies. If this causes an incorrect version to be found, [check_all_requirements]source:ticket1306/src/allmydata/init.py@4963#L163 on the ticket1306 branch will tell you to remove the incorrect version from the shared directory.)

Replying to [zooko](/tahoe-lafs/trac/issues/27270): > David-Sarah [proposed](http://tahoe-lafs.org/pipermail/tahoe-dev/2011-January/005867.html) a different approach, in which `python setup.py build` would build a local private copy of `foolscap >= 0.6.0` even if the system already have `foolscap-0.5.1` installed. There must be some confusion -- this is what trunk *already* does. If having an installed foolscap-0.5.1 breaks the ability to build a private foolscap 0.6.0 in `support/.../site-packages`, that's a bug. It works fine for me, and on the buildbots, many of which have foolscap 0.5.1 installed. (Potentially #1258 could cause such breakage, but only if the installed foolscap is in a directory shared with other packages that are Tahoe dependencies. If this causes an incorrect version to be found, [check_all_requirements]source:ticket1306/src/allmydata/*init*.py@4963#L163 on the ticket1306 branch will tell you to remove the incorrect version from the shared directory.)

zooko commented

2011-01-16 20:46:40 +00:00

Author

Replying to [davidsarah]comment:2:

Replying to zooko:

David-Sarah proposed a different approach, in which python setup.py build would build a local private copy of foolscap >= 0.6.0 even if the system already have foolscap-0.5.1 installed.

There must be some confusion -- this is what trunk already does.

Hm, yes I didn't understand your proposal. On that mailing list message, you wrote:
'setup.py build', by default, should either:

have built a private copy of twisted < 10.2 (always used by bin/tahoe)
that won't be affected by upgrades of the installed version, or
have built a private copy of foolscap 0.6.0 (always used by bin/tahoe).

So if I understand correctly, you prefer for Tahoe-LAFS v1.8.2 to require foolscap >= 0.6.0 even if Twisted is < 10.2 (or alternately to require Twisted <= 10.2) for installation, because if Tahoe-LAFS v1.8.2 allows foolscap-0.5.1 at install time, then the user may later upgrade their Twisted to >= 10.2 and then execute bin/tahoe without re-running python setup.py build and they would then get a failure to start due to conflicting version requirements.

This is what trunk currently does.

The drawback to this approach is Brian's original complaint -- that it means Tahoe-LAFS requires foolscap >= 0.6.0 when the foolscap-0.5.1 that comes with Ubuntu Lucid should really be good enough.

Now, I don't feel so strongly about this, so I'm basically -0 on leaving trunk as it is, because I can live with it, but I would like to understand your motivation better in order to work with you on related issues.

My idea of packaging is that there is a "build/install" step where the packaging system can do stuff having to do with dependencies--either automatically resolve dependencies (complicated and controversial, but handy for users who don't know how to resolve dependencies themselves), or at least detect unresolved dependencies and clearly alert the user to the problem and exit loudly. Then, at run-time the packaging system could re-examine dependencies in order to give a better error message if a dependency is missing or is of an incompatible version, but it must not resolve dependencies at that time.

So I have the feeling that if the user changes installed deps after build time and before run-time, then we have no responsibility beyond erroring out cleanly, but you seem to think we should pay a cost (in the form of a higher version requirement on foolscap) in order to make Tahoe-LAFS run correctly even if the user has done that,.

Replying to [davidsarah]comment:2: > Replying to [zooko](/tahoe-lafs/trac/issues/27270): > > David-Sarah [proposed](http://tahoe-lafs.org/pipermail/tahoe-dev/2011-January/005867.html) a different approach, in which `python setup.py build` would build a local private copy of `foolscap >= 0.6.0` even if the system already have `foolscap-0.5.1` installed. > > There must be some confusion -- this is what trunk *already* does. Hm, yes I didn't understand your proposal. On that mailing list message, you wrote: 'setup.py build', by default, should either: - have built a private copy of twisted < 10.2 (always used by bin/tahoe) that won't be affected by upgrades of the installed version, or - have built a private copy of foolscap 0.6.0 (always used by bin/tahoe). So if I understand correctly, you prefer for Tahoe-LAFS v1.8.2 to require `foolscap >= 0.6.0` even if `Twisted` is `< 10.2` (or alternately to require `Twisted <= 10.2`) for installation, because if Tahoe-LAFS v1.8.2 allows `foolscap-0.5.1` at install time, then the user may later upgrade their `Twisted` to `>= 10.2` and then execute `bin/tahoe` without re-running `python setup.py build` and they would then get a failure to start due to conflicting version requirements. This is what trunk currently does. The drawback to this approach is [Brian's original complaint](http://tahoe-lafs.org/pipermail/tahoe-dev/2011-January/005842.html) -- that it means Tahoe-LAFS requires `foolscap >= 0.6.0` when the `foolscap-0.5.1` that comes with Ubuntu Lucid should really be good enough. Now, I don't feel so strongly about this, so I'm basically `-0` on leaving trunk as it is, because I can live with it, but I would like to understand your motivation better in order to work with you on related issues. My idea of packaging is that there is a "build/install" step where the packaging system can do stuff having to do with dependencies--either automatically resolve dependencies (complicated and controversial, but handy for users who don't know how to resolve dependencies themselves), or at least detect unresolved dependencies and clearly alert the user to the problem and exit loudly. Then, at *run-time* the packaging system *could* re-examine dependencies in order to give a better error message if a dependency is missing or is of an incompatible version, but it *must not* resolve dependencies at that time. So I have the feeling that if the user changes installed deps after build time and before run-time, then we have no responsibility beyond erroring out cleanly, but you seem to think we should pay a cost (in the form of a higher version requirement on `foolscap`) in order to make Tahoe-LAFS run correctly even if the user has done that,.

daira commented

2011-01-16 21:20:17 +00:00

After further thought and some IRC discussion, my complaint would be addressed if in the case where bin/tahoe fails because twisted >= 10.2 has been installed, it gives a message saying to re-run python setup.py build. I'll prepare a patch for that.

After further thought and some IRC discussion, my complaint would be addressed if in the case where `bin/tahoe` fails because twisted >= 10.2 has been installed, it gives a message saying to re-run `python setup.py build`. I'll prepare a patch for that.

warner commented

2011-01-17 05:42:53 +00:00

Given that #1329 can be fixed by depending upon the new (just released a few hours ago) foolscap-0.6.1, perhaps this ticket has been rendered obsolete.

zooko commented

2011-01-18 08:22:43 +00:00

Author

We agree to fix this by applying attachment:foolscap051060.darcs.patch foolscap051060.darcs.patch, adding code to catch version conflict exceptions at start-up time and print out an error message suggesting to the user to re-run python setup.py build, and fixing #1329.

Next step is for David-Sarah to write that patch to print a useful error message if there is a version error at start-up time.

We agree to fix this by applying [attachment:foolscap051060.darcs.patch](/tahoe-lafs/trac/attachments/000078ac-ee3e-f934-29ab-a77bfb7955c4) foolscap051060.darcs.patch, adding code to catch version conflict exceptions at start-up time and print out an error message suggesting to the user to re-run `python setup.py build`, and fixing #1329. Next step is for David-Sarah to write that patch to print a useful error message if there is a version error at start-up time.

daira commented

2011-01-19 08:11:09 +00:00

It turned out to be much more difficult than I or zooko had thought to make the foolscap requirement dependent on the installed version of Twisted.

The problem is that:

the first import of either pkg_resources or setuptools in source:setup.py must be after *requires* has been set (because that import has side effects on the sys.path that depend on *requires*);
*requires* is set from the requirements computed by source:src/allmydata/_auto_deps.py (loaded using execfile);
if _auto_deps.py imports pkg_resources in order to get the version of Twisted (for the current working set), then that first import would have to be both before and after setting *requires*: contradiction.

If _auto_deps.py were to import twisted directly, OTOH, that wouldn't necessarily be the same Twisted version that would later be imported at run time.

So we decided to just require foolscap >= 0.6.1 instead. The difference between 0.6.0 and 0.6.1 is to quiet a DeprecationWarning (#1329).

It turned out to be much more difficult than I or zooko had thought to make the foolscap requirement dependent on the installed version of Twisted. The problem is that: * the first import of either `pkg_resources` or `setuptools` in source:setup.py must be after `*requires*` has been set (because that import has side effects on the `sys.path` that depend on `*requires*`); * `*requires*` is set from the requirements computed by source:src/allmydata/_auto_deps.py (loaded using `execfile`); * if `_auto_deps.py` imports `pkg_resources` in order to get the version of Twisted (for the current working set), then that first import would have to be both before *and* after setting `*requires*`: contradiction. If `_auto_deps.py` were to import `twisted` directly, OTOH, that wouldn't necessarily be the same Twisted version that would later be imported at run time. So we decided to just require `foolscap >= 0.6.1` instead. The difference between 0.6.0 and 0.6.1 is to quiet a DeprecationWarning (#1329).