tahoe-lafs/trac

use setuptools's --multi-version mode #530

New issue

Closed

opened 2008-10-22 02:27:05 +00:00 by zooko · 16 comments

zooko commented

2008-10-22 02:27:05 +00:00

(http://peak.telecommunity.com/DevCenter/setuptools#develop-deploy-the-project-source-in-development-mode)

This would make it so that if there is already an older version of a dependency, e.g. pyutil, present on the system and we require a newer version, that it would go ahead and install the newer version, and Tahoe would use the newer version, without requiring the user to delete the older version from her system.

(http://peak.telecommunity.com/DevCenter/setuptools#develop-deploy-the-project-source-in-development-mode) This would make it so that if there is already an older version of a dependency, e.g. pyutil, present on the system and we require a newer version, that it would go ahead and install the newer version, and Tahoe would use the newer version, without requiring the user to delete the older version from her system.

zooko added the

labels 2008-10-22 02:27:05 +00:00

zooko added this to the undecided milestone 2008-10-22 02:27:05 +00:00

zooko commented

2008-11-25 20:06:49 +00:00

Author

This was implemented in changeset:0de6e616e0d0890d but it caused a mysterious problem on most buildbots. I think the problem might be due to an interaction with our "fake setuptools" code in our "Trial" class in source:setup.py and that the problem might be fixed by Chris Galvan's solution to #505 (wanted: a setuptools plugin to make unit tests be executed with trial instead of with pyunit). Anyway, I undid this change in changeset:1d377cc2d9871e39 in order to make the buildbots green again. By the way, this issue seems to be interfering with Francois's attempt to fix #534 ("tahoe cp" command encoding issue) since an older version of simplejson is installed system-wide on the buildslave that he is experimenting with.

warner commented

2008-12-07 22:01:43 +00:00

I'm not sure I see how --multi-version would help this. We know that the tahoe .deb is not supposed to contain anything but tahoe code. What --multi-version might do is to make it easier to build tahoe from source on a host that has an older version of e.g. simplejson installed from a deb. But that seems to work anyways right now.

Zooko mentioned, in response to #555 ("tahoe .deb cannot be installed on hardy because of simplejson dependency"), that this #530 ticket "would have meant that simplejson upgrade happened automatically". I'm not sure I see how --multi-version would help this. We know that the tahoe .deb is not supposed to contain anything but tahoe code. What --multi-version might do is to make it easier to build tahoe from source on a host that has an older version of e.g. simplejson installed from a deb. But that seems to work anyways right now.

zooko commented

2008-12-08 18:15:05 +00:00

Author

Oh I guess I confused #555 with the deployment issue that arose this weekend. That issue involved building from source (or, equivalently, running easy_install allmydata-tahoe. This ticket -- #530 -- would have made it so that deployment attempt would have automatically succeeded by automatically downloading and installing a newer simplejson in egg or source form. Which isn't what #555 is requesting.

Oh I guess I confused #555 with the deployment issue that arose this weekend. That issue involved building from source (or, equivalently, running `easy_install allmydata-tahoe`. This ticket -- #530 -- would have made it so that deployment attempt would have automatically succeeded by automatically downloading and installing a newer `simplejson` in egg or source form. Which isn't what #555 is requesting.

zooko commented

2009-01-20 19:01:40 +00:00

Author

Sigh, changeset:8148366d93c34dbe unapplies this patch, again, because setuptools doesn't create the easy-install.pth, setuptools.pth, and site.py files when we add --multi-version.

http://bugs.python.org/setuptools/issue57 # develop doesn't create .pth files and site.py if --multi-version

Sigh, changeset:8148366d93c34dbe unapplies this patch, again, because setuptools doesn't create the `easy-install.pth`, `setuptools.pth`, and `site.py` files when we add `--multi-version`. <http://bugs.python.org/setuptools/issue57> # develop doesn't create .pth files and site.py if --multi-version

daira commented

2009-12-07 03:32:34 +00:00

Replying to zooko:

http://bugs.python.org/setuptools/issue57 # develop doesn't create .pth files and site.py if --multi-version

which got closed with the following comment:

Behavior is as intended. More precisely, it is an intended feature of
--multi-version that it does not require .pth or site.py support in the target
directory.

Can someone translate this to English for those unfamiliar with the vagaries of Python package management? Is this just more of the same "setuptools is broken-by-design" that seems to be causing so many other problems?

Replying to [zooko](/tahoe-lafs/trac/issues/530#issuecomment-368849): > <http://bugs.python.org/setuptools/issue57> # develop doesn't create .pth files and site.py if --multi-version which got closed with the following comment: > Behavior is as intended. More precisely, it is an intended feature of `--multi-version` that it does not require `.pth` or `site.py` support in the target directory. Can someone translate this to English for those unfamiliar with the vagaries of Python package management? Is this just more of the same "setuptools is broken-by-design" that seems to be causing so many other problems?

zooko commented

2010-10-03 07:31:59 +00:00

Author

See http://tahoe-lafs.org/pipermail/tahoe-dev/2010-October/005324.html

See <http://tahoe-lafs.org/pipermail/tahoe-dev/2010-October/005324.html>

zooko commented

2010-10-05 20:28:42 +00:00

Author

In changeset:98ffbfb31faccdaf:

setup: add --multi-version to the "setup.py develop" command-line
fixes #530. I earlier tried this twice (see #530 for history) and then twice rolled it back due to some problems that arose. However, I didn't write down what the problems were in enough detail on the ticket that I can tell today whether those problems are still issues, so here goes the third attempt. (I did write down on the ticket that it would not create site.py or .pth files in the target directory with --multi-version mode, but I didn't explain why *that* was a problem.)

In changeset:98ffbfb31faccdaf: ``` setup: add --multi-version to the "setup.py develop" command-line fixes #530. I earlier tried this twice (see #530 for history) and then twice rolled it back due to some problems that arose. However, I didn't write down what the problems were in enough detail on the ticket that I can tell today whether those problems are still issues, so here goes the third attempt. (I did write down on the ticket that it would not create site.py or .pth files in the target directory with --multi-version mode, but I didn't explain why *that* was a problem.) ```

zooko added the

r/fixed

label 2010-10-05 20:28:42 +00:00

zooko closed this issue

2010-10-05 20:28:42 +00:00

warner commented

2010-10-06 18:26:09 +00:00

Using /home/warner/stuff/tahoe/darcs-trunk/support/lib/python2.6/site-packages/zbase32-1.1.2-py2.6.egg

Because this distribution was installed --multi-version, before you can
import modules from this package in an application, you will need to
'import pkg_resources' and then use a 'require()' call similar to one of
these examples, in order to select the desired version:

    pkg_resources.require("zbase32")  # latest installed version
    pkg_resources.require("zbase32==1.1.2")  # this exact version
    pkg_resources.require("zbase32>=1.1.2")  # this version or higher


Note also that the installation directory must be on sys.path at runtime for
this to work.  (e.g. by being the application's script directory, by being on
PYTHONPATH, or by being added to sys.path by your code.)

I suppose that this is a benign warning, but there's too much output there
for me to tell: I first assumed that it was a fatal error.

After building it, trying to run setup.py test on my debian box failed
(after emitting the same 353 lines of warnings):

pkg_resources.VersionConflict: (pycryptopp 0.5.17 (/usr/lib/pymodules/python2.6), Requirement.parse('pycryptopp>=0.5.20'))

That 0.5.17 is from the current debian/sid python-pycryptopp .deb package, which
includes .egg-info data (and pkg_resources.require("pycryptopp")
returns 0.5.17).

Does --multi-version break if there is a version on PYTHONPATH that was
installed without --multi-version?

After this change, a "python setup.py build" that shouldn't do anything (i.e. do it twice and capture the output from the second run) emits 353 lines, including 16 copies of a warning about how to use versioned dependencies in the importing code: ``` Using /home/warner/stuff/tahoe/darcs-trunk/support/lib/python2.6/site-packages/zbase32-1.1.2-py2.6.egg Because this distribution was installed --multi-version, before you can import modules from this package in an application, you will need to 'import pkg_resources' and then use a 'require()' call similar to one of these examples, in order to select the desired version: pkg_resources.require("zbase32") # latest installed version pkg_resources.require("zbase32==1.1.2") # this exact version pkg_resources.require("zbase32>=1.1.2") # this version or higher Note also that the installation directory must be on sys.path at runtime for this to work. (e.g. by being the application's script directory, by being on PYTHONPATH, or by being added to sys.path by your code.) ``` I suppose that this is a benign warning, but there's too much output there for me to tell: I first assumed that it was a fatal error. After building it, trying to run `setup.py test` on my debian box failed (after emitting the same 353 lines of warnings): ``` pkg_resources.VersionConflict: (pycryptopp 0.5.17 (/usr/lib/pymodules/python2.6), Requirement.parse('pycryptopp>=0.5.20')) ``` That 0.5.17 is from the current debian/sid python-pycryptopp `.deb` package, which includes .egg-info data (and `pkg_resources.require("pycryptopp")` returns 0.5.17). Does `--multi-version` break if there is a version on PYTHONPATH that was installed **without** `--multi-version`?

warner commented

2010-10-06 19:34:21 +00:00

Running python -c 'import pkg_resources; print pkg_resources.require("pycryptopp")', either from inside or outside my tahoe tree, reports:

`[0.5.17 (/usr/lib/pymodules/python2.6)]pycryptopp`

Amending that to use "pycryptopp>=0.5.20" results in an exception in both places:

64:warner@fluxx% python -c 'import pkg_resources; print pkg_resources.require("pycryptopp>=0.5.20")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 654, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 556, in resolve
    raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (pycryptopp 0.5.17 (/usr/lib/pymodules/python2.6), Requirement.parse('pycryptopp>=0.5.20'))

Adding support/lib/python2.6/site-packages to my PYTHONPATH (which is what I vaguely remember we used to do inside bin/tahoe) doesn't change its behavior. This is the same error I'm seeing when I do setup.py test.

I started with a setup.py build, so my support/lib/python2.6/site-packages directory contains:

55:warner@fluxx% ls support/lib/python2.6/site-packages/
allmydata-tahoe.egg-link  foolscap-0.5.1-py2.6.egg/                  setuptools-0.6c16dev2.egg/
argparse-1.1-py2.6.egg/   pycryptopp-0.5.25-py2.6-linux-x86_64.egg/  zbase32-1.1.2-py2.6.egg/
darcsver-1.6.3.egg/       pyutil-1.7.12-py2.6.egg/                   zfec-1.4.7-py2.6-linux-x86_64.egg/

I wonder if that -linux-x86_64 in there is causing problems.

Running `python -c 'import pkg_resources; print pkg_resources.require("pycryptopp")'`, either from inside or outside my tahoe tree, reports: `[0.5.17 (/usr/lib/pymodules/python2.6)]pycryptopp` Amending that to use `"pycryptopp>=0.5.20"` results in an exception in both places: ``` 64:warner@fluxx% python -c 'import pkg_resources; print pkg_resources.require("pycryptopp>=0.5.20")' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 654, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 556, in resolve raise VersionConflict(dist,req) # XXX put more info here pkg_resources.VersionConflict: (pycryptopp 0.5.17 (/usr/lib/pymodules/python2.6), Requirement.parse('pycryptopp>=0.5.20')) ``` Adding support/lib/python2.6/site-packages to my `PYTHONPATH` (which is what I vaguely remember we used to do inside `bin/tahoe`) doesn't change its behavior. This is the same error I'm seeing when I do `setup.py test`. I started with a `setup.py build`, so my support/lib/python2.6/site-packages directory contains: ``` 55:warner@fluxx% ls support/lib/python2.6/site-packages/ allmydata-tahoe.egg-link foolscap-0.5.1-py2.6.egg/ setuptools-0.6c16dev2.egg/ argparse-1.1-py2.6.egg/ pycryptopp-0.5.25-py2.6-linux-x86_64.egg/ zbase32-1.1.2-py2.6.egg/ darcsver-1.6.3.egg/ pyutil-1.7.12-py2.6.egg/ zfec-1.4.7-py2.6-linux-x86_64.egg/ ``` I wonder if that `-linux-x86_64` in there is causing problems.

daira commented

2010-10-30 00:29:55 +00:00

Reopening until the warnings in comment:368855 are fixed.

Reopening until the warnings in [comment:368855](/tahoe-lafs/trac/issues/530#issuecomment-368855) are fixed.

daira removed the

r/fixed

label 2010-10-30 00:29:55 +00:00

daira reopened this issue

2010-10-30 00:29:55 +00:00

zooko commented

2010-10-30 04:20:59 +00:00

Author

Maybe we don't need --multi-version? Now that we have a reliable test of this behavior (see the step "test-with-fake-pkg" on buildbot, or run python misc/build_helpers/test-with-fake-pkg.py), maybe we should try removing --multi-version and see if that test stays green.

Maybe we don't need `--multi-version`? Now that we have a reliable test of this behavior (see the step "test-with-fake-pkg" on buildbot, or run `python misc/build_helpers/test-with-fake-pkg.py`), maybe we should try removing `--multi-version` and see if that test stays green.

daira commented

2010-10-30 05:57:27 +00:00

Replying to zooko:

Maybe we don't need --multi-version? Now that we have a reliable test of this behavior (see the step "test-with-fake-pkg" on buildbot, or run python misc/build_helpers/test-with-fake-pkg.py), maybe we should try removing --multi-version and see if that test stays green.

Replying to [zooko](/tahoe-lafs/trac/issues/530#issuecomment-368860): > Maybe we don't need `--multi-version`? Now that we have a reliable test of this behavior (see the step "test-with-fake-pkg" on buildbot, or run `python misc/build_helpers/test-with-fake-pkg.py`), maybe we should try removing `--multi-version` and see if that test stays green. +1

zooko commented

2010-11-15 09:45:41 +00:00

Author

I created [a ticket530 branch]source:ticket530 to test patches on the buildbot. Then I forced a build on all buildbots with the current ticket530 (which was unchanged from the current trunk), and now I will push a patch to remove --multi-version (rolling back that part of changeset:98ffbfb31faccdaf) to see if it fares better or worse than trunk.

I created [a ticket530 branch]source:ticket530 to test patches on the buildbot. Then I forced a build on all buildbots with the current ticket530 (which was unchanged from the current trunk), and now I will push a patch to remove `--multi-version` (rolling back that part of changeset:98ffbfb31faccdaf) to see if it fares better or worse than trunk.

zooko commented

2010-11-15 09:49:54 +00:00

Author

Okay I committed changeset:5f61bad92d8766e7 to the source:ticket530 branch.

zooko commented

2010-11-15 21:03:28 +00:00

Author

Removing --multi-version didn't induce any regressions on buildbot. I guess I would like to add a unit test making sure that build doesn't emit certain scary warnings, which test will go from red to green when I commit changeset:5f61bad92d8766e7 to trunk...

Removing `--multi-version` didn't induce any regressions on buildbot. I guess I would like to add a unit test making sure that build doesn't emit certain scary warnings, which test will go from red to green when I commit changeset:5f61bad92d8766e7 to trunk...

zooko commented

2010-11-16 07:33:06 +00:00

Author

Fixed in changeset:5f61bad92d8766e7. I decided not to add a unit tests as I couldn't think of a way to check whether the output of build was too scary without "overfitting" and just checking for this particular message text.

Rows
Columns