tahoe-lafs/trac

figure out why "FreeStorm Win7-amd64-mingw py2.6" is red on the "test-with-fake-pkg" step #1246

New issue

Closed

opened 2010-11-02 14:52:36 +00:00 by zooko · 26 comments

zooko commented

2010-11-02 14:52:36 +00:00

(Note that the two failures here have different causes. From comment:4 onwards, this ticket is about the second one.)

MM's buildslave "MM netbsd5 i386 warp" fails the test-from-egg step, like this:

Traceback (most recent call last):
  File "/home/tahoe/buildslave/tahoe/netbsd5-i386/build/misc/build_helpers/run_trial.py", line 52, in <module>
    __import__(modulename)
ImportError: No module named allmydata.test

FS's buildslave "FreeStorm Win7-amd64-mingw py2.6" fails the test-with-fake-pkg step, like this:

Traceback (most recent call last):
  File "c:\buildbot_tahoe\FreeStorm_Win7-amd64-mingw_py2.6\build\src\..\misc\build_helpers\run_trial.py", line 98, in <module>
    raise AssertionError(msg)
AssertionError: We seem to be testing the code at 'c:\\python26\\lib'
(according to the source filename 'c:\\python26\\lib\\site-packages\\allmydata\\test\\test_base62.pyc'),
but expected to be testing the code at 'c:\\buildbot_tahoe\\freestorm_win7-amd64-mingw_py2.6\\build'.
This script needs to be run from the source directory to be tested.

(In a subsequent build, this buildslave hung, and then in subsequent builds it was unable to start due to a process holding a handle to the build directory.)

(Note that the two failures here have different causes. From comment:4 onwards, this ticket is about the second one.) MM's buildslave "MM netbsd5 i386 warp" fails the test-from-egg step, like this: ``` Traceback (most recent call last): File "/home/tahoe/buildslave/tahoe/netbsd5-i386/build/misc/build_helpers/run_trial.py", line 52, in <module> __import__(modulename) ImportError: No module named allmydata.test ``` FS's buildslave "FreeStorm Win7-amd64-mingw py2.6" fails the test-with-fake-pkg step, like this: ``` Traceback (most recent call last): File "c:\buildbot_tahoe\FreeStorm_Win7-amd64-mingw_py2.6\build\src\..\misc\build_helpers\run_trial.py", line 98, in <module> raise AssertionError(msg) AssertionError: We seem to be testing the code at 'c:\\python26\\lib' (according to the source filename 'c:\\python26\\lib\\site-packages\\allmydata\\test\\test_base62.pyc'), but expected to be testing the code at 'c:\\buildbot_tahoe\\freestorm_win7-amd64-mingw_py2.6\\build'. This script needs to be run from the source directory to be tested. ``` (In [a subsequent build](http://tahoe-lafs.org/buildbot/builders/FreeStorm%20Win7-amd64-mingw%20py2.6/builds/149), this buildslave [hung](http://tahoe-lafs.org/buildbot/builders/FreeStorm%20Win7-amd64-mingw%20py2.6/builds/149/steps/test/logs/stdio), and then in subsequent builds it was [unable to start](http://tahoe-lafs.org/buildbot/builders/FreeStorm%20Win7-amd64-mingw%20py2.6/builds/150/steps/darcs/logs/stdio) due to a process holding a handle to the `build` directory.)

zooko added the

labels 2010-11-02 14:52:36 +00:00

zooko added this to the soon milestone 2010-11-02 14:52:36 +00:00

midnightmagic commented

2010-11-03 18:42:34 +00:00

Owner

For the warp client, there isn't an allmydata egg except inside the egginstalldir itself; therefore the issue seems to be:

Code highlighting:

import glob,os,subprocess,sys
trial=os.path.join(os.getcwd(), 'misc', 'build_helpers', 'run_trial.py')
os.chdir('.')

**Right in here... **

If I change the above os.chdir('.') to os.chdir('egginstalldir') then the test proceeds as normal. Since this is ostensibly a test from the egginstalldir location, I fail to see why it should be globbing the eggs from the enclosing directory prior to changing directory into the egginstalldir and then running the trial from that directory after overriding PYTHONPATH.

So I guess I am failing to pierce the meaning behind the way we are prepending the eggs' realpath() to PYTHONPATH from pre- to post- chdir().

eggs=[os.path.realpath(egg) for egg in
      glob.glob('*.egg')]
testsuite='allmydata.test'
os.chdir('egginstalldir')
os.environ['PATH']=os.getcwd()+os.pathsep+os.environ['PATH']
os.environ['PYTHONPATH']=os.pathsep.join(eggs)+os.pathsep+os.environ.get('PYTHONPATH','')
sys.exit(subprocess.call([sys.executable, trial, '--rterror', testsuite], env=os.environ))

For the warp client, there isn't an allmydata egg except inside the egginstalldir itself; therefore the issue seems to be: <div style="font-size: 80%"> Code highlighting: ```#!python import glob,os,subprocess,sys trial=os.path.join(os.getcwd(), 'misc', 'build_helpers', 'run_trial.py') os.chdir('.') ``` </div> **Right in here... ** If I change the above os.chdir('.') to os.chdir('egginstalldir') then the test proceeds as normal. Since this is ostensibly a test **from** the egginstalldir location, I fail to see why it should be globbing the eggs from the **enclosing** directory prior to changing directory into the egginstalldir and then running the trial from that directory after overriding PYTHONPATH. So I guess I am failing to pierce the meaning behind the way we are prepending the eggs' realpath() to PYTHONPATH from pre- to post- chdir(). <div style="font-size: 80%"> ```#!python eggs=[os.path.realpath(egg) for egg in glob.glob('*.egg')] testsuite='allmydata.test' os.chdir('egginstalldir') os.environ['PATH']=os.getcwd()+os.pathsep+os.environ['PATH'] os.environ['PYTHONPATH']=os.pathsep.join(eggs)+os.pathsep+os.environ.get('PYTHONPATH','') sys.exit(subprocess.call([sys.executable, trial, '--rterror', testsuite], env=os.environ)) ```

daira commented

2010-11-04 01:16:25 +00:00

Attachment buildmaster-search-parent-for-eggs.darcs.patch (81043 bytes) added

buildmaster bbsupport.py: have the test-from-egg step look for eggs in the egginstalldir as well as its parent. refs #1246

**Attachment** buildmaster-search-parent-for-eggs.darcs.patch (81043 bytes) added buildmaster bbsupport.py: have the test-from-egg step look for eggs in the egginstalldir as well as its parent. refs #1246

buildmaster-search-parent-for-eggs.darcs.patch

79 KiB

daira commented

2010-11-04 01:39:01 +00:00

zooko: please apply attachment:buildmaster-search-parent-for-eggs.darcs.patch and restart the buildmaster.

zooko: please apply [attachment:buildmaster-search-parent-for-eggs.darcs.patch](/tahoe-lafs/trac/attachments/000078ac-26ce-d0b5-ea2b-f5d533993b99) and restart the buildmaster.

zooko commented

2010-11-04 06:34:34 +00:00

Author

Replying to davidsarah:

zooko: please apply attachment:buildmaster-search-parent-for-eggs.darcs.patch and restart the buildmaster.

Done:

http://tahoe-lafs.org/buildbot/builders/MM%20netbsd5%20i386%20warp/builds/135

It looks like MM's buildslave is all fixed up now. FreeStorm's Win67-amd64-mingw-py2.6 is not, though:

http://tahoe-lafs.org/buildbot/builders/FreeStorm%20Win7-amd64-mingw%20py2.6/builds/154/steps/test-with-fake-pkg/logs/stdio

Replying to [davidsarah](/tahoe-lafs/trac/issues/1246#issuecomment-381928): > zooko: please apply [attachment:buildmaster-search-parent-for-eggs.darcs.patch](/tahoe-lafs/trac/attachments/000078ac-26ce-d0b5-ea2b-f5d533993b99) and restart the buildmaster. Done: <http://tahoe-lafs.org/buildbot/builders/MM%20netbsd5%20i386%20warp/builds/135> It looks like MM's buildslave is all fixed up now. FreeStorm's Win67-amd64-mingw-py2.6 is not, though: <http://tahoe-lafs.org/buildbot/builders/FreeStorm%20Win7-amd64-mingw%20py2.6/builds/154/steps/test-with-fake-pkg/logs/stdio>

zooko changed title from ~~figure out why MM's and FS's buildslaves are red on some steps~~ to figure out why "FreeStorm Win7-amd64-mingw py2.6" is red on the "test-with-fake-pkg" step

2010-11-04 06:34:34 +00:00

daira commented

2010-11-05 00:59:15 +00:00

Replying to [zooko]comment:3:

FreeStorm's Win67-amd64-mingw-py2.6 is not fixed, though:

http://tahoe-lafs.org/buildbot/builders/FreeStorm%20Win7-amd64-mingw%20py2.6/builds/154/steps/test-with-fake-pkg/logs/stdio

The Requirements: ['allmydata-tahoe==1.8.0-r4803', ...] line indicates what source:misc/build_helpers/run_trial.py set the *requires*_ variable to. That is supposed to tell pkg_resources to insert entries on sys.path so that only modules from exactly that version of Tahoe-LAFS will be imported. But in fact the installed version (which cannot be 1.8.0-r4803 because that number increases with each build) is being run.

I asked freestorm to modify run_trial.py to print out sys.path just before and just after importing pkg_resources. The results show no specific version of the allmydata-tahoe package on either of these paths. In particular, the src directory of the Tahoe-LAFS distribution is not on the "before" path, as I would have expected. (That is the current directory when executing run_trial.py from source:misc/build_helpers/test-with-fake-pkg.py.)

The code in pkg_resources that inserts path entries is difficult to follow, but I think that it only reorders entries and adds .egg files from directories that are already on the sys.path, which would explain why it's not able to find the correct version of allymydata-tahoe. Perhaps it would help to explicitly add the src directory to the PYTHONPATH variable in test-with-fake-pkg.py; I'll ask freestorm to try that.

Replying to [zooko]comment:3: > FreeStorm's Win67-amd64-mingw-py2.6 is not fixed, though: > > <http://tahoe-lafs.org/buildbot/builders/FreeStorm%20Win7-amd64-mingw%20py2.6/builds/154/steps/test-with-fake-pkg/logs/stdio> The `Requirements: ['allmydata-tahoe==1.8.0-r4803', ...]` line indicates what source:misc/build_helpers/run_trial.py set the `*requires*_` variable to. That is supposed to tell `pkg_resources` to insert entries on `sys.path` so that only modules from exactly that version of Tahoe-LAFS will be imported. But in fact the installed version (which cannot be 1.8.0-r4803 because that number increases with each build) is being run. I asked freestorm to modify `run_trial.py` to print out `sys.path` just before and just after importing `pkg_resources`. The results show no specific version of the `allmydata-tahoe` package on either of these paths. In particular, the `src` directory of the Tahoe-LAFS distribution is not on the "before" path, as I would have expected. (That is the current directory when executing `run_trial.py` from source:misc/build_helpers/test-with-fake-pkg.py.) The code in `pkg_resources` that inserts path entries is difficult to follow, but I think that it only reorders entries and adds .egg files from directories that are already on the `sys.path`, which would explain why it's not able to find the correct version of `allymydata-tahoe`. Perhaps it would help to explicitly add the `src` directory to the `PYTHONPATH` variable in `test-with-fake-pkg.py`; I'll ask freestorm to try that.

daira commented

2010-11-05 01:00:40 +00:00

Attachment freestorm-test-with-fake-pkg-output.txt (6336 bytes) added

output of test-with-fake-pkg.py on freestorm's builder, with printing of sys.path before and after 'import pkg_resources' in run_trial.py

**Attachment** freestorm-test-with-fake-pkg-output.txt (6336 bytes) added output of test-with-fake-pkg.py on freestorm's builder, with printing of sys.path before and after 'import pkg_resources' in run_trial.py

freestorm-test-with-fake-pkg-output.txt

6.2 KiB

daira commented

2010-11-05 01:40:49 +00:00

Replying to [davidsarah]comment:4:

I asked freestorm to modify run_trial.py to print out sys.path just before and just after importing pkg_resources. The results show no specific version of the allmydata-tahoe package on either of these paths. In particular, the src directory of the Tahoe-LAFS distribution is not on the "before" path, as I would have expected.

Sorry, I was mistaken. It is on line 24 "before" and line 68 "after". The reason why the installed Tahoe is being picked up instead is that c:\python26\lib\site-packages is earlier than c:\buildbot_tahoe_0.8.2\freestorm_win7-amd64-mingw_py2.6\build\src on the "after" path, even though it is later on the "before" path. So importing pkg_resources incorrectly changed the order of these two entries.

Replying to [davidsarah]comment:4: > I asked freestorm to modify `run_trial.py` to print out `sys.path` just before and just after importing `pkg_resources`. The results show no specific version of the `allmydata-tahoe` package on either of these paths. In particular, the `src` directory of the Tahoe-LAFS distribution is not on the "before" path, as I would have expected. Sorry, I was mistaken. It is on [line 24](http://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/1246/freestorm-test-with-fake-pkg-output.txt#L24) "before" and line 68 "after". The reason why the installed Tahoe is being picked up instead is that `c:\python26\lib\site-packages` is earlier than `c:\buildbot_tahoe_0.8.2\freestorm_win7-amd64-mingw_py2.6\build\src` on the "after" path, even though it is later on the "before" path. So importing `pkg_resources` incorrectly changed the order of these two entries.

daira commented

2010-11-05 04:54:51 +00:00

Replying to [davidsarah]comment:5:

The reason why the installed Tahoe is being picked up instead is that c:\\python26\\lib\\site-packages is earlier than c:\\buildbot_tahoe_0.8.2\\freestorm_win7-amd64-mingw_py2.6\\build\\src on the "after" path, even though it is later on the "before" path. So importing pkg_resources incorrectly changed the order of these two entries.

Freestorm and I had a long debugging session and worked out what was causing this. The buildslave has three distributions that are installed directly in C:\Python26\lib\site-packages: allmydata_tahoe-1.8.0c1_r4687-py2.6, pywin32-214-py2.6.egg-info and Twisted-10.1.0-py2.6. Each is installed as an .egg-info directory and source files with the same root (e.g. Twisted-10.1.0-py2.6 and twisted both in site-packages).

That meant that when pkg_resources tried to add an entry for Twisted (10.1.0) to sys.path, it added the location C:\Python26\lib\site-packages, which effectively also added Tahoe 1.8.0c1-r4687 to sys.path with the same priority. The correct Tahoe 1.8.0-r4803 was also later on the path, but pkg_resources makes no attempt to order the requirements specified in *requires* relative to each other.

.egg-info installations directly into site-packages are apparently quite common :-( On my machine I have Tahoe (several .egg-infos for different versions), pywin32, Twisted 9.0.0, pycairo, pyflakes, and zetuptoolz installed in that way. The pywin32 and Twisted .exe installers apparently do this by default.

According to [http://stackoverflow.com/questions/256417/python-packages-and-egg-info-directories this stackoverflow answer], ".egg-info directories get only created if --single-version-externally-managed was used to install the egg". My conclusion from this is that we should never use --single-version-externally-managed to install Tahoe into any directory where other distributions might also be installed.

Anyway, Freestorm deleted the allmydata and allmydata_tahoe-1.8.0c1_r4687-py2.6.egg-info directories from site-packages (after taking a copy), and that [http://tahoe-lafs.org/buildbot/builders/FreeStorm%20Win7-amd64-mingw%20py2.6/builds/157/steps/test-with-fake-pkg/logs/stdio made test-with-fake-pkg go green].

Replying to [davidsarah]comment:5: > The reason why the installed Tahoe is being picked up instead is that `c:\\python26\\lib\\site-packages` is earlier than `c:\\buildbot_tahoe_0.8.2\\freestorm_win7-amd64-mingw_py2.6\\build\\src` on the "after" path, even though it is later on the "before" path. So importing `pkg_resources` incorrectly changed the order of these two entries. Freestorm and I had a long debugging session and worked out what was causing this. The buildslave has three distributions that are installed *directly* in `C:\Python26\lib\site-packages`: `allmydata_tahoe-1.8.0c1_r4687-py2.6`, `pywin32-214-py2.6.egg-info` and `Twisted-10.1.0-py2.6`. Each is installed as an `.egg-info` directory and source files with the same root (e.g. `Twisted-10.1.0-py2.6` and `twisted` both in `site-packages`). That meant that when `pkg_resources` tried to add an entry for Twisted (10.1.0) to `sys.path`, it added the location `C:\Python26\lib\site-packages`, which effectively also added Tahoe 1.8.0c1-r4687 to `sys.path` with the same priority. The correct Tahoe 1.8.0-r4803 was also later on the path, but `pkg_resources` makes no attempt to order the requirements specified in `*requires*` relative to each other. `.egg-info` installations directly into `site-packages` are apparently quite common :-( On my machine I have Tahoe (several .egg-infos for different versions), pywin32, Twisted 9.0.0, pycairo, pyflakes, and zetuptoolz installed in that way. The pywin32 and Twisted `.exe` installers apparently do this by default. According to [http://stackoverflow.com/questions/256417/python-packages-and-egg-info-directories this stackoverflow answer], ".egg-info directories get only created if `--single-version-externally-managed` was used to install the egg". My conclusion from this is that we should never use `--single-version-externally-managed` to install Tahoe into any directory where other distributions might also be installed. Anyway, Freestorm deleted the `allmydata` and `allmydata_tahoe-1.8.0c1_r4687-py2.6.egg-info` directories from `site-packages` (after taking a copy), and that [http://tahoe-lafs.org/buildbot/builders/FreeStorm%20Win7-amd64-mingw%20py2.6/builds/157/steps/test-with-fake-pkg/logs/stdio made test-with-fake-pkg go green].

daira added

c/packaging

and removed

c/dev-infrastructure

labels 2010-11-05 04:59:26 +00:00

zooko commented

2010-11-05 13:24:13 +00:00

Author

So if I understand correctly, the problem you are focussing on is that sometimes distributions get installed by copying their modules directly into some directory which is on sys.path. This makes it so that import $MODULENAME will subsequently work but it is a problem, if I understand you correctly, if more than one distribution is installed into the same directory because then there is no way to prioritize one of the distributions without thereby also prioritizing all the other distributions in the same directory. By "prioritize" here I guess I mean move it closer to the front of sys.path. Other methods of manipulating its precedence before importing would be moving it closer to the end of the sys.path or removing the directory from the sys.path entirely. None of these can be done on only a single distribution if more than one distribution has been installed "directly" (by dint of copying its modules into place) in a single directory.

One thing to note is that the standard distutils, without setuptools/zetuptoolz/distribute, always installs things in this way! If you're using the standard distutils, the only way to avoid doing this would be to create a new empty directory and install into that (with --prefix) and then add that to the sys.path. You would have to do that for every distribution that you install.

Okay, so now I can appreciate David-Sarah's conclusion: "We should never use --single-version-externally-managed to install Tahoe into any directory where other distributions might also be installed.". That makes sense. We already do avoid that, as far as I know. For example, quickstart.html doesn't advise the user to do that.

So if I understand correctly, the problem you are focussing on is that sometimes distributions get installed by copying their modules directly into some directory which is on `sys.path`. This makes it so that `import $MODULENAME` will subsequently work but it is a problem, if I understand you correctly, if more than one distribution is installed into the same directory because then there is no way to prioritize one of the distributions without thereby also prioritizing all the other distributions in the same directory. By "prioritize" here I guess I mean move it closer to the front of `sys.path`. Other methods of manipulating its precedence before importing would be moving it closer to the end of the `sys.path` or removing the directory from the `sys.path` entirely. None of these can be done on only a single distribution if more than one distribution has been installed "directly" (by dint of copying its modules into place) in a single directory. One thing to note is that the standard distutils, without setuptools/zetuptoolz/distribute, *always* installs things in this way! If you're using the standard distutils, the only way to avoid doing this would be to create a new empty directory and install into that (with `--prefix`) and then add that to the `sys.path`. You would have to do that for every distribution that you install. Okay, so now I can appreciate David-Sarah's conclusion: "We should never use `--single-version-externally-managed` to install Tahoe into any directory where other distributions might also be installed.". That makes sense. We already do avoid that, as far as I know. For example, [quickstart.html](http://tahoe-lafs.org/source/tahoe-lafs/trunk/docs/quickstart.html) doesn't advise the user to do that.

daira commented

2010-11-05 17:25:44 +00:00

Replying to [zooko]comment:8:

So if I understand correctly, the problem you are focussing on is that sometimes distributions get installed by copying their modules directly into some directory which is on sys.path. This makes it so that import $MODULENAME will subsequently work but it is a problem, if I understand you correctly, if more than one distribution is installed into the same directory because then there is no way to prioritize one of the distributions without thereby also prioritizing all the other distributions in the same directory. By "prioritize" here I guess I mean move it closer to the front of sys.path. Other methods of manipulating its precedence before importing would be moving it closer to the end of the sys.path or removing the directory from the sys.path entirely. None of these can be done on only a single distribution if more than one distribution has been installed "directly" (by dint of copying its modules into place) in a single directory.

Exactly.

One thing to note is that the standard distutils, without setuptools/zetuptoolz/distribute, always installs things in this way!

However, the problem in this ticket only applies to Tahoe-LAFS dependencies (direct or indirect, including the allmydata-tahoe package itself) that are installed in this way.

This could be worked around in several ways:

a) change zetuptoolz to order "development" dists (directories containing Python code and an .egg-info subdirectory) after non-development dists when satisfying *requires* lists. Note: the pkg_resources code involved is complicated and I don't understand it.

b) change zetuptoolz so that it does not view development dists as satisfying a dependency. That would cause a non-development dist to be used automatically instead. (pywin32 is a potential spanner in the works because it can't currently be installed as an egg, but maybe #142 could be fixed.) Fixing #1233 is important here, because otherwise you would have regressions where a package is already installed but only as a development dist, and building a new version of it fails because it can't be compiled.

c) change the wrapper scripts not to use the *requires* mechanism and to put the dependencies directly at the start of sys.path itself. This would still have to avoid viewing development dists as satisfying a dependency, or else it would run into the same problem.

[...]

Okay, so now I can appreciate David-Sarah's conclusion: "We should never use --single-version-externally-managed to install Tahoe into any directory where other distributions might also be installed.". That makes sense. We already do avoid that, as far as I know. For example, quickstart.html doesn't advise the user to do that.

As far as I can see, nothing in our docs advises against installing Tahoe using setup.py install or easy_install. That's a standard way of installing Python packages.

Besides, there's no good reason why that shouldn't work (that is, make the installed version the default, but still allow running other versions). Design failings in distutils and/or setuptools are a bad reason.

Replying to [zooko]comment:8: > So if I understand correctly, the problem you are focussing on is that sometimes distributions get installed by copying their modules directly into some directory which is on `sys.path`. This makes it so that `import $MODULENAME` will subsequently work but it is a problem, if I understand you correctly, if more than one distribution is installed into the same directory because then there is no way to prioritize one of the distributions without thereby also prioritizing all the other distributions in the same directory. By "prioritize" here I guess I mean move it closer to the front of `sys.path`. Other methods of manipulating its precedence before importing would be moving it closer to the end of the `sys.path` or removing the directory from the `sys.path` entirely. None of these can be done on only a single distribution if more than one distribution has been installed "directly" (by dint of copying its modules into place) in a single directory. Exactly. > One thing to note is that the standard distutils, without setuptools/zetuptoolz/distribute, *always* installs things in this way! However, the problem in this ticket only applies to Tahoe-LAFS dependencies (direct or indirect, including the `allmydata-tahoe` package itself) that are installed in this way. This could be worked around in several ways: a) change zetuptoolz to order "development" dists (directories containing Python code and an `.egg-info` subdirectory) after non-development dists when satisfying `*requires*` lists. Note: the `pkg_resources` code involved is complicated and I don't understand it. b) change zetuptoolz so that it does not view development dists as satisfying a dependency. That would cause a non-development dist to be used automatically instead. (pywin32 is a potential spanner in the works because it can't currently be installed as an egg, but maybe #142 could be fixed.) Fixing #1233 is important here, because otherwise you would have regressions where a package is already installed but only as a development dist, and building a new version of it fails because it can't be compiled. c) change the wrapper scripts not to use the `*requires*` mechanism and to put the dependencies directly at the start of `sys.path` itself. This would still have to avoid viewing development dists as satisfying a dependency, or else it would run into the same problem. [...] > Okay, so now I can appreciate David-Sarah's conclusion: "We should never use `--single-version-externally-managed` to install Tahoe into any directory where other distributions might also be installed.". That makes sense. We already do avoid that, as far as I know. For example, [quickstart.html](http://tahoe-lafs.org/source/tahoe-lafs/trunk/docs/quickstart.html) doesn't advise the user to do that. As far as I can see, nothing in our docs advises *against* installing Tahoe using `setup.py install` or `easy_install`. That's a standard way of installing Python packages. Besides, there's no **good** reason why that shouldn't work (that is, make the installed version the default, but still allow running other versions). Design failings in distutils and/or setuptools are a bad reason.

daira commented

2010-11-05 17:34:45 +00:00

Replying to [davidsarah]comment:9:

As far as I can see, nothing in our docs advises against installing Tahoe using setup.py install or easy_install. That's a standard way of installing Python packages.

Besides, there's no good reason why that shouldn't work (that is, make the installed version the default, but still allow running other versions). Design failings in distutils and/or setuptools are a bad reason.

Hmm, one way to make this work is for setup.py install (and therefore easy_install) to actually do a build rather than an install, but also install a tahoe wrapper script for that build to the global scripts directory.

Replying to [davidsarah]comment:9: > As far as I can see, nothing in our docs advises *against* installing Tahoe using `setup.py install` or `easy_install`. That's a standard way of installing Python packages. > > Besides, there's no **good** reason why that shouldn't work (that is, make the installed version the default, but still allow running other versions). Design failings in distutils and/or setuptools are a bad reason. Hmm, one way to make this work is for `setup.py install` (and therefore `easy_install`) to actually do a build rather than an install, but also install a `tahoe` wrapper script for that build to the global scripts directory.

daira commented

2010-11-05 17:51:41 +00:00

Replying to [davidsarah]comment:10:

Replying to [davidsarah]comment:9:

As far as I can see, nothing in our docs advises against installing Tahoe using setup.py install or easy_install. That's a standard way of installing Python packages.

Besides, there's no good reason why that shouldn't work (that is, make the installed version the default, but still allow running other versions). Design failings in distutils and/or setuptools are a bad reason.

Hmm, one way to make this work is for setup.py install (and therefore easy_install) to actually do a build rather than an install, but also install a tahoe wrapper script for that build to the global scripts directory.

Maybe this is as simple as putting "install = darcsver --count-all-patches develop --prefix=support make_executable build install_scripts" in place of the existing install alias in Tahoe's source:setup.cfg.

Replying to [davidsarah]comment:10: > Replying to [davidsarah]comment:9: > > As far as I can see, nothing in our docs advises *against* installing Tahoe using `setup.py install` or `easy_install`. That's a standard way of installing Python packages. > > > > Besides, there's no **good** reason why that shouldn't work (that is, make the installed version the default, but still allow running other versions). Design failings in distutils and/or setuptools are a bad reason. > > Hmm, one way to make this work is for `setup.py install` (and therefore `easy_install`) to actually do a build rather than an install, but also install a `tahoe` wrapper script for that build to the global scripts directory. Maybe this is as simple as putting "`install = darcsver --count-all-patches develop --prefix=support make_executable build install_scripts`" in place of the existing `install` alias in Tahoe's source:setup.cfg.

daira commented

2010-11-05 23:21:26 +00:00

Replying to [davidsarah]comment:11:

Maybe this is as simple as putting "install = darcsver --count-all-patches develop --prefix=support make_executable build install_scripts" in place of the existing install alias in Tahoe's source:setup.cfg.

Apparently not. That does cause python setup.py install to build Tahoe successfully and generate C:\Python26\Scripts\tahoe.pyscript, and the resulting script has a correct *requires* line. Nevertheless it fails (on this machine) with the error

>tahoe --version
Traceback (most recent call last):
  File "C:\Python26\Scripts\tahoe.pyscript", line 6, in <module>
    from pkg_resources import load_entry_point
  File "C:\Python26\lib\site-packages\setuptools-0.6c16dev.egg\pkg_resources.py", line 2607, in <module>
    parse_requirements(__requires__), Environment()
  File "C:\Python26\lib\site-packages\setuptools-0.6c16dev.egg\pkg_resources.py", line 565, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
pkg_resources.DistributionNotFound: pycryptopp>=0.5.20

This is understandable because pycryptopp>=0.5.20 is not globally installed, the support dir of the Tahoe build is not on the PYTHONPATH, and C:\Python26\Scripts\tahoe.pyscript does not put it onto the PYTHONPATH (since it is a zetuptoolz-generated script rather than the one from Tahoe's bin directory). Also note that it is using pkg_resources from setuptools-0.6c16dev, which is the globally installed version.

Getting setup.py install to work as I want it to is possible, but might have to wait for Brian's "unsuck" branch, since it's probably not a good idea to make the zetuptoolz-generated script as complicated as [the 'bin/tahoe' one]source:bin/tahoe-script.template.

Replying to [davidsarah]comment:11: > Maybe this is as simple as putting "`install = darcsver --count-all-patches develop --prefix=support make_executable build install_scripts`" in place of the existing `install` alias in Tahoe's source:setup.cfg. Apparently not. That does cause `python setup.py install` to build Tahoe successfully and generate `C:\Python26\Scripts\tahoe.pyscript`, and the resulting script has a correct `*requires*` line. Nevertheless it fails (on this machine) with the error ``` >tahoe --version Traceback (most recent call last): File "C:\Python26\Scripts\tahoe.pyscript", line 6, in <module> from pkg_resources import load_entry_point File "C:\Python26\lib\site-packages\setuptools-0.6c16dev.egg\pkg_resources.py", line 2607, in <module> parse_requirements(__requires__), Environment() File "C:\Python26\lib\site-packages\setuptools-0.6c16dev.egg\pkg_resources.py", line 565, in resolve raise DistributionNotFound(req) # XXX put more info here pkg_resources.DistributionNotFound: pycryptopp>=0.5.20 ``` This is understandable because `pycryptopp>=0.5.20` is not globally installed, the support dir of the Tahoe build is not on the `PYTHONPATH`, and `C:\Python26\Scripts\tahoe.pyscript` does not put it onto the `PYTHONPATH` (since it is a zetuptoolz-generated script rather than the one from Tahoe's `bin` directory). Also note that it is using `pkg_resources` from `setuptools-0.6c16dev`, which is the globally installed version. Getting `setup.py install` to work as I want it to is possible, but might have to wait for Brian's "unsuck" branch, since it's probably not a good idea to make the zetuptoolz-generated script as complicated as [the 'bin/tahoe' one]source:bin/tahoe-script.template.

daira commented

2010-11-06 00:29:08 +00:00

Attachment install-should-only-install-scripts.darcs.patch (19728 bytes) added

setup.py, setup.cfg, bin/tahoe-script.template: make it so that 'python setup.py install' just does a build and installs a script pointing to that build. refs #1246

**Attachment** install-should-only-install-scripts.darcs.patch (19728 bytes) added setup.py, setup.cfg, bin/tahoe-script.template: make it so that 'python setup.py install' just does a build and installs a script pointing to that build. refs #1246

install-should-only-install-scripts.darcs.patch

19 KiB

daira commented

2010-11-06 00:36:01 +00:00

Replying to [davidsarah]comment:10:

Hmm, one way to make this work is for setup.py install (and therefore easy_install) to actually do a build rather than an install, but also install a tahoe wrapper script for that build to the global scripts directory.

attachment:install-should-only-install-scripts.darcs.patch implements that. If we agree that it's a good idea, we would need a buildbot step to test it.

Replying to [davidsarah]comment:10: > Hmm, one way to make this work is for `setup.py install` (and therefore `easy_install`) to actually do a build rather than an install, but also install a `tahoe` wrapper script for that build to the global scripts directory. [attachment:install-should-only-install-scripts.darcs.patch](/tahoe-lafs/trac/attachments/000078ac-26ce-d0b5-ea2b-e3a61f43e3ed) implements that. If we agree that it's a good idea, we would need a buildbot step to test it.

daira commented

2010-11-06 01:23:25 +00:00

Replying to [davidsarah]comment:13:

Replying to [davidsarah]comment:10:

Hmm, one way to make this work is for setup.py install (and therefore easy_install) to actually do a build rather than an install, but also install a tahoe wrapper script for that build to the global scripts directory.

attachment:install-should-only-install-scripts.darcs.patch implements that. If we agree that it's a good idea, we would need a buildbot step to test it.

... and reconsider how to implement install-to-prefix, which this patch breaks.

Replying to [davidsarah]comment:13: > Replying to [davidsarah]comment:10: > > Hmm, one way to make this work is for `setup.py install` (and therefore `easy_install`) to actually do a build rather than an install, but also install a `tahoe` wrapper script for that build to the global scripts directory. > > [attachment:install-should-only-install-scripts.darcs.patch](/tahoe-lafs/trac/attachments/000078ac-26ce-d0b5-ea2b-e3a61f43e3ed) implements that. If we agree that it's a good idea, we would need a buildbot step to test it. ... and reconsider how to implement install-to-prefix, which this patch breaks.

daira commented

2011-01-01 09:14:25 +00:00

Replying to [davidsarah]comment:6:

According to this stackoverflow answer, ".egg-info directories get only created if --single-version-externally-managed was used to install the egg". My conclusion from this is that we should never use --single-version-externally-managed to install Tahoe into any directory where other distributions might also be installed.

Unfortunately, that's probably not sufficient (although it is necessary). The bug requires that at least one Tahoe dependency be installed as an .egg-info, but the other distribution in the same directory that gets imported by mistake does not itself have to have an .egg-info directory.

So, we should never use --single-version-externally-managed or equivalent to install Tahoe or any of its dependencies into any directory where other distributions might also be installed. But we can't control how users install, say, Twisted or pywin32 (which have installers that are equivalent to using --single-version-externally-managed).

Replying to [davidsarah]comment:6: > According to [this stackoverflow answer](http://stackoverflow.com/questions/256417/python-packages-and-egg-info-directories), ".egg-info directories get only created if `--single-version-externally-managed` was used to install the egg". My conclusion from this is that we should never use `--single-version-externally-managed` to install Tahoe into any directory where other distributions might also be installed. Unfortunately, that's probably not sufficient (although it is necessary). The bug requires that at least one Tahoe dependency be installed as an `.egg-info`, but the other distribution in the same directory that gets imported by mistake does not itself have to have an `.egg-info` directory. So, we should never use `--single-version-externally-managed` or equivalent to install Tahoe **or any of its dependencies** into any directory where other distributions might also be installed. But we can't control how users install, say, Twisted or pywin32 (which have installers that are equivalent to using `--single-version-externally-managed`).

zooko commented

2011-01-01 23:55:43 +00:00

Author

Attachment misc-build_helper-test-1246.darcs.patch (12919 bytes) added

**Attachment** misc-build_helper-test-1246.darcs.patch (12919 bytes) added

misc-build_helper-test-1246.darcs.patch

13 KiB

zooko commented

2011-01-02 00:04:46 +00:00

Author

attachment:misc-build_helper-test-1246.darcs.patch is my attempt to exercise this problem. However, I don't think this attempt succeeds. When I run python misc/build_helpers/test-dists-in-shared-directory.py, I get this result:


Requirements: ['allmydata-tahoe==1.8.1-r4897', 'setuptools >= 0.6c6', 'zfec >= 1.1.0', 'simplejson >= 1.4', 'zope.interface', 'Twisted == 10.1.0', 'foolscap[secure_connections] >= 0.6.0', 'Nevow >= 0.6.0', 'pycrypto == 2.0.1, == 2.1, >= 2.3', 'pyasn1 >= 0.0.8a', 'pycryptopp >= 0.5.20', 'mock']
Inserting egg on sys.path: '/Users/zooko/playground/tahoe-lafs/trunk/setuptools-0.6c16dev3.egg'
buildtest.test_dists_in_shared_directory
  T
    test_version ...                                                    [ERROR]

===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/Users/zooko/playground/tahoe-lafs/trunk/src/buildtest/test_dists_in_shared_directory.py", line 8, in test_version
    pkg_resources.require('pycryptopp >= 0.5.14')
  File "/Users/zooko/playground/tahoe-lafs/trunk/setuptools-0.6c16dev3.egg/pkg_resources.py", line 676, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/zooko/playground/tahoe-lafs/trunk/setuptools-0.6c16dev3.egg/pkg_resources.py", line 575, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
pkg_resources.DistributionNotFound: pycryptopp>=0.5.14

buildtest.test_dists_in_shared_directory.T.test_version
-------------------------------------------------------------------------------
Ran 1 tests in 0.045s

FAILED (errors=1)

Maybe this means that the current zetuptoolz-based build system passes this test? Maybe the test that we want has two versions of the package, both of which are of a sufficiently high version number, and we are trying to show that the build system mistakenly imports one when pkg_resources reports that it uses the other.

Note that this test works only if you don't have a sufficient version of pycryptopp installed in your system. If there is already a sufficient version of pycryptopp, then you'll get a skip.

Whoops, in fact you get some sort of failure-to-run trial:

Traceback (most recent call last):
  File "/Users/zooko/playground/tahoe-lafs/trunk/src/../misc/build_helpers/run_trial.py", line 52, in <module>
    __import__(modulename)
ImportError: No module named test_dists_in_shared_directory

[attachment:misc-build_helper-test-1246.darcs.patch](/tahoe-lafs/trac/attachments/000078ac-26ce-d0b5-ea2b-472bef993455) is my attempt to exercise this problem. However, I don't think this attempt succeeds. When I run `python misc/build_helpers/test-dists-in-shared-directory.py`, I get this result: ``` Requirements: ['allmydata-tahoe==1.8.1-r4897', 'setuptools >= 0.6c6', 'zfec >= 1.1.0', 'simplejson >= 1.4', 'zope.interface', 'Twisted == 10.1.0', 'foolscap[secure_connections] >= 0.6.0', 'Nevow >= 0.6.0', 'pycrypto == 2.0.1, == 2.1, >= 2.3', 'pyasn1 >= 0.0.8a', 'pycryptopp >= 0.5.20', 'mock'] Inserting egg on sys.path: '/Users/zooko/playground/tahoe-lafs/trunk/setuptools-0.6c16dev3.egg' buildtest.test_dists_in_shared_directory T test_version ... [ERROR] =============================================================================== [ERROR] Traceback (most recent call last): File "/Users/zooko/playground/tahoe-lafs/trunk/src/buildtest/test_dists_in_shared_directory.py", line 8, in test_version pkg_resources.require('pycryptopp >= 0.5.14') File "/Users/zooko/playground/tahoe-lafs/trunk/setuptools-0.6c16dev3.egg/pkg_resources.py", line 676, in require needed = self.resolve(parse_requirements(requirements)) File "/Users/zooko/playground/tahoe-lafs/trunk/setuptools-0.6c16dev3.egg/pkg_resources.py", line 575, in resolve raise DistributionNotFound(req) # XXX put more info here pkg_resources.DistributionNotFound: pycryptopp>=0.5.14 buildtest.test_dists_in_shared_directory.T.test_version ------------------------------------------------------------------------------- Ran 1 tests in 0.045s FAILED (errors=1) ``` Maybe this means that the current `zetuptoolz`-based build system passes this test? Maybe the test that we want has two versions of the package, both of which are of a sufficiently high version number, and we are trying to show that the build system mistakenly imports one when `pkg_resources` reports that it uses the other. Note that this test works only if you don't have a sufficient version of `pycryptopp` installed in your system. If there is already a sufficient version of pycryptopp, then you'll get a skip. Whoops, in fact you get some sort of failure-to-run trial: ``` Traceback (most recent call last): File "/Users/zooko/playground/tahoe-lafs/trunk/src/../misc/build_helpers/run_trial.py", line 52, in <module> __import__(modulename) ImportError: No module named test_dists_in_shared_directory ```

daira commented

2011-01-02 00:19:53 +00:00

In at least some cases (#1258), setup.py finds the correct code but bin/tahoe does not. Perhaps that difference is involved in the attachment:misc-build_helper-test-1246.darcs.patch test's failure to reproduce the problem (since the test invokes run_trial.py which invokes setup.py).

In at least some cases (#1258), `setup.py` finds the correct code but `bin/tahoe` does not. Perhaps that difference is involved in the [attachment:misc-build_helper-test-1246.darcs.patch](/tahoe-lafs/trac/attachments/000078ac-26ce-d0b5-ea2b-472bef993455) test's failure to reproduce the problem (since the test invokes `run_trial.py` which invokes `setup.py`).

daira commented

2011-01-02 01:22:11 +00:00

Replying to davidsarah:

In at least some cases (#1258), setup.py finds the correct code but bin/tahoe does not. Perhaps that difference is involved in the attachment:misc-build_helper-test-1246.darcs.patch test's failure to reproduce the problem (since the test invokes run_trial.py which invokes setup.py).

Here is a untried test that imports pycryptopp in a subprocess, similar to test_import_from_repl in attachment🎫1258:add-test-import-in-repl.darcs.patch. Unfortunately it is complicated and depends on more than we want to depend on for this test, but let's see whether it helps to reproduce the problem.

from twisted.trial import unittest
import subprocess, sys

# TODO: move these to common or common_util
from allmydata.test.test_runner import bintahoe, SkipMixin

class T(unittest.TestCase, SkipMixin):
    def test_version(self):
        # based on test_import_in_repl
        self.skip_if_cannot_run_bintahoe()

        p = subprocess.Popen([sys.executable, bintahoe, "debug", "repl"],
                             stdin=subprocess.PIPE, stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE)
        (out, err) = p.communicate("import pycryptopp; "+
                                   "print pycryptopp.__version__")

        self.failUnlessEqual(p.returncode, 0)
        self.failIfIn("Traceback", err)
        first = out.splitlines()[0]
        self.failUnless(first.startswith('>>> '))
        pycryptopp_version = first[4:]

        # If you tried to import 0.5.13 then the repl would have gotten
        # an exception and printed a Traceback to stderr, causing a
        # failure above. If there is already a pycryptopp installed in
        # the system then the import might have gotten that, instead of
        # the fake 99.0.0 that we intended. In that case, we can't easily
        # tell whether we got this far because we correctly skipped over
        # the insufficient 0.5.13 and imported the sufficient 99.0.0,
        # or because the system-provided pycryptopp was found before
        # the insufficient 0.5.13 was found, so this test can't draw a
        # conclusion about whether our import machinery is working
        # right.
        if pycryptopp_version != '99.0.0':
            raise unittest.SkipTest(
"""We can't tell if this worked because this system has a different
version of pycryptopp (%r) already installed. See comment in
misc/build_helpers/test_dists_in_shared_directory.py .
""" % (pycryptopp_version,))

[to fix an import error.]Edited

Replying to [davidsarah](/tahoe-lafs/trac/issues/1246#issuecomment-381945): > In at least some cases (#1258), `setup.py` finds the correct code but `bin/tahoe` does not. Perhaps that difference is involved in the [attachment:misc-build_helper-test-1246.darcs.patch](/tahoe-lafs/trac/attachments/000078ac-26ce-d0b5-ea2b-472bef993455) test's failure to reproduce the problem (since the test invokes `run_trial.py` which invokes `setup.py`). Here is a untried test that imports pycryptopp in a subprocess, similar to `test_import_from_repl` in attachment:ticket:1258:add-test-import-in-repl.darcs.patch. Unfortunately it is complicated and depends on more than we want to depend on for this test, but let's see whether it helps to reproduce the problem. ``` from twisted.trial import unittest import subprocess, sys # TODO: move these to common or common_util from allmydata.test.test_runner import bintahoe, SkipMixin class T(unittest.TestCase, SkipMixin): def test_version(self): # based on test_import_in_repl self.skip_if_cannot_run_bintahoe() p = subprocess.Popen([sys.executable, bintahoe, "debug", "repl"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) (out, err) = p.communicate("import pycryptopp; "+ "print pycryptopp.__version__") self.failUnlessEqual(p.returncode, 0) self.failIfIn("Traceback", err) first = out.splitlines()[0] self.failUnless(first.startswith('>>> ')) pycryptopp_version = first[4:] # If you tried to import 0.5.13 then the repl would have gotten # an exception and printed a Traceback to stderr, causing a # failure above. If there is already a pycryptopp installed in # the system then the import might have gotten that, instead of # the fake 99.0.0 that we intended. In that case, we can't easily # tell whether we got this far because we correctly skipped over # the insufficient 0.5.13 and imported the sufficient 99.0.0, # or because the system-provided pycryptopp was found before # the insufficient 0.5.13 was found, so this test can't draw a # conclusion about whether our import machinery is working # right. if pycryptopp_version != '99.0.0': raise unittest.SkipTest( """We can't tell if this worked because this system has a different version of pycryptopp (%r) already installed. See comment in misc/build_helpers/test_dists_in_shared_directory.py . """ % (pycryptopp_version,)) ``` [to fix an import error.]Edited

daira commented

2011-01-02 02:26:33 +00:00

The test in comment:18 does reproduce the error for me:

$ python misc/build_helpers/test-dists-in-shared-directory.py
Requirements: ['allmydata-tahoe==1.8.1-r4913', 'setuptools >= 0.6c6', 'zfec >= 1.1.0', 'simplejson >= 1.4', 'zope.interf
ace', 'Twisted == 10.2.0', 'foolscap[secure_connections] >= 0.6.0', 'Nevow >= 0.6.0', 'pycrypto == 2.0.1, == 2.1, >= 2.3
', 'pyasn1 >= 0.0.8a', 'pycryptopp >= 0.5.20', 'mock']
Inserting egg on sys.path: 'd:\\tahoe\\tahoe-clean\\setuptools-0.6c16dev3.egg'
Traceback (most recent call last):
  File "d:\tahoe\tahoe-clean\src\..\misc\build_helpers\run_trial.py", line 52, in <module>
    __import__(modulename)
  File "d:\tahoe\tahoe-clean\src\buildtest\test_dists_in_shared_directory.py", line 7, in <module>
    from allmydata.test.test_runner import bintahoe, SkipMixin
  File "d:\tahoe\tahoe-clean\src\allmydata\test\__init__.py", line 31, in <module>
    initialize()
  File "d:\tahoe\tahoe-clean\src\allmydata\windows\fixups.py", line 15, in initialize
    from allmydata.util.encodingutil import canonical_encoding
  File "d:\tahoe\tahoe-clean\src\allmydata\util\encodingutil.py", line 13, in <module>
    from allmydata.util.fileutil import abspath_expanduser_unicode
  File "d:\tahoe\tahoe-clean\src\allmydata\util\fileutil.py", line 9, in <module>
    from pycryptopp.cipher.aes import AES
  File "d:\tahoe\tahoe-clean\tmpdir-for-test-dists-in-shared-directory\packages1\pycryptopp\__init__.py", line 1, in <module>
    raise Exception('Aha I caught you trying to import me. I am a fake pycryptopp 0.5.13 package and you should be require a newer version than me.')
Exception: Aha I caught you trying to import me. I am a fake pycryptopp 0.5.13 package and you should be require a newer version than me.

It still needs work, though. If I delete the temp directory and the code that creates the fake pycryptopp 0.5.13, I get a spurious failure:

...
  File "d:\tahoe\tahoe-clean\src\allmydata\util\fileutil.py", line 9, in <module>
    from pycryptopp.cipher.aes import AES
ImportError: No module named cipher.aes

That is, the repl indirectly attempts to import pycryptopp.cipher.aes. Working around that by removing the dependency on AES from fileutil.py, I get a skip:

[SKIPPED]: buildtest.test_dists_in_shared_directory.T.test_version

We can't tell if this worked because this system has a different
version of pycryptopp ('0.5.25-r757') already installed. See comment in
misc/build_helpers/test_dists_in_shared_directory.py .

(0.5.25-!r757 is the version of pycryptopp that normally gets used when running Tahoe on this system. It is not installed in site-packages.)

The reason why this happens is that the bin/tahoe script adds the support/{Lib,lib/pythonx.x}/site-packages directory to PYTHONPATH before the entries that were inherited from test-dists-in-shared-directory.py. Therefore, it finds the real pycryptopp before the fake pycryptopp 99.0.0. I confirmed that by modifying the test to print PYTHONPATH in the repl, which gives (one entry on each line for clarity):

d:\tahoe\tahoe-clean\support\Lib\site-packages;
;
d:\tahoe\tahoe-clean\tmpdir-for-test-dists-in-shared-directory\packages1;
d:\tahoe\tahoe-clean\tmpdir-for-test-dists-in-shared-directory\packages2;
d:\tahoe\tahoe-clean\darcsver-1.7.1.egg;
d:\tahoe\tahoe-clean\foolscap-0.6.0-py2.6.egg;
d:\tahoe\tahoe-clean\setuptools-0.6c16dev3.egg;
d:\tahoe\tahoe-clean\setuptools_darcs-1.2.11-py2.6.egg;
d:\tahoe\tahoe-clean\setuptools_trial-0.5.9.egg;

(Ignore the foolscap entry, that's a workaround for foolscap 0.6.0 not being found on my system.)

The test in comment:18 does reproduce the error for me: ``` $ python misc/build_helpers/test-dists-in-shared-directory.py Requirements: ['allmydata-tahoe==1.8.1-r4913', 'setuptools >= 0.6c6', 'zfec >= 1.1.0', 'simplejson >= 1.4', 'zope.interf ace', 'Twisted == 10.2.0', 'foolscap[secure_connections] >= 0.6.0', 'Nevow >= 0.6.0', 'pycrypto == 2.0.1, == 2.1, >= 2.3 ', 'pyasn1 >= 0.0.8a', 'pycryptopp >= 0.5.20', 'mock'] Inserting egg on sys.path: 'd:\\tahoe\\tahoe-clean\\setuptools-0.6c16dev3.egg' Traceback (most recent call last): File "d:\tahoe\tahoe-clean\src\..\misc\build_helpers\run_trial.py", line 52, in <module> __import__(modulename) File "d:\tahoe\tahoe-clean\src\buildtest\test_dists_in_shared_directory.py", line 7, in <module> from allmydata.test.test_runner import bintahoe, SkipMixin File "d:\tahoe\tahoe-clean\src\allmydata\test\__init__.py", line 31, in <module> initialize() File "d:\tahoe\tahoe-clean\src\allmydata\windows\fixups.py", line 15, in initialize from allmydata.util.encodingutil import canonical_encoding File "d:\tahoe\tahoe-clean\src\allmydata\util\encodingutil.py", line 13, in <module> from allmydata.util.fileutil import abspath_expanduser_unicode File "d:\tahoe\tahoe-clean\src\allmydata\util\fileutil.py", line 9, in <module> from pycryptopp.cipher.aes import AES File "d:\tahoe\tahoe-clean\tmpdir-for-test-dists-in-shared-directory\packages1\pycryptopp\__init__.py", line 1, in <module> raise Exception('Aha I caught you trying to import me. I am a fake pycryptopp 0.5.13 package and you should be require a newer version than me.') Exception: Aha I caught you trying to import me. I am a fake pycryptopp 0.5.13 package and you should be require a newer version than me. ``` It still needs work, though. If I delete the temp directory and the code that creates the fake pycryptopp 0.5.13, I get a spurious failure: ``` ... File "d:\tahoe\tahoe-clean\src\allmydata\util\fileutil.py", line 9, in <module> from pycryptopp.cipher.aes import AES ImportError: No module named cipher.aes ``` That is, the repl indirectly attempts to import `pycryptopp.cipher.aes`. Working around that by removing the dependency on AES from `fileutil.py`, I get a skip: ``` [SKIPPED]: buildtest.test_dists_in_shared_directory.T.test_version We can't tell if this worked because this system has a different version of pycryptopp ('0.5.25-r757') already installed. See comment in misc/build_helpers/test_dists_in_shared_directory.py . ``` (0.5.25-!r757 is the version of pycryptopp that normally gets used when running Tahoe on this system. It is *not* installed in site-packages.) The reason why this happens is that the `bin/tahoe` script adds the `support/{Lib,lib/pythonx.x}/site-packages` directory to PYTHONPATH before the entries that were inherited from `test-dists-in-shared-directory.py`. Therefore, it finds the real pycryptopp before the fake pycryptopp 99.0.0. I confirmed that by modifying the test to print PYTHONPATH in the repl, which gives (one entry on each line for clarity): ``` d:\tahoe\tahoe-clean\support\Lib\site-packages; ; d:\tahoe\tahoe-clean\tmpdir-for-test-dists-in-shared-directory\packages1; d:\tahoe\tahoe-clean\tmpdir-for-test-dists-in-shared-directory\packages2; d:\tahoe\tahoe-clean\darcsver-1.7.1.egg; d:\tahoe\tahoe-clean\foolscap-0.6.0-py2.6.egg; d:\tahoe\tahoe-clean\setuptools-0.6c16dev3.egg; d:\tahoe\tahoe-clean\setuptools_darcs-1.2.11-py2.6.egg; d:\tahoe\tahoe-clean\setuptools_trial-0.5.9.egg; ``` (Ignore the foolscap entry, that's a workaround for foolscap 0.6.0 not being found on my system.)

daira commented

2011-01-02 02:28:38 +00:00

My conclusion at this point is that we need to know why the problem is only occurring in subprocesses, before we can write a reliable test.

zooko commented

2011-01-14 23:30:20 +00:00

Author

Removing review-needed. I will try to write a test.

Removing `review-needed`. I will try to write a test.

daira commented

2011-01-17 05:59:01 +00:00

In changeset:e52d97a917364cb0:

Undo the temporary hack to check the foolscap version. refs #1246

In changeset:e52d97a917364cb0: ``` Undo the temporary hack to check the foolscap version. refs #1246 ```

daira commented

2011-01-17 22:45:20 +00:00

Replying to davidsarah:

My conclusion at this point is that we need to know why the problem is only occurring in subprocesses, before we can write a reliable test.

This is due to a different order for entries in *requires*. With help from abadger1999 (Toshio Kuratomi), we worked out a reliable test, using only three dists (two versions of one package and one of another), that will be independent of any globally installed packages. I'll prepare a patch for that.

I'm now sure that this is the same issue as #1258, so I'm marking it as a duplicate (but all of the discussion in this ticket from comment:4 on is still relevant).

Replying to [davidsarah](/tahoe-lafs/trac/issues/1246#issuecomment-381948): > My conclusion at this point is that we need to know why the problem is only occurring in subprocesses, before we can write a reliable test. This is due to a different order for entries in `*requires*`. With help from abadger1999 (Toshio Kuratomi), we worked out a reliable test, using only three dists (two versions of one package and one of another), that will be independent of any globally installed packages. I'll prepare a patch for that. I'm now sure that this is the same issue as #1258, so I'm marking it as a duplicate (but all of the discussion in this ticket from comment:4 on is still relevant).

daira added the

r/duplicate

label 2011-01-17 22:45:20 +00:00

daira closed this issue

2011-01-17 22:45:20 +00:00