"make test" tests the installed version of allmydata, not the local sandbox version of allmydata #145

New issue

Closed

opened 2007-09-23 13:30:28 +00:00 by zooko · 22 comments

zooko commented

2007-09-23 13:30:28 +00:00

It seems like .pth paths get searched earlier than PYTHONPATH, so if you install allmydata and then change something in your local sandbox and then run "make test", it will test the installed version, not the local one that you changed.

zooko added the

labels 2007-09-23 13:30:28 +00:00

zooko added this to the 0.6.1 milestone 2007-09-23 13:30:28 +00:00

warner commented

2007-09-28 02:35:10 +00:00

Hmm. I went to confirm this, and realized that I can't seem to find any .pth
files on my system that contain code, and I can't seem to find a reference to
'.pth' in the site.py files there.

eek. I thought that .pth files had path-manipulation logic in them to make sure that everything which gets added is really inserted into the middle of sys.path, in the same place that the .pth file was found. Hmm. I went to confirm this, and realized that I can't seem to find any .pth files on my system that contain code, and I can't seem to find a reference to '.pth' in the site.py files there.

warner added

c/code

and removed

c/unknown

labels 2007-09-28 02:35:10 +00:00

zooko commented

2007-10-13 06:40:31 +00:00

Author

This one bit me again, as I sleepily couldn't figure out why my new unit test wasn't failing as it should. When I deleted the installed version of allmydata, this was fixed.

One workaround to this which might not be a terrible idea in any case is to exclude test files from the installed files.

This one bit me again, as I sleepily couldn't figure out why my new unit test wasn't failing as it should. When I deleted the installed version of allmydata, this was fixed. One workaround to this which might not be a terrible idea in any case is to exclude test files from the installed files.

warner commented

2007-10-13 23:11:54 +00:00

I'd like to understand the reasons first.. what was the PYTHONPATH used for that test?

I'm also inclined to leave the test sub-package installed, since that allows users who have just installed tahoe (perhaps from some binary installer) to run the unit tests on their own. FWIW, Twisted does the same thing, and following their lead I did the same for Buildbot and Foolscap.

I'd like to understand the reasons first.. what was the PYTHONPATH used for that test? I'm also inclined to leave the test sub-package installed, since that allows users who have just installed tahoe (perhaps from some binary installer) to run the unit tests on their own. FWIW, Twisted does the same thing, and following their lead I did the same for Buildbot and Foolscap.

zooko commented

2007-10-15 13:54:04 +00:00

Author

When tahoe isn't installed, then make test starts with:

PYTHONPATH="/Users/wonwinmcbrootles/playground/allmydata/tahoe/src::" \
 python -u "/usr/local/bin/trial" --rterrors   allmydata.test
Running 233 tests.

I'll investigate more later...

When tahoe isn't installed, then `make test` starts with: ``` PYTHONPATH="/Users/wonwinmcbrootles/playground/allmydata/tahoe/src::" \ python -u "/usr/local/bin/trial" --rterrors allmydata.test Running 233 tests. ``` I'll investigate more later...

zooko commented

2007-10-15 16:33:41 +00:00

Author

Okay here is the difference between sys.path when running unit tests when tahoe is installed vs. when it is not installed:

-    test_options ... sys.path: ['/Users/wonwinmcbrootles/playground/allmydata/tahoe', '/usr/local/stow/python-release25-maint/bin', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/zope.interface-3.4.1-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/zfec-unknown-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/foolscap-0.1.6-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/simplejson-1.7.1-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/pyutil-1.3.2-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/z_base_32-0.9.14-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/Nevow-0.9.18-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/pyflakes-0.2.1-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/allmydata_tahoe-0.6.0_76-py2.5-macosx-10.3-i386.egg', '/Users/wonwinmcbrootles/playground/allmydata/tahoe/src', '/usr/local/stow/python-release25-maint/lib/python25.zip', '/usr/local/stow/python-release25-maint/lib/python2.5', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-darwin', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-mac', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-mac/lib-scriptpackages', '/usr/local/stow/python-release25-maint/lib/python2.5/lib-tk', '/usr/local/stow/python-release25-maint/lib/python2.5/lib-dynload', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages']
+    test_options ... sys.path: ['/Users/wonwinmcbrootles/playground/allmydata/tahoe', '/usr/local/stow/python-release25-maint/bin', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/zope.interface-3.4.1-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/zfec-unknown-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/foolscap-0.1.6-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/simplejson-1.7.1-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/pyutil-1.3.2-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/z_base_32-0.9.14-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/Nevow-0.9.18-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/pyflakes-0.2.1-py2.5.egg', '/Users/wonwinmcbrootles/playground/allmydata/tahoe/src', '/usr/local/stow/python-release25-maint/lib/python25.zip', '/usr/local/stow/python-release25-maint/lib/python2.5', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-darwin', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-mac', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-mac/lib-scriptpackages', '/usr/local/stow/python-release25-maint/lib/python2.5/lib-tk', '/usr/local/stow/python-release25-maint/lib/python2.5/lib-dynload', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages']

So all of the installed .eggs from the Python site-packages dir, including the allmydata-tahoe .egg, are earlier on the path than $PWD/src is.

Okay here is the difference between sys.path when running unit tests when tahoe is installed vs. when it is not installed: ``` - test_options ... sys.path: ['/Users/wonwinmcbrootles/playground/allmydata/tahoe', '/usr/local/stow/python-release25-maint/bin', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/zope.interface-3.4.1-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/zfec-unknown-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/foolscap-0.1.6-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/simplejson-1.7.1-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/pyutil-1.3.2-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/z_base_32-0.9.14-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/Nevow-0.9.18-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/pyflakes-0.2.1-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/allmydata_tahoe-0.6.0_76-py2.5-macosx-10.3-i386.egg', '/Users/wonwinmcbrootles/playground/allmydata/tahoe/src', '/usr/local/stow/python-release25-maint/lib/python25.zip', '/usr/local/stow/python-release25-maint/lib/python2.5', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-darwin', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-mac', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-mac/lib-scriptpackages', '/usr/local/stow/python-release25-maint/lib/python2.5/lib-tk', '/usr/local/stow/python-release25-maint/lib/python2.5/lib-dynload', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages'] + test_options ... sys.path: ['/Users/wonwinmcbrootles/playground/allmydata/tahoe', '/usr/local/stow/python-release25-maint/bin', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/setuptools-0.6c7-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/zope.interface-3.4.1-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/zfec-unknown-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/foolscap-0.1.6-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/simplejson-1.7.1-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/pyutil-1.3.2-py2.5-macosx-10.3-i386.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/z_base_32-0.9.14-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/Nevow-0.9.18-py2.5.egg', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages/pyflakes-0.2.1-py2.5.egg', '/Users/wonwinmcbrootles/playground/allmydata/tahoe/src', '/usr/local/stow/python-release25-maint/lib/python25.zip', '/usr/local/stow/python-release25-maint/lib/python2.5', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-darwin', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-mac', '/usr/local/stow/python-release25-maint/lib/python2.5/plat-mac/lib-scriptpackages', '/usr/local/stow/python-release25-maint/lib/python2.5/lib-tk', '/usr/local/stow/python-release25-maint/lib/python2.5/lib-dynload', '/usr/local/stow/python-release25-maint/lib/python2.5/site-packages'] ``` So all of the installed .eggs from the Python site-packages dir, including the allmydata-tahoe .egg, are earlier on the path than `$PWD/src` is.

zooko removed their assignment 2007-10-15 16:33:41 +00:00

warner was assigned by zooko

2007-10-15 16:33:41 +00:00

zooko commented

2007-10-15 20:26:51 +00:00

Author

Bumping to v0.7.

zooko modified the milestone from 0.6.1 to 0.7.0

2007-10-15 20:26:51 +00:00

warner commented

2007-10-24 00:53:34 +00:00

this is currently giving us false-positive test results on the Solaris buildslave, and possibly
others, so I'm raising the priority.

One way to lower the priority would be to modify the test process and buildbot config to emit the
version number of the code actually being tested (as extracted from the _trial_temp/twistd.log file) in the test step, so we could spot the discrepancy. At the moment, the buildbot is silently testing the wrong versions on platforms where tahoe was installed into a normal system path.

this is currently giving us false-positive test results on the Solaris buildslave, and possibly others, so I'm raising the priority. One way to lower the priority would be to modify the test process and buildbot config to emit the version number of the code actually being tested (as extracted from the _trial_temp/twistd.log file) in the test step, so we could spot the discrepancy. At the moment, the buildbot is silently testing the wrong versions on platforms where tahoe was installed into a normal system path.

warner added

p/critical

and removed

p/major

labels 2007-10-24 00:53:34 +00:00

warner commented

2007-10-24 01:27:06 +00:00

from the following mailing list articles:
http://www.eby-sarna.com/pipermail/peak/2006-june/002582.html
http://mail.python.org/pipermail/distutils-sig/2006-July/006492.html

it looks like using eggs always results in the following import order:

eggs on PYTHONPATH
eggs in site-packages
non-eggs on PYTHONPATH
non-eggs in site-packages

and for us, the tahoe "egg in site-packages" is overriding the code-under-test "non-egg on PYTHONPATH".

It appears that the recommended solution might be to use "setup.py develop", to create an .egg (really a directory, rather than a .zip file) that contains symlinks into our source tree. Then put that .egg in a local directory on PYTHONPATH.

from the following mailing list articles: <http://www.eby-sarna.com/pipermail/peak/2006-june/002582.html> <http://mail.python.org/pipermail/distutils-sig/2006-July/006492.html> it looks like using eggs always results in the following import order: * eggs on PYTHONPATH * eggs in site-packages * non-eggs on PYTHONPATH * non-eggs in site-packages and for us, the tahoe "egg in site-packages" is overriding the code-under-test "non-egg on PYTHONPATH". It appears that the recommended solution might be to use "setup.py develop", to create an .egg (really a directory, rather than a .zip file) that contains symlinks into our source tree. Then put that .egg in a local directory on PYTHONPATH.

zooko commented

2007-11-01 17:09:07 +00:00

Author

We're focussing on an imminent v0.7.0 which hopefully has [#197 Small Distributed Mutable Files] and also a fix for [#199 bad SHA-256]. So I'm bumping less urgent tickets to v0.7.1.

zooko added

and removed

labels 2007-11-01 17:09:07 +00:00

zooko commented

2007-11-13 17:48:26 +00:00

Author

I uninstalled the allmydata-tahoe package from the system on nooxie, so now the buildslave on nooxie is testing its local checkout of the trunk source.

So while this is still a bug, it is no longer the cause of any build failures on nooxie.

I uninstalled the allmydata-tahoe package from the system on nooxie, so now the buildslave on nooxie is testing its local checkout of the trunk source. So while this is still a bug, it is no longer the cause of any build failures on nooxie.

zooko commented

2007-11-13 19:41:52 +00:00

Author

Brian, do I understand correctly that the fix is for "make test" to do "./setup.py develop" to produce an egg which symlinks into the directory and put that egg on the PYTHONPATH?

zooko added

p/minor

and removed

p/major

labels 2007-11-13 19:42:00 +00:00

zooko added this to the undecided milestone 2008-01-23 02:47:59 +00:00

zooko commented

2008-02-14 04:12:49 +00:00

Author

Nowadays make build runs ./setup.py develop, so this might fix this bug. I haven't tested it yet, though.

Nowadays `make build` runs `./setup.py develop`, so this might fix this bug. I haven't tested it yet, though.

warner was unassigned by zooko

2008-02-14 04:12:49 +00:00

zooko self-assigned this 2008-02-14 04:12:49 +00:00

zooko commented

2008-03-04 22:29:49 +00:00

Author

Yes, this is fixed.

zooko added the

r/fixed

label 2008-03-04 22:29:49 +00:00

zooko closed this issue

2008-03-04 22:29:49 +00:00

zooko modified the milestone from undecided to 0.9.0 (Allmydata 3.0 final)

2008-03-13 17:10:50 +00:00

zooko commented

2008-03-16 08:01:28 +00:00

Author

Hm. This is apparently not fixed. If you easy_install tahoe then make test tests that installed version instead of the one in ./support.

Hm. This is apparently not fixed. If you easy_install tahoe then `make test` tests that installed version instead of the one in ./support.

zooko removed the

r/fixed

label 2008-03-16 08:01:28 +00:00

zooko reopened this issue

2008-03-16 08:01:28 +00:00

warner modified the milestone from 1.1.0 to 1.2.0

2008-05-29 22:31:39 +00:00

zooko added

c/packaging

and removed

c/code

labels 2008-09-22 20:12:08 +00:00

zooko commented

2008-10-22 01:32:41 +00:00

Author

No, I checked and this seems to be fixed. If anyone else can reproduce the problem -- make test testing some other copy of the allmydata-tahoe source code instead of the source code in the current directory -- please let us know.

Also, I guess, if ./setup.py trial tests any allmydata package other than the one in the current directory, that too would mean this ticket should be re-opened.

No, I checked and this seems to be fixed. If anyone else can reproduce the problem -- `make test` testing some other copy of the allmydata-tahoe source code instead of the source code in the current directory -- please let us know. Also, I guess, if `./setup.py trial` tests any `allmydata` package other than the one in the current directory, that too would mean this ticket should be re-opened.

zooko added the

r/fixed

label 2008-10-22 01:32:41 +00:00

zooko closed this issue

2008-10-22 01:32:41 +00:00

daira commented

2010-07-27 06:07:38 +00:00

See #1137 for a reoccurrence of this problem, in the test-from-egg and test-from-prefixdir buildbot steps.

Why do we think it was fixed for setup.py trial? I see no test that would fail if we were testing the wrong code. ([test_path in test_runner.py]source:src/allmydata/test/test_runner.py@4581#L35 checks that we run the same code by invoking bin/tahoe that we are currently testing, but if bin/tahoe and setup.py trial were running the same installed version -- which is entirely plausible because they both work by setting PYTHONPATH -- then we wouldn't detect that.)

See #1137 for a reoccurrence of this problem, in the test-from-egg and test-from-prefixdir buildbot steps. Why do we think it was fixed for `setup.py trial`? I see no test that would fail if we were testing the wrong code. ([test_path in test_runner.py]source:src/allmydata/test/test_runner.py@4581#L35 checks that we run the *same* code by invoking `bin/tahoe` that we are currently testing, but if `bin/tahoe` and `setup.py trial` were running the same installed version -- which is entirely plausible because they both work by setting PYTHONPATH -- then we wouldn't detect that.)

daira added

p/major

and removed

p/minor

r/fixed

labels 2010-07-27 06:07:38 +00:00

daira modified the milestone from 1.5.0 to soon

2010-07-27 06:07:38 +00:00

daira reopened this issue

2010-07-27 06:07:38 +00:00

zooko commented

2010-07-27 06:55:39 +00:00

Author

Hm, [test_path in test_runner.py]source:src/allmydata/test/test_runner.py@4581#L35 detects whether starting from the allmydata directory that contains the allmydata/*init*.py file and going ../../bin/tahoe finds a working tahoe executable that prints out the same path. Hm, and I guess this will fail if we are running the tests of an installed version, for example on my Mac:

 Zooko-Ofsimplegeos-MacBook-Pro:~$ python -i
Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import allmydata
>>> import os
>>> os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(allmydata.__file__))), 'bin', 'tahoe')
'/Library/Python/2.6/site-packages/bin/tahoe'
>>> ^D
 Zooko-Ofsimplegeos-MacBook-Pro:~$ ls -al /Library/Python/2.6/site-packages/bin/tahoe
ls: /Library/Python/2.6/site-packages/bin/tahoe: No such file or directory
 Zooko-Ofsimplegeos-MacBook-Pro:~$ python -c 'import allmydata;print allmydata.__file__'
/Library/Python/2.6/site-packages/allmydata_tahoe-1.7.1-py2.6.egg/allmydata/__init__.pyc

Ah, but in this case the test is skipped:

HACK Zooko-Ofsimplegeos-MacBook-Pro:~/playground/tahoe-lafs/trunk/xyz$ trial allmydata.test.test_runner.TheRightCode.test_path
allmydata.test.test_runner
  TheRightCode
    test_path ...                                                     [SKIPPED]

===============================================================================
[SKIPPED]
The bin/tahoe script isn't to be found in the expected location (/Library/Python/2.6/site-packages/bin/tahoe), and I don't want to test a 'tahoe' executable that I find somewhere else, in case it isn't the right executable for this version of Tahoe. Perhaps running 'setup.py build' again will help.

allmydata.test.test_runner.TheRightCode.test_path
-------------------------------------------------------------------------------
Ran 1 tests in 0.011s

PASSED (skips=1)

So this test does fail in (some) cases that it is running a test_runner.py from an installed version instead of from a source tree, but this is not what we want! I want to be able to run the tests on an installed version and for them to be green, and I also want a different way to ensure that we are testing the right versions of the source in normal tests, test-from-egg and test-from-prefix-installdir on the buildbots. Hm.

Hm, [test_path in test_runner.py]source:src/allmydata/test/test_runner.py@4581#L35 detects whether starting from the `allmydata` directory that contains the `allmydata/*init*.py` file and going `../../bin/tahoe` finds a working `tahoe` executable that prints out the same path. Hm, and I guess this will fail if we are running the tests of an installed version, for example on my Mac: ``` Zooko-Ofsimplegeos-MacBook-Pro:~$ python -i Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) [GCC 4.2.1 (Apple Inc. build 5646)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import allmydata >>> import os >>> os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(allmydata.__file__))), 'bin', 'tahoe') '/Library/Python/2.6/site-packages/bin/tahoe' >>> ^D Zooko-Ofsimplegeos-MacBook-Pro:~$ ls -al /Library/Python/2.6/site-packages/bin/tahoe ls: /Library/Python/2.6/site-packages/bin/tahoe: No such file or directory Zooko-Ofsimplegeos-MacBook-Pro:~$ python -c 'import allmydata;print allmydata.__file__' /Library/Python/2.6/site-packages/allmydata_tahoe-1.7.1-py2.6.egg/allmydata/__init__.pyc ``` Ah, but in this case the test is skipped: ``` HACK Zooko-Ofsimplegeos-MacBook-Pro:~/playground/tahoe-lafs/trunk/xyz$ trial allmydata.test.test_runner.TheRightCode.test_path allmydata.test.test_runner TheRightCode test_path ... [SKIPPED] =============================================================================== [SKIPPED] The bin/tahoe script isn't to be found in the expected location (/Library/Python/2.6/site-packages/bin/tahoe), and I don't want to test a 'tahoe' executable that I find somewhere else, in case it isn't the right executable for this version of Tahoe. Perhaps running 'setup.py build' again will help. allmydata.test.test_runner.TheRightCode.test_path ------------------------------------------------------------------------------- Ran 1 tests in 0.011s PASSED (skips=1) ``` So this test does fail in (some) cases that it is running a `test_runner.py` from an installed version instead of from a source tree, but this is not what we want! I want to be able to run the tests on an installed version and for them to be green, and I also want a different way to ensure that we are testing the right versions of the source in normal tests, test-from-egg and test-from-prefix-installdir on the buildbots. Hm.

daira commented

2010-07-29 03:31:23 +00:00

Replying to zooko:

Hm, [test_path in test_runner.py]source:src/allmydata/test/test_runner.py@4581#L35 detects whether starting from the allmydata directory that contains the allmydata/*init*.py file and going ../../bin/tahoe finds a working tahoe executable that prints out the same path. Hm, and I guess this will fail if we are running the tests of an installed version, for example on my Mac:
[...]
Ah, but in this case the test is skipped:
HACK Zooko-Ofsimplegeos-MacBook-Pro:~/playground/tahoe-lafs/trunk/xyz$ trial allmydata.test.test_runner.TheRightCode.test_path
allmydata.test.test_runner
  [TheRightCode](wiki/TheRightCode)
    test_path ...                                                     SKIPPED

Actually the test will always be skipped rather than failing because of this.
It is skipped in the case in #1137, for example.

So this test does fail in (some) cases that it is running a test_runner.py from an installed version instead of from a source tree, but this is not what we want! I want to be able to run the tests on an installed version and for them to be green, ...

I think running the tests on an installed version (as opposed to a version that is built at a prefixdir) isn't currently a supported thing to do; you have to be in a directory that contains at least a setuptools-*.egg, and presumably that also looks like a Tahoe-LAFS source distribution in other ways (the missing setuptools-*.egg is as far as I got). That it passes when you import an installed version instead of the one in src/allmydata is an accident; I'll open another ticket to support that case.

and I also want a different way to ensure that we are testing the right versions of the source in normal tests, test-from-egg and test-from-prefix-installdir on the buildbots. Hm.

The test_the_right_code test that I added on the ticket1074 branch should fail in this case, but an additional problem is that the version we're incorrectly testing might not have that test (or, we might fix a bug in that the test in future, and be confused by its difference in behaviour in the version we're actually testing).

Making a change in setuptools_trial would have the same problem because we might be using an installed version of that. Making a change in build_helpers/run_trial.py would work, though. I'm trying that approach now.

Replying to [zooko](/tahoe-lafs/trac/issues/145#issuecomment-362838): > Hm, [test_path in test_runner.py]source:src/allmydata/test/test_runner.py@4581#L35 detects whether starting from the `allmydata` directory that contains the `allmydata/*init*.py` file and going `../../bin/tahoe` finds a working `tahoe` executable that prints out the same path. Hm, and I guess this will fail if we are running the tests of an installed version, for example on my Mac: [...] > Ah, but in this case the test is skipped: > ``` > HACK Zooko-Ofsimplegeos-MacBook-Pro:~/playground/tahoe-lafs/trunk/xyz$ trial allmydata.test.test_runner.TheRightCode.test_path > allmydata.test.test_runner > [TheRightCode](wiki/TheRightCode) > test_path ... SKIPPED Actually the test will always be skipped rather than failing because of this. It is skipped in the [case](http://tahoe-lafs.org/buildbot/builders/Kyle%20OpenBSD-4.6%20amd64/builds/316/steps/test-from-prefixdir/logs/stdio) in #1137, for example. > So this test does fail in (some) cases that it is running a `test_runner.py` from an installed version instead of from a source tree, but this is not what we want! I want to be able to run the tests on an installed version and for them to be green, ... I think running the tests on an installed version (as opposed to a version that is built at a prefixdir) isn't currently a supported thing to do; you have to be in a directory that contains at least a `setuptools-*.egg`, and presumably that also looks like a Tahoe-LAFS source distribution in other ways (the missing `setuptools-*.egg` is as far as I got). That it passes when you import an installed version instead of the one in `src/allmydata` is an accident; I'll open another ticket to support that case. > and I also want a different way to ensure that we are testing the right versions of the source in normal tests, test-from-egg and test-from-prefix-installdir on the buildbots. Hm. The `test_the_right_code` test that I [added on the ticket1074 branch](http://tahoe-lafs.org/trac/tahoe-lafs/changeset/4607/ticket1074/) should fail in this case, but an additional problem is that the version we're incorrectly testing might not have that test (or, we might fix a bug in that the test in future, and be confused by its difference in behaviour in the version we're actually testing). Making a change in setuptools_trial would have the same problem because we might be using an installed version of that. Making a change in `build_helpers/run_trial.py` would work, though. I'm trying that approach now.

zooko commented

2010-07-29 05:29:07 +00:00

Author

Hm, let's list the supported ways to run tests:

cd $SOURCEDIR && python setup.py test
cd $SOURCEDIR && trial allmydata.test
mkdir tempempty && cd tempempty && trial allmydata.test # should run unit tests of the installed source code, might skip some tests -- test_runner -- that depend on the test code figuring our where the 'tahoe' executable lives
mkdir -p egginstalldir && PYTHONPATH=egginstalldir:${PYTHONPATH} easy_install -d egginstalldir dist/*.egg && cd egginstalldir && PYTHONPATH=.:${PYTHONPATH} trial allmydata.test # should run unit tests of the code in this egginstalldir including test_runner. This is what install-to-egg and test-from-egg on the buildbot is trying to do.
python setup.py install --single-version-externally-managed --record=/dev/null --prefix=prefixinstalldir && cd prefixinstalldir && PYTHONPATH=lib/python2.6/site-packages:${PYTHONPATH} PATH=bin:${PATH} trial allmydata.test # should run unit tests of the code in this prefixinstalldir including test_runner. This is what install-to-prefixdir and test-from-prefixdir on the buildbot is trying to do

Is that it?

Note: I just tried each of these manually and here are my results:

works
does not work; It runs the unit tests of the installed version in /Library/Python/2.6/site-packages/allmydata_tahoe-1.7.1-py2.6.egg instead of the current source tree, even if I set the PYTHONPATH before running trial.
works
works
does not work; It runs the unit tests of the installed version in /Library/Python/2.6/site-packages/allmydata_tahoe-1.7.1-py2.6.egg instead of the current source tree, even if I set the PYTHONPATH before running trial.

Hm, let's list the supported ways to run tests: 1. `cd $SOURCEDIR && python setup.py test` 2. `cd $SOURCEDIR && trial allmydata.test` 3. `mkdir tempempty && cd tempempty && trial allmydata.test` # should run unit tests of the installed source code, might skip some tests -- test_runner -- that depend on the test code figuring our where the 'tahoe' executable lives 4. `mkdir -p egginstalldir && PYTHONPATH=egginstalldir:${PYTHONPATH} easy_install -d egginstalldir dist/*.egg && cd egginstalldir && PYTHONPATH=.:${PYTHONPATH} trial allmydata.test` # should run unit tests of the code in this egginstalldir including test_runner. This is what install-to-egg and test-from-egg on the buildbot is trying to do. 5. `python setup.py install --single-version-externally-managed --record=/dev/null --prefix=prefixinstalldir && cd prefixinstalldir && PYTHONPATH=lib/python2.6/site-packages:${PYTHONPATH} PATH=bin:${PATH} trial allmydata.test` # should run unit tests of the code in this prefixinstalldir including test_runner. This is what install-to-prefixdir and test-from-prefixdir on the buildbot is trying to do Is that it? Note: I just tried each of these manually and here are my results: 1. works 2. *does not work*; It runs the unit tests of the installed version in `/Library/Python/2.6/site-packages/allmydata_tahoe-1.7.1-py2.6.egg` instead of the current source tree, even if I set the `PYTHONPATH` before running trial. 3. works 4. works 5. *does not work*; It runs the unit tests of the installed version in `/Library/Python/2.6/site-packages/allmydata_tahoe-1.7.1-py2.6.egg` instead of the current source tree, even if I set the `PYTHONPATH` before running trial.

daira commented

2011-01-22 03:13:23 +00:00

Note that as part of #1296, all of the supported ways to run tests (including the Makefile targets) now directly or indirectly invoke bin/tahoe debug trial. The ways to run tests described in comment:26 that didn't work, still don't work, but they are no longer used on the buildbots and I don't think they should be considered supported.

It is still possible for --single-version-externally-managed installs (and other installs that copy code to site directories, such as the Twisted installers for Windows) to mess things up, although we try very hard to detect such cases. The only such cases I'm aware of are caused by #1258. Can we consider this ticket to be a duplicate of that one?

Note that as part of #1296, all of the supported ways to run tests (including the Makefile targets) now directly or indirectly invoke `bin/tahoe debug trial`. The ways to run tests described in comment:26 that didn't work, still don't work, but they are no longer used on the buildbots and I don't think they should be considered supported. It is still possible for `--single-version-externally-managed` installs (and other installs that copy code to site directories, such as the Twisted installers for Windows) to mess things up, although we try very hard to detect such cases. The only such cases I'm aware of are caused by #1258. Can we consider this ticket to be a duplicate of that one?

zooko commented

2011-01-22 05:27:39 +00:00

Author

Let's document the "supported and unsupported ways to run tests" from comment:26. Perhaps in a new file named source:docs/tests.rst?

daira commented

2011-07-23 00:32:49 +00:00

Replying to zooko:

Let's document the "supported and unsupported ways to run tests" from comment:26. Perhaps in a new file named source:docs/tests.rst?

Filed as #1439. I haven't seen any other cases than #1258 where we test the wrong code, so I'm marking this as a duplicate.

Replying to [zooko](/tahoe-lafs/trac/issues/145#issuecomment-362842): > Let's document the "supported and unsupported ways to run tests" from comment:26. Perhaps in a new file named source:docs/tests.rst? Filed as #1439. I haven't seen any other cases than #1258 where we test the wrong code, so I'm marking this as a duplicate.

daira added the

r/duplicate

label 2011-07-23 00:32:49 +00:00

daira closed this issue

2011-07-23 00:32:49 +00:00