the test-from-egg buildstep has started failing on all builders #2378

New issue

Closed

opened 2015-02-06 23:26:28 +00:00 by daira · 10 comments

daira commented

2015-02-06 23:26:28 +00:00

It was passing in [5218e87ed148e92353e03f86e8949d227b5af731/trunk] and has started failing since e4149496d267d39b61d93ececcb014010235f850/trunk. (There are 14 non-merge commits between those revisions, but it should be an easy bisection.)

daira added the

labels 2015-02-06 23:26:28 +00:00

daira added this to the 1.10.1 milestone 2015-02-06 23:26:28 +00:00

daira self-assigned this 2015-02-06 23:26:28 +00:00

daira commented

2015-02-06 23:32:16 +00:00

Author

This is caused by cf0f64be98c42b4c18c2dccf297d67b44de53a0d/trunk. (I hadn't noticed that source:misc/build_helpers/run_trial.py was still used by the test-from-egg step. Mea culpa.)

warner commented

2015-02-25 19:07:30 +00:00

We changed the buildbot config to run sys.executable tahoe debug trial testsuite. Unfortunately that runs into the bug described in #2242, in which a function inside pkg_resources is interpreting a *requires* version specification incorrectly. The symptom is:

ValueError: ('Missing distribution spec', '=', '=', {'thisreqstr': '='})

So I think we must fix #2242 first, either by fixing our zetuptoolz fork, or getting rid of it altogether (#2044).

We changed the buildbot config to run `sys.executable tahoe debug trial testsuite`. Unfortunately that runs into the bug described in #2242, in which a function inside `pkg_resources` is interpreting a `*requires*` version specification incorrectly. The symptom is: ``` ValueError: ('Missing distribution spec', '=', '=', {'thisreqstr': '='}) ``` So I think we must fix #2242 first, either by fixing our zetuptoolz fork, or getting rid of it altogether (#2044).

warner added

c/packaging

and removed

c/code

labels 2015-02-25 19:07:30 +00:00

warner commented

2015-02-25 22:33:32 +00:00

I was able to reproduce this locally, eventually, with the following (paraphrased slightly):

cd ~/tahoe (contains setuptools.egg, really zetuptoolz)
python setup.py bdist_egg -> produces "tahoe.egg"
mkdir ~/tmp ~/tmp/egginst
cd ~/tmp
PYTHONPATH=egginst easy_install -d egginst ~/tahoe/dist/tahoe.egg
cd egginst
PYTHONPATH=~/tahoe/setuptools.egg:./*.egg python tahoe debug trial

The PYTHONPATH in step 5 is necessary to appease easy_install, which is trying to protect you against writing into a directory that you wouldn't normally read from. The PYTHONPATH in step 7 does two things: it uses tahoe's zetuptoolz fork, and it makes the dependencies available.

Step 7 produces the "Missing distribution spec" exception.

The egginst directory, as populated by easy_install, contains an .egg for every dependency, an easy-install.pth which would add them to the path (if you included egginst on PYTHONPATH), and an executable entry-point script for everything that produced one (tahoe, flappserver, flogtool, some zfec ones, etc). It is this tahoe entrypoint script that we execute. On my machine, it contains the following:

# EASY-INSTALL-ENTRY-SCRIPT: 'allmydata-tahoe==1.10.0.post272','console_scripts','tahoe'
__requires__ = 'allmydata-tahoe==1.10.0.post272'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('allmydata-tahoe==1.10.0.post272', 'console_scripts', 'tahoe')()
    )

(this matches what I'm seeing on freestorm's buildslave).

If I omit tahoe's zetuptoolz egg from the PYTHONPATH in step 7, the problem goes away, and the tests run normally.

Note that step 5 is using my system-supplied easy_install, which reports its --version as setuptools 12.1. I think the problem here is a mismatch between the version of setuptools (zetuptoolz) that produced the .egg file in step 2, the version of easy_install that installed the egg (setuptools-12.1) in step 5, and the version of setuptools (zetuptoolz again) which provides the pkg_resources used for execution in step 7.

Our toolchain takes great pains to work even if you don't have setuptools installed on your system (perhaps due to my early complaints about not liking setuptools :-), by providing a setuptools/zetuptoolz .egg in the source tree, and importing it early in setup.py. However, step 5 in the test-from-egg process depends upon having a system-supplied easy_install script. (Tahoe includes setuptools as an egg, but not easy_install).

Digging into the zetuptoolz pkg_resources, it seems that the single-string *requires* in the entrypoint script is, sometimes, being interpreted as a list, causing it to parse each letter of the string as a separate requirement string. Most of these ("a", "l", "l", "m", etc) are ignored, but when it finally gets to "=", the parser throws an exception, because "=" is supposed to have a package name on the left. However, in other places within pkg_resources, the string is correctly interpreted as a string.

I've seen other entrypoint scripts that provide a list for *requires*, like the one tahoe's setup.py build installs into support/bin/tahoe:

# EASY-INSTALL-ENTRY-SCRIPT: 'allmydata-tahoe==1.10.0.post272.dev0','console_scripts','tahoe'
# generated by zetuptoolz 0.6c16dev5
__requires__ = ['allmydata-tahoe==1.10.0.post272.dev0', 'setuptools>=0.6c6', 'zfec>=1.1.0', 'simplejson>=1.4', 'zope.interface>=3.6.0,!=3.6.3,!=3.6.4', 'foolscap>=0.6.3', 'pycrypto>=2.1.0,!=2.2,!=2.4', 'mock>=0.8.0', 'pycryptopp>=0.6.0', 'Twisted>=13.0.0', 'Nevow>=0.11.1', 'service-identity', 'characteristic>=14.0.0', 'pyasn1>=0.1.4', 'pyasn1-modules', 'pyOpenSSL>=0.13', 'cryptography', 'cffi>=0.8', 'six>=1.4.1', 'enum34', 'pycparser']
import sys
from pkg_resources import load_entry_point

sys.exit(
   load_entry_point('allmydata-tahoe==1.10.0.post272.dev0', 'console_scripts', 'tahoe')()
)

And if I make easy_install use tahoe's zetuptoolz egg, by replacing step 5 with:

5b. PYTHONPATH=~/tahoe/setuptools.egg:egginst easy_install -d egginst ~/tahoe/dist/tahoe.egg

then I get the list-*requires* entrypoint script like the one in support/bin/tahoe.

So my hunches are:

zetuptoolz produces, and requires, a *requires* list
in modern setuptools:
- pkg_resources accepts a string or a list
- easy_install produces a string

and the mismatch is between the version of easy_install that creates the entrypoint script, and the version of pkg_resources that gets loaded by that script.

Solutions

So one solution for this ticket might be to use zetuptoolz for the easy_install, like step 5b. I'm experimenting with this to see if it would work.

A second would be to fix zetuptoolz to look more like modern setuptools.

But I suspect this test is no longer exercising the kind of functionality we really care about. Nobody installing a tahoe .egg is going to use 5b: they'll use easy_install tahoe.egg. And they won't include tahoe's zetuptools in PYTHONPATH when they run tahoe, so they'll get their pkg_resources from the system setuptools. To be relevant, our test should match real-world usage.

So a third fix is to remove the non-realistic PYTHONPATH= from step 7. The easy-install.pth in egginstalldir means we could probably use:

7b. PYTHONPATH=~/tmp/egginst python tahoe debug trial

(This reflects reality, since eggs are usually installed into something on the built-in PYTHONPATH, like /usr/local/lib).

Doing that would make the test dependent upon a system-installed setuptools, but it was already depending on that for easy_install.

I'm going to try this third option and see if it works.

I was able to reproduce this locally, eventually, with the following (paraphrased slightly): 1. cd ~/tahoe (contains setuptools.egg, really zetuptoolz) 2. python setup.py bdist_egg -> produces "tahoe.egg" 3. mkdir ~/tmp ~/tmp/egginst 4. cd ~/tmp 5. PYTHONPATH=egginst easy_install -d egginst ~/tahoe/dist/tahoe.egg 6. cd egginst 7. PYTHONPATH=~/tahoe/setuptools.egg:./*.egg python tahoe debug trial The PYTHONPATH in step 5 is necessary to appease `easy_install`, which is trying to protect you against writing into a directory that you wouldn't normally read from. The PYTHONPATH in step 7 does two things: it uses tahoe's zetuptoolz fork, and it makes the dependencies available. Step 7 produces the "Missing distribution spec" exception. The `egginst` directory, as populated by `easy_install`, contains an .egg for every dependency, an `easy-install.pth` which would add them to the path (if you included `egginst` on PYTHONPATH), and an executable entry-point script for everything that produced one (tahoe, flappserver, flogtool, some zfec ones, etc). It is this `tahoe` entrypoint script that we execute. On my machine, it contains the following: ```/usr/local/opt/python/bin/python2.7 # EASY-INSTALL-ENTRY-SCRIPT: 'allmydata-tahoe==1.10.0.post272','console_scripts','tahoe' __requires__ = 'allmydata-tahoe==1.10.0.post272' import sys from pkg_resources import load_entry_point if __name__ == '__main__': sys.exit( load_entry_point('allmydata-tahoe==1.10.0.post272', 'console_scripts', 'tahoe')() ) ``` (this matches what I'm seeing on freestorm's buildslave). If I omit tahoe's zetuptoolz egg from the PYTHONPATH in step 7, the problem goes away, and the tests run normally. Note that step 5 is using my system-supplied `easy_install`, which reports its `--version` as `setuptools 12.1`. I think the problem here is a mismatch between the version of setuptools (zetuptoolz) that produced the .egg file in step 2, the version of easy_install that installed the egg (setuptools-12.1) in step 5, and the version of setuptools (zetuptoolz again) which provides the `pkg_resources` used for execution in step 7. Our toolchain takes great pains to work even if you don't have setuptools installed on your system (perhaps due to my early complaints about not liking setuptools :-), by providing a setuptools/zetuptoolz .egg in the source tree, and importing it early in `setup.py`. However, step 5 in the test-from-egg process depends upon having a system-supplied `easy_install` script. (Tahoe includes setuptools as an egg, but not easy_install). Digging into the zetuptoolz `pkg_resources`, it seems that the single-string `*requires*` in the entrypoint script is, *sometimes*, being interpreted as a list, causing it to parse each letter of the string as a separate requirement string. Most of these ("a", "l", "l", "m", etc) are ignored, but when it finally gets to "=", the parser throws an exception, because "=" is supposed to have a package name on the left. However, in other places within `pkg_resources`, the string is correctly interpreted as a string. I've seen other entrypoint scripts that provide a list for `*requires*`, like the one tahoe's `setup.py build` installs into `support/bin/tahoe`: ```/usr/local/opt/python/bin/python2.7 # EASY-INSTALL-ENTRY-SCRIPT: 'allmydata-tahoe==1.10.0.post272.dev0','console_scripts','tahoe' # generated by zetuptoolz 0.6c16dev5 __requires__ = ['allmydata-tahoe==1.10.0.post272.dev0', 'setuptools>=0.6c6', 'zfec>=1.1.0', 'simplejson>=1.4', 'zope.interface>=3.6.0,!=3.6.3,!=3.6.4', 'foolscap>=0.6.3', 'pycrypto>=2.1.0,!=2.2,!=2.4', 'mock>=0.8.0', 'pycryptopp>=0.6.0', 'Twisted>=13.0.0', 'Nevow>=0.11.1', 'service-identity', 'characteristic>=14.0.0', 'pyasn1>=0.1.4', 'pyasn1-modules', 'pyOpenSSL>=0.13', 'cryptography', 'cffi>=0.8', 'six>=1.4.1', 'enum34', 'pycparser'] import sys from pkg_resources import load_entry_point sys.exit( load_entry_point('allmydata-tahoe==1.10.0.post272.dev0', 'console_scripts', 'tahoe')() ) ``` And if I make `easy_install` use tahoe's zetuptoolz egg, by replacing step 5 with: 5b. PYTHONPATH=~/tahoe/setuptools.egg:egginst easy_install -d egginst ~/tahoe/dist/tahoe.egg then I get the list-`*requires*` entrypoint script like the one in `support/bin/tahoe`. So my hunches are: * zetuptoolz produces, and requires, a `*requires*` list * in modern setuptools: * `pkg_resources` accepts a string or a list * `easy_install` produces a string and the mismatch is between the version of easy_install that creates the entrypoint script, and the version of pkg_resources that gets loaded by that script. ## Solutions So one solution for this ticket might be to use zetuptoolz for the `easy_install`, like step 5b. I'm experimenting with this to see if it would work. A second would be to fix zetuptoolz to look more like modern setuptools. But I suspect this test is no longer exercising the kind of functionality we really care about. Nobody installing a tahoe .egg is going to use 5b: they'll use `easy_install tahoe.egg`. And they won't include tahoe's zetuptools in PYTHONPATH when they run tahoe, so they'll get their pkg_resources from the system setuptools. To be relevant, our test should match real-world usage. So a third fix is to remove the non-realistic PYTHONPATH= from step 7. The `easy-install.pth` in egginstalldir means we could probably use: 7b. PYTHONPATH=~/tmp/egginst python tahoe debug trial (This reflects reality, since eggs are usually installed into something on the built-in PYTHONPATH, like /usr/local/lib). Doing that would make the test dependent upon a system-installed setuptools, but it was already depending on that for easy_install. I'm going to try this third option and see if it works.

warner commented

2015-02-25 22:46:19 +00:00

Hm, one potential problem with that third option: I think the Twisted egg comes from the source directory (is it used as a setup_requires=? or was that a workaround for some dependency's packaging bug?), and doesn't seem to be installed to egginstdir along with everything else. So without a PYTHONPATH that points back to the source dir, this might not work. Haven't tried it yet.

Hm, one potential problem with that third option: I think the Twisted egg comes from the source directory (is it used as a `setup_requires=`? or was that a workaround for some dependency's packaging bug?), and doesn't seem to be installed to `egginstdir` along with everything else. So without a PYTHONPATH that points back to the source dir, this might not work. Haven't tried it yet.

warner commented

2015-02-25 23:01:34 +00:00

Indeed that failed, as it couldn't find the Twisted egg.
Our setup.py explains why it setup_requires= on Twisted (it involves bugs in Nevow packaging). Since removing our dependency on Nevow doesn't sound feasible for this release, I'm going to make the test slightly less realistic and include the Twisted .egg in the PYTHONPATH.

Indeed that failed, as it couldn't find the Twisted egg. Our setup.py explains why it `setup_requires=` on Twisted (it involves bugs in Nevow packaging). Since removing our dependency on Nevow doesn't sound feasible for this release, I'm going to make the test slightly less realistic and include the Twisted .egg in the PYTHONPATH.

zooko commented

2015-02-25 23:13:52 +00:00

I think that bug has been fixed in nevow:

https://github.com/twisted/nevow/issues/7

I think that bug has been fixed in nevow: <https://github.com/twisted/nevow/issues/7>

warner commented

2015-02-26 01:56:24 +00:00

Cool. I'm hesitant to change our setup.py too much right now, but after the release let's remove that stuff and see what happens.

I've experimented some more, and I don't know why easy_install isn't putting a Twisted egg into the egginstdir. It might have to do with the buildslave using distribute-0.6.10 (I'm using setuptools-12.1 here and it installs a twisted egg when Twisted isn't already installed in /usr/lib). It might be that running easy_install from the source directory, where there's a twisted.egg sitting, might trick it into thinking that twisted is generally available. It might also involve an apparent bug in buildbot, since I see the InstallToEgg buildstep is using:

kwargs['env'] = {"PYTHONPATH": egginstalldir}

but easy_install sees PYTHONPATH=egginstalldir: (probably the result of a lazy join.. egginstalldir+":"+os.environ.get("PYTHONPATH","")). The extra colon (or, rather, the empty component once you pp.split(":")) causes the current directory to be added to the PYTHONPATH. And maybe it's somehow seeing the twisted .egg there, so not bothering to install a new one into the install dir.

I'm going to try doing the install from a temporary directory, to see if that helps.

Cool. I'm hesitant to change our setup.py too much right now, but after the release let's remove that stuff and see what happens. I've experimented some more, and I don't know why easy_install isn't putting a Twisted egg into the egginstdir. It might have to do with the buildslave using distribute-0.6.10 (I'm using setuptools-12.1 here and it installs a twisted egg when Twisted isn't already installed in /usr/lib). It might be that running easy_install from the source directory, where there's a twisted.egg sitting, might trick it into thinking that twisted is generally available. It might also involve an apparent bug in buildbot, since I see the `InstallToEgg` buildstep is using: ``` kwargs['env'] = {"PYTHONPATH": egginstalldir} ``` but easy_install sees `PYTHONPATH=egginstalldir:` (probably the result of a lazy join.. `egginstalldir+":"+os.environ.get("PYTHONPATH","")`). The extra colon (or, rather, the empty component once you `pp.split(":")`) causes the current directory to be added to the PYTHONPATH. And maybe it's somehow seeing the twisted .egg there, so not bothering to install a new one into the install dir. I'm going to try doing the install from a temporary directory, to see if that helps.

warner commented

2015-02-26 02:04:59 +00:00

Hey, that fixed it. Ok, time to pull out all the temporary debug stuff and make sure it all works.

warner commented

2015-02-26 06:32:02 +00:00

The tempdir-on-easy_install seems to have fixed it.. the orange buildslaves are turning green. Calling this one fixed.

warner added the

r/fixed

label 2015-02-26 06:32:02 +00:00

warner closed this issue

2015-02-26 06:32:02 +00:00

daira commented

2015-02-27 17:43:53 +00:00

Author

Replying to zooko:

I think that bug has been fixed in nevow:

https://github.com/twisted/nevow/issues/7

It has, and removing the workaround will work on Unix. It won't work on Windows, because on Windows we still require an unfixed version of Nevow (because the fixed version depends on Twisted >= 13 and that in turn depends on pywin32).

Replying to [zooko](/tahoe-lafs/trac/issues/2378#issuecomment-398632): > I think that bug has been fixed in nevow: > > <https://github.com/twisted/nevow/issues/7> It has, and removing the workaround will work on Unix. It won't work on Windows, because on Windows we still require an unfixed version of Nevow (because the fixed version depends on Twisted >= 13 and that in turn depends on pywin32).