tahoe-lafs/trac

`pip` managed install is broken. #2032

New issue

Closed

opened 2013-07-20 19:33:59 +00:00 by nejucomo · 14 comments

nejucomo commented

2013-07-20 19:33:59 +00:00

Synopsis:

The command pip install allmydata-tahoe fails because of a bug in the Nevow dependency.

Known workaround:

Install twisted first with pip. This works for me:

$ pip install twisted && pip install allmydata-tahoe

Details:

It appears related to Nevow. I'm hopping on #twisted to ask for advice.

$ mkdir ~/tmp/tahoe-install-test

$ cd !$
cd ~/tmp/tahoe-install-test

$ virtualenv .
Using real prefix '/usr/local/bin/../stow/Python-2.7.3-setuptools-0.6c11'
New python executable in ./bin/python
Installing setuptools............done.
Installing pip...............done.

$ source ./bin/activate
(tahoe-install-test)
$ pip install allmydata-tahoe
Downloading/unpacking allmydata-tahoe
  Downloading allmydata-tahoe-1.10.0.tar.gz (1.4MB): 1.4MB downloaded
  Running setup.py egg_info for package allmydata-tahoe
    Not found: tahoe-deps
    Not found: ../tahoe-deps

    Installed /home/n/tmp/tahoe-install-test/build/allmydata-tahoe/Twisted-13.0.0-py2.7-linux-x86_64.egg
    Searching for zope.interface==3.6.0,==3.6.1,==3.6.2,>=3.6.5
    Reading http://pypi.python.org/simple/zope.interface/
    Best match: zope.interface 4.0.5
    Downloading https://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-sdists/zope.interface-4.0.5.zip
    Processing zope.interface-4.0.5.zip
    Running zope.interface-4.0.5/setup.py -q bdist_egg --dist-dir /home/n/tmp/easy_install-B8abnY/zope.interface-4.0.5/egg-dist-tmp-7VsGgq
    warning: no previously-included files matching '*.dll' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '*.pyo' found anywhere in distribution
    warning: no previously-included files matching '*.so' found anywhere in distribution

    Installed /home/n/tmp/tahoe-install-test/build/allmydata-tahoe/zope.interface-4.0.5-py2.7-linux-x86_64.egg

    package init file 'src/allmydata/web/static/__init__.py' not found (or not a regular file)
    package init file 'src/allmydata/web/static/css/__init__.py' not found (or not a regular file)
    warning: no previously-included files matching '*~' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
Requirement already satisfied (use --upgrade to upgrade): setuptools>=0.6c6 in ./lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg (from allmydata-tahoe)
Downloading/unpacking zfec>=1.1.0 (from allmydata-tahoe)
  Downloading zfec-1.4.24.tar.gz (175kB): 175kB downloaded
  Running setup.py egg_info for package zfec

Downloading/unpacking simplejson>=1.4 (from allmydata-tahoe)
  Downloading simplejson-3.3.0.tar.gz (67kB): 67kB downloaded
  Running setup.py egg_info for package simplejson

Downloading/unpacking zope.interface==3.6.0,==3.6.1,==3.6.2,>=3.6.5 (from allmydata-tahoe)
  Downloading zope.interface-4.0.5.zip (173kB): 173kB downloaded
  Running setup.py egg_info for package zope.interface

    warning: no previously-included files matching '*.dll' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '*.pyo' found anywhere in distribution
    warning: no previously-included files matching '*.so' found anywhere in distribution
Downloading/unpacking Twisted>=11.0.0 (from allmydata-tahoe)
  Downloading Twisted-13.1.0.tar.bz2 (2.7MB): 2.7MB downloaded
  Running setup.py egg_info for package Twisted

Downloading/unpacking foolscap>=0.6.3 (from allmydata-tahoe)
  Downloading foolscap-0.6.4.tar.gz (479kB): 479kB downloaded
  Running setup.py egg_info for package foolscap

Downloading/unpacking pyOpenSSL (from allmydata-tahoe)
  Downloading pyOpenSSL-0.13.tar.gz (250kB): 250kB downloaded
  Running setup.py egg_info for package pyOpenSSL

    warning: no previously-included files matching '*.pyc' found anywhere in distribution
Downloading/unpacking Nevow>=0.6.0 (from allmydata-tahoe)
  Downloading Nevow-0.10.0.tar.gz (518kB): 518kB downloaded
  Running setup.py egg_info for package Nevow
    Traceback (most recent call last):
      File "<string>", line 16, in <module>
      File "/home/n/tmp/tahoe-install-test/build/Nevow/setup.py", line 3, in <module>
        from nevow import __version__ as version
      File "nevow/__init__.py", line 5, in <module>
        from nevow._version import version
      File "nevow/_version.py", line 2, in <module>
        from twisted.python import versions
    ImportError: No module named twisted.python
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 16, in <module>

  File "/home/n/tmp/tahoe-install-test/build/Nevow/setup.py", line 3, in <module>

    from nevow import __version__ as version

  File "nevow/__init__.py", line 5, in <module>

    from nevow._version import version

  File "nevow/_version.py", line 2, in <module>

    from twisted.python import versions

ImportError: No module named twisted.python

----------------------------------------
Command python setup.py egg_info failed with error code 1 in /home/n/tmp/tahoe-install-test/build/Nevow
Storing complete log in /home/n/.pip/pip.log

**Synopsis:** The command `pip install allmydata-tahoe` fails because of a bug in the `Nevow` dependency. **Known workaround:** Install `twisted` first with pip. This works for me: ``` $ pip install twisted && pip install allmydata-tahoe ``` **Details:** It appears related to `Nevow`. I'm hopping on `#twisted` to ask for advice. ``` $ mkdir ~/tmp/tahoe-install-test $ cd !$ cd ~/tmp/tahoe-install-test $ virtualenv . Using real prefix '/usr/local/bin/../stow/Python-2.7.3-setuptools-0.6c11' New python executable in ./bin/python Installing setuptools............done. Installing pip...............done. $ source ./bin/activate (tahoe-install-test) $ pip install allmydata-tahoe Downloading/unpacking allmydata-tahoe Downloading allmydata-tahoe-1.10.0.tar.gz (1.4MB): 1.4MB downloaded Running setup.py egg_info for package allmydata-tahoe Not found: tahoe-deps Not found: ../tahoe-deps Installed /home/n/tmp/tahoe-install-test/build/allmydata-tahoe/Twisted-13.0.0-py2.7-linux-x86_64.egg Searching for zope.interface==3.6.0,==3.6.1,==3.6.2,>=3.6.5 Reading http://pypi.python.org/simple/zope.interface/ Best match: zope.interface 4.0.5 Downloading https://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-sdists/zope.interface-4.0.5.zip Processing zope.interface-4.0.5.zip Running zope.interface-4.0.5/setup.py -q bdist_egg --dist-dir /home/n/tmp/easy_install-B8abnY/zope.interface-4.0.5/egg-dist-tmp-7VsGgq warning: no previously-included files matching '*.dll' found anywhere in distribution warning: no previously-included files matching '*.pyc' found anywhere in distribution warning: no previously-included files matching '*.pyo' found anywhere in distribution warning: no previously-included files matching '*.so' found anywhere in distribution Installed /home/n/tmp/tahoe-install-test/build/allmydata-tahoe/zope.interface-4.0.5-py2.7-linux-x86_64.egg package init file 'src/allmydata/web/static/__init__.py' not found (or not a regular file) package init file 'src/allmydata/web/static/css/__init__.py' not found (or not a regular file) warning: no previously-included files matching '*~' found anywhere in distribution warning: no previously-included files matching '*.pyc' found anywhere in distribution Requirement already satisfied (use --upgrade to upgrade): setuptools>=0.6c6 in ./lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg (from allmydata-tahoe) Downloading/unpacking zfec>=1.1.0 (from allmydata-tahoe) Downloading zfec-1.4.24.tar.gz (175kB): 175kB downloaded Running setup.py egg_info for package zfec Downloading/unpacking simplejson>=1.4 (from allmydata-tahoe) Downloading simplejson-3.3.0.tar.gz (67kB): 67kB downloaded Running setup.py egg_info for package simplejson Downloading/unpacking zope.interface==3.6.0,==3.6.1,==3.6.2,>=3.6.5 (from allmydata-tahoe) Downloading zope.interface-4.0.5.zip (173kB): 173kB downloaded Running setup.py egg_info for package zope.interface warning: no previously-included files matching '*.dll' found anywhere in distribution warning: no previously-included files matching '*.pyc' found anywhere in distribution warning: no previously-included files matching '*.pyo' found anywhere in distribution warning: no previously-included files matching '*.so' found anywhere in distribution Downloading/unpacking Twisted>=11.0.0 (from allmydata-tahoe) Downloading Twisted-13.1.0.tar.bz2 (2.7MB): 2.7MB downloaded Running setup.py egg_info for package Twisted Downloading/unpacking foolscap>=0.6.3 (from allmydata-tahoe) Downloading foolscap-0.6.4.tar.gz (479kB): 479kB downloaded Running setup.py egg_info for package foolscap Downloading/unpacking pyOpenSSL (from allmydata-tahoe) Downloading pyOpenSSL-0.13.tar.gz (250kB): 250kB downloaded Running setup.py egg_info for package pyOpenSSL warning: no previously-included files matching '*.pyc' found anywhere in distribution Downloading/unpacking Nevow>=0.6.0 (from allmydata-tahoe) Downloading Nevow-0.10.0.tar.gz (518kB): 518kB downloaded Running setup.py egg_info for package Nevow Traceback (most recent call last): File "<string>", line 16, in <module> File "/home/n/tmp/tahoe-install-test/build/Nevow/setup.py", line 3, in <module> from nevow import __version__ as version File "nevow/__init__.py", line 5, in <module> from nevow._version import version File "nevow/_version.py", line 2, in <module> from twisted.python import versions ImportError: No module named twisted.python Complete output from command python setup.py egg_info: Traceback (most recent call last): File "<string>", line 16, in <module> File "/home/n/tmp/tahoe-install-test/build/Nevow/setup.py", line 3, in <module> from nevow import __version__ as version File "nevow/__init__.py", line 5, in <module> from nevow._version import version File "nevow/_version.py", line 2, in <module> from twisted.python import versions ImportError: No module named twisted.python ---------------------------------------- Command python setup.py egg_info failed with error code 1 in /home/n/tmp/tahoe-install-test/build/Nevow Storing complete log in /home/n/.pip/pip.log ```

nejucomo added the

labels 2013-07-20 19:33:59 +00:00

nejucomo added this to the undecided milestone 2013-07-20 19:33:59 +00:00

nejucomo commented

2013-07-20 19:42:59 +00:00

Author

This is a known Nevow vs pip issue: https://bugs.launchpad.net/nevow/+bug/1091055

This is a known `Nevow` vs `pip` issue: <https://bugs.launchpad.net/nevow/+bug/1091055>

nejucomo commented

2013-07-20 20:15:43 +00:00

Author

I've submitted a patch to Nevow which fixes this issue (at least on my system): https://bugs.launchpad.net/nevow/+bug/1091055/comments/5

I've submitted a patch to Nevow which fixes this issue (at least on my system): <https://bugs.launchpad.net/nevow/+bug/1091055/comments/5>

nejucomo added

p/major

and removed

p/normal

labels 2013-07-20 20:23:18 +00:00

nejucomo commented

2013-07-20 20:25:07 +00:00

Author

I consider this major because I'm part of the PyPI + pip hegemony which intends to take over the (python) world. Also, check out this claim of >40,000 PyPI downloads!

I consider this major because I'm part of the PyPI + pip hegemony which intends to take over the (python) world. Also, check out [this claim](https://crate.io/packages/allmydata-tahoe/) of >40,000 PyPI downloads!

nejucomo commented

2013-07-20 20:26:27 +00:00

Author

See also #1963 "replace nevow with twisted.web.template"

See also #1963 `"replace nevow with twisted.web.template"`

nejucomo added

p/normal

and removed

p/major

labels 2013-07-20 20:26:27 +00:00

nejucomo added

p/major

and removed

p/normal

labels 2013-07-20 20:27:35 +00:00

nejucomo commented

2013-07-20 20:57:42 +00:00

Author

Replying to nejucomo:

I've submitted a patch to Nevow which fixes this issue (at least on my system): https://bugs.launchpad.net/nevow/+bug/1091055/comments/5

Also, to keep the ball rolling I attempted to translate the patch into a bzr / lp merge request.

Replying to [nejucomo](/tahoe-lafs/trac/issues/2032#issuecomment-394343): > I've submitted a patch to Nevow which fixes this issue (at least on my system): <https://bugs.launchpad.net/nevow/+bug/1091055/comments/5> Also, to keep the ball rolling I attempted to translate the patch into a [bzr / lp merge request](https://code.launchpad.net/~nejucomo/divmod.org/fix-nevow-setup/+merge/176026).

daira commented

2013-08-28 11:16:25 +00:00

I believe that, due to this Nevow commit, the bug only happens when import setuptools fails (as it will in a clean virtualenv).

I believe that, due to [this Nevow commit](http://bazaar.launchpad.net/~divmod-dev/divmod.org/build-nevow-without-importing-2699/revision/2601#Nevow/setup.py), the bug only happens when `import setuptools` fails (as it will in a clean virtualenv).

daira commented

2013-08-28 11:24:10 +00:00

... and Zooko correctly anticipated the problem two years ago. How annoying.

... and Zooko correctly anticipated the problem [two years ago](https://bugs.launchpad.net/nevow/+bug/812537). How annoying.

daira commented

2013-08-28 11:34:44 +00:00

[12:28] <dstufft> setuptools should always be importable when you're installing with pip (unless you hit the pip bug that uninstalls setuptools but your entire install will break then)

[12:31] <dstufft> daira1: pip is still broken with Nevow because the last release of Nevow was made in 2009 and it does not contain that commit

[12:32] <daira1> wow. okay

`[12:28] <dstufft>` setuptools should always be importable when you're installing with pip (unless you hit the pip bug that uninstalls setuptools but your entire install will break then) `[12:31] <dstufft>` daira1: pip is still broken with Nevow because the last release of Nevow was made in 2009 and it does not contain that commit `[12:32] <daira1>` wow. okay

nejucomo commented

2013-08-29 00:23:09 +00:00

Author

Replying to daira:

I believe that, due to this Nevow commit, the bug only happens when import setuptools fails (as it will in a clean virtualenv).

There may be more than one bug. I don't understand what the presence of setuptools has to do with the bug I find when running pip install nevow, which is that if setup.py imports nevow, then it will import twisted.

If twisted is not installed, then setup.py will raise an ImportError. This prevents a user from running "python ./setup.py " unless they've already installed a dependency. If this is true then I believe it would also break easy_install or any other case where setup.py is executed prior to satisfying dependencies.

Replying to [daira](/tahoe-lafs/trac/issues/2032#issuecomment-394353): > I believe that, due to [this Nevow commit](http://bazaar.launchpad.net/~divmod-dev/divmod.org/build-nevow-without-importing-2699/revision/2601#Nevow/setup.py), the bug only happens when `import setuptools` fails (as it will in a clean virtualenv). There may be more than one bug. I don't understand what the presence of `setuptools` has to do with the bug I find when running `pip install nevow`, which is that if `setup.py` imports `nevow`, then it will import `twisted`. If `twisted` is not installed, then `setup.py` will raise an `ImportError`. This prevents a user from running "python ./setup.py <anything>" unless they've already installed a dependency. If this is true then I believe it would also break `easy_install` or any other case where `setup.py` is executed *prior* to satisfying dependencies.

daira commented

2013-09-01 05:16:29 +00:00

Replying to [nejucomo]comment:13:

Replying to daira:

I believe that, due to this Nevow commit, the bug only happens when import setuptools fails (as it will in a clean virtualenv).

There may be more than one bug. I don't understand what the presence of setuptools has to do with the bug I find when running pip install nevow, which is that if setup.py imports nevow, then it will import twisted.

I was mistaken: the Nevow commit isn't in any release yet [isn't on trunk, since the branch was never merged]and. pip depends on setuptools so that will always be present.

Replying to [nejucomo]comment:13: > Replying to [daira](/tahoe-lafs/trac/issues/2032#issuecomment-394353): > > I believe that, due to [this Nevow commit](http://bazaar.launchpad.net/~divmod-dev/divmod.org/build-nevow-without-importing-2699/revision/2601#Nevow/setup.py), the bug only happens when `import setuptools` fails (as it will in a clean virtualenv). > > There may be more than one bug. I don't understand what the presence of `setuptools` has to do with the bug I find when running `pip install nevow`, which is that if `setup.py` imports `nevow`, then it will import `twisted`. I was mistaken: the Nevow commit isn't in any release yet [isn't on trunk, since the branch was never merged]and. pip depends on setuptools so that will always be present.

zooko commented

2014-01-06 19:19:52 +00:00

I've added a comment on https://bugs.launchpad.net/nevow/+bug/812537 saying that this issue is causing Tahoe-LAFS users to be unable to install it.

I've added a comment on <https://bugs.launchpad.net/nevow/+bug/812537> saying that this issue is causing Tahoe-LAFS users to be unable to install it.

warner commented

2014-06-23 19:41:42 +00:00

Nevow-0.11.1 was released a few days ago (https://github.com/twisted/nevow/blob/nevow-0.11.1/NEWS.txt), and supports install from 'pip'. I just confirmed that a simple pip install allmydata-tahoe in a clean virtualenv gets me a working tahoe binary (and tahoe debug repl can import nevow without complaint).

Closing this one out. Yay! Crediting this to the 1.11 milestone (even though it was fixed by something outside of tahoe's source tree) so we remember to mention it in the release notes.

Nevow-0.11.1 was released a few days ago (<https://github.com/twisted/nevow/blob/nevow-0.11.1/NEWS.txt>), and supports install from 'pip'. I just confirmed that a simple `pip install allmydata-tahoe` in a clean virtualenv gets me a working `tahoe` binary (and `tahoe debug repl` can `import nevow` without complaint). Closing this one out. Yay! Crediting this to the 1.11 milestone (even though it was fixed by something outside of tahoe's source tree) so we remember to mention it in the release notes.

warner added the

r/fixed

label 2014-06-23 19:41:42 +00:00

warner modified the milestone from undecided to 1.11.0

2014-06-23 19:41:42 +00:00

warner closed this issue

2014-06-23 19:41:42 +00:00

nejucomo commented

2014-07-02 03:56:28 +00:00

Author

Daira and I noticed we do not yet depend on Nevow >= 0.11.1. We're not yet sure if it's still possible to run into this problem, but it might be prudent to bump the requirement up.

I'm going to reopen this to ensure we bump the dependency version or discuss why not to do so.

Daira and I noticed we do not yet depend on `Nevow >= 0.11.1`. We're not yet sure if it's still possible to run into this problem, but it might be prudent to bump the requirement up. I'm going to reopen this to ensure we bump the dependency version or discuss why not to do so.

nejucomo removed the

r/fixed

label 2014-07-02 03:56:28 +00:00

nejucomo reopened this issue

2014-07-02 03:56:28 +00:00

daira commented

2014-09-02 16:57:33 +00:00

Replying to nejucomo:

Daira and I noticed we do not yet depend on Nevow >= 0.11.1. We're not yet sure if it's still possible to run into this problem, but it might be prudent to bump the requirement up.

I'm going to reopen this to ensure we bump the dependency version or discuss why not to do so.

Moved to #2291. The original problem is fixed just by the availability of Nevow 0.11.1.

Replying to [nejucomo](/tahoe-lafs/trac/issues/2032#issuecomment-394363): > Daira and I noticed we do not yet depend on `Nevow >= 0.11.1`. We're not yet sure if it's still possible to run into this problem, but it might be prudent to bump the requirement up. > > I'm going to reopen this to ensure we bump the dependency version or discuss why not to do so. Moved to #2291. The original problem is fixed just by the availability of Nevow 0.11.1.

pip managed install is broken. #2032

`pip` managed install is broken. #2032