pypiwin32 not picked up in Windows wheel install #2763

New issue

Closed

opened 2016-03-31 13:14:33 +00:00 by crd · 6 comments

crd commented

2016-03-31 13:14:33 +00:00

Owner

I just installed tahoe-lafs 1.11.0 on Windows 10 and ran into a minor problem in that it did not automatically pick up the pypiwin32 package:

Microsoft Windows [Version 10.0.10586]
(c) 2015 Microsoft Corporation. All rights reserved.

C:\Users\crd>\Python27\Scripts\virtualenv tahoe-lafs
New python executable in C:\Users\crd\tahoe-lafs\Scripts\python.exe
Installing setuptools, pip, wheel...done.

C:\Users\crd>tahoe-lafs\Scripts\activate

(tahoe-lafs) C:\Users\crd>pip list
pip (8.1.1)
setuptools (20.4)
wheel (0.29.0)

(tahoe-lafs) C:\Users\crd>pip install --find-links=https://tahoe-lafs.org/deps/ tahoe-lafs
Collecting tahoe-lafs
  Downloading tahoe_lafs-1.11.0-py2-none-any.whl (1.1MB)
    100% |################################| 1.1MB 522kB/s
Requirement already satisfied (use --upgrade to upgrade): setuptools>=11.3 in c:\users\crd\tahoe-lafs\lib\site-packages (from tahoe-lafs)
Collecting simplejson>=1.4 (from tahoe-lafs)
  Downloading simplejson-3.8.2-cp27-cp27m-win_amd64.whl (66kB)
    100% |################################| 71kB 1.7MB/s
Collecting service-identity (from tahoe-lafs)
  Downloading service_identity-16.0.0-py2.py3-none-any.whl
Collecting zope.interface!=3.6.3,!=3.6.4,>=3.6.0 (from tahoe-lafs)
  Downloading https://tahoe-lafs.org/deps/zope.interface-4.1.3-cp27-cp27m-win_amd64.whl (130kB)
    100% |################################| 133kB 497kB/s
Collecting pyasn1>=0.1.8 (from tahoe-lafs)
  Downloading pyasn1-0.1.9-py2.py3-none-any.whl
Collecting foolscap>=0.10.1 (from tahoe-lafs)
  Downloading foolscap-0.11.0-py2-none-any.whl (290kB)
    100% |################################| 296kB 1.2MB/s
Collecting pycrypto!=2.2,!=2.4,>=2.1.0 (from tahoe-lafs)
  Downloading https://tahoe-lafs.org/deps/pycrypto-2.6.1-cp27-cp27m-win_amd64.whl (478kB)
    100% |################################| 481kB 860kB/s
Collecting pyOpenSSL>=0.14 (from tahoe-lafs)
  Downloading pyOpenSSL-16.0.0-py2.py3-none-any.whl (45kB)
    100% |################################| 51kB 1.8MB/s
Collecting characteristic>=14.0.0 (from tahoe-lafs)
  Downloading characteristic-14.3.0-py2.py3-none-any.whl
Collecting zfec>=1.1.0 (from tahoe-lafs)
  Downloading https://tahoe-lafs.org/deps/zfec-1.4.24-cp27-cp27m-win_amd64.whl (70kB)
    100% |################################| 71kB 669kB/s
Collecting pycryptopp>=0.6.0 (from tahoe-lafs)
  Downloading https://tahoe-lafs.org/deps/pycryptopp-0.7.1.869544967005693312591928092448767568728501330214-cp27-cp27m-win_amd64.whl (1.5MB)
    100% |################################| 1.5MB 380kB/s
Collecting pyasn1-modules>=0.0.5 (from tahoe-lafs)
  Downloading pyasn1_modules-0.0.8-py2.py3-none-any.whl
Collecting Nevow>=0.11.1 (from tahoe-lafs)
  Downloading Nevow-0.13.0-py2-none-any.whl (448kB)
    100% |################################| 450kB 898kB/s
Collecting Twisted[tls]>=15.1.0 (from tahoe-lafs)
  Downloading Twisted-16.0.0-cp27-none-win_amd64.whl (3.2MB)
    100% |################################| 3.2MB 215kB/s
Collecting attrs (from service-identity->tahoe-lafs)
  Downloading attrs-15.2.0-py2.py3-none-any.whl
Collecting cryptography>=1.3 (from pyOpenSSL>=0.14->tahoe-lafs)
  Downloading cryptography-1.3.1-cp27-none-win_amd64.whl (1.3MB)
    100% |################################| 1.3MB 451kB/s
Collecting six>=1.5.2 (from pyOpenSSL>=0.14->tahoe-lafs)
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting pyutil>=1.3.19 (from zfec>=1.1.0->tahoe-lafs)
  Downloading https://tahoe-lafs.org/deps/pyutil-2.0.0-py2-none-any.whl (180kB)
    100% |################################| 184kB 747kB/s
Collecting idna>=0.6 (from Twisted[tls]>=15.1.0->tahoe-lafs)
  Downloading idna-2.1-py2-none-any.whl (54kB)
    100% |################################| 61kB 1.8MB/s
Collecting enum34 (from cryptography>=1.3->pyOpenSSL>=0.14->tahoe-lafs)
  Downloading https://tahoe-lafs.org/deps/enum34-1.1.2-cp27-none-any.whl (57kB)
    100% |################################| 61kB 516kB/s
Collecting ipaddress (from cryptography>=1.3->pyOpenSSL>=0.14->tahoe-lafs)
  Downloading ipaddress-1.0.16-py27-none-any.whl
Collecting cffi>=1.4.1 (from cryptography>=1.3->pyOpenSSL>=0.14->tahoe-lafs)
  Downloading cffi-1.5.2-cp27-none-win_amd64.whl (150kB)
    100% |################################| 153kB 1.5MB/s
Collecting zbase32>=1.0 (from pyutil>=1.3.19->zfec>=1.1.0->tahoe-lafs)
  Downloading https://tahoe-lafs.org/deps/zbase32-1.1.5-py2-none-any.whl
Collecting pycparser (from cffi>=1.4.1->cryptography>=1.3->pyOpenSSL>=0.14->tahoe-lafs)
  Downloading https://tahoe-lafs.org/deps/pycparser-2.14-cp27-none-any.whl (188kB)
    100% |################################| 194kB 1.2MB/s
Installing collected packages: simplejson, enum34, ipaddress, pyasn1, six, idna, pycparser, cffi, cryptography, pyOpenSSL, pyasn1-modules, attrs, service-identity, zope.interface, Twisted, foolscap, pycrypto, characteristic, zbase32, pyutil, zfec, pycryptopp, Nevow, tahoe-lafs
Successfully installed Nevow-0.13.0 Twisted-16.0.0 attrs-15.2.0 cffi-1.5.2 characteristic-14.3.0 cryptography-1.3.1 enum34-1.1.2 foolscap-0.11.0 idna-2.1 ipaddress-1.0.16 pyOpenSSL-16.0.0 pyasn1-0.1.9 pyasn1-modules-0.0.8 pycparser-2.14 pycrypto-2.6.1 pycryptopp-0.7.1.869544967005693312591928092448767568728501330214 pyutil-2.0.0 service-identity-16.0.0 simplejson-3.8.2 six-1.10.0 tahoe-lafs-1.11.0 zbase32-1.1.5 zfec-1.4.24 zope.interface-4.1.3

(tahoe-lafs) C:\Users\crd>tahoe --version
Traceback (most recent call last):
  File "c:\python27\Lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\python27\Lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Users\crd\tahoe-lafs\Scripts\tahoe.exe\__main__.py", line 5, in <module>
  File "c:\users\crd\tahoe-lafs\lib\site-packages\allmydata\__init__.py", line 412, in <module>
    _vers_and_locs_list, _cross_check_errors = get_package_versions_and_locations()
  File "c:\users\crd\tahoe-lafs\lib\site-packages\allmydata\__init__.py", line 222, in get_package_versions_and_locations
    for p in pkg_resources.require(install_requires)])
  File "c:\users\crd\tahoe-lafs\lib\site-packages\pkg_resources\__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "c:\users\crd\tahoe-lafs\lib\site-packages\pkg_resources\__init__.py", line 830, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pypiwin32' distribution was not found and is required by the application

(tahoe-lafs) C:\Users\crd>
(tahoe-lafs) C:\Users\crd>pip install --find-links=https://tahoe-lafs.org/deps/ pypiwin32
Collecting pypiwin32
  Downloading pypiwin32-219-cp27-none-win_amd64.whl (7.3MB)
    100% |################################| 7.3MB 98kB/s
Installing collected packages: pypiwin32
Successfully installed pypiwin32-219

(tahoe-lafs) C:\Users\crd>tahoe --version
tahoe-lafs: 1.11.0 [master: 04a3e7993f70ac87c208d61f4387a59f5f419367]
foolscap: 0.11.0
pycryptopp: 0.7.1.869544967005693312591928092448767568728501330214
zfec: 1.4.24
Twisted: 16.0.0
Nevow: 0.13.0
zope.interface: unknown
python: 2.7.11
platform: Windows-10-10.0.10586
pyOpenSSL: 16.0.0
OpenSSL: 1.0.2g [ 1 Mar 2016]
simplejson: 3.8.2
pycrypto: 2.6.1
pyasn1: 0.1.9
service-identity: 16.0.0
characteristic: 14.3.0
pyasn1-modules: 0.0.8
cryptography: 1.3.1
cffi: 1.5.2
six: 1.10.0
enum34: 1.1.2
pycparser: 2.14
setuptools: 20.4
pypiwin32: 219
attrs: 15.2.0 [according to pkg_resources]
ipaddress: 1.0.16 [according to pkg_resources]
idna: 2.1 [according to pkg_resources]

I just installed tahoe-lafs 1.11.0 on Windows 10 and ran into a minor problem in that it did not automatically pick up the pypiwin32 package: ``` Microsoft Windows [Version 10.0.10586] (c) 2015 Microsoft Corporation. All rights reserved. C:\Users\crd>\Python27\Scripts\virtualenv tahoe-lafs New python executable in C:\Users\crd\tahoe-lafs\Scripts\python.exe Installing setuptools, pip, wheel...done. C:\Users\crd>tahoe-lafs\Scripts\activate (tahoe-lafs) C:\Users\crd>pip list pip (8.1.1) setuptools (20.4) wheel (0.29.0) (tahoe-lafs) C:\Users\crd>pip install --find-links=https://tahoe-lafs.org/deps/ tahoe-lafs Collecting tahoe-lafs Downloading tahoe_lafs-1.11.0-py2-none-any.whl (1.1MB) 100% |################################| 1.1MB 522kB/s Requirement already satisfied (use --upgrade to upgrade): setuptools>=11.3 in c:\users\crd\tahoe-lafs\lib\site-packages (from tahoe-lafs) Collecting simplejson>=1.4 (from tahoe-lafs) Downloading simplejson-3.8.2-cp27-cp27m-win_amd64.whl (66kB) 100% |################################| 71kB 1.7MB/s Collecting service-identity (from tahoe-lafs) Downloading service_identity-16.0.0-py2.py3-none-any.whl Collecting zope.interface!=3.6.3,!=3.6.4,>=3.6.0 (from tahoe-lafs) Downloading https://tahoe-lafs.org/deps/zope.interface-4.1.3-cp27-cp27m-win_amd64.whl (130kB) 100% |################################| 133kB 497kB/s Collecting pyasn1>=0.1.8 (from tahoe-lafs) Downloading pyasn1-0.1.9-py2.py3-none-any.whl Collecting foolscap>=0.10.1 (from tahoe-lafs) Downloading foolscap-0.11.0-py2-none-any.whl (290kB) 100% |################################| 296kB 1.2MB/s Collecting pycrypto!=2.2,!=2.4,>=2.1.0 (from tahoe-lafs) Downloading https://tahoe-lafs.org/deps/pycrypto-2.6.1-cp27-cp27m-win_amd64.whl (478kB) 100% |################################| 481kB 860kB/s Collecting pyOpenSSL>=0.14 (from tahoe-lafs) Downloading pyOpenSSL-16.0.0-py2.py3-none-any.whl (45kB) 100% |################################| 51kB 1.8MB/s Collecting characteristic>=14.0.0 (from tahoe-lafs) Downloading characteristic-14.3.0-py2.py3-none-any.whl Collecting zfec>=1.1.0 (from tahoe-lafs) Downloading https://tahoe-lafs.org/deps/zfec-1.4.24-cp27-cp27m-win_amd64.whl (70kB) 100% |################################| 71kB 669kB/s Collecting pycryptopp>=0.6.0 (from tahoe-lafs) Downloading https://tahoe-lafs.org/deps/pycryptopp-0.7.1.869544967005693312591928092448767568728501330214-cp27-cp27m-win_amd64.whl (1.5MB) 100% |################################| 1.5MB 380kB/s Collecting pyasn1-modules>=0.0.5 (from tahoe-lafs) Downloading pyasn1_modules-0.0.8-py2.py3-none-any.whl Collecting Nevow>=0.11.1 (from tahoe-lafs) Downloading Nevow-0.13.0-py2-none-any.whl (448kB) 100% |################################| 450kB 898kB/s Collecting Twisted[tls]>=15.1.0 (from tahoe-lafs) Downloading Twisted-16.0.0-cp27-none-win_amd64.whl (3.2MB) 100% |################################| 3.2MB 215kB/s Collecting attrs (from service-identity->tahoe-lafs) Downloading attrs-15.2.0-py2.py3-none-any.whl Collecting cryptography>=1.3 (from pyOpenSSL>=0.14->tahoe-lafs) Downloading cryptography-1.3.1-cp27-none-win_amd64.whl (1.3MB) 100% |################################| 1.3MB 451kB/s Collecting six>=1.5.2 (from pyOpenSSL>=0.14->tahoe-lafs) Downloading six-1.10.0-py2.py3-none-any.whl Collecting pyutil>=1.3.19 (from zfec>=1.1.0->tahoe-lafs) Downloading https://tahoe-lafs.org/deps/pyutil-2.0.0-py2-none-any.whl (180kB) 100% |################################| 184kB 747kB/s Collecting idna>=0.6 (from Twisted[tls]>=15.1.0->tahoe-lafs) Downloading idna-2.1-py2-none-any.whl (54kB) 100% |################################| 61kB 1.8MB/s Collecting enum34 (from cryptography>=1.3->pyOpenSSL>=0.14->tahoe-lafs) Downloading https://tahoe-lafs.org/deps/enum34-1.1.2-cp27-none-any.whl (57kB) 100% |################################| 61kB 516kB/s Collecting ipaddress (from cryptography>=1.3->pyOpenSSL>=0.14->tahoe-lafs) Downloading ipaddress-1.0.16-py27-none-any.whl Collecting cffi>=1.4.1 (from cryptography>=1.3->pyOpenSSL>=0.14->tahoe-lafs) Downloading cffi-1.5.2-cp27-none-win_amd64.whl (150kB) 100% |################################| 153kB 1.5MB/s Collecting zbase32>=1.0 (from pyutil>=1.3.19->zfec>=1.1.0->tahoe-lafs) Downloading https://tahoe-lafs.org/deps/zbase32-1.1.5-py2-none-any.whl Collecting pycparser (from cffi>=1.4.1->cryptography>=1.3->pyOpenSSL>=0.14->tahoe-lafs) Downloading https://tahoe-lafs.org/deps/pycparser-2.14-cp27-none-any.whl (188kB) 100% |################################| 194kB 1.2MB/s Installing collected packages: simplejson, enum34, ipaddress, pyasn1, six, idna, pycparser, cffi, cryptography, pyOpenSSL, pyasn1-modules, attrs, service-identity, zope.interface, Twisted, foolscap, pycrypto, characteristic, zbase32, pyutil, zfec, pycryptopp, Nevow, tahoe-lafs Successfully installed Nevow-0.13.0 Twisted-16.0.0 attrs-15.2.0 cffi-1.5.2 characteristic-14.3.0 cryptography-1.3.1 enum34-1.1.2 foolscap-0.11.0 idna-2.1 ipaddress-1.0.16 pyOpenSSL-16.0.0 pyasn1-0.1.9 pyasn1-modules-0.0.8 pycparser-2.14 pycrypto-2.6.1 pycryptopp-0.7.1.869544967005693312591928092448767568728501330214 pyutil-2.0.0 service-identity-16.0.0 simplejson-3.8.2 six-1.10.0 tahoe-lafs-1.11.0 zbase32-1.1.5 zfec-1.4.24 zope.interface-4.1.3 (tahoe-lafs) C:\Users\crd>tahoe --version Traceback (most recent call last): File "c:\python27\Lib\runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "c:\python27\Lib\runpy.py", line 72, in _run_code exec code in run_globals File "C:\Users\crd\tahoe-lafs\Scripts\tahoe.exe\__main__.py", line 5, in <module> File "c:\users\crd\tahoe-lafs\lib\site-packages\allmydata\__init__.py", line 412, in <module> _vers_and_locs_list, _cross_check_errors = get_package_versions_and_locations() File "c:\users\crd\tahoe-lafs\lib\site-packages\allmydata\__init__.py", line 222, in get_package_versions_and_locations for p in pkg_resources.require(install_requires)]) File "c:\users\crd\tahoe-lafs\lib\site-packages\pkg_resources\__init__.py", line 943, in require needed = self.resolve(parse_requirements(requirements)) File "c:\users\crd\tahoe-lafs\lib\site-packages\pkg_resources\__init__.py", line 830, in resolve raise DistributionNotFound(req, requirers) pkg_resources.DistributionNotFound: The 'pypiwin32' distribution was not found and is required by the application (tahoe-lafs) C:\Users\crd> (tahoe-lafs) C:\Users\crd>pip install --find-links=https://tahoe-lafs.org/deps/ pypiwin32 Collecting pypiwin32 Downloading pypiwin32-219-cp27-none-win_amd64.whl (7.3MB) 100% |################################| 7.3MB 98kB/s Installing collected packages: pypiwin32 Successfully installed pypiwin32-219 (tahoe-lafs) C:\Users\crd>tahoe --version tahoe-lafs: 1.11.0 [master: 04a3e7993f70ac87c208d61f4387a59f5f419367] foolscap: 0.11.0 pycryptopp: 0.7.1.869544967005693312591928092448767568728501330214 zfec: 1.4.24 Twisted: 16.0.0 Nevow: 0.13.0 zope.interface: unknown python: 2.7.11 platform: Windows-10-10.0.10586 pyOpenSSL: 16.0.0 OpenSSL: 1.0.2g [ 1 Mar 2016] simplejson: 3.8.2 pycrypto: 2.6.1 pyasn1: 0.1.9 service-identity: 16.0.0 characteristic: 14.3.0 pyasn1-modules: 0.0.8 cryptography: 1.3.1 cffi: 1.5.2 six: 1.10.0 enum34: 1.1.2 pycparser: 2.14 setuptools: 20.4 pypiwin32: 219 attrs: 15.2.0 [according to pkg_resources] ipaddress: 1.0.16 [according to pkg_resources] idna: 2.1 [according to pkg_resources] ```

tahoe-lafs added the

labels 2016-03-31 13:14:33 +00:00

tahoe-lafs added this to the undecided milestone 2016-03-31 13:14:33 +00:00

daira was assigned by tahoe-lafs

2016-03-31 13:14:33 +00:00

warner commented

2016-03-31 17:48:11 +00:00

Huh. That looks like tahoe didn't express it's need for pypiwin32 the first time it was asked (during the install), but did express it the second time (at runtime).

Oh.. it's because I uploaded tahoe_lafs-1.11.0-py2-none-any.whl , and the wheel was built on a unix box. That wheel doesn't express the dependency: only a source tarball has the conditional "am I on windows? then ask for pypiwin32" code.

I need to delete that wheel.

Huh. That looks like tahoe didn't express it's need for pypiwin32 the first time it was asked (during the install), but did express it the second time (at runtime). Oh.. it's because I uploaded tahoe_lafs-1.11.0-py2-none-any.whl , and the wheel was built on a unix box. That wheel doesn't express the dependency: only a source tarball has the conditional "am I on windows? then ask for pypiwin32" code. I need to delete that wheel.

warner added

p/critical

and removed

p/normal

labels 2016-03-31 17:48:11 +00:00

warner modified the milestone from undecided to soon

2016-03-31 17:48:11 +00:00

daira was unassigned by warner

2016-03-31 17:48:11 +00:00

warner self-assigned this 2016-03-31 17:48:11 +00:00

warner commented

2016-03-31 17:51:50 +00:00

I've deleted the wheel from pypi. Could you try a new installation?

If you see it trying to use that same wheel again, it may be in your HTTP cache (~\AppData\Local\pip\cache). If so, just delete that directory.

I've deleted the wheel from pypi. Could you try a new installation? If you see it trying to use that same wheel again, it may be in your HTTP cache (`~\AppData\Local\pip\cache`). If so, just delete that directory.

warner commented

2016-03-31 23:15:37 +00:00

crd reported on IRC that this fixed the problem.

Changing the focus of this ticket: to be consistent about not building wheels (which is a shame), or find a way to make wheels that won't be used on windows where the dependencies aren't sufficient. Could/should we use a tahoe-lafswindows target?

crd reported on IRC that this fixed the problem. Changing the focus of this ticket: to be consistent about not building wheels (which is a shame), or find a way to make wheels that won't be used on windows where the dependencies aren't sufficient. Could/should we use a `tahoe-lafswindows` target?

warner added

p/major

and removed

p/critical

labels 2016-03-31 23:15:37 +00:00

warner commented

2016-12-24 22:57:33 +00:00

The right fix is to use a platform-specific "extra", with a setup.py syntax like:

      extras_require={
          ':sys_platform=="win32"': ["pypiwin32"],

The right fix is to use a platform-specific "extra", with a `setup.py` syntax like: ``` extras_require={ ':sys_platform=="win32"': ["pypiwin32"], ```

warner modified the milestone from soon to 1.13.0

2016-12-24 22:57:33 +00:00

warner commented

2016-12-24 23:04:54 +00:00

(https://github.com/tahoe-lafs/tahoe-lafs/pull/388)

warner modified the milestone from 1.13.0 to 1.12.1

2016-12-25 18:56:38 +00:00

Brian Warner <warner@lothar.com> commented

2016-12-25 19:12:37 +00:00

Author

Owner

In a06cf2e/trunk:

express pypiwin32 dep via a setup.py extra

Previously this looked at sys.platform to decide what dependencies to
include. The problem with that approach is that wheels built on a unix box
won't work on windows (and vice versa), when the (pure-python) Tahoe wheels
aren't supposed to be platform-dependent.

setup.py provides a syntax to express this properly, so wheels created on
either platform will include the pypiwin32 dependency in the metadata, but
marked as only being relevant when installing on a win32 platform.

closes ticket:2763

In [a06cf2e/trunk](/tahoe-lafs/trac/commit/a06cf2ea4c966e74ad63dca2abd8a58cc2a6e33e): ``` express pypiwin32 dep via a setup.py extra Previously this looked at sys.platform to decide what dependencies to include. The problem with that approach is that wheels built on a unix box won't work on windows (and vice versa), when the (pure-python) Tahoe wheels aren't supposed to be platform-dependent. setup.py provides a syntax to express this properly, so wheels created on either platform will include the pypiwin32 dependency in the metadata, but marked as only being relevant when installing on a win32 platform. closes ticket:2763 ```