tahoe-lafs/trac-2024-07-25

updating setuptools with setup_requires causes build failure #2910

New issue

Closed

opened 2018-03-27 18:56:06 +00:00 by warner · 2 comments

warner commented

2018-03-27 18:56:06 +00:00

Owner

That caused a second set of build failures: when tox builds the sdist (before installing it into the new virtualenv), it does so with the sam e python that Tox itself uses (and using whatever version of setuptools was available to that python). Then it sees the setup_requires= in tahoe's setup.py and attempts to install it. Something doesn't get reloaded properly, because we wind up with an error that looks like:

tox -- --reporter=timing allmydata
 in dir /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build (timeout 1200 secs)
 watching logfiles {'test.log': '_trial_temp/test.log'}
 argv: ['tox', '--', '--reporter=timing', 'allmydata']
 environment:
  HOME=/var/lib/buildslave
  LOGNAME=buildbot
  PATH=/var/lib/buildslave/.local/bin:/usr/local/bin:/sbin:/usr/sbin:/bin:/usr/bin
  PWD=/private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build
  SHELL=/bin/bash
  SHLVL=1
  TMPDIR=/var/folders/ql/2vmcxkls1tl1w71h9_1wvkch000035/T/
  USER=buildbot
  VERSIONER_PYTHON_PREFER_32_BIT=no
  VERSIONER_PYTHON_VERSION=2.7
  XPC_FLAGS=0x0
  XPC_SERVICE_NAME=0
  _=/var/lib/buildslave/.local/bin/buildslave
  __CF_USER_TEXT_ENCODING=0x65:0:0
 using PTY: False
GLOB sdist-make: /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/setup.py
ERROR: invocation failed (exit code 1), logfile: /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/.tox/log/tox-0.log
ERROR: actionid: tox
msg: packaging
cmdargs: ['/usr/bin/python', local('/private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/.tox/dist')]
env: None

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'long_description_content_type'
  warnings.warn(msg)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'project_urls'
  warnings.warn(msg)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)

Installed /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/.eggs/setuptools-39.0.1-py2.7.egg
Traceback (most recent call last):
  File "setup.py", line 291, in <module>
    **setup_args
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/dist.py", line 272, in __init__
    _Distribution.__init__(self,attrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 287, in __init__
    self.finalize_options()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/dist.py", line 327, in finalize_options
    ep.load()(self, ep.name, value)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 2370, in load
    return self.resolve()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 2380, in resolve
    raise ImportError(str(exc))
ImportError: 'module' object has no attribute 'check_specifier'

ERROR: FAIL could not package project - v = InvocationError('/usr/bin/python /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/setup.py sdist --formats=zip --dist-dir /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/.tox/dist (see /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/.tox/log/tox-0.log)', 1)

This seems related to https://github.com/tox-dev/tox/issues/334

We think we can fix this with a workaround described in that ticket: add skipsdist = True to the tox section of tox.ini. Our testenv section already has skip_install = True because we (ab)use the deps feature to get editable installs (which are faster than building an sdist) with the "test" feature enabled (which gets us coverage and mock and stuff).

[testenv]
skip_install = True
deps = --editable=.[test]

Since we're already not using the default install step, I don't think we need that sdist tarball either. I'm hoping that the pip install used for the deps will use the new virtualenv's pip, which means it can upgrade setuptools (according to our setup.py's setup_requires=) without triggering this bug.

We had one set of build failures: the LGTM static-analysis tool was using the lowest-acceptable version of all dependencies, including setuptools-11.3, which was too old to understand the `python_requires="<3.0"` clause that tells the world how ashamed we are to not support py3. We fixed this by adding both an `install_requires=` and `setup_requires=` of `>=28.8.0`. That caused a second set of build failures: when tox builds the sdist (before installing it into the new virtualenv), it does so with the sam e python that Tox itself uses (and using whatever version of setuptools was available to that python). Then it sees the `setup_requires=` in tahoe's setup.py and attempts to install it. Something doesn't get reloaded properly, because we wind up with an error that looks like: ``` tox -- --reporter=timing allmydata in dir /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build (timeout 1200 secs) watching logfiles {'test.log': '_trial_temp/test.log'} argv: ['tox', '--', '--reporter=timing', 'allmydata'] environment: HOME=/var/lib/buildslave LOGNAME=buildbot PATH=/var/lib/buildslave/.local/bin:/usr/local/bin:/sbin:/usr/sbin:/bin:/usr/bin PWD=/private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build SHELL=/bin/bash SHLVL=1 TMPDIR=/var/folders/ql/2vmcxkls1tl1w71h9_1wvkch000035/T/ USER=buildbot VERSIONER_PYTHON_PREFER_32_BIT=no VERSIONER_PYTHON_VERSION=2.7 XPC_FLAGS=0x0 XPC_SERVICE_NAME=0 _=/var/lib/buildslave/.local/bin/buildslave __CF_USER_TEXT_ENCODING=0x65:0:0 using PTY: False GLOB sdist-make: /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/setup.py ERROR: invocation failed (exit code 1), logfile: /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/.tox/log/tox-0.log ERROR: actionid: tox msg: packaging cmdargs: ['/usr/bin/python', local('/private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/.tox/dist')] env: None /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'long_description_content_type' warnings.warn(msg) /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'project_urls' warnings.warn(msg) /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires' warnings.warn(msg) Installed /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/.eggs/setuptools-39.0.1-py2.7.egg Traceback (most recent call last): File "setup.py", line 291, in <module> **setup_args File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 111, in setup _setup_distribution = dist = klass(attrs) File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/dist.py", line 272, in __init__ _Distribution.__init__(self,attrs) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 287, in __init__ self.finalize_options() File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/setuptools/dist.py", line 327, in finalize_options ep.load()(self, ep.name, value) File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 2370, in load return self.resolve() File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 2380, in resolve raise ImportError(str(exc)) ImportError: 'module' object has no attribute 'check_specifier' ERROR: FAIL could not package project - v = InvocationError('/usr/bin/python /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/setup.py sdist --formats=zip --dist-dir /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/.tox/dist (see /private/var/lib/buildslave/bb-tahoe/OS-X_10_13/build/.tox/log/tox-0.log)', 1) ``` This seems related to <https://github.com/tox-dev/tox/issues/334> We think we can fix this with a workaround described in that ticket: add `skipsdist = True` to the `tox` section of `tox.ini`. Our `testenv` section already has `skip_install = True` because we (ab)use the `deps` feature to get editable installs (which are faster than building an sdist) with the "test" feature enabled (which gets us `coverage` and `mock` and stuff). ``` [testenv] skip_install = True deps = --editable=.[test] ``` Since we're already not using the default install step, I don't think we need that sdist tarball either. I'm hoping that the `pip install` used for the deps will use the new virtualenv's `pip`, which means it can upgrade setuptools (according to our `setup.py`'s `setup_requires=`) without triggering this bug.

tahoe-lafs added the

labels 2018-03-27 18:56:06 +00:00

tahoe-lafs added this to the soon milestone 2018-03-27 18:56:06 +00:00

Brian Warner <warner@lothar.com> commented

2018-03-27 21:15:21 +00:00

Author

Owner

In 526b97c/trunk:

tox: add 'skipsdist=True', hoping this will fix buildbot

There appears to be a bug in setuptools, triggered by running "python
setup.py sdist" with setuptools==11.3 in that python's environment, on a
project whose setup.py has a setup_requires= that requests setuptools >=
28.8.0. When setuptools is upgraded from inside setup.py, it gets into a
weird hybrid state where it's using setup() keyword-argument plugins from the
newer setuptools, but those plugins reference functions that aren't present
in the older setuptools, and the sdist command fails with an import
error (module object has no attribute 'check_specifier').

We don't actually need the sdist: all our tox test environments use
"skip_install = true", because we install tahoe via the "deps" line (so we
can get the `[test]` extra, and get a faster symlink-ish "editable" install).
That install uses "pip", which uses the pip inside the new virtualenv, which
either uses a newer version of setuptools (dependent upon what version of
"virtualenv" was installed in the parent environment, next to tox) or somehow
allows setuptools to be upgraded without exposing this weird broken hybrid
state.

Either way, skipping the sdist seems to fix this problem.

refs ticket:2910
https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2910

In [526b97c/trunk](/tahoe-lafs/trac-2024-07-25/commit/526b97c7531e12e379f0c577bc15a07b5c2d6446): ``` tox: add 'skipsdist=True', hoping this will fix buildbot There appears to be a bug in setuptools, triggered by running "python setup.py sdist" with setuptools==11.3 in that python's environment, on a project whose setup.py has a setup_requires= that requests setuptools >= 28.8.0. When setuptools is upgraded from inside setup.py, it gets into a weird hybrid state where it's using setup() keyword-argument plugins from the newer setuptools, but those plugins reference functions that aren't present in the older setuptools, and the sdist command fails with an import error (module object has no attribute 'check_specifier'). We don't actually need the sdist: all our tox test environments use "skip_install = true", because we install tahoe via the "deps" line (so we can get the `[test]` extra, and get a faster symlink-ish "editable" install). That install uses "pip", which uses the pip inside the new virtualenv, which either uses a newer version of setuptools (dependent upon what version of "virtualenv" was installed in the parent environment, next to tox) or somehow allows setuptools to be upgraded without exposing this weird broken hybrid state. Either way, skipping the sdist seems to fix this problem. refs ticket:2910 https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2910 ```

warner commented

2018-03-27 21:41:52 +00:00

Author

Owner

Yup, that seems to have fixed the problem.

I also had to change our Makefile to delegate tarball generation to tox, and add a tox environment to run python setup.py sdist as the Makefile used to do. This gives us a newer/updatable python/setuptools with which to run the sdist command. Without that change, the make tarballs target was failing with the same check_specifier exception from above.

Yup, that seems to have fixed the problem. I also had to change our Makefile to delegate tarball generation to tox, and add a tox environment to run `python setup.py sdist` as the Makefile used to do. This gives us a newer/updatable python/setuptools with which to run the sdist command. Without that change, the `make tarballs` target was failing with the same `check_specifier` exception from above.

tahoe-lafs added the

fixed

label 2018-03-27 21:41:52 +00:00

tahoe-lafs modified the milestone from soon to 1.13.0

2018-03-27 21:41:52 +00:00

warner closed this issue

2018-03-27 21:41:52 +00:00