tox won't run on ubuntu xenial: python3 #2876

New issue

Closed

opened 2017-06-05 10:42:10 +00:00 by warner · 16 comments

warner commented

2017-06-05 10:42:10 +00:00

We recently landed a change to our setup.py, to bail with a clear error message if you try to run it under python3 (since tahoe only supports python2). Unfortunately, the tox provided in Ubuntu-16.04 "xenial" (the most recent LTS release) is a python3 executable, and when it runs setup.py to build a .zip package, it fails:

$ tox -e py27
GLOB sdist-make: /home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/setup.py
ERROR: invocation failed (exit code 1), logfile: /home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/.tox/log/tox-0.log
ERROR: actionid: tox
msg: packaging
cmdargs: ['/usr/bin/python3', local('/home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/.tox/dist')]
env: None

Traceback (most recent call last):
  File "setup.py", line 32, in <module>
    raise RuntimeError("Python version 2 is required")
RuntimeError: Python version 2 is required

ERROR: FAIL could not package project - v = InvocationError('/usr/bin/python3 /home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/setup.py sdist --formats=zip --dist-dir /home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/.tox/dist (see /home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/.tox/log/tox-0.log)', 1)

As a result, our xenial buildbot is failing, and folks using xenial can't test tahoe.

I'm a bit surprised that tox is using it's native python3 to build a package, even though the -e py27 should cause it to use python2 to actually run the tests. Is there some way to tell tox which python to use for packaging purposes?

We recently landed a change to our `setup.py`, to bail with a clear error message if you try to run it under python3 (since tahoe only supports python2). Unfortunately, the `tox` provided in Ubuntu-16.04 "xenial" (the most recent LTS release) is a python3 executable, and when it runs `setup.py` to build a .zip package, it fails: ``` $ tox -e py27 GLOB sdist-make: /home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/setup.py ERROR: invocation failed (exit code 1), logfile: /home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/.tox/log/tox-0.log ERROR: actionid: tox msg: packaging cmdargs: ['/usr/bin/python3', local('/home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/.tox/dist')] env: None Traceback (most recent call last): File "setup.py", line 32, in <module> raise RuntimeError("Python version 2 is required") RuntimeError: Python version 2 is required ERROR: FAIL could not package project - v = InvocationError('/usr/bin/python3 /home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/setup.py sdist --formats=zip --dist-dir /home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/.tox/dist (see /home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/.tox/log/tox-0.log)', 1) ``` As a result, our xenial buildbot is failing, and folks using xenial can't test tahoe. I'm a bit surprised that tox is using it's native python3 to build a package, even though the `-e py27` should cause it to use python2 to actually run the tests. Is there some way to tell tox which python to use for packaging purposes?

warner added the

labels 2017-06-05 10:42:10 +00:00

warner added this to the 1.13.0 milestone 2017-06-05 10:42:10 +00:00

daira was assigned by warner

2017-06-05 10:42:10 +00:00

warner commented

2017-06-05 10:56:23 +00:00

Author

Looking at the tox source code, in tox/session.py (around line 412, in Session._makesdist()), there's a hard-coded use of sys.executable to run the sdist command:

action.popen([sys.executable, setup, "sdist", "--formats=zip",
              "--dist-dir", self.config.distdir, ],
             cwd=self.config.setupdir)

which suggests to me that a py3-based tox just can't run a py2-requiring setup.py.

I suppose we could change our setup.py check to tolerate an sdist command, but reject all others. Or we could soften it to a warning.

I think we ought to allow our published "just run tox" test procedure to work on the current ubuntu LTS (or any other distribution that has python2.7 available, even if tox or the system default python are py3).

Looking at the `tox` source code, in `tox/session.py` (around line 412, in `Session._makesdist()`), there's a hard-coded use of `sys.executable` to run the sdist command: ``` action.popen([sys.executable, setup, "sdist", "--formats=zip", "--dist-dir", self.config.distdir, ], cwd=self.config.setupdir) ``` which suggests to me that a py3-based tox just can't run a py2-requiring setup.py. I suppose we could change our setup.py check to tolerate an `sdist` command, but reject all others. Or we could soften it to a warning. I think we ought to allow our published "just run `tox`" test procedure to work on the current ubuntu LTS (or any other distribution that has python2.7 available, even if `tox` or the system default `python` are py3).

warner commented

2017-06-05 10:57:02 +00:00

Author

BTW setting basepython=python2.7 in tox.ini didn't fix it.

BTW setting `basepython=python2.7` in `tox.ini` didn't fix it.

warner commented

2017-06-05 11:04:09 +00:00

Author

Also, it looks like travis didn't catch this when it was a pull request because travis runs everything in a py2 virtualenv, whereas our xenial buildbot tries harder to mimic a real user following our build instructions (using /usr/bin/tox).

Also, it looks like travis didn't catch this when it was a [pull request](https://github.com/tahoe-lafs/tahoe-lafs/pull/414) because travis runs everything in a py2 virtualenv, whereas our xenial buildbot tries harder to mimic a real user following our build instructions (using `/usr/bin/tox`).

warner commented

2017-06-05 11:11:01 +00:00

Author

Many of our buildbot boxes appear to be using native tox. But several are using a buildbot-specific installation: Centos7, Fedora2.4, and !Debian/Jesse all have tool-versions logs that show something like ~/.local/bin/tox being used (quite possibly installed specifically to overcome problems like this one). So those builders might be masking problems that would affect users who follow our instructions and just run tox.

Many of our buildbot boxes appear to be using native `tox`. But several are using a buildbot-specific installation: Centos7, Fedora2.4, and !Debian/Jesse all have `tool-versions` logs that show something like `~/.local/bin/tox` being used (quite possibly installed specifically to overcome problems like this one). So those builders might be masking problems that would affect users who follow our instructions and just run `tox`.

warner commented

2017-07-04 19:15:51 +00:00

Author

Another approach would be to change the unconditional not-py3 check to instead only run in the "build" subcommand, or "build_py". Would that accomplish the same goals as the original issue? We're looking to raise an error as early as possible, for folks who mistakenly use py3 to try to install Tahoe. So we need to make sure that pip3 install tahoe-lafs fails in a good way.

warner commented

2017-07-04 19:34:28 +00:00

Author

Ah, part of the issue is py3-incompatible syntax in tahoe's setup.py itself, or (worse) in those of the dependencies. When I do a pip install . inside a py3 virtualenv (so pip is really pip3), the first exception I get is a syntax error in zfec's setup.py, which contains an old-style print "something" statement.

Tahoe's setup.py is first called as setup.py egg-info, then if this succeeds, pip(3) starts doing the same for its dependencies. To deliver a useful error message early, we need egg-info to throw.

When tox runs, the first thing it does is to run sdist. Only later, inside the new (py2) virtualenv, does it run egg_info. So maybe changing the egg_info command to check the version, but making sure that sdist is py3-tolerant, would allow tox3 to work and still retain the useful error when pip3 is used by mistake.

Ah, part of the issue is py3-incompatible syntax in tahoe's setup.py itself, or (worse) in those of the dependencies. When I do a `pip install .` inside a py3 virtualenv (so `pip` is really `pip3`), the first exception I get is a syntax error in zfec's setup.py, which contains an old-style `print "something"` statement. Tahoe's setup.py is first called as `setup.py egg-info`, then if this succeeds, pip(3) starts doing the same for its dependencies. To deliver a useful error message early, we need `egg-info` to throw. When `tox` runs, the first thing it does is to run `sdist`. Only later, inside the new (py2) virtualenv, does it run `egg_info`. So maybe changing the `egg_info` command to check the version, but making sure that `sdist` *is* py3-tolerant, would allow tox3 to work and still retain the useful error when pip3 is used by mistake.

warner commented

2017-07-04 19:38:55 +00:00

Author

But.. the sdist command invokes egg-info internally. Rats.

But.. the `sdist` command invokes `egg-info` internally. Rats.

warner commented

2017-07-04 19:57:13 +00:00

Author

Ah, so I think the simplest solution is to have our setup.py look at sys.argv, and apply the py2-only check iff setup.py was invoked as setup.py egg_info or setup.py install. A pip install will do the former, and users may do the latter directly.

PR in https://github.com/tahoe-lafs/tahoe-lafs/pull/422

Ah, so I think the simplest solution is to have our setup.py look at `sys.argv`, and apply the py2-only check iff setup.py was invoked as `setup.py egg_info` or `setup.py install`. A pip install will do the former, and users may do the latter directly. PR in <https://github.com/tahoe-lafs/tahoe-lafs/pull/422>

warner commented

2017-07-12 15:43:38 +00:00

Author

At yesterday's devchat, exarkun pointed out that this is really a Tox bug (it ought to use the virtualenv python for the sdist step, not whatever python tox itself is using). So:

we should file a Tox bug about it
they might WONTFIX it, because maybe there's a good reason for Tox working the way it does
if so, we should get them to commit to exactly which setup.py subcommands might be run by the tox-python, and land a tahoe patch that enables py3 for just those verbs and no others

Meanwhile, we can monkeypatch something to let our CI continue to work until that Tox bug gets fixed. And we should probably find a way to make a zfec release with a py3-tolerant setup.py (which means changing the print "foo" into a print("foo"))

At yesterday's devchat, exarkun pointed out that this is really a Tox bug (it ought to use the virtualenv python for the sdist step, not whatever python tox itself is using). So: * we should file a Tox bug about it * they might WONTFIX it, because maybe there's a good reason for Tox working the way it does * if so, we should get them to commit to exactly which setup.py subcommands might be run by the tox-python, and land a tahoe patch that enables py3 for just those verbs and no others Meanwhile, we can monkeypatch something to let our CI continue to work until that Tox bug gets fixed. And we should probably find a way to make a zfec release with a py3-tolerant setup.py (which means changing the `print "foo"` into a `print("foo")`)

warner commented

2017-07-12 18:59:11 +00:00

Author

The upstream Tox bug is https://github.com/tox-dev/tox/issues/507 , and I've added a note about how it affects us.

The upstream Tox bug is <https://github.com/tox-dev/tox/issues/507> , and I've added a note about how it affects us.

warner commented

2017-07-13 00:01:48 +00:00

Author

Note: xenial (and other platforms) are unlikely to backport a newer Tox, so even if/when this gets fixed upstream, we'll still have the problem that Tahoe can't run tests on current py3-based systems. So I'm going to argue for landing PR 422 (or something similar) anyways.

warner commented

2017-07-25 17:57:25 +00:00

Author

A quick workaround is to create a py2 virtualenv, activate it, and install tox:

virtualenv ve
source ve/bin/activate
pip install tox
tox

A quick workaround is to create a py2 virtualenv, activate it, and install tox: ``` virtualenv ve source ve/bin/activate pip install tox tox ```

warner commented

2017-07-25 18:11:18 +00:00

Author

The narrow-focussed monkeypatch could be:

have travis install a py2 virtualenv and run tox from that (this doesn't help folks at home, though)
have setup.py somehow detect that it's running under tox, and not enforce the py3 check if that's the case

The narrow-focussed monkeypatch could be: * have travis install a py2 virtualenv and run tox from that (this doesn't help folks at home, though) * have setup.py somehow detect that it's running under tox, and not enforce the py3 check if that's the case

warner commented

2017-08-01 16:49:28 +00:00

Author

Some other ideas that exarkun had in today's devchat:

add a tox.py script to the tahoe source tree, which would:
check that it's running under py2 (and explain the situation if not)
find the tox entrypoint (by asking pkg_resources, probably, or maybe just hard-coding the current from tox import cmdline)
invoke the entrypoint
or find some module that tox imports and add a .py file of the same name to tahoe's top-level directory
as in, "that'll teach them to not use absolute imports"
warner was.. not fond of this one

If we did the tox.py thing, we could put a note in the "how to run tests" doc that said "on many systems you can just run tox, but if you see this complain about py3, then you'll need to run python2 tox.py instead". The error message that our setup.py raises (when run under py3) would be a good place for these instructions too.

Some other ideas that exarkun had in today's devchat: * add a `tox.py` script to the tahoe source tree, which would: * check that it's running under py2 (and explain the situation if not) * find the `tox` entrypoint (by asking `pkg_resources`, probably, or maybe just hard-coding the current `from tox import cmdline`) * invoke the entrypoint * or find some module that tox imports and add a `.py` file of the same name to tahoe's top-level directory * as in, "that'll teach them to not use absolute imports" * warner was.. not fond of this one If we did the `tox.py` thing, we could put a note in the "how to run tests" doc that said "on many systems you can just run `tox`, but if you see this complain about py3, then you'll need to run `python2 tox.py` instead". The error message that our setup.py raises (when run under py3) would be a good place for these instructions too.

warner commented

2017-08-15 21:20:13 +00:00

Author

I saw a presentation at PyBay this last weekend, by one of the Jupyter developers, about their transition from py2+py3 to py3-only. He encouraged everyone to add a python_requires= argument to their setup() call, which adds metadata that specifies which versions of python the package/distribution is compatible with. For their needs, they made sure to publish a version that claimed both 2 and 3 before their switchover. Later, when a newer version was marked as py3-only, py2-based sites which had installed the old version will correctly refuse to upgrade to the newer one. He mentioned that older versions of pip and setuptools do not pay attention to the python_requires= field, and they had some workaround and monitoring to keep track of how many installs were affected.

So another approach for us would be to mark our setup() with python_requires=, then remove the manual py3-crowbar from our setup.py. I'll do some testing: my hope is that a pip install will see the metadata and give a useful error (and pip will own that error, not us), but when tox3 does setup.py wheel, it won't complain.

If that works, I'll cancel PR-422 and replace it with one that uses python_requires=.

I saw a presentation at PyBay this last weekend, by one of the Jupyter developers, about their transition from py2+py3 to py3-only. He encouraged everyone to add a `python_requires=` argument to their `setup()` call, which adds metadata that specifies which versions of python the package/distribution is compatible with. For their needs, they made sure to publish a version that claimed both 2 and 3 before their switchover. Later, when a newer version was marked as py3-only, py2-based sites which had installed the old version will correctly refuse to upgrade to the newer one. He mentioned that older versions of pip and setuptools do not pay attention to the `python_requires=` field, and they had some workaround and monitoring to keep track of how many installs were affected. So another approach for us would be to mark our `setup()` with `python_requires=`, then remove the manual py3-crowbar from our setup.py. I'll do some testing: my hope is that a `pip install` will see the metadata and give a useful error (and pip will own that error, not us), but when tox3 does `setup.py wheel`, it won't complain. If that works, I'll cancel PR-422 and replace it with one that uses `python_requires=`.

Brian Warner <warner@lothar.com> commented

2017-08-16 06:07:48 +00:00

Owner

In 04fc0e4/trunk:

setup.py: use python_requires= to complain about py3, not an exception

This allows a python3-based tox (as is common on modern debian/ubuntu
systems) to test our py2-only package. The first thing Tox does is to build a
wheel to install into the target virtualenv (which is a py2-based venv, for
tahoe). But Tox bug (https://github.com/tox-dev/tox/issues/507) in which this
wheel is built with the same python that Tox is using, instead of the python
from the target environment. Our setup.py would see sys.version_info with py3
and launch a crowbar into the works.

With python_requires=, pip is smart enough to know that it's ok to build
wheels with the wrong python, but "pip install" still throws a sensible error
message:

(ve36) ~/stuff/tahoe/tahoe$ pip install .
Processing /home/warner/stuff/tahoe/tahoe
tahoe-lafs requires Python '<3.0' but the running Python is 3.6.1


Closes ticket:2876

In [04fc0e4/trunk](/tahoe-lafs/trac/commit/04fc0e43f721acb00b0af7544a1a44340acd93ae): ``` setup.py: use python_requires= to complain about py3, not an exception This allows a python3-based tox (as is common on modern debian/ubuntu systems) to test our py2-only package. The first thing Tox does is to build a wheel to install into the target virtualenv (which is a py2-based venv, for tahoe). But Tox bug (https://github.com/tox-dev/tox/issues/507) in which this wheel is built with the same python that Tox is using, instead of the python from the target environment. Our setup.py would see sys.version_info with py3 and launch a crowbar into the works. With python_requires=, pip is smart enough to know that it's ok to build wheels with the wrong python, but "pip install" still throws a sensible error message: ``` (ve36) ~/stuff/tahoe/tahoe$ pip install . Processing /home/warner/stuff/tahoe/tahoe tahoe-lafs requires Python '<3.0' but the running Python is 3.6.1 ``` Closes ticket:2876 ```