tahoe-lafs/trac-2024-07-25

"tox" on debian (py3) can't run our setup.py #2747

New issue

Closed

opened 2016-03-21 18:38:55 +00:00 by warner · 6 comments

warner commented

2016-03-21 18:38:55 +00:00

Owner

rkirshnan reported that running "tox" on current debian/testing fails.

On that platform (as well as some Ubuntu ones I've seen), the system-packaged /usr/bin/tox is shbang'ed to use python3. Tox runs the project's setup.py to make an sdist tarball (which it then installs into the virtualenv). Tahoe's setup.py has a line that explicitly fails when run with python3, to avoid confusion, since Tahoe only works with python-2.7 . When tox hits this, it fails.

To fix this, I think we should take out the hard-fail. It might be good to leave in a warning (e.g. check sys.version_info), but move the version-based hard-fail to early in the entry-point script.

We should do an experiment and try to install tahoe in a py3 virtualenv, and see what happens: I suspect that some of the dependencies (zfec? pycryptopp? foolscap?) will fail to install, so nobody would ever get to the point of even running tahoe. If they made it that far, the next question would be whether importing our dependencies would succeed: that might produce different errors depending upon which subcommand was run (since we sometimes avoid importing things before we really need them).

rkirshnan reported that running "tox" on current debian/testing fails. On that platform (as well as some Ubuntu ones I've seen), the system-packaged `/usr/bin/tox` is shbang'ed to use python3. Tox runs the project's `setup.py` to make an sdist tarball (which it then installs into the virtualenv). Tahoe's `setup.py` has a line that explicitly fails when run with python3, to avoid confusion, since Tahoe only works with python-2.7 . When tox hits this, it fails. To fix this, I think we should take out the hard-fail. It might be good to leave in a warning (e.g. check `sys.version_info`), but move the version-based hard-fail to early in the entry-point script. We should do an experiment and try to install tahoe in a py3 virtualenv, and see what happens: I suspect that some of the dependencies (zfec? pycryptopp? foolscap?) will fail to install, so nobody would ever get to the point of even running `tahoe`. If they made it that far, the next question would be whether importing our dependencies would succeed: that might produce different errors depending upon which subcommand was run (since we sometimes avoid importing things before we really need them).

tahoe-lafs added the

labels 2016-03-21 18:38:55 +00:00

warner commented

2016-03-22 04:55:51 +00:00

Author

Owner

I started working on this, but it's not trivial. We basically need to make our setup.py work correctly under both py2 and py3. The problems:

execfile(../_auto_deps.py)
the code that runs git to compute a version string has bytes-vs-str issues

If I bypass those two issues, tox is able to install an sdist, and then run the tests.

The second issue will be fixed when we move to Versioneer (#2748).

The first needs us to either find a py2/py3-compatible equivalent of execfile(), or (my preference) remove _auto_deps.py and move the install_requires= list into setup.py. That will also entail removing a lot of the code from *init*.py that checks what versions of things are installed and complains if they don't look right (and frequently complains even if they do). #2749 covers this (#2283 is related, but doesn't go far enough).

I'm torn whether to try to address this in the upcoming release (for the sake of developers using system "tox" on recent debian/ubuntu), or to punt until the next release (to avoid the deep *init*.py changes it would require).

I started working on this, but it's not trivial. We basically need to make our `setup.py` work correctly under both py2 and py3. The problems: * `execfile(../_auto_deps.py)` * the code that runs `git` to compute a version string has bytes-vs-str issues If I bypass those two issues, tox is able to install an sdist, and then run the tests. The second issue will be fixed when we move to Versioneer (#2748). The first needs us to either find a py2/py3-compatible equivalent of `execfile()`, or (my preference) remove `_auto_deps.py` and move the `install_requires=` list into `setup.py`. That will also entail removing a lot of the code from `*init*.py` that checks what versions of things are installed and complains if they don't look right (and frequently complains even if they do). #2749 covers this (#2283 is related, but doesn't go far enough). I'm torn whether to try to address this in the upcoming release (for the sake of developers using system "tox" on recent debian/ubuntu), or to punt until the next release (to avoid the deep `*init*.py` changes it would require).

warner commented

2016-03-22 05:03:36 +00:00

Author

Owner

Milestone renamed

tahoe-lafs added this to the 1.11.0 milestone 2016-03-22 05:03:36 +00:00

warner commented

2016-03-22 19:20:31 +00:00

Author

Owner

We could fix this after 1.11.0 ships, under the theory that most people inclined to run unit tests will also be inclined to run from a git checkout.

But if we don't fix this before 1.11.0, most debian/ubuntu users will be unable to run the release's unit tests with our recommended process. They'll have to create a py2-based virtualenv, pip install tox into that, then run the resulting executable. Or, since they're going to make a virtualenv anyways (to run tahoe), they can just do "trial allmydata" from within the virtualenv.

That doesn't sound like a great experience. It doesn't affect people's ability to use tahoe, just their ability to run the tests in a standard way.

I surveyed a couple of releases. All current versions of debian ("sid", "testing", and even the latest release "jesse") use a python3-based tox. On ubuntu, trusty (14.04LTS) uses py2, but vivid (15.04) and beyond use py3. The next Ubuntu LTS (16.04 Xenial) is coming out in a month, and will use py3-based tox too. We could fix this after 1.11.0 ships, under the theory that most people inclined to run unit tests will also be inclined to run from a git checkout. But if we don't fix this before 1.11.0, most debian/ubuntu users will be unable to run the release's unit tests with our recommended process. They'll have to create a py2-based virtualenv, `pip install tox` into that, then run the resulting executable. Or, since they're going to make a virtualenv anyways (to run tahoe), they can just do "trial allmydata" from within the virtualenv. That doesn't sound like a great experience. It doesn't affect people's ability to *use* tahoe, just their ability to run the tests in a standard way.

warner commented

2016-03-22 19:32:28 +00:00

Author

Owner

Oh, I think we can do this. I figured out a 2+3 cross-compatible execfile() replacement to avoid changing the way we use _auto_deps.py for now. And I think I can 2+3-ify the version string code too (I've got a py3-compatible flavor now, but doing 2+3 shouldn't be too hard).

Oh, I think we can do this. I figured out a 2+3 cross-compatible `execfile()` replacement to avoid changing the way we use `_auto_deps.py` for now. And I think I can 2+3-ify the version string code too (I've got a py3-compatible flavor now, but doing 2+3 shouldn't be too hard).

Brian Warner <warner@lothar.com> commented

2016-03-23 05:36:06 +00:00

Author

Owner

In dd84abd/trunk:

setup.py: work with tox under py3

This allows a python3-based "tox" (as shipped with modern debian and
ubuntu systems) to run setup.py egg_info, update_version, and sdist
commands. It moves the main "tahoe requires py2" check out of setup.py
and into allmydata.scripts.runner.run, where it gets applied at runtime
rather than build time.

It also changes the execfile(_auto_deps.py) and Versioneer-like "ask git
what our version string should be" code to work under both py2 and py3.

fixes ticket:2747

In [dd84abd/trunk](/tahoe-lafs/trac-2024-07-25/commit/dd84abd9f22db56cdce610cd882ab8337041cf4a): ``` setup.py: work with tox under py3 This allows a python3-based "tox" (as shipped with modern debian and ubuntu systems) to run setup.py egg_info, update_version, and sdist commands. It moves the main "tahoe requires py2" check out of setup.py and into allmydata.scripts.runner.run, where it gets applied at runtime rather than build time. It also changes the execfile(_auto_deps.py) and Versioneer-like "ask git what our version string should be" code to work under both py2 and py3. fixes ticket:2747 ```

tahoe-lafs added the

fixed

label 2016-03-23 05:36:06 +00:00

Brian Warner <warner@lothar.com> closed this issue

2016-03-23 05:36:06 +00:00

vu3rdd commented

2016-03-23 09:07:22 +00:00

Author

Owner

Very nice! Thanks. I can confirm that tox indeed works fine on Debian testing now on the latest master branch.