tox won't run on ubuntu xenial: python3

[warner]

We recently landed a change to our setup.py, to bail with a clear error message if you try to run it under python3 (since tahoe only supports python2). Unfortunately, the tox provided in Ubuntu-16.04 "xenial" (the most recent LTS release) is a python3 executable, and when it runs setup.py to build a .zip package, it fails:

$ tox -e py27
GLOB sdist-make: /home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/setup.py
ERROR: invocation failed (exit code 1), logfile: /home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/.tox/log/tox-0.log
ERROR: actionid: tox
msg: packaging
cmdargs: ['/usr/bin/python3', local('/home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/.tox/dist')]
env: None

Traceback (most recent call last):
  File "setup.py", line 32, in <module>
    raise RuntimeError("Python version 2 is required")
RuntimeError: Python version 2 is required

ERROR: FAIL could not package project - v = InvocationError('/usr/bin/python3 /home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/setup.py sdist --formats=zip --dist-dir /home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/.tox/dist (see /home/bb-tahoe/bb-tahoe/Ubuntu_xenial_16_04/build/.tox/log/tox-0.log)', 1)

As a result, our xenial buildbot is failing, and folks using xenial can't test tahoe.

I'm a bit surprised that tox is using it's native python3 to build a package, even though the -e py27 should cause it to use python2 to actually run the tests. Is there some way to tell tox which python to use for packaging purposes?

[warner]

Looking at the tox source code, in tox/session.py (around line 412, in Session._makesdist()), there's a hard-coded use of sys.executable to run the sdist command:

action.popen([sys.executable, setup, "sdist", "--formats=zip",
              "--dist-dir", self.config.distdir, ],
             cwd=self.config.setupdir)

which suggests to me that a py3-based tox just can't run a py2-requiring setup.py.

I suppose we could change our setup.py check to tolerate an sdist command, but reject all others. Or we could soften it to a warning.

I think we ought to allow our published "just run tox" test procedure to work on the current ubuntu LTS (or any other distribution that has python2.7 available, even if tox or the system default python are py3).

[warner]

BTW setting basepython=python2.7 in tox.ini didn't fix it.

[warner]

Also, it looks like travis didn't catch this when it was a ​pull request because travis runs everything in a py2 virtualenv, whereas our xenial buildbot tries harder to mimic a real user following our build instructions (using /usr/bin/tox).

[warner]

Many of our buildbot boxes appear to be using native tox. But several are using a buildbot-specific installation: Centos7, Fedora2.4, and Debian/Jesse all have tool-versions logs that show something like ~/.local/bin/tox being used (quite possibly installed specifically to overcome problems like this one). So those builders might be masking problems that would affect users who follow our instructions and just run tox.

[warner]

Another approach would be to change the unconditional not-py3 check to instead only run in the "build" subcommand, or "build_py". Would that accomplish the same goals as the original issue? We're looking to raise an error as early as possible, for folks who mistakenly use py3 to try to install Tahoe. So we need to make sure that pip3 install tahoe-lafs fails in a good way.

[warner]

Ah, part of the issue is py3-incompatible syntax in tahoe's setup.py itself, or (worse) in those of the dependencies. When I do a pip install . inside a py3 virtualenv (so pip is really pip3), the first exception I get is a syntax error in zfec's setup.py, which contains an old-style print "something" statement.

Tahoe's setup.py is first called as setup.py egg-info, then if this succeeds, pip(3) starts doing the same for its dependencies. To deliver a useful error message early, we need egg-info to throw.

When tox runs, the first thing it does is to run sdist. Only later, inside the new (py2) virtualenv, does it run egg_info. So maybe changing the egg_info command to check the version, but making sure that sdist *is* py3-tolerant, would allow tox3 to work and still retain the useful error when pip3 is used by mistake.

[warner]

But.. the sdist command invokes egg-info internally. Rats.

[warner]

Ah, so I think the simplest solution is to have our setup.py look at sys.argv, and apply the py2-only check iff setup.py was invoked as setup.py egg_info or setup.py install. A pip install will do the former, and users may do the latter directly.

PR in ​https://github.com/tahoe-lafs/tahoe-lafs/pull/422

[warner]

At yesterday's devchat, exarkun pointed out that this is really a Tox bug (it ought to use the virtualenv python for the sdist step, not whatever python tox itself is using). So:

• we should file a Tox bug about it
• they might WONTFIX it, because maybe there's a good reason for Tox working the way it does
• if so, we should get them to commit to exactly which setup.py subcommands might be run by the tox-python, and land a tahoe patch that enables py3 for just those verbs and no others

Meanwhile, we can monkeypatch something to let our CI continue to work until that Tox bug gets fixed. And we should probably find a way to make a zfec release with a py3-tolerant setup.py (which means changing the print "foo" into a print("foo"))

[warner]

The upstream Tox bug is ​https://github.com/tox-dev/tox/issues/507 , and I've added a note about how it affects us.

[warner]

Note: xenial (and other platforms) are unlikely to backport a newer Tox, so even if/when this gets fixed upstream, we'll still have the problem that Tahoe can't run tests on current py3-based systems. So I'm going to argue for landing PR 422 (or something similar) anyways.

[warner]

A quick workaround is to create a py2 virtualenv, activate it, and install tox:

virtualenv ve
source ve/bin/activate
pip install tox
tox

[warner]

The narrow-focussed monkeypatch could be:

• have travis install a py2 virtualenv and run tox from that (this doesn't help folks at home, though)
• have setup.py somehow detect that it's running under tox, and not enforce the py3 check if that's the case

[warner]

Some other ideas that exarkun had in today's devchat:

• add a tox.py script to the tahoe source tree, which would:
  • check that it's running under py2 (and explain the situation if not)
  • find the tox entrypoint (by asking pkg_resources, probably, or maybe just hard-coding the current from tox import cmdline)
  • invoke the entrypoint
• or find some module that tox imports and add a .py file of the same name to tahoe's top-level directory
  • as in, "that'll teach them to not use absolute imports"
  • warner was.. not fond of this one

If we did the tox.py thing, we could put a note in the "how to run tests" doc that said "on many systems you can just run tox, but if you see this complain about py3, then you'll need to run python2 tox.py instead". The error message that our setup.py raises (when run under py3) would be a good place for these instructions too.

[warner]

I saw a presentation at PyBay this last weekend, by one of the Jupyter developers, about their transition from py2+py3 to py3-only. He encouraged everyone to add a python_requires= argument to their setup() call, which adds metadata that specifies which versions of python the package/distribution is compatible with. For their needs, they made sure to publish a version that claimed both 2 and 3 before their switchover. Later, when a newer version was marked as py3-only, py2-based sites which had installed the old version will correctly refuse to upgrade to the newer one. He mentioned that older versions of pip and setuptools do not pay attention to the python_requires= field, and they had some workaround and monitoring to keep track of how many installs were affected.

So another approach for us would be to mark our setup() with python_requires=, then remove the manual py3-crowbar from our setup.py. I'll do some testing: my hope is that a pip install will see the metadata and give a useful error (and pip will own that error, not us), but when tox3 does setup.py wheel, it won't complain.

If that works, I'll cancel PR-422 and replace it with one that uses python_requires=.

[Brian Warner <warner@…>]

In 04fc0e4/trunk:

setup.py: use python_requires= to complain about py3, not an exception

This allows a python3-based tox (as is common on modern debian/ubuntu
systems) to test our py2-only package. The first thing Tox does is to build a
wheel to install into the target virtualenv (which is a py2-based venv, for
tahoe). But Tox bug (​https://github.com/tox-dev/tox/issues/507) in which this
wheel is built with the same python that Tox is using, instead of the python
from the target environment. Our setup.py would see sys.version_info with py3
and launch a crowbar into the works.

With python_requires=, pip is smart enough to know that it's ok to build
wheels with the wrong python, but "pip install" still throws a sensible error
message:

`
(ve36) ~/stuff/tahoe/tahoe$ pip install .
Processing /home/warner/stuff/tahoe/tahoe
tahoe-lafs requires Python '<3.0' but the running Python is 3.6.1
`

Closes ticket:2876