the test-from-egg buildstep has started failing on all builders

[daira]

It was passing in [5218e87ed148e92353e03f86e8949d227b5af731/trunk] and has started failing since [e4149496d267d39b61d93ececcb014010235f850/trunk]. (There are 14 non-merge commits between those revisions, but it should be an easy bisection.)

[daira]

This is caused by [cf0f64be98c42b4c18c2dccf297d67b44de53a0d/trunk]. (I hadn't noticed that misc/build_helpers/run_trial.py was still used by the test-from-egg step. Mea culpa.)

[warner]

We changed the buildbot config to run sys.executable tahoe debug trial [testsuite]. Unfortunately that runs into the bug described in #2242, in which a function inside pkg_resources is interpreting a __requires__ version specification incorrectly. The symptom is:

ValueError: ('Missing distribution spec', '=', '=', {'thisreqstr': '='})

So I think we must fix #2242 first, either by fixing our zetuptoolz fork, or getting rid of it altogether (#2044).

[warner]

I was able to reproduce this locally, eventually, with the following (paraphrased slightly):

1. cd ~/tahoe (contains setuptools.egg, really zetuptoolz)
2. python setup.py bdist_egg -> produces "tahoe.egg"
3. mkdir ~/tmp ~/tmp/egginst
4. cd ~/tmp
5. PYTHONPATH=egginst easy_install -d egginst ~/tahoe/dist/tahoe.egg
6. cd egginst
7. PYTHONPATH=~/tahoe/setuptools.egg:./*.egg python tahoe debug trial

The PYTHONPATH in step 5 is necessary to appease easy_install, which is trying to protect you against writing into a directory that you wouldn't normally read from. The PYTHONPATH in step 7 does two things: it uses tahoe's zetuptoolz fork, and it makes the dependencies available.

Step 7 produces the "Missing distribution spec" exception.

The egginst directory, as populated by easy_install, contains an .egg for every dependency, an easy-install.pth which would add them to the path (if you included egginst on PYTHONPATH), and an executable entry-point script for everything that produced one (tahoe, flappserver, flogtool, some zfec ones, etc). It is this tahoe entrypoint script that we execute. On my machine, it contains the following:

#!/usr/local/opt/python/bin/python2.7
# EASY-INSTALL-ENTRY-SCRIPT: 'allmydata-tahoe==1.10.0.post272','console_scripts','tahoe'
__requires__ = 'allmydata-tahoe==1.10.0.post272'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('allmydata-tahoe==1.10.0.post272', 'console_scripts', 'tahoe')()
    )

(this matches what I'm seeing on freestorm's buildslave).

If I omit tahoe's zetuptoolz egg from the PYTHONPATH in step 7, the problem goes away, and the tests run normally.

Note that step 5 is using my system-supplied easy_install, which reports its --version as setuptools 12.1. I think the problem here is a mismatch between the version of setuptools (zetuptoolz) that produced the .egg file in step 2, the version of easy_install that installed the egg (setuptools-12.1) in step 5, and the version of setuptools (zetuptoolz again) which provides the pkg_resources used for execution in step 7.

Our toolchain takes great pains to work even if you don't have setuptools installed on your system (perhaps due to my early complaints about not liking setuptools :-), by providing a setuptools/zetuptoolz .egg in the source tree, and importing it early in setup.py. However, step 5 in the test-from-egg process depends upon having a system-supplied easy_install script. (Tahoe includes setuptools as an egg, but not easy_install).

Digging into the zetuptoolz pkg_resources, it seems that the single-string __requires__ in the entrypoint script is, sometimes, being interpreted as a list, causing it to parse each letter of the string as a separate requirement string. Most of these ("a", "l", "l", "m", etc) are ignored, but when it finally gets to "=", the parser throws an exception, because "=" is supposed to have a package name on the left. However, in other places within pkg_resources, the string is correctly interpreted as a string.

I've seen other entrypoint scripts that provide a list for __requires__, like the one tahoe's setup.py build installs into support/bin/tahoe:

#!/usr/local/opt/python/bin/python2.7
# EASY-INSTALL-ENTRY-SCRIPT: 'allmydata-tahoe==1.10.0.post272.dev0','console_scripts','tahoe'
# generated by zetuptoolz 0.6c16dev5
__requires__ = ['allmydata-tahoe==1.10.0.post272.dev0', 'setuptools>=0.6c6', 'zfec>=1.1.0', 'simplejson>=1.4', 'zope.interface>=3.6.0,!=3.6.3,!=3.6.4', 'foolscap>=0.6.3', 'pycrypto>=2.1.0,!=2.2,!=2.4', 'mock>=0.8.0', 'pycryptopp>=0.6.0', 'Twisted>=13.0.0', 'Nevow>=0.11.1', 'service-identity', 'characteristic>=14.0.0', 'pyasn1>=0.1.4', 'pyasn1-modules', 'pyOpenSSL>=0.13', 'cryptography', 'cffi>=0.8', 'six>=1.4.1', 'enum34', 'pycparser']
import sys
from pkg_resources import load_entry_point

sys.exit(
   load_entry_point('allmydata-tahoe==1.10.0.post272.dev0', 'console_scripts', 'tahoe')()
)

And if I make easy_install use tahoe's zetuptoolz egg, by replacing step 5 with:

5b. PYTHONPATH=~/tahoe/setuptools.egg:egginst easy_install -d egginst ~/tahoe/dist/tahoe.egg

then I get the list-__requires__ entrypoint script like the one in support/bin/tahoe.

So my hunches are:

• zetuptoolz produces, and requires, a __requires__ list
• in modern setuptools:
  • pkg_resources accepts a string or a list
  • easy_install produces a string

and the mismatch is between the version of easy_install that creates the entrypoint script, and the version of pkg_resources that gets loaded by that script.

Solutions

So one solution for this ticket might be to use zetuptoolz for the easy_install, like step 5b. I'm experimenting with this to see if it would work.

A second would be to fix zetuptoolz to look more like modern setuptools.

But I suspect this test is no longer exercising the kind of functionality we really care about. Nobody installing a tahoe .egg is going to use 5b: they'll use easy_install tahoe.egg. And they won't include tahoe's zetuptools in PYTHONPATH when they run tahoe, so they'll get their pkg_resources from the system setuptools. To be relevant, our test should match real-world usage.

So a third fix is to remove the non-realistic PYTHONPATH= from step 7. The easy-install.pth in egginstalldir means we could probably use:

7b. PYTHONPATH=~/tmp/egginst python tahoe debug trial

(This reflects reality, since eggs are usually installed into something on the built-in PYTHONPATH, like /usr/local/lib).

Doing that would make the test dependent upon a system-installed setuptools, but it was already depending on that for easy_install.

I'm going to try this third option and see if it works.

[warner]

Hm, one potential problem with that third option: I think the Twisted egg comes from the source directory (is it used as a setup_requires=? or was that a workaround for some dependency's packaging bug?), and doesn't seem to be installed to egginstdir along with everything else. So without a PYTHONPATH that points back to the source dir, this might not work. Haven't tried it yet.

[warner]

Indeed that failed, as it couldn't find the Twisted egg. Our setup.py explains why it setup_requires= on Twisted (it involves bugs in Nevow packaging). Since removing our dependency on Nevow doesn't sound feasible for this release, I'm going to make the test slightly less realistic and include the Twisted .egg in the PYTHONPATH.

[zooko]

I think that bug has been fixed in nevow:

​https://github.com/twisted/nevow/issues/7

[warner]

Cool. I'm hesitant to change our setup.py too much right now, but after the release let's remove that stuff and see what happens.

I've experimented some more, and I don't know why easy_install isn't putting a Twisted egg into the egginstdir. It might have to do with the buildslave using distribute-0.6.10 (I'm using setuptools-12.1 here and it installs a twisted egg when Twisted isn't already installed in /usr/lib). It might be that running easy_install from the source directory, where there's a twisted.egg sitting, might trick it into thinking that twisted is generally available. It might also involve an apparent bug in buildbot, since I see the InstallToEgg buildstep is using:

kwargs['env'] = {"PYTHONPATH": egginstalldir}

but easy_install sees PYTHONPATH=egginstalldir: (probably the result of a lazy join.. egginstalldir+":"+os.environ.get("PYTHONPATH","")). The extra colon (or, rather, the empty component once you pp.split(":")) causes the current directory to be added to the PYTHONPATH. And maybe it's somehow seeing the twisted .egg there, so not bothering to install a new one into the install dir.

I'm going to try doing the install from a temporary directory, to see if that helps.

[warner]

Hey, that fixed it. Ok, time to pull out all the temporary debug stuff and make sure it all works.

[warner]

The tempdir-on-easy_install seems to have fixed it.. the orange buildslaves are turning green. Calling this one fixed.

[daira]

Replying to zooko:

I think that bug has been fixed in nevow:

​https://github.com/twisted/nevow/issues/7

It has, and removing the workaround will work on Unix. It won't work on Windows, because on Windows we still require an unfixed version of Nevow (because the fixed version depends on Twisted >= 13 and that in turn depends on pywin32).