Tahoe-lafs fails to start with python 2.7 due to missing argparse module

[vrusinov]

Traceback (most recent call last):
  File "/usr/bin/tahoe", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2707, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 686, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 584, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: argparse>=0.8

But since python 2.7 argparse is included in standard library: ​http://www.python.org/dev/peps/pep-0389/, so my system does not have separate module for argparse.

[warner]

Out of curiosity, what operating system is this?

(I suspect that this is another setuptools-ish problem. I don't believe that the python standard library is expected to provide pkg-info metadata for its contents, which makes it hard to use pkg_resources to detect availability of such modules).

[davidsarah]

It is not setuptools' responsibility to detect the standard library version of argparse, or the responsibility of that version to provide pkg-info metadata. What is supposed to happen, I believe, is that the argparse package is always built (if not installed *as a package*) when building dependencies of zfec, and that the version so built ​should work fine on Python 2.7. It is not intended that the standard library version be used by zfec (but if it were, as a result of a strangely configured sys.path for example, it wouldn't matter because the standard library version of argparse is compatible with the package version; and besides Tahoe doesn't use the zfec command-line tools).

vrusinov: how did you install Tahoe? If you installed it from an OS package, then that package is broken because it should have declared a dependency on the argparse package regardless of whether the OS has Python 2.7.

If you installed it using 'python setup.py build', then it should have downloaded and built the argparse package, again regardless of whether the OS has Python 2.7.

[zooko]

We can also change zfec to declare a dependency on the argparse package only if the argparse module can't be imported. This would be analogous to the way Tahoe-LAFS declares a dependency on the pysqlite package only if the sqlite3 module can't be imported.

A simpler solution to this problem is to require Python 2.7! I'm excited about Python 2.7 because the Python developers have sworn to stop improving the Python 2 line.

Realistically, though, it will probably be years before we're willing to tell people they can't run Tahoe-LAFS with the Python 2.6 that came with their operating system. See #1658 (drop support for Python < 2.6) for a ticket where we are dithering about when to drop support for Python < 2.6. To simplify the subject of *this* ticket we would *also* have to drop support for Python < 2.7.

[zooko]

Hey waitaminute. I went to implement my idea from comment:4 to have zfec declare a dependency on the argparse package only if the argparse module couldn't be imported. However, I discovered that zfec already declares a dependency on argparse only if Python < 2.7. So how did you get a version of zfec installed into your system that says that it requires the argparse package? My guess is that you installed zfec with Python 2.6 and then upgraded your system to Python 2.7. I guess this shows that the hack of "try to import the argparse module and, if that doesn't work, declare a dependency on the argparse package" is more robust.

Assigning to vrusinov to see if he can explain how his install of zfec has this dependency on argparse even though he has Python 2.7. (Hint: where is zfec located? You can tell with $ python -c 'import zfec;print zfec'.)

Oh, another explanation would be that you have a version of zfec from before its dependency on argparse was conditionalized. That would be... let's see...

https://tahoe-lafs.org/trac/zfec/changeset/390/trunk

Oh dear... I guess I haven't made a new release of zfec in the 13 months since I committed that patch.

Okay, so a good next step would be for me to edit zfec to use the "try import and then fallback to require" approach instead of the "Python < 2.7" approach, and then make a new release of zfec.

[davidsarah]

Replying to zooko:

Okay, so a good next step would be for me to edit zfec to use the "try import and then fallback to require" approach instead of the "Python < 2.7" approach, and then make a new release of zfec.

Why is that change needed? Just making a new release of zfec as-is would solve the problem, no?

(I'm generally -1 on attempting to import modules that are normally only runtime dependencies at build time.)

[davidsarah]

BTW, making a new release of zfec will not help if vrusinov's (or anyone else's with the same problem) system is using the OS-packaged zfec. So we still need to know how vrusinov installed Tahoe.

[davidsarah]

Oh, and zooko: you have a habit of getting halfway through a comment and finding out something that negates the evidence for a previous hypothesis (e.g. "My guess is that you installed zfec with Python 2.6..." above), and then not editing or striking out the hypothesis. It's confusing!

[zooko]

Okay, I've posted zfec v1.4.23 which hopefully fixes this problem. vrusinov: please try again!

[zooko]

Replying to davidsarah:

Oh, and zooko: you have a habit of getting halfway through a comment and finding out something that negates the evidence for a previous hypothesis (e.g. "My guess is that you installed zfec with Python 2.6..." above), and then not editing or striking out the hypothesis. It's confusing!

Sorry! You're right.

[zooko]

Replying to davidsarah:

Why is that change needed? Just making a new release of zfec as-is would solve the problem, no?

I guess my main reason was that I wanted zfec's build system to be more like Tahoe-LAFS's. Also, I'm not sure if there could be some situation where Python 2.7 is in use, but argparse isn't importable. This happened recently, for example, when a Tahoe-LAFS user reported that their Python 2.7 didn't have the sqlite module. It wasn't entirely clear if this was the *BSD packaging of Python or some custom build of that user that had managed to omit this module from the Python standard library.

In general, I prefer the approach of "try the most specific test that is accurate" over "determine the functionality from a version number", when practical.

(I'm generally -1 on attempting to import modules that are normally only runtime dependencies at build time.)

That's a good consideration, too. In this case I currently prefer the above considerations, but I guess I could go either way.

[vrusinov]

Out of curiosity, what operating system is this?

It's Fedora 16.

vrusinov: how did you install Tahoe? If you installed it from an OS package, then that package is broken because it should have declared a dependency on the argparse package regardless of whether the OS has Python 2.7.

If you installed it using 'python setup.py build', then it should have downloaded and built the argparse package, again regardless of whether the OS has Python 2.7.

I'm using my own spec to build tahoe-lafs rpm and some additional specs to build missing dependencies (including zfec). Specs does nothing more than python setup.py build and python setup.py install.

It works pefectly on RHEL 5 & 6 (python 2.4 and 2.6), but does not works on Fedora 16.

Of course, I can port python-argparse (which presents on RHEL) to Fedora 16, but it will look ugly and I'm sure it's not necessary.

A simpler solution to this problem is to require Python 2.7! I'm excited about Python 2.7 because the Python developers have sworn to stop improving the Python 2 line.

Yep, but it will break existing installs on all RHEL/CentOS/SL versions (not sure about status of 2.7 on other distros), and nobody really want to install 2.7 on such distro. Personally I think any wide-used project should support at least 2.4. I share you pain as a developer, but as a sysadmin I understand the reason why there are lots of users with older long-term support systems.

My guess is that you installed zfec with Python 2.6 and then upgraded your system to Python 2.7.

No, I did not. I've build zfec 1.4.22 just some days ago. And, oh, Thank you! I see you've just released a new version. I will try it.

BTW, making a new release of zfec will not help if vrusinov's (or anyone else's with the same problem) system is using the OS-packaged zfec. So we still need to know how vrusinov installed Tahoe.

It will help if tahoe-lafs will declare >= zfec-1.4.23 dependency on python 2.7. At least it will be clear from error message what needs to be upgraded.

[vrusinov]

I will try it.

Well, it not so easy: ​https://tahoe-lafs.org/trac/zfec/ticket/15

I've built it, but I don't want to install manually (as it will mess with system packaging) and don't want to use virtualenv (as I want tahoe & zfec installed system-wide).

[zooko]

What's the status of this ticket? Did the fact that I made a new release of zfec solve it and now it can be closed?

Or should we implement one of the suggestions from the comments, such as "It will help if tahoe-lafs will declare >= zfec-1.4.23 dependency" (from comment:12)?

[daira]

I still don't understand why the original DistributionNotFound: argparse>=0.8 occurred, because python setup.py build should have downloaded and built argparse, which should work absolutely fine on Python 2.7. Of course it might have failed to do so due to one of the numerous setuptools bugs (or some other network or packaging error), but we don't have enough information to tell which one.

[daira]

By the way, if the zfec requirement is bumped, it should be to >=1.4.24, not >=1.4.23 (due to ​https://tahoe-lafs.org/trac/zfec/ticket/15 ).

[zooko]

Okay, it sounds like our options are:

• a) close this ticket as "invalid", "wontfix", "irreproducible", or "someone else's problem"
• b) drop support for any version of Python that isn't Python v2.7.x! ☺
• c) bump the requirement on zfec to be ≥ 1.4.24

I vote for a).

[zooko]

Addendum: I propose to close this ticket because, as people upgrade to zfec ≥ 1.4.24 and as they upgrade to Python ≥ 2.7, this problem will stop happening. Also, since we don't understand exactly how it happened in the first place, it could have been something very specific to vrusinov's system, that won't affect other users.

[daira]

vrusinov (or anyone else), please feel free to reopen this ticket if it happens again with any recent version of Tahoe-LAFS.