"tahoe --version" shouldn't show scary diagnostic warnings

[warner]

It is very common for tahoe --version to report diagnostics warnings about the difference between what pkg_resources claims to provide, and what the actual imported modules (and their __version__) reports. These warnings have convinced several users that their tahoe installation has failed, when in fact everything will would work normally apart from those warnings.

Zooko and I agreed on IRC to change tahoe --version to report the versions of Tahoe itself and its dependencies (foolscap, pycryptopp, etc), but to remove the import-vs-pkg_resources warnings. We'll add a separate tahoe debug versions command that emits complete diagnostics.

For reference, this is what the problematic output looks like:

% bin/tahoe -V
allmydata-tahoe: 1.10.0
foolscap: 0.8.0
pycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958
zfec: 1.4.24
Twisted: 13.0.0
Nevow: 0.11.1
zope.interface: unknown
python: 2.7.9
platform: Linux-fedora_22-x86_64-64bit_ELF
pyOpenSSL: 0.14
simplejson: 3.7.2
pycrypto: 2.6.1
pyasn1: 0.1.7
mock: 1.0.1
setuptools: 0.6c16dev4

Warning: dependency 'pycparser' (version '2.10') found by pkg_resources not found by import.
Warning: dependency 'six' (version '1.9.0') found by pkg_resources not found by import.
Warning: dependency 'cryptography' (version '0.8.2') found by pkg_resources not found by import.
Warning: dependency 'cffi' (version '0.8.6') found by pkg_resources not found by import.
Warning: dependency 'pyasn1-modules' (version '0.0.5') found by pkg_resources not found by import.
Warning: dependency 'enum34' (version '1.0') found by pkg_resources not found by import.
Warning: dependency 'characteristic' (version '14.3.0') found by pkg_resources not found by import.
Warning: dependency 'service-identity' (version '14.0.0') found by pkg_resources not found by import.

For debugging purposes, the PYTHONPATH was
  '/home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages'
install_requires was
  ['setuptools >= 0.6c6', 'zfec >= 1.1.0', 'simplejson >= 1.4', 'zope.interface == 3.6.0, == 3.6.1, == 3.6.2, >= 3.6.5', 'Twisted >= 11.0.0', 'foolscap >= 0.6.3', 'pyOpenSSL', 'Nevow >= 0.6.0', 'pycrypto == 2.1.0, == 2.3, >= 2.4.1', 'pyasn1 >= 0.0.8a', 'mock >= 0.8.0', 'pycryptopp >= 0.6.0']
sys.path after importing pkg_resources was
  /home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/support/bin:
  /home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/setuptools-0.6c16dev4.egg:
  /home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/src:
  /home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/pycryptopp-0.6.0.1206569328141510525648634803928199668821045408958-py2.7-linux-x86_64.egg:
  /home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/mock-1.0.1-py2.7.egg:
  /home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/Nevow-0.11.1-py2.7.egg:
  /home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/foolscap-0.8.0-py2.7.egg:
  /home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/zfec-1.4.24-py2.7-linux-x86_64.egg:
  /home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/service_identity-14.0.0-py2.7.egg:
  /home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/pyutil-2.0.0-py2.7.egg:
  /home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/pyasn1_modules-0.0.5-py2.7.egg:
  /home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/characteristic-14.3.0-py2.7.egg:
  /home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/zbase32-1.1.5-py2.7.egg:
  /home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/Twisted-13.0.0-py2.7-linux-x86_64.egg:
  /home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages/zope.interface-4.1.2-py2.7-linux-x86_64.egg:
  /home/olabini/workspace/SPEAR/tahoe/allmydata-tahoe-1.10.0/support/lib/python2.7/site-packages:
  /usr/lib64/python27.zip:
  /usr/lib64/python2.7:
  /usr/lib64/python2.7/plat-linux2:
  /usr/lib64/python2.7/lib-tk:
  /usr/lib64/python2.7/lib-old:
  /usr/lib64/python2.7/lib-dynload:
  /usr/lib64/python2.7/site-packages:
  /usr/lib64/python2.7/site-packages/gst-0.10:
  /usr/lib64/python2.7/site-packages/gtk-2.0:
  /usr/lib/python2.7/site-packages

The output is large enough that, depending upon the size of your screen, you may wind up with the scary "Warning:... not found by import" lines at the top of the screen, which sounds an awful lot like an ImportError.

[zooko]

On IRC Daira said that she thinks all of the warnings shown in the user-supplied output above have been fixed in trunk. I'll ask the user to try the 10.1a1post9 tarball.

[zooko]

​https://twitter.com/olabini/status/604300869832597504

"the printing of sys.path REALLY looks like a stack trace btw."

I think a lesson here is never to show output to a user that looks like a stack trace, i.e. has a list of lines where the beginning of each line is a filename/pathname. I've seen this before: if a user sees something that looks like a stack trace, they immediately avert their gaze and assume that the software is malfunctioning.

[warner]

What if tahoe --version emitted just the versions and a single line that said "warning: not all dependency versions matched expectations. Run 'tahoe debug versions' for more info." ? And then we move all that other (scary) stuff to the other command?

[warner]

Moving this out of 1.10.1, insufficient consensus to make a last-minute change.

[daira]

The issue I have with not displaying a warning is that the printed versions may not be correct. However, I'm okay with printing a short warning on the line that displays each offending version (in the "comment" field).

[daira]

So in the example in the description, for instance, the output would be:

allmydata-tahoe: 1.10.0
foolscap: 0.8.0
pycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958
zfec: 1.4.24
Twisted: 13.0.0
Nevow: 0.11.1
zope.interface: unknown
python: 2.7.9
platform: Linux-fedora_22-x86_64-64bit_ELF
pyOpenSSL: 0.14
simplejson: 3.7.2
pycrypto: 2.6.1
pyasn1: 0.1.7
mock: 1.0.1
setuptools: 0.6c16dev4
pycparser: 2.10 [according to pkg_resources]
six: 1.9.0 [according to pkg_resources]
cryptography: 0.8.2 [according to pkg_resources]
cffi: 0.8.6 [according to pkg_resources]
pyasn1-modules: 0.0.5 [according to pkg_resources]
enum34: 1.0 [according to pkg_resources]
characteristic: 14.3.0 [according to pkg_resources]
service-identity: 14.0.0 [according to pkg_resources]

[warner]

That sounds like a good compromise to me. I think I'll use () instead of [].

[daira]

[] is already used for comments; () is used for paths when --version-and-path is given. I want the output to remain consistently parsable.

[daira]

Review needed for ​https://github.com/tahoe-lafs/tahoe-lafs/pull/176.

[Daira Hopwood <daira@…>]

In 431728f8f854e02d9ab2f731675f12ce18cda122/trunk:

Don't show scary diagnostic warnings from --version[-and-path]
if the only problem is that extra packages are found by pkg_resources.
fixes ticket:2436

Signed-off-by: Daira Hopwood <daira@…>

[daira]

Oops, I just spotted a bug. get_package_versions_string extends _vers_and_locs_list each time it is called (assuming not hasattr(sys, 'frozen')), but it may be called more than once.

I will fix it.

[Daira Hopwood <daira@…>]

In 2c99294a4faa0e0f94fdb53ed60954e1e0724955/trunk:

Don't show scary diagnostic warnings from --version[-and-path] (corrected). refs ticket:2436

The previous version would incorrectly add to the output of
get_package_versions_string each time it was called.

Signed-off-by: Daira Hopwood <daira@…>

[warner]

Landed in 28399b6.