apparent serious integrity problem in build system - setuptools bug?

[Greg Troxel]

Zooko Wilcox-OHearn <zooko at leastauthority.com> writes:

> 1. The Tahoe-LAFS build system (which is based on setuptools) will
> download and execute any package which pypi.python.org says is the
> "foo" package, if any part of Tahoe-LAFS transitively depends on a
> package named "foo". To change it so that it *doesn't* automatically
> download packages over the Net is ticket #1220.

OK - this is the basic bug.  By default, it should fail with an error,
and do this autofetching if given the --autofetch or whatever option.

> 2. pyOpenSSL just updated from 0.13 to 0.14, and newly depends on a
> new project named cryptography.io: https://cryptography.io . To deal
> with this is ticket #2193.

Indeed.  pyOpenSSL 0.14 is now in pkgsrc, and now that all the 0.14
fallout is mostly fixed, tahoe is failing:

    File "/usr/pkg/lib/python2.7/site-packages/twisted/application/app.py", line 452, in getApplication
      application = service.loadApplication(filename, style, passphrase)
    File "/usr/pkg/lib/python2.7/site-packages/twisted/application/service.py", line 405, in loadApplication
      application = sob.loadValueFromFile(filename, 'application', passphrase)
    File "/usr/pkg/lib/python2.7/site-packages/twisted/persisted/sob.py", line 210, in loadValueFromFile
      exec fileObj in d, d
    File "tahoe-client.tac", line 10, in <module>
      c = client.Client()
    File "/usr/pkg/lib/python2.7/site-packages/allmydata/client.py", line 130, in __init__
      node.Node.__init__(self, basedir)
    File "/usr/pkg/lib/python2.7/site-packages/allmydata/node.py", line 82, in __init__
      self.create_tub()
    File "/usr/pkg/lib/python2.7/site-packages/allmydata/node.py", line 174, in create_tub
      self.tub = Tub(certFile=certfile)
    File "/usr/pkg/lib/python2.7/site-packages/foolscap/pb.py", line 240, in __init__
      self.setupEncryptionFile(certFile)
    File "/usr/pkg/lib/python2.7/site-packages/foolscap/pb.py", line 252, in setupEncryptionFile
      self.setupEncryption(certData)
    File "/usr/pkg/lib/python2.7/site-packages/foolscap/pb.py", line 261, in setupEncryption
      raise RuntimeError("crypto for PB is not available, "
  exceptions.RuntimeError: crypto for PB is not available, try importing foolscap.crypto and see what happens

> My current plan for #2193 is to pin Tahoe-LAFS's dependency on
> pyOpenSSL ¹ to pyOpenSSL == 0.13.

Well, that's only semi ok, because packaging systems think it's ok to
update to new releases of python packages, and really it should be,
unless they are marked as horribly incompatible and able to be parallel
installed with the previous one.

I guess the real problem is that 0.14 is not backwards compatible.
(I don't know that, but if tahoe-lafs needs to change to use it when it
works with 0.13.1, that's a clue.)

> I have a question for you: how did "cryptography" and "six" get into
> the dependencies that setuptools is trying to satisfy (in your
> transcript above)? I would guess that "cryptography" got in there
> because of "pyOpenSSL", and that "six" got in there because of
> "cryptography", but why didn't your build use the pyOpenSSL package
> that was already installed!?

I have now figured that out.  py-OpenSSL was updated to 0.14 in pkgsrc,
but it was not marked as depending on six and cryptography.  When
*building* py-OpenSSL, these weren't used, so one got a package.  But
when using it (via "import OpenSSL"), or trying to test it (via
setuptools test), it failed.

So somehow, setuptools grovelled over all of this and figured out that
cryptography and six (and then cffi which is required by cryptography)
were needed, and tried to get them.

> I have a suggestion for you: go ahead and patch Tahoe-LAFS's
> src/allmydata/_auto_deps.py to specify "pyOpenSSL == 0.13" instead of
> just "pyOpenSSL" if it helps.

Well, that should make the package fail to build when 0.14 is installed,
which is better than building and not working.   But what actually
happened was:

  at build time, it downloaded 0.13.1

  this was not installed to the destdir as part of the install stage

  after installing the built package, tahoe fails to start because it
  can't load 0.13.1:

    File "/usr/pkg/lib/python2.7/site-packages/pkg_resources.py", line
    628, in resolve
     raise DistributionNotFound(req)
    pkg_resources.DistributionNotFound: pyOpenSSL==0.13.1


So my big question is: is there any checksum/cryptographic protection on
getting valid versions of these libraries?   It seems completely unsafe
for build processes to just go dowloading things.

> See also relevant tickets #1582, #2055, and #2077.

Thanks, will add to reading list ;-)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 180 bytes
Desc: not available
URL: <http://tahoe-lafs.org/pipermail/tahoe-dev/attachments/20140321/a95f93c2/attachment.pgp>