pyOpenSSL 0.14 pulls in a bunch of new dependencies

[daira]

$ cat pyOpenSSL-0.14-py2.7.egg/EGG-INFO/requires.txt
cryptography>=0.2.1
six>=1.5.2

cryptography in turn depends on cffi and pycparser, resulting in the following warnings:

$ tahoe --version-and-path
allmydata-tahoe: 1.10.0 (/usr/local/lib/python2.7/dist-packages)
foolscap: 0.6.4 (/usr/local/lib/python2.7/dist-packages)
pycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958 (/usr/local/lib/python2.7/dist-packages)
zfec: 1.4.24 (/usr/local/lib/python2.7/dist-packages)
Twisted: 13.2.0 (/usr/local/lib/python2.7/dist-packages)
Nevow: 0.10.0 (/usr/local/lib/python2.7/dist-packages)
zope.interface: unknown (/usr/local/lib/python2.7/dist-packages/zope)
python: 2.7.3 (/usr/bin/python)
platform: Linux-debian_7.4-x86_64-64bit_ELF (None)
pyOpenSSL: 0.14 (/usr/local/lib/python2.7/dist-packages)
simplejson: 3.3.3 (/usr/local/lib/python2.7/dist-packages)
pycrypto: 2.6.1 (/usr/local/lib/python2.7/dist-packages)
pyasn1: 0.1.7 (/usr/local/lib/python2.7/dist-packages)
mock: 1.0.1 (/usr/local/lib/python2.7)
setuptools: 2.2 (/usr/local/lib/python2.7/dist-packages/setuptools-2.2-py2.7.egg)
 
Warning: dependency 'cryptography' (version '0.2.2') found by pkg_resources not found by import.
Warning: dependency 'six' (version '1.5.2') found by pkg_resources not found by import.
Warning: dependency 'cffi' (version '0.8.1') found by pkg_resources not found by import.
Warning: dependency 'pycparser' (version '2.10') found by pkg_resources not found by import.
 
For debugging purposes, the PYTHONPATH was
  None
install_requires was
  ['setuptools >= 0.6c6', 'zfec >= 1.1.0', 'simplejson >= 1.4', 'zope.interface == 3.6.0, == 3.6.1, == 3.6.2, >= 3.6.5', 'Twisted >= 11.0.0', 'foolscap >= 0.6.3', 'pyOpenSSL', 'Nevow >= 0.6.0', 'pycrypto == 2.1.0, == 2.3, >= 2.4.1', 'pyasn1 >= 0.0.8a', 'mock >= 0.8.0', 'pycryptopp >= 0.6.0']
sys.path after importing pkg_resources was
  /usr/local/bin:
  /usr/local/lib/python2.7/dist-packages/pip-1.5.4-py2.7.egg:
  /usr/local/lib/python2.7/dist-packages/distribute-0.7.3-py2.7.egg:
  /usr/local/lib/python2.7/dist-packages/setuptools-2.2-py2.7.egg:
  /usr/lib/python2.7:
  /usr/lib/python2.7/plat-linux2:
  /usr/lib/python2.7/lib-tk:
  /usr/lib/python2.7/lib-old:
  /usr/lib/python2.7/lib-dynload:
  /usr/local/lib/python2.7/dist-packages:
  /usr/lib/python2.7/dist-packages

This is annoying, since it's going backwards with respect to our goal of reducing the number of crypto library dependencies.

[daira]

daira: SwedFTP: line 286 or thereabouts of src/allmydata/__init__.py should say:
daira:     ignorable = set(['argparse', 'pyutil', 'zbase32', 'distribute', 'twisted-web', 'twisted-core', 'twisted-conch'])
daira: change it to:
daira:     ignorable = set(['argparse', 'pyutil', 'zbase32', 'distribute', 'twisted-web', 'twisted-core', 'twisted-conch', 'cryptography', 'six', 'cffi', 'pycparser'])
daira: and reinstall tahoe
daira: then try 'tahoe --version-and-path' again
[...]
SwedFTP: daira1, daira , that fixes the issue http://pastebin.com/vXLGXfem
[...]
daira1: excellent, thanks for testing that

[daira]

Note that the change in comment:2 only hides the warning. We need to decide whether the additional dependency on cryptography is acceptable, and require pyOpenSSL == 0.13 if it is not. (The cryptography dependency is via pyOpenSSL 0.14.)

[Zancas]

Relevant to this issue, is my experience provisioning new S4 subscriptions. The provisioning process began failing unexpectedly on Tuesday March 03, 2014.

I don't immediately recall when I last ran it prior to that, but it was probably within a week (i.e. since Februrary 24th).

I ssh'd in to the SSEC2 which had been launched during the failed provisioning. I manually repeated the provisioning steps until I reached the first step that failed.

The failing step was: sudo python ./setup.py install run in the context of txAWS-0.2.1.post4 's 'base' directory.

I upgraded the version of txAWS used during deployment to post5 , by applying a commit daira wrote a while back. post5 depends on python-dateutil instead of Epsilon.

Here's the relevant commit: https://github.com/LeastAuthority/leastauthority.com/commit/05ce3d44708c89158c6ae828d2a11827b74001f7

After making _only_ that change, the S4 provisioning process began functioning as expected again. (As evidenced by a "smoke-test" on an S4 instance provisioned as above.)

The following is the version info from the provisioned S4:

customer@ip-10-185-216-61:~$ /home/customer/LAFS_source/bin/tahoe --version                                                                          
allmydata-tahoe: 1.9.1.dev1,
foolscap: 0.6.3,
pycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958,
zfec: 1.4.24,
Twisted: 12.2.0,
Nevow: 0.10.0,
zope.interface: unknown,
python: 2.7.3,
platform: Linux-Ubuntu_12.04-i686-32bit_ELF,
pyOpenSSL: 0.12,
simplejson: 2.3.2,
pycrypto: 2.4.1,
pyasn1: 0.1.7,
mock: 1.0.1,
sqlite3: 2.6.0 [sqlite 3.7.9],
setuptools: 0.6c16dev3

customer@ip-10-185-216-61:~$ 

And the version of txAWS:

customer@ip-10-185-216-61:~$ python -c 'import txaws;print txaws'
<module 'txaws' from '/usr/local/lib/python2.7/dist-packages/txAWS-0.2.1.post5-py2.7.egg/txaws/__init__.pyc'>
customer@ip-10-185-216-61:~$ 

As you can see from the tahoe version information, the _current_ version deployed with S4, does _not_ depend on cryptography because the version of pyOpenSSL it uses is: 0.12. When we upgrade to newer versions of Tahoe-LAFS (an urgent needed upgrade, to fulfill customer-driven feature requests) however, this issue will effect S4.

Finally the upgrade to txAWS-0.2.1.post5 is _not_ in production, and we can therefore expect new signups to fail.

[Zancas]

Also, as Daira pointed out elsewhere, the version of pyOpenSSL selected by setuptools when building ticket999-S3-backend, is liable to unpredictably change.

[zooko]

dstufft asked on IRC why this is such a big deal to us, and my answer was that for us resolving a dependency is a *user-facing* action, so if a dependency of ours (pyOpenSSL) gains a new dependency (cryptography.io), then that is not just a developer-facing issue, but is in fact a *regression* for our users. That's because our users start with this document: quickstart.rst, and follow the instructions written there, and at the end they have a working bin/tahoe script. Before the pyOpenSSL v0.14 release came out, following those instructions worked for most users, after the pyOpenSSL v0.14 release came out, it did not work for most users (because the new dependencies of pyOpenSSL cannot be automatically resolved by setuptools, especially considering that most users do not have a C compiler and Python header files installed). So it is a regression for us.

I intend to pin our dependency on pyOpenSSL to pyOpenSSL < 0.14 for now.

(We don't really rely on pyOpenSSL for much anyway, so if we could in fact *remove* the dependency on pyOpenSSL entirely, that would be nice. See also tickets with "openssl" in their tags.)

[glyph]

Replying to zooko:

dstufft asked on IRC why this is such a big deal to us

If you are telling end-users and not developers to install a tool with a development toolchain (and setuptools is definitely a development toolchain) then perhaps the problem is with the instructions? Your dependencies' dependencies should not be a user-visible change.

Have you considered creating distributions for end-users that bundle everything together into a single file, bundle, or linux distro package, so that dependency issues like this aren't exposed? Or perhaps at least updating quickstart.rst to use contemporary tools, i.e. pip and virtualenv, rather than ez_setup?

These dependencies can be automatically resolved by pip. There are already binary wheels for Windows so those folks don't need a C compiler. And in the coming months my understanding is that this will be extended to OS X as well.

I intend to pin our dependency on pyOpenSSL

By pinning this dependency you're opting out of all potential future security updates for pyOpenSSL which seems like a bad idea, if you depend on it at all. And the move to Cryptography and thereby cffi is a huge upgrade to the simplicity and security of the basic implementation strategy of pyOpenSSL itself.

(We don't really rely on pyOpenSSL for much anyway, so if we could in fact *remove* the dependency on pyOpenSSL entirely, that would be nice.)

Removing the dependency might be nice. The OpenSSL API is rightly universally reviled. Although I would suggest that Cryptography is a promising new project to provide backend agility for cryptographic primitives and you should be depending upon it directly at some point in the future :-).

It's quite likely that Twisted will acquire a hard dependency on Cryptography or some other cffi-based project in the future, so this is probably worth working out now.

[gdt]

In a packaging environment, having a pinned dependency on 0.13 is really not ok, because upstream py-OpenSSL doesn't seem to provide for parallel-installable multiple versions. So I think tahoe has to cope with whatever upstream releases, or stop using py-OpenSSL. But py-twisted uses pyOpenSSL, so it's not clear that what tahoe does matters in the grand scheme of things.

longer version after I retyped when the above was lost to spam filtering:

Regarding pinnning the dependency to 0.13, I don't think that's a resaonable approach. First, I think that any software that is broadly successful is used almost entirely from prebuilt packages in various packaging systems (or for us odd pkgsrc types, automatically built from source using the packaging system, which amounts to the same thing).

My impression is that python generally does not support installing multiple versions of a package, e.g py-OpenSSL 0.13.1 and 0.14 both. (While pkgsrc has made python27 and python33 parallel installable, and most libraries can have both at once, I haven't seen this extend to individual libraries.) So a packaging system has to just pick a version. Given that upstreams do not do security maintenance on obsolete versions, the only reasonable choice is the latest, except that taking a few months to move to a new release is also reasonable. So packaging systems soon (within a year) will no longer offer py-OpenSSL 0.13.1.

Twisted also requires py-OpenSSL, so I think fighting this implies pulling py-Twisted and py-OpenSSL into tahoe sources. That seems crazy in terms of maintenance burden.

Overall, my impresssion is that we're seeing a big hiccup because of a poorly-documented unexpected rototill, and that once packaging systems catch up with the new dependencies, things will be ok as far as building tahoe in a packaging system context (or installing deps from packaging system and then building tahoe itself from source).

[daira]

I'm annoyed and frustrated by pyOpenSSL adding a dependency on yet another fucking cryptography library. It's going in entirely the wrong direction. The actual mechanics of how to make Tahoe tolerate the dependency are not so much the issue here.

[daira]

Note that the only overlap between what cryptography.io provides, and what Tahoe-LAFS needs (and can't get from the Python standard lib) is AES and RSA. So we can't use it to remove any of our other crypto dependencies.

[glyph]

Replying to daira:

The actual mechanics of how to make Tahoe tolerate the dependency are not so much the issue here.

I strongly disagree. The mechanics of how to make Tahoe tolerate the dependency are the only issue here. If you want to talk about how pyOpenSSL should revert to being a giant pile of ad-hoc C code rather than a tiny wrapper around some statically-verified and ​well-tested wrappers for the OpenSSL API, the right place for that discussion would be ​the pyOpenSSL tracker, not here. (However, I can tell you that you're not likely to meet with much success with that proposal.)

pyOpenSSL implements a thing, and the hope here is that the way in which it implements this is abstracted and its clients, like Tahoe, should not need to care how it does it. Having dependencies is one way it might choose to implement this. It may acquire new dependencies in the future (as might Twisted, for that matter, for reasons having nothing to do with cryptography). It's deeply unfortunate that this addition broke things for Tahoe, and hopefully that won't happen in the future, but "your dependencies are part of your API contract and you can't ever add new ones" is not an acceptable constraint for Twisted; nor, I suspect, for any other thing that Tahoe depends on. We should work out how to do this non-destructively in the future. (Note that I believe the Cryptography 0.3 release resolves the installation issue at least for Windows users because it fixed the thing that would cause binary wheels to need to be rebuilt on the installer's system if a new release of cffi came out.)

I'm annoyed and frustrated by pyOpenSSL adding a dependency on yet another fucking cryptography library. It's going in entirely the wrong direction.

However, since you seem frustrated let me try to address that. It might be helpful to read ​the Cryptography team's rationale for creating something new rather than trying to maintain one of the existing options, but that's pretty brief.

First of all, Cryptography is not really "yet another ... cryptography library". My understanding is that it implements exactly no cryptographic primitives itself. Instead, its mission is to be a comprehensive wrapper around and common interface to multiple back-ends which allow for using your platform's existing cryptography engine, allowing you to access things like hardware support and platform security updates without updating all of your application's code. For example, it's not just a wrapper around OpenSSL; it also wraps ​the CommonCrypto API from OS X's security framework.

(Please be aware that this is my interpretation and I'm not a Cryptography core developer.)

In light of this substantially improved approach, pyOpenSSL is really just a light wrapper around one of these layers. It would be very premature to say pyOpenSSL is "deprecated" – lots of existing software uses it, it'll likely be around for a very long time, and Cryptography doesn't implement nearly enough high-level functionality to replace it – but as Cryptography implements higher level functionality I think there's no real reason why one would rely on the pyOpenSSL version of that functionality rather than Cryptography's. If you use Cryptography, it will be doing your crypto functionality in a way that is multi-backend and doesn't, for example, require you to compile OpenSSL on Windows; instead you could just use existing Windows APIs for these algorithms.

For Twisted, all this stuff is a godsend. Instead of having some stdlib stuff and some pyOpenSSL stuff and some PyCrypto? stuff (with known exploits) we can (eventually) just use one library to provide all of our security features.

Replying to daira:

Note that the only overlap between what cryptography.io provides, and what Tahoe-LAFS needs (and can't get from the Python standard lib) is AES and RSA. So we can't use it to remove any of our other crypto dependencies.

It's pretty early days for Cryptography. This is why I said "at some point in the future". They only added RSA like a couple of weeks ago, and the only high-level (i.e. non-"hazmat") thing implemented so far is Fernet. So there's a huge swath of functionality they're going to be adding already. If some of the Tahoe developers were to talk to the Cryptography folks, it might be easy to convince them to add all of your needs to the library in some form.

Since Tahoe uses more novel cryptographic primitives than many applications, it's possible that the Cryptography developers won't be interested in adding everything you need, but I really think it would be worthwhile to try; given the effort that those developers are investing in fixing their build and deployment story, it might make issues like this a lot easier to deal with in the future.

[daira]

Replying to glyph:

Replying to daira:

I'm annoyed and frustrated by pyOpenSSL adding a dependency on yet another fucking cryptography library. It's going in entirely the wrong direction.

However, since you seem frustrated let me try to address that. It might be helpful to read ​the Cryptography team's rationale for creating something new rather than trying to maintain one of the existing options, but that's pretty brief.

First of all, Cryptography is not really "yet another ... cryptography library". My understanding is that it implements exactly no cryptographic primitives itself. Instead, its mission is to be a comprehensive wrapper around and common interface to multiple back-ends which allow for using your platform's existing cryptography engine, allowing you to access things like hardware support and platform security updates without updating all of your application's code. For example, it's not just a wrapper around OpenSSL; it also wraps ​the CommonCrypto API from OS X's security framework.

Well exactly, and this is the core of the issue. The decision to change the scope of pyOpenSSL from being a wrapper around just OpenSSL, to being something that depends on a considerably more ambitious project, came entirely out of the blue from the point of view of Tahoe-LAFS developers. Perhaps we should have been paying more attention, I don't know. I certainly don't have any objection to the cryptography project; what I have an objection to is the lack of any heads-up to projects depending on pyOpenSSL (a list of them is easily determined from the package metadata on PyPI) of a rather major design change.

[daira]

In any case, I guess we have no choice but to add cffi as another binary dependency. (The pure Python dependencies aren't so much a problem.) Sigh.

[glyph]

Replying to daira:

Replying to glyph:

Replying to daira:

I'm annoyed and frustrated by pyOpenSSL adding a dependency on yet another fucking cryptography library. It's going in entirely the wrong direction.

However, since you seem frustrated let me try to address that. It might be helpful to read ​the Cryptography team's rationale for creating something new rather than trying to maintain one of the existing options, but that's pretty brief.

First of all, Cryptography is not really "yet another ... cryptography library". My understanding is that it implements exactly no cryptographic primitives itself. Instead, its mission is to be a comprehensive wrapper around and common interface to multiple back-ends which allow for using your platform's existing cryptography engine, allowing you to access things like hardware support and platform security updates without updating all of your application's code. For example, it's not just a wrapper around OpenSSL; it also wraps ​the CommonCrypto API from OS X's security framework.

Well exactly, and this is the core of the issue. The decision to change the scope of pyOpenSSL from being a wrapper around just OpenSSL, to being something that depends on a considerably more ambitious project, came entirely out of the blue from the point of view of Tahoe-LAFS developers. Perhaps we should have been paying more attention, I don't know. I certainly don't have any objection to the cryptography project; what I have an objection to is the lack of any heads-up to projects depending on pyOpenSSL (a list of them is easily determined from the package metadata on PyPI) of a rather major design change.

This change was announced ​in January on the pyOpenSSL mailing list, ​and then on the Twisted list, ​and then again on the Twisted list, specifically stressing how major the change would be, not to mention the 4 other alpha pre-release announcements on those lists. I feel like this was reasonably broadly communicated. What more do you feel we should have done?

[glyph]

Replying to daira:

In any case, I guess we have no choice but to add cffi as another binary dependency. (The pure Python dependencies aren't so much a problem.) Sigh.

While I understand that the task of adding cffi itself is yet more frustrating package churn, there are a number of reasons to be enthusiastic about it:

1. Writing cffi modules is far easier, safer, etc, than writing any other kind of extension module. Unless you've used it, it's hard to explain just how much easier and safer it is, but one personal anecdote I can give you is that I once spent about 20 hours wrapping a fairly complex C library with cffi, including callbacks, reentrancy, and pointer math, and literally experienced zero segfaults in the process. One segfault I had near the end turned out to be a bug in the C library itself and not a problem with my extension.
2. cffi modules are JIT-visible in PyPy? and will therefore actually speed up applications on PyPy?, unlike cpyext-built extensions, which generally have enough overhead that they slow things down.
3. the cffi team provides packaging and building tools for extension modules to normalize the process of distributing binary dependencies, in a way which is challenging to do for arbitrary extension modules, and (in combination with the PyPA developers) is working on more. This has been somewhat rocky to get off the ground, but once the binary compatibility story for wheels has been worked out for OS X and Linux, this kind of uproar from users when a binary dependency is added should get much rarer, because they'll actually be able to install them from PyPI and have them actually work without a C compiler.
4. Jython developers are working on a cffi runtime support system now as well, making it by far the most portable way to distribute an extension module; a number of "it only works on CPython because of a C dependency that doesn't work on any other VMs" projects will be able to migrate to other interesting environments.

Finally, there's a ctypes backend for cffi; I'm not sure how complete it is, but in principle it ought to allow for (slow) invocation of arbitrary cffi-based extensions with no C compiler installed. If Tahoe still has users having problems before The Great Packaging Singularity Where Everything Finally Starts Working, it might be worthwhile to figure out if this can be enabled for your users somehow if their C compiler isn't installed or doesn't work.

[dstufft]

For what it's worth, I'm both a cryptography developer (kind of, I'm probably the least active one of us) and a PyPA developer. I also have PyNaCl? and bcrypt which also use the cffi backends. cffi has some pitfalls with packaging right now, however they are being worked on both on the cffi side with work to remove the implicit compile stuff, in the project themselves with work arounds, and in the packaging tools to better handle binary dependencies in general.

What tahoe-lafs decides to do with it's dependencies will affect y'all far more than it affects me, but I would suggest that pinning to an old release of pyOpenSSL will end up causing you more grief in the long run. I concur with that you should either get rid of it or adopt to the newer style of things.

I also think it'd probably be reasonable for cryptography.io to look to handle more of what tahoe needs from a crypto library too if you're all interested in that. We have very stringent requirements for what gets committed to our code base as far as code quality, test coverage, documentation etc goes and we have goals of being somewhat of a standard library of crypto :).

[daira]

OK, well, the immediate problem is that cryptography depends on cffi which depends (on Debian at least) on the libffi-dev package. Is there any way to build a cffi egg that (at run-time) doesn't depend on libffi-dev having been installed?

[dstufft]

Any binary package of cffi (Wheel or Egg) should only require libffi installed not libffi-dev if that's what you mean.

[daira]

Replying to dstufft:

Any binary package of cffi (Wheel or Egg) should only require libffi installed not libffi-dev if that's what you mean.

Sorry, I didn't mean to make a distinction between libffi and libffi-dev here. I meant to ask, is there any way to build a cffi egg that at run-time doesn't depend on libffi having been manually installed?

[dstufft]

Probably if you link it statically instead of dynamically, but I'm not sure that the cffi setup.py supports that ATM.

[glyph]

Replying to dstufft:

Probably if you link it statically instead of dynamically, but I'm not sure that the cffi setup.py supports that ATM.

Tempting as it is to ask “but then how the heck does it work on Windows!?” I think maybe ​this issue on the CFFI tracker is the place to keep discussing this stuff, and make sure that the cffi developers are aware of tahoe's requirements.

[daira]

Zooko and I agreed on the Dev Chat to temporarily work around this by fixing the pyOpenSSL dependency to == 0.13. We're aware that this can only be a short-term work-around, but cryptography simply has too many build problems at the moment (#2217 for example).

[daira]

The ticket for a longer-term solution to allowing pyOpenSSL >= 0.14 is #2221.

[daira]

Re-reading the above comments, I think I may have given a false impression that I don't approve of the overall technical direction that the cryptography and pyOpenSSL developers are taking. That's not the case; I absolutely appreciate the need for FFIs between memory-safe and non-memory-safe languages to reduce the amount of glue code required, and to allow most of the remaining glue code to be on the memory-safe side. cffi seems to be on the right track here, and it's clear from comparing the code of pyOpenSSL 0.13 and 0.14 that it should be a great deal more maintainable and less error-prone. (I've long been a fan of Standard ML's ​NLFFI which takes a similar approach.) Also, the ability to use multiple backends from cryptography is useful and important.

[gdt]

I'm not entirely following the pinning argument. tahoe is building ok in pkgsrc with 0.14. So if you require 0.13, the package will fail and no longer be available, and I'll just mark it BROKEN. What's wrong with a situation where pyOpenSSL 0.14 is properly installed, with all its dependencies? If the new code handles that gracefully, there's no issue - if all you're talking about is forcing a choice of 0.13 when the build of pyOpenSSL is triggered from within tahoe-lafs that sounds fine.

[daira]

Replying to gdt:

I'm not entirely following the pinning argument. tahoe is building ok in pkgsrc with 0.14. So if you require 0.13, the package will fail and no longer be available, and I'll just mark it BROKEN. What's wrong with a situation where pyOpenSSL 0.14 is properly installed, with all its dependencies?

That's a good question. The issue is that the Python package-distribution metadata conflates the question "what dependent package versions are acceptable to meet this package's requirement?", with the question "what dependent package versions should I attempt to fetch and build in order to meet this package's requirement (when not using an OS packaging system)?" That's a problem in situations where the fetch-and-build process is unreliable.

We could hack around this conflation by doing the following:

• From Tahoe's setup.py, attempt "from OpenSSL import SSL".
• If that succeeds and passes version sanity checks (including checks on the linked OpenSSL version for #2215), then use the requirement pyOpenSSL == $WHATEVER_VERSION_WAS_IMPORTED.
• If that does not succeed or does not pass version sanity checks, then use the requirement pyOpenSSL == 0.13 (or == 0.13.1 if we decide to do the OpenSSL version check that way).

That would implement the following:

If the new code handles that gracefully, there's no issue - if all you're talking about is forcing a choice of 0.13 when the build of pyOpenSSL is triggered from within tahoe-lafs that sounds fine.

However, attempting to import dependent packages from setup.py has caused problems in the past and I'm not entirely sure it's a good idea. If we do it, we should have some exit strategy to stop doing it in a future Tahoe-LAFS version.

[gdt]

That sounds reasonable.

Alternatively, there could be two setup.py versions, and a pre-build stage that symlinks one in. The standard one would be for packaging environments, and fail if any prereqs are not installed, never trying to download or build anything else. The developer one could do whatever you want.

I've said this before, but it's important to realize that for software that has widespread usage, almost all uses of it are via a packaging system. Developers live in a bubble where they build the programs they like to hack on directly from source. But I bet nearly all if not all of you are using even python from a packaging system, not a by-hand build.

[daira]

Replying to daira:

However, attempting to import dependent packages from setup.py has caused problems in the past and I'm not entirely sure it's a good idea.

One important problem is that if the setup.py process imports OpenSSL but it turns out to be broken or vulnerable, then there is no way to "unimport" it (well, not reliably), and this may cause problems later in the build. It would be possible to shell out to a check_pyopenssl.py script in a separate process, but that's getting rather complicated.

gdt: would it be sufficient for you to just patch the pyOpenSSL == 0.13 requirement in src/allmydata/_auto_deps.py in your pkgsrc packaging of Tahoe-LAFS 0.11?

[gdt]

Yes, probably it would suffice to patch the control file.

[daira]

Replying to dstufft:

Any binary package of cffi (Wheel or Egg) should only require libffi installed not libffi-dev if that's what you mean.

The Debian package name seems to be libffi6, not libffi. That's annoying because it'll probably change in future.

[zooko]

diff --git a/src/allmydata/_auto_deps.py b/src/allmydata/_auto_deps.py
index b7be321..2b82cf5 100644
--- a/src/allmydata/_auto_deps.py
+++ b/src/allmydata/_auto_deps.py
@@ -46,9 +46,13 @@ install_requires = [
     #   pyOpenSSL, such as foolscap requiring a specific version of
     #   pyOpenSSL, or foolscap switching from pyOpenSSL to a different crypto
     #   library, we need to update this declaration here.
+    # * pyOpenSSL 0.14 and newer depends on a new library named 
+    #   cryptography.io, which uses cffi to build. This causes problems for 
+    #   our deployments currently (see ticket #2193), so for now we require 
+    #   pyOpenSSL <= 0.13.
     #
     "foolscap >= 0.6.3",
-    "pyOpenSSL",
+    "pyOpenSSL <= 0.13",
 
     "Nevow >= 0.6.0",
 

[zooko]

Replying to daira:

Replying to gdt:

I'm not entirely following the pinning argument. tahoe is building ok in pkgsrc with 0.14. So if you require 0.13, the package will fail and no longer be available, and I'll just mark it BROKEN. What's wrong with a situation where pyOpenSSL 0.14 is properly installed, with all its dependencies?

That's a good question. The issue is that the Python package-distribution metadata conflates the question "what dependent package versions are acceptable to meet this package's requirement?", with the question "what dependent package versions should I attempt to fetch and build in order to meet this package's requirement (when not using an OS packaging system)?" That's a problem in situations where the fetch-and-build process is unreliable.

We could hack around this conflation by doing the following:

• From Tahoe's setup.py, attempt "from OpenSSL import SSL".
• If that succeeds and passes version sanity checks (including checks on the linked OpenSSL version for #2215), then use the requirement pyOpenSSL == $WHATEVER_VERSION_WAS_IMPORTED.
• If that does not succeed or does not pass version sanity checks, then use the requirement pyOpenSSL == 0.13 (or == 0.13.1 if we decide to do the OpenSSL version check that way).

That would implement the following:

If the new code handles that gracefully, there's no issue - if all you're talking about is forcing a choice of 0.13 when the build of pyOpenSSL is triggered from within tahoe-lafs that sounds fine.

However, attempting to import dependent packages from setup.py has caused problems in the past and I'm not entirely sure it's a good idea. If we do it, we should have some exit strategy to stop doing it in a future Tahoe-LAFS version.

I don't think we should add all this logic into the Tahoe-LAFS build system. I agree that there is a problem here, but I don't think we should try to address that problem by adding code into the Tahoe-LAFS build system. The problem, as I understand it, is that there are two different audiences listening to Tahoe-LAFS's announcements about what version of pyOpenSSL it depends on. One audience is packaging engineers like gdt, and the other is the automatic build mechanisms such as pip.

There is no single statement we can make that will give the right idea to both of these audiences. So I propose that we let the machine-parseable declaration in src/allmydata/_auto_deps.py be optimized for the automatic build mechanisms like pip, and we use an attendant comment to communicate to the human engineer like gdt. So I propose a solution along these lines:

diff --git a/src/allmydata/_auto_deps.py b/src/allmydata/_auto_deps.py
index b7be321..93a9994 100644
--- a/src/allmydata/_auto_deps.py
+++ b/src/allmydata/_auto_deps.py
@@ -38,6 +38,9 @@ install_requires = [
     # * foolscap < 0.6.3 is incompatible with Twisted-11.1.0 and newer. Since
     #   current Twisted is 12.0, any build which needs twisted will grab a
     #   version that requires foolscap>=0.6.3
+    #
+    "foolscap >= 0.6.3",
+
     # * pyOpenSSL is required by foolscap for it (foolscap) to provide secure
     #   connections. Foolscap doesn't reliably declare this dependency in a
     #   machine-readable way, so we need to declare a dependency on pyOpenSSL
@@ -47,8 +50,21 @@ install_requires = [
     #   pyOpenSSL, or foolscap switching from pyOpenSSL to a different crypto
     #   library, we need to update this declaration here.
     #
-    "foolscap >= 0.6.3",
-    "pyOpenSSL",
+    # * pyOpenSSL >= 0.14 uses a new implementation which depends on the new
+    #   "cryptography.io" library, which depends on cffi and other *
+    #   things. Our automated build system (as run by typical users on their
+    #   * typical systems, which may not have a compiler or Python header
+    #   files * installed), can't handle those new dependencies, so for now
+    #   we're pinning this dependency to pyOpenSSL == 0.13.
+    #
+    #   Note that if you are a build engineer who already has pyOpenSSL >=
+    #   v0.14 working and wants to use Tahoe-LAFS with pyOpenSSL >= v0.14, go
+    #   right ahead! You'll have to edit or comment-out this pinning
+    #   declaration here. This declaration is not for your benefit, but for
+    #   the benefit of the automatic build system (i.e. pip or easy_install)
+    #   as used by our non-expert users.
+    # 
+    "pyOpenSSL == 0.13",
 
     "Nevow >= 0.6.0",
 

[zooko]

Relatedly, I find it pretty annoying that Tahoe-LAFS needs to mention pyOpenSSL at all in its packaging metadata. pyOpenSSL is not a direct dependency of Tahoe-LAFS, it is a dependency of Foolscap. So, I feel like it ought to be Foolscap's responsibility to deal with all this crap. Hopefully in the future Foolscap will stop depending on pyOpenSSL at all (perhaps because Foolscap has switched to ​nacl/​libsodium), and at that point there should not need to be any change made to Tahoe-LAFS, because Tahoe-LAFS should just continue to say "I depend on Foolscap.".

Related tickets on the foolscap trac:

• ​Foolscap ticket #215 (change "install_requires" in setup.py to reflect requirement of 2.5.0 of Twisted)
• ​Foolscap ticket #67 (remove UnauthenticatedTub)
• ​Foolscap ticket #174 [closed: fixed] (foolscap 0.6.1 has no such extra feature 'secure_connections')
• ​Foolscap ticket #66 [closed: fixed] (install requires pyOpenSSL (for secure mode))
• ​Foolscap ticket #162 [closed: invalid] (No dependency declared on PyOpenSSL)

[zooko]

Here's another thing that Foolscap might switch to instead of SSL: ​https://github.com/trevp/noise/wiki

[daira]

The combination of this problem, #2028, and #2249, are making it almost impossible to install or build on Windows :-(

[daira]

(This problem isn't Windows-specific but it particularly affects Windows systems because most do not have a working compiler.)

[Daira Hopwood <daira@…>]

In b0b76a7c5b89c3fed5a65ef6732dc45e578f12f4/trunk:

Improve comments in _auto_deps.py. refs #2249, #2028, #2193, #2005, #1258

Signed-off-by: Daira Hopwood <daira@…>

[daira]

Fixed by [18ffc29f4949b6098b8b89e6e89c89923121cda2/trunk].

[daira]

​https://caremad.io/2014/11/distributing-a-cffi-project/ explains what is wrong with cffi from a packaging point of view.