add dependency on `Twisted[tls]` to overcome pip's non-resolver

[warner]

Tahoe itself doesn't strictly depend on TLS support. It depends on Foolscap, which *does*, but in a perfect world Tahoe wouldn't have to know about that.

To ask Twisted to be TLS-capable, a package uses a square-bracketed "extra" in its dependencies, like Twisted[tls] >= 15.2.1 instead of just Twisted >= 15.2.1.

In that perfect world, we'd have:

Tahoe: Twisted >= 13.0.0, Foolscap
Foolscap: Twisted[tls] >= 16.0.0

But unless/until we bypass the issue by using a requirements.txt file (maybe for #2055), we're affected by a missing pip feature: it lacks a full resolver.

A full resolver is what lets Debian's "apt" try all possible combinations of packages and versions to find any (hopefully the "best") that will meet the given constraints. It takes a lot more work, and is scarily powerful (someone once proved that the constraint solver is Turing-complete).

Pip installs the highest allowable version of the first thing that it encounters, and then explores the dependencies. It doesn't go back to try something different if the subsequent dependencies don't fit. And in particular, if it installs a package without any "extras", it won't go back and re-install it (with the extras) if it sees a later dependency that wants them.

So in the above example, when we install tahoe, pip will first install the latest version of Twisted it can find (e.g. 16.0.0, with no extras), then it installs foolscap, then, it sees that Foolscap wants Twisted[tls] >= 16.0.0. It knows it can satisfy the >= 16.0.0 requirement, but it can't go back and re-install the [tls] extra. It emits a warning, but can't fix it.

To overcome this, for the 1.11.0 release, we made Tahoe aware of the need for TLS, by using:

Tahoe: Twisted[tls] >= 13.0.0, Foolscap

But this hits another problem, which is that Twisted didn't start offering the [tls] extra until 15.2.1 . So we're actually using:

Tahoe: Twisted[tls] >= 15.2.1, Foolscap

even though Tahoe, itself, doesn't need that recent of a Twisted version.

(incidentally, ​foolscap#249 is what added Twisted[tls] to Foolscap)

This version bump might be inconvenient for OS packagers who are backporting the new tahoe-1.11.0 to older distributions that don't have the more recent Twisted. As I mentioned on the ​mailing list just now, porters who find themselves in this situation (and can't upgrade their Twisted packages) should consider modifying Tahoe's src/allmydata/_auto_deps.py to reduce this constraint back to the previous version. As long as there are OS-package-level dependency constraints on everything that the older version of Twisted needs for TLS support, things should work.

(although note that the most recent version of Foolscap does, in fact, depend upon Twisted >= 16.0.0, so if you're upgrading that, you should probably go all-in and upgrade everything).

This ticket is just to record the reasons for this version bump.

[Brian Warner <warner@…>]

In d57c8d5/trunk:

bump Twisted dependency (>=15.1.0) to get the [tls] extra

We only really need "Twisted >= 13.0.0", but we must add "[tls]" because
otherwise pip won't install it when Foolscap asks for it later, and we
need ">= 15.1.0" because that's the first version that provided "[tls]".

Fixes ticket:2760.