make Tahoe-LAFS work under PyPy

[davidsarah]

If I remember correctly, the main obstacle was extension modules. Current PyPy? ​supports extension modules compiled for CPython, but is not 100% compatible with the CPython extension API.

​Twisted apparently now builds correctly with PyPy.

[davidsarah]

fijal wrote in #pypy:

davidsarah: pyopenssl works as long as you don't have multiple threads

[hawkowl]

From an IRC discussion about why it doesn't work (pycryptopp), I decided to look at Cryptography and pynacl, and what uses pycryptopp in tahoe.

• ed25519 is supported by pynacl.
• AES is supported by Cryptography.
• RSA keys (in many formats) are supported by Cryptography.
• SHA256 hashes are supported by Cryptography.

Zooko mentioned that Cryptography has a hard dependency on OpenSSL (which he rightfully dislikes) but my unqualified self thinks that it might not be so bad. Worth a shot, maybe? I have (some) time on Tuesday's PyCon? AU sprint session to maybe take a hack on it, if the maintainers don't find the current-hard-dep on OpenSSL a dealbreaker.

[warner]

Foolscap depends upon OpenSSL too, and even though we're planning on replacing Foolscap (#510), for backwards compatibility we'll probably need it for quite a while yet. Also when we replace it, we may wind up wanting TLS-protected HTTP instead of just plain HTTP (to maintain privacy of which shares you're fetching), which might be another thing that depends upon OpenSSL. I'm not a fan of OpenSSL either, but I don't think depending upon it would be the worst direction to go. Maybe our policy could be to use pynacl for new features, and migrate old ones from pycryptopp to cryptography?

[hawkowl]

That policy sounds reasonable -- although pynacl doesn't expose much more than ed22519 keys and "boxes" (Curve25519-pubpriv, salsa20+Poly1305 symmetric), so, unless the new features use that, you'll have to use Cryptography.

To replace pycryptopp in Tahoe, both Cryptography and pynacl will have to be used (pynacl for the ed22519, cryptography for AES, RSA, and SHA-2), so new features could possibly rely on either. The best case is that Cryptography later comes with a non-OpenSSL backend (that being said, LibreSSL works, if I'm not mistaken), which is improbable (because it's a lot of work) but possible (because it's been written from the start to have multiple backends -- on OS X it uses CommonCrypto? for instance).

I'll see what I can get from a day of hacking in the sprints.

[exarkun]

Note that efforts are underway to switch from pycryptopp to cryptography (​https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3031) as part of the Python 3 porting effort.

[exarkun]

There is a PR that seems to get us most of the way there now - ​https://github.com/tahoe-lafs/tahoe-lafs/pull/669

[exarkun]

Merged in ​https://github.com/tahoe-lafs/tahoe-lafs/commit/466369dd6948d8fb4ecc3b9fea0c98ba6aca140d (some while ago)