Design a replacement for the Foolscap-based storage node API

[exarkun]

Storage nodes expose their functionality by way of a Foolscap-based network interface. Foolscap offers a very rich set of functionality for defining such interfaces but this comes with certain costs.

For example, Foolscap is presently only implement in Python 2. As the planned de-support of Python 2 by python-dev looms (presently less than 2 years of support officially remain and _practically_ Python 2 is already orphaned from a number of interesting perspectives), this poses an increasingly relevant challenge to the Tahoe-LAFS project (will Tahoe-LAFS require an unsupported runtime? Will Tahoe-LAFS abandon CPython and require PyPy?? etc).

Further, as there are no non-Python implementations at all, any efforts to create a non-Python Tahoe-LAFS implementation must start at the low level of implementing the Foolscap protocol (and due to the rich featureset of Foolscap, this is no simple task).

Then, despite the rich feature set of Foolscap, Tahoe-LAFS does not actually make extensive use of these features. Instead, Tahoe-LAFS network interactions are largely request/response-driven. Certain security properties granted automatically by Foolscap are interesting and important but many other features (asynchronous server-side requests, for example) are essentially ignored. However, despite not using these features, much of the complexity related to implementing them is still implicated in Tahoe-LAFS. This manifests, for example, as reduced runtime efficiency compared to similar operations performed with a simpler protocol. Consequently, significant CPU resources must be allocated to allow a Tahoe-LAFS storage node to make full use of relatively modest storage resources.

The complexity of Foolscap comes with other burdens. The complexity makes a security audit more difficult. This is largely a consequence of the complexity making the code more difficult to understand - partly a result of the inherent complexity of Foolscap, partly a result of Foolscap simply being a relatively narrowly used protocol (and so almost no developers coming to Tahoe-LAFS will have prior experience with it).

For all these reasons, it would be greatly beneficial if Tahoe-LAFS storage nodes exposed their functionality via a different protocol - a protocol which avoids some of these shortcomings of Foolscap.

To sum up, the dimensions for consideration are (in no particular order):

• the existence of implementations for many languages/runtimes
• the degree to which the protocol is considered "standard"
• the runtime performance of the protocol for the necessary operations
• the implementation complexity of the protocol
• the provision by the protocol of the security properties Tahoe-LAFS requires (or the ability to build these on top of the protocol with minimal additional complexity)

With all these in mind, only one serious candidate really comes to mind. TLS (with a certain peer validation and verification configuration) carrying HTTP (either 1.2 or 2) carrying a raw data and/or a compact general-purpose serialization format (such as BSON - but given standardized content negotiation features of HTTP, multiple formats may be possible).

Supposing this line of reasoning is valid, it then remains to detail the precise network interface which can replace the current Foolscap-based interface (while addressing the above outlined concerns). This is the focus of this ticket.

Then, as a follow-up, the protocol can be implemented for both the storage and client nodes, and released (alongside the existing Foolscap protocol). After an acceptable transitional period, the Foolscap protocol can be removed (presumably with some period of deprecation-style notice to end users to ensure they don't upgrade themselves into a broken configuration accidentally).