Could a gatewayless architecture improve performance or avoid some webapi security issues?

[davidsarah]

If clients contacted storage servers directly rather than via the gateway, to what extent could that improve performance and scalability by removing a potential bottleneck?

Possible downsides:

• the gateway cannot act as a cache
• the clients become more complicated
• the clients need direct connectivity to the storage servers
• the clients become dependent on the storage server protocol.

[warner]

I'm confused.. in our current terminology, the "gateway" is the component which implements the Tahoe protocols (specifically the encryption, integrity-checking, and erasure-coding parts). Clients speak with the gateway using source:docs/frontends/webapi.txt, and the gateway speaks with storage servers about getting/putting/modifying shares.

For clients to "contact storage servers directly", they would have to implement all of the tahoe protocols/formats/algorithms that are currently in the "gateway", so instead of e.g. a javascript application speaking webapi GETs and POSTs, such an application would have to perform AES, SHA256, RSA, FEC, and then speak Foolscap to the servers.

We've discussed changing the storage server protocol to be more HTTP like, so it's at least feasible to lower the bar for talking to storage servers (i.e. I'd like some day to implement tahoe in javascript, and changing the server protocol would be one fewer barrier to that task).

So I don't think I understand what you're proposing, since "clients" (as distinguished from a "gateway") would have to be enhanced drastically to allow them to speak directly to storage servers and still achieve the normal tahoe properties. Perhaps I'm misunderstanding what you mean by "gateway"?

[davidsarah]

Replying to warner:

For clients to "contact storage servers directly", they would have to implement all of the tahoe protocols/formats/algorithms that are currently in the "gateway", so instead of e.g. a javascript application speaking webapi GETs and POSTs, such an application would have to perform AES, SHA256, RSA, FEC, and then speak Foolscap to the servers.

Yes, that's what I meant.

This doesn't have any advantage unless the gateway is a performance bottleneck (in terms of either CPU or bandwidth), because it is shared by too many clients. OTOH, in that case it is possible for each client to run a gateway on localhost, although that potentially runs into security problems such as #870.

[zooko]

I also don't understand. Suppose you put the current gateway implementation into the same process as the client, and had the client communicate to the gateway through direct in-process communication instead of through HTTP. Would that be a "gatewayless architecture"? If so, then suppose the security problems of using HTTP over localhost such as #870 were solved. Would then HTTP-over-localhost be just as good as the gateway-in-process?

[davidsarah]

Replying to zooko:

I also don't understand. Suppose you put the current gateway implementation into the same process as the client, and had the client communicate to the gateway through direct in-process communication instead of through HTTP. Would that be a "gatewayless architecture"?

Yes.

If so, then suppose the security problems of using HTTP over localhost such as #870 were solved. Would then HTTP-over-localhost be just as good as the gateway-in-process?

Yes -- possibly with slightly worse latency, but that's probably not significant.

If the webapi could be eliminated completely, that would be an overall simplification, but it probably can't be (because we still want to support the WUI).

I was prompted to write this ticket because ​Octavia is gatewayless, and claims a performance improvement and other benefits from that, but I think it is too different from Tahoe for the same arguments to apply. Feel free to wontfix.

[zooko]

I would like to understand better what performance advantages or other advantages are alleged to redound from a gatewayless architecture. I'm skeptical that there are fundamental advantages to it that a gateway architecture can't match.

On the other hand, there are some current issues that our system has that a gatewayless architecture would not have that we ought to pay attention to. These include the #870 that David-Sarah just opened, and #320 (add streaming (on-line) upload to HTTP interface). Tahoe-LAFS will always incur a performance penalty compared to a gatewayless architecture like Octavia unless #320 is fixed.

[davidsarah]

Replying to zooko:

I would like to understand better what performance advantages or other advantages are alleged to redound from a gatewayless architecture. I'm skeptical that there are fundamental advantages to it that a gateway architecture can't match.

The main page on Octavia seems to be comparing to systems that are more centralized than Tahoe, where you can't just run another gateway.

On the other hand, there are some current issues that our system has that a gatewayless architecture would not have that we ought to pay attention to.

Another important one is #822 ('Web API should use a more reliable, out-of-band means of reporting errors when a server connection is lost during a download').

[warner]

So, I think it's a question of use cases. We could imagine a "libtahoe.a", some piece of code which is compiled into the application of your choice, and implements the Tahoe protocol. Now, how would you use this? How much work would it take to modify your favorite storage-hungry application to use libtahoe? Who would have to do that work?

Our "gateway" process is effectively a webserver with a specific API that's been linked against "libtahoe", lowering the bar for integration with other apps: instead of learning about the libtahoe interface and writing code to use it and integrating the necessary runtime code (including an event loop), the app authors learn about the webapi conventions and use their probably-preexisting HTTP client code to achieve the interface.

We built tahoe in python and twisted because we knew it would be the best way to get it done quickly and cleanly (and reliably and maintainably, etc). Authors of applications which happen to already be written in python+twisted can import allmydata.client and use it like a "libtahoe", but applications written in other languages are not so lucky. It's worth thinking about writing an actual "libtahoe", say in C++ or something similar, but I'd expect it to be more difficult, partly because I'm not aware of a well-accepted event-driven reactor framework like Twisted, partly because python is just so durn easy. But first we'd want to talk to some likely application authors and find out what would be easiest for them to use. It might actually be easier for them to speak HTTP to localhost (or perhaps some other protocol which could be made more secure than basic HTTP) than to provide the sorts of runtime support that an externally-chosen event loop library demanded.

The "gateway" node was never intended as a performance-improving component.. it's always just been the user's agent that speaks the Tahoe protocol on their behalf. It should be run on nearby, trusted hardware, and only lives in a separate process than the applications because none of those applications were written in the right language.

[zooko]

Closing as wontfix. There is a lot of valuable information about the architecture in this ticket, and I hope that future Tahoe-LAFS hackers read it. As far as performance the main issue as far as I can tell is #320 (add streaming (on-line) upload to HTTP interface). As far as security, there are a surprisingly large number of subtle security issues in our gateway architecture (see the comments in this ticket), but it seems like those issues can be fixed without changing the gateway architecture. As far as ease-of-use, ease-of-programming, I would definitely be open to providing different APIs depending on what programmers ask for. So far our RESTful WAPI is very popular, and as far as I know nobody uses the in-process Python interface or the CLI or FUSE or anything else than the WAPI for programmatic use.