publish-and-subscribe to change events on mutables

[zooko]

Currently if a client wants to keep track of changes being made by someone else to a mutable, it has to poll that mutable. In Magic Folder, for example, we poll every few seconds. Polling is terrible! It has bad latency (a few seconds) and also bad load (new requests every few seconds).

Instead we should implement a publish-and-subscribe notification system by which clients can maintain an open (but idle) connection to a server, and then get low-latency, cheap notifications from servers about events.

The storage server should provide an API which does something like this:

remote_registerWatcher(watcher, storage_index, previous_watcher=None)

[zooko]

Passing the previous_watcher to the storage server is just a way to let the storage server free up resources by ceasing to send notifications to that old one. The client does *not* rely on the storage server's behavior with regard to that! The client implements a purely client-side ordering guarantee in which it synchronously disables the old watcher (the old watcher is effectively a revocable forwarder) before it sends this request to the server and gives the server access to the new watcher.

[warner]

To be specific:

• registerWatch would be a method of RIStorageServer, next to get_buckets() and slot_readv()
• the watcher would be called with no arguments, or maybe just the storage index as an argument: in particular, it would *not* be called with any contents of the file. It's purely a "something has changed, you might want to come back and read it soon" notification.
  • Daira pointed out that calling it with *some* contents of the file might let us optimize out a roundtrip from the subsequent mapupdate/read operation. To avoid making the server know about how mutable shares are arranged, we'd have to specify a range or a read-vector of which parts of the share to deliver along with the "something has changed" notification
  • I countered that it would take some difficult code changes to take advantage of that. Mutable reads do two share reads: one for the "mapupdate" phase (which reads at least the signed roothash/seqnum, and might optimistically read and cache a little bit more), then a second that reads the main data. To remove one of these, we'd need to change the mutable downloader to accept a blob of data from outside the usual control flow.
  • Also, really the notification would need to deliver the entire contents of the file, since there are almost never situations where some data has changed, but then you *don't* end up reading the entire share
• we're trying to avoid using notifyOnDisconnect here
• the idea is that every 5 minutes we do two things:
  • establish a watcher
  • once established, read the share
• replacing the watcher with a new one is just to avoid needing to pay attention to whether the old one is still working or not

We think it'd be a good idea to require that the mutable share exists before a caller can establish a watcher on it. Ideally we'd require that the caller proves knowledge of a readcap before allowing them to add the watcher, but our share formats don't really make that possible. Also maybe the API should let you watch multiple storage indicies at once. But maybe not.

At the next higher level up, I think we'll add a IMutableFileNode.watch(callback) object, and maybe unwatch, which will do a mapupdate of the mutable file, locate all the servers that currently have shares, then register a watcher on each one. Actually it should probably establish a loop which does:

• does a mapupdate on the file
• adds a watcher on each share that was found
• read the file contents
• wait 5 minutes
• re-establish the watchers
• re-read the file contents
• wait, etc

Basically we want a max-5-minutes-latency fallback in case the watcher connection fails, without trying to react to Foolscap's (unreliable / racy) notifyOnDisconnect feature (which we're trying to remove).

Also, note that when we move to an HTTP-based storage server, this will probably be implemented with an EventSource or WebSocket. The IMutableFileNode API should hide the details.

[exarkun]

There is currently a WIP PR for a WebSocket?-based design - ​https://github.com/tahoe-lafs/tahoe-lafs/pull/513