Benchmarks for basic immutable Tahoe operations (upload and download)

[itamarst]

Motivation:

1. We don't want HTTP to be slower than Foolscap.
2. It's not clear how fast these operations are in general.

Thus: there should be a way to measure how long these operations take.

First pass will run over loopback, but latency can be added-on on Linux (e.g. ​https://bencane.com/2012/07/16/tc-adding-simulated-network-latency-to-your-linux-server/).

[itamarst]

For a small 2000 byte file with a single share uploaded to a single node, an upload is 3 sequential HTTP requests, and a download is 4 sequential HTTP requests. So with high latency this gets quite slow.

Next, I tested over loopback, but with RTT latency of 60ms (30ms one way) artificially added. Bandwidth is quite high, so not likely to be a bottleneck.

• When uploading the 2000 byte file, upload time was 300ms (30ms CPU), and download was 300ms (65ms CPU).
• When uploading a 2MB file, upload time was 700ms (70ms are CPU), and download time was 3200ms (680ms is CPU).
• For 20MB file: upload 3,550ms (CPU 600ms), download 31,200ms (6,200ms CPU).

This is all sharing a single CPU core for both client and server, it's one big process.

So at first glance:

• Immutable uploads use parallelism, and main bottleneck is latency. Upload time is not linear with size, . However, sequential uploads will have minimal time no matter what the bandwidth.
• Immutable downloads have a massive latency hit, probably due to not utilizing parallelism well, so large files suffer from latency hit even when bandwidth is not a bottleneck. Also using a lot of CPU.

[itamarst]

Changing default segment size to be bigger helps a lot (mostly a workaround).

[itamarst]

The problems with increasing default segment size:

• Some code actually relies on it (e.g. ​https://github.com/tahoe-lafs/tahoe-lafs/blob/master/src/allmydata/immutable/downloader/node.py#L74) though _hopefully_ that's just an optimization. Would need testing of all relevant places that rely on it, at least.
• It doesn't help with existing downloads.