[tahoe-dev] Tahoe benchmarking data

[Kyle Markley]

Brian,

> The fastest data rate you're seeing here is 64MiB/14.80s, so about
> 4.47MB/s or roughly 35-40Mbps, which is probably about the middle of
> what you'd expect out of a 100Mbps ethernet (maybe a bit on the low
> side, but not by much). Was the client CPU pegged during the upload? I
> suspect you're CPU bound, but that overall your network is pretty close
> to being saturated too.

Yeah, I think we're approximately saturating the network during the large
file transfers.  But for the small files, both network and CPU load are
very low (under 10%).

I used systrace to count kernel calls during the large and small file
transfers.  The heavy end of the histograms are:

tahoe backup large files:
native-recvfrom(29),: 1593
native-fstat(292),: 2542
native-mmap(197),: 2668
native-sigprocmask(48),: 3183
native-stat(291),: 3367
native-gettimeofday(116),: 4122
native-mprotect(74),: 4167
native-fcntl(92),: 4646
native-read(3),: 5162
native-poll(252),: 8206
native-open(5),: 11813
native-sendto(133): 12296

tahoe backup small files:
native-gettimeofday(116),: 5700
native-mprotect(74),: 6205
native-poll(252),: 7344
native-stat(291),: 7443
native-read(3),: 9235
native-fstat(292),: 11692
native-sendto(133): 11922
native-open(5),: 14857
native-fcntl(92),: 34077
native-write(4),: 38708
native-lseek(199),: 45015
native-recvfrom(29),: 186860

daemon process, combined for both transfers:
native-recvfrom(29),: 9260
native-getpid(20),: 10350
native-open(5),: 27059
native-fcntl(92),: 27200
native-mmap(197),: 29567
native-read(3),: 30886
native-munmap(73),: 47465
native-write(4),: 76731
native-poll(252),: 124552
native-gettimeofday(116),: 291893


The large number of recvfrom calls on the tahoe backup of small files
looks odd because it's a huge mismatch against the sendto calls, but
nothing else stands out.

I wonder whether temporary file creation for the small file transfers
might be part of the problem.  I know that temporary files are created on
occasion; could someone explain precisely when?


> The pipeline will really become important when your client-to-server
> latency is more than the fraction of a millisecond that you probably get
> on your LAN. How do ping times compare between the wired and the
> wireless connections? On this network I'd expect them to be too close to
> have a huge effect on tahoe's performance, although clearly there *some*
> kind of effect. The one thing that I know wireless networks are more
> susceptible to is up/down interference: the ACK coming back from the
> server will contend with the next data packet coming from the client
> (it'd be worse if both client and server are on wireless connections).

Ping to the storage node through the wired interface is about 0.66ms.
Ping to the storage node through the wireless interface is about 1.25ms.

-- 
Kyle Markley