[tahoe-dev] Tahoe benchmarking data

[Zooko O'Whielacronx]

Dear Kyle:

I'm grateful to you for spending your time to run these benchmarks and
report the results to us.

I'm wondering what we can do to make sure that these benchmarks serve
their purpose and don't just get wasted.

The best thing, of course, would be if we could make them automated so
that you could re-run them (for example, using Brian's New Downloader)
without spending too much time on it, and so that other people could
re-run them to confirm your results.

Also, perhaps we should put them up on a wiki page, with a link to
this mailing list thread.

I really don't have an explanation about the anomalies you noted:

On Sun, Jul 25, 2010 at 2:02 PM, Kyle Markley <kyle at arbyte.us> wrote:
>
> For small files, the results are surprising!  The default settings
> (pipeline 50000 segsize 128KiB) give significantly better performance on
> the wired LAN than anything else I tried -- this is a huge outlier when
> compared to the other data points.  What explains this?  Why does the
> wireless network perform so much better than the wired network for small
> files (except for the single outlier)?  Why are small segments so much
> worse on wireless for small files but not for large files?
>
> Hopefully someone can get some answers out of the .tcp files in the
> attached archive.

When you do a measurement like this then one of two things happen:
either you learn some nice understandable pattern that you can act on,
such as saying "Oh, I guess we should set the default pipeline size to
be bigger.", or you get a mystery. If you get a mystery, then either
you give up, or you eventually solve the mystery. If you solve it,
then either you get a disappointing realization that your measurement
was wrong, or you get a really great breakthrough for improving the
code. :-)

So, what can we do to solve this mystery? I can think of two possibilities:

a. Look at the packet trace your captured. Do you remember which run
it was from?

b. Reproduce this. Make it scripted so that it can be reproduced with
less work. You reproduce it on your network and see if you get the
same results. Other people reproduce it on their networks. Everyone
apply Brian's New Downloader patches
http://tahoe-lafs.org/trac/tahoe-lafs/ticket/798 and reproduce it.

Regards,

Zooko