[tahoe-dev] Perf-related architecture question

[Brian Warner]

On 7/23/10 7:18 PM, Kyle Markley wrote:
> 
> I'll vary these parameters:
> 
> 1) pipeline_size (in WriteBucketProxy.__init__)
>   - What values would be interesting, and in what step size?
> 2) Segment size (ticket #809) (in class Client?)
>   - I'll try each power of two from 32KiB to 4MiB.
> 3) Any other parameters worth playing with?

Yeah, as Zooko said, I'd go with pipeline_size in [1,2,3,4,8,16] * segsize/K
for each value of segsize you mentioned. You might also make it somewhat
adaptive: for a given segsize, keep doubling pipeline size until the
throughput stops improving, do one more double for good measure, then fill
in some of the intermediate values so we can see the shape of the curve
a bit better. Then increase segsize and try it again. At some point
the increase in segsize won't improve things very much.

Note that an increased segsize is visible to downloaders in the form of
increased "alacrity": even if you just want to download a few bytes of
the file, you have to fetch a whole segment (to check the integrity
hashes), so a giant segsize may be maximally efficient for upload, but
less efficient for partial-file downloads. The pipeline size does affect
downloaders in this way.

looking forward to seeing the results.. thanks for doing this!
 -Brian