[tahoe-dev] blocks instead of files?

[Brian Warner]

David-Sarah Hopwood wrote:
> Kevan Carstensen wrote:
>> At 2010-03-10 23:27 (+0000), David-Sarah Hopwood wrote:
>>
>>> That it isn't true for mutable files is, as far as I know, just a
>>> temporary limitation of the current mutable download/upload code
>>> (which does work in terms of segments, but hasn't been optimized to
>>> reduce its memory usage by doing so).
>> This is true in the sense that the entire mutable file is treated as
>> one segment, AFAIK. Is that what you meant?
> 
> In the sense that the storage format and the 'mutable retrieve' code
> still works in terms of arbitrary-length segments (as far as I can
> tell), but the 'mutable publish' code happens to choose the segment
> size so that there is only one segment.

I tried to make that true when I wrote the mutable code, but I'm not
sure I pulled it off. There's a hash tree that's prepared to handle
multiple blocks (just like CHK), which would let you use normal-sized
128KiB segments (just like CHK), but since the download-size Retrieve
class is hard-wired to expect a single segment, we couldn't change that
without breaking existing clients. So even though the protocol nominally
has room for this improvement, in practice we'll probably have to have
some sort of version bump.

Also, if we want to be able to update just a portion of the file
(without rewriting the whole thing), then the protocol will need to be
extended to have a unique salt per segment. (this would of course reveal
which segments you're modifying to the server and everybody else). That
would certainly require a big version bump.

So it may be the case that SDMF is O(filesize) RAM, and we need to
define+build+deploy MDMF to fix that.


cheers,
 -Brian