[tahoe-dev] blocks instead of files?

[Zooko O'Whielacronx]

On Wed, Mar 10, 2010 at 11:14 AM, Jody Harris <havoc at harrisdev.com> wrote:
> Has the Tahoe Dev ever discussed the possibility of encrypting and storing
> blocks instead of files?
> I don't know:
> 1) if this has already been discussed, or

Yes. Tahoe-LAFS's predecessor -- a proprietary product code-named
"Mountain View" which was written by allmydata.com -- and Mountain
View's predecessors -- Mnet and Mojo Nation -- did this.

> If the encrypt/store algorithms encrypt in (for example) 128 KB blocks,
> then
>  - you would never hit the large memory overhead problems seen with 100 MB
> and larger files.
>  - processing and storage would just take the file one chunk at a time
>  - retrieval and decryption (seems) that it would stream nicely since
> allocation of the next block(s) could take place while the current block is
> streaming out.

This is all currently the case for immutable files. Our plan is to
make it be the case for mutable files as well. (This is one of the
reasons why we recommend immutable files instead of mutable files for
most uses. Another is that immutability is a nice property that can
make it easier to keep your stuff organized.)

Note that the question of processing and transferring segments of the
file is somewhat independent of the question of storing segments of
the file on different servers. We want to continue storing shares of a
file all on one server in order to correlate failures (there is an
interesting technical reason why it is safer to do it this way than to
spread out blocks from one share to multiple servers) and in order to
reduce the number of different servers that the client has to talk to
and the number of different objects that the clients and servers have
to keep track of.

Regards,

Zooko