[tahoe-dev] [tahoe-lafs] #393: mutable: implement MDMF

[tahoe-lafs]

#393: mutable: implement MDMF
------------------------------+---------------------------------------------
     Reporter:  warner        |       Owner:  kevan                                                                    
         Type:  enhancement   |      Status:  assigned                                                                 
     Priority:  major         |   Milestone:  1.8.0                                                                    
    Component:  code-mutable  |     Version:  1.0.0                                                                    
   Resolution:                |    Keywords:  newcaps performance random-access privacy gsoc mdmf mutable review-needed
Launchpad Bug:                |  
------------------------------+---------------------------------------------

Comment (by kevan):

 Brian pointed out on IRC today that writing a mutable file segment-by-
 segment leads to regressions relative to SDMF behavior. If a writer dies
 in the middle of a mutable file update (i.e.: before writing all of the
 blocks), we can have a situation where there are no recoverable versions
 of the mutable file on the grid. In SDMF, each share is written all at
 once, so we have the property that each storage server with a share for a
 particular mutable file has a complete share (barring disk errors, network
 problems, etc), which makes SDMF more resilient to writer death than MDMF
 as currently implemented.

 Brian suggested that this could be resolved by making MDMF write shares
 all at once, as SDMF does. This preserves the semantics described above
 from SDMF, but means that modifying a mutable file would consume memory
 proportional to the size of the modification.

 Most of the user-level tools (at least the ones I'm familiar with) only
 modify mutable files by completely overwriting them, so many/most user-
 level mutable file operations will still consume memory proportional to
 the size of the file, as with SDMF. We can still use (unless I'm missing
 another regression) the block-by-block downloader that I'm working on now,
 so MDMF would give users streaming and memory-friendly downloads of large
 files, just not uploads.

 (part of MDMF is writing webapi functions to allow outside code to easily
 modify portions of mutable files, which would be easier on memory than
 modifying all of the mutable file, but many command-line tools have no
 easy way of taking advantage of such functionality)

 I'd really like to keep the memory efficient uploads, but I can't see a
 way to do it without that regression. Any ideas, audience at home?

-- 
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/393#comment:38>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized file storage grid