[tahoe-dev] [tahoe-lafs] #670: large file download has bad alacrity

[tahoe-lafs]

#670: large file download has bad alacrity
---------------------------+------------------------------------------------
 Reporter:  zooko          |           Owner:       
     Type:  defect         |          Status:  new  
 Priority:  major          |       Milestone:  1.3.2
Component:  code-encoding  |         Version:  1.3.0
 Keywords:                 |   Launchpad_bug:       
---------------------------+------------------------------------------------
 A 10 GB file has about 80,000 segments, and the download process first
 downloads the entire Merkle Tree, which means 80,000 32-byte hashes.  For
 some reason building the Merkle Hash Tree object with
 http://allmydata.org/trac/tahoe/browser/src/allmydata/hashtree.py takes
 more than an hour.  Could be slowness in our SHA256d implementation, could
 be something in the hash tree data structure.  Brian is busy measuring
 exactly how long it takes to build hash trees of various sizes even as I
 write this.

 This means it takes a long time before it even begins downloading the file
 contents.  Apache's reverse proxy sometimes times-out while waiting for
 the file download to start, and wget can time-out on this as well.

 The best fix is probably to download and use only the subset of the hashes
 of Merkle Tree that we actually need for the current block.  Another
 improvement would be to optimize the construction of the hash tree object
 with those hashes.

-- 
Ticket URL: <http://allmydata.org/trac/tahoe/ticket/670>
tahoe-lafs <http://allmydata.org>
secure decentralized file storage grid