Opened at 2007-10-11T03:36:54Z
Last modified at 2009-12-13T03:01:59Z
#173 new enhancement
How does tahoe filesystem layer (lafs) handle lots of file-upload tasks?
| Reported by: | zooko | Owned by: | somebody |
|---|---|---|---|
| Priority: | major | Milestone: | eventually |
| Component: | dev-infrastructure | Version: | 0.6.1 |
| Keywords: | scalability upload performance memory | Cc: | |
| Launchpad Bug: |
Description
Rob, Peter, Justin and I just fixed up the commercial, proprietary cousin of Tahoe -- the http://allmydata.com product -- to behave better when the user asks it to upload 100,000 files ASAP. How would Tahoe handle a similar situation? To close this ticket, implement an automated measurement (akin to the memory and upload/download rate measurements) which measures how much RAM, wall-clock time, CPU load, etc. a Tahoe node requires to simultaneously upload 1000, 10,000, or 100,000 files.
Change History (9)
comment:1 Changed at 2007-10-19T23:11:38Z by zooko
- Version changed from 0.6.0 to 0.6.1
comment:2 Changed at 2007-12-18T00:08:53Z by zooko
- Keywords scaling added
comment:3 Changed at 2008-02-28T01:58:33Z by zooko
- Milestone changed from undecided to 0.9.0 (Allmydata 3.0 final)
comment:4 Changed at 2008-03-10T19:23:28Z by zooko
- Milestone changed from 0.9.0 (Allmydata 3.0 final) to 0.10.0
comment:5 Changed at 2008-03-10T19:57:45Z by zooko
- Summary changed from How does tahoe handle lots of simultaneous file-upload tasks? to How does tahoe filesystem layer (laughfs) handle lots of file-upload tasks?
This ticket is just for the tahoe filesystem layer ("laughfs"). See also #337 (how does the whole system including the Windows native client handle lots of file-upload tasks?).
This ticket is hereby vagueified to be unclear about whether we're talking about lots of parallel upload tasks or a lot of sequential upload tasks or some combination thereof.
comment:6 Changed at 2008-03-10T20:04:18Z by warner
incidentally, I'm -1 on the "laughfs" term. I have no idea what it refers to.
comment:7 Changed at 2008-03-21T22:31:09Z by zooko
- Summary changed from How does tahoe filesystem layer (laughfs) handle lots of file-upload tasks? to How does tahoe filesystem layer (lafs) handle lots of file-upload tasks?
comment:8 Changed at 2008-03-21T22:31:29Z by zooko
- Milestone changed from 0.10.0 to undecided
comment:9 Changed at 2009-12-13T03:01:59Z by davidsarah
- Keywords upload performance memory added; scaling removed
Fabrice, the entrepeneur behind Allmydata, says that our reference test should be 25,000 files and 100 GB -- "VERY WELL and RELIABLY!". He says this is necessary to be competitive with Carbonite, Mozy, etc.. He promises to raise the bar to 100,000 files and 500 GB as soon as we reach 25,000 files and 100 GB. ;-)
Putting this into the 0.9.0 (Allmydata 3.0 final) Milestone.