Ticket #1291: perf-doc.darcs.patch

1 patch for repository zooko@dev.allmydata.org:/home/darcs/tahoe-lafs/trunk:

Mon Jan  3 23:54:55 MST 2011  zooko@zooko.com
  * docs: update performance.rst to describe the difference between already-uploaded and not-already-uploaded, to parameterize segment size, and to use "~A" to mean "approximately A"

New patches:

[docs: update performance.rst to describe the difference between already-uploaded and not-already-uploaded, to parameterize segment size, and to use "~A" to mean "approximately A"
zooko@zooko.com**20110104065455
 Ignore-this: 8df0d79a062ee19854c0211bd202f606
] {
hunk ./docs/performance.rst 17
 10. `Performing a file-verify on an A-byte file`_
 11. `Repairing an A-byte file (mutable or immutable)`_
 
+``K`` indicates the number of shares required to reconstruct the file
+(default: 3)
+
+``N`` indicates the total number of shares produced (default: 10)
+
+``S`` indicates the segment size (default: 128 KiB)
+
+``A`` indicates the number of bytes in a file
+
+``B`` indicates the number of bytes of a file which are being read or
+written
+
+``G`` indicates the number of storage servers on your grid
+
 Publishing an ``A``-byte immutable file
 =======================================
 
hunk ./docs/performance.rst 34
-network: A
+when the file is already uploaded
+---------------------------------
+
+If the file is already uploaded with the exact same contents, same
+erasure coding parameters (K, N), and same added convergence secret,
+then it reads the whole file from disk one time while hashing it to
+compute the storage index, then contacts about N servers to ask each
+one to store a share. All of the servers reply that they already have
+a copy of that share, and the upload is done.
+
+disk: A
+
+cpu: ~A
+
+network: ~N
+
+memory footprint: N/K*S
+
+when the file is not already uploaded
+-------------------------------------
+
+If the file is not already uploaded with the exact same contents, same
+erasure coding parameters (K, N), and same added convergence secret,
+then it reads the whole file from disk one time while hashing it to
+compute the storage index, then contacts about N servers to ask each
+one to store a share. Then it uploads each share to a storage server.
+
+disk: 2*A
+
+cpu: 2*~A
 
hunk ./docs/performance.rst 65
-memory footprint: N/k*128KiB
+network: ~N + ~A
 
hunk ./docs/performance.rst 67
-notes: An immutable file upload requires an additional I/O pass over the entire
-source file before the upload process can start, since convergent
-encryption derives the encryption key in part from the contents of the
-source file.
+memory footprint: N/K*S
 
 Publishing an ``A``-byte mutable file
 =====================================
hunk ./docs/performance.rst 74
 
 network: A
 
-memory footprint: N/k*A
+memory footprint: N/K*A
 
hunk ./docs/performance.rst 76
-cpu: O(A) + a large constant for RSA keypair generation
+cpu: ~A + a large constant for RSA keypair generation
 
 notes: Tahoe-LAFS generates a new RSA keypair for each mutable file that it
 publishes to a grid. This takes up to 1 or 2 seconds on a typical desktop PC.
hunk ./docs/performance.rst 91
 
 network: B
 
-memory footprint: 128KiB
+cpu: ~A
 
hunk ./docs/performance.rst 93
-notes: When Tahoe-LAFS 1.8.0 or later is asked to read an arbitrary range
-of an immutable file, only the 128-KiB segments that overlap the
+notes: When Tahoe-LAFS 1.8.0 or later is asked to read an arbitrary
+range of an immutable file, only the S-byte segments that overlap the
 requested range will be downloaded.
 
 (Earlier versions would download from the beginning of the file up
hunk ./docs/performance.rst 117
 
 network: A
 
-memory footprint: N/k*A
+memory footprint: N/K*A
 
 notes: If you upload a changed version of a mutable file that you
 earlier put onto your grid with, say, 'tahoe put --mutable',
hunk ./docs/performance.rst 132
 
 network: A
 
-memory footprint: N/k*A
+memory footprint: N/K*A
 
 notes: Modifying any part of a mutable file in Tahoe-LAFS requires that
 the entire file be downloaded, modified, held in memory while it is
hunk ./docs/performance.rst 147
 Adding an entry to an ``A``-entry directory
 ===========================================
 
-network: O(A)
+network: ~A
 
hunk ./docs/performance.rst 149
-memory footprint: N/k*A
+memory footprint: N/K*A
 
 notes: In Tahoe-LAFS, directories are implemented as specialized mutable
 files. So adding an entry to a directory is essentially adding B
hunk ./docs/performance.rst 158
 Listing an ``A`` entry directory
 ================================
 
-network: O(A)
+network: ~A
 
hunk ./docs/performance.rst 160
-memory footprint: N/k*A
+memory footprint: N/K*A
 
 notes: Listing a directory requires that the mutable file storing the
 directory be downloaded from the grid. So listing an A entry
hunk ./docs/performance.rst 170
 Performing a file-check on an ``A``-byte file
 =============================================
 
-network: O(S), where S is the number of servers on your grid
+network: ~G, where G is the number of servers on your grid
 
 memory footprint: negligible
 
hunk ./docs/performance.rst 182
 Performing a file-verify on an ``A``-byte file
 ==============================================
 
-network: N/k*A
+network: N/K*A
 
hunk ./docs/performance.rst 184
-memory footprint: N/k*128KiB
+memory footprint: N/K*S
 
 notes: To verify a file, Tahoe-LAFS downloads all of the ciphertext
 shares that were originally uploaded to the grid and integrity
hunk ./docs/performance.rst 195
 Repairing an ``A``-byte file (mutable or immutable)
 ===================================================
 
-network: variable; up to around O(A)
+network: variable; up to around ~A
 
hunk ./docs/performance.rst 197
-memory footprint: from 128KiB to (1+N/k)*128KiB
+memory footprint: from S to (1+N/K)*S
 
 notes: To repair a file, Tahoe-LAFS downloads the file, and generates/uploads
 missing shares in the same way as when it initially uploads the file.
}

Context:

[NEWS: update entry for removal of Mac and Windows apps. refs #1282
david-sarah@jacaranda.org**20101226042245
 Ignore-this: c8099bc6e8235718d042c9a13c1e2425
] 
[Move dependency imports from windows/depends.py (which has gone away) into src/allmydata/windows/tahoesvc.py. Also fix a pyflakes warning, and change the service display name from 'Allmydata Tahoe Node' to 'Tahoe-LAFS node'. refs #1282
david-sarah@jacaranda.org**20101226042100
 Ignore-this: ee45f324934e1251380206dbee6346d0
] 
[Remove unmaintained Windows GUI app, except for windows/tahoesvc.py which is moved to src/allmydata/windows. refs #1282
david-sarah@jacaranda.org**20101226040237
 Ignore-this: cae37b6622a7dd5940acc7d3e6a98b90
] 
[Remove the Makefile targets relating to the Mac GUI app. refs #1282
david-sarah@jacaranda.org**20101226025859
 Ignore-this: 75303be783974b41138744ec62b07965
] 
[NEWS: remove unmaintained Mac GUI app. refs #1282
david-sarah@jacaranda.org**20101226020858
 Ignore-this: 40474a07f4a550b48563d35350be7ab5
] 
[Remove unmaintained Mac GUI app. fixes #1282
david-sarah@jacaranda.org**20101226020508
 Ignore-this: b3613bf1abfd284d542bf7c753ec557a
] 
[Remove src/allmydata/util/find_exe.py which is no longer used. fixes #1150
david-sarah@jacaranda.org**20101226023206
 Ignore-this: 7436c9b53bf210aed34a1a973cd9cace
] 
[status_web_pages_review.darcs.patch 
freestorm77@gmail.com**20110102034214
 Ignore-this: 29f1ecb36177f10f3f846b3d56b313b2
 
 I make some changes on status web pages
 
 status.xhtml:
 - Delete unused webform_css link
 - Align tables on the left
 
 tahoe-css:
 - Do some minor changes on code synthax
 - changes table.status-download-events style to look like other tables
 
 status.py:
 - Align table on the left
 - Changes table header
 - Add heading tags
 - Modify google api graph: add image border, calculate height to feet data
 
 signed-off-by: zooko@zooko.com
 fixes #1219
] 
[test_storage.py: fix a pyflakes unused import warning.
david-sarah@jacaranda.org**20101231220756
 Ignore-this: df08231540cb7dff9d2b038e47ab30ee
] 
[test_storage.py: leave at least 512 MiB free when running test_large_share. refs #1195
david-sarah@jacaranda.org**20101231203215
 Ignore-this: b2144c0341c3452b5d4ba219e284ea0e
] 
[storage: use fileutil's version of get_disk_stats() and get_available_space(), use mockery/fakery in tests, enable large share test on platforms with sparse files and if > 4 GiB of disk space is currently available
zooko@zooko.com**20100910173629
 Ignore-this: 1304f1164c661de6d5304f993eb9b27b
] 
[fileutil: copy in the get_disk_stats() and get_available_space() functions from storage/server.py
zooko@zooko.com**20100910173520
 Ignore-this: 8b15569715f710f4fc5092f7ca109253
] 
[Update foolscap version requirement to 0.6.0, to address http://foolscap.lothar.com/trac/ticket/167
david-sarah@jacaranda.org**20101231060039
 Ignore-this: 98d2b8086a1a500b9f4565bca5a3810
] 
[docs/webapi.rst: typos.
david-sarah@jacaranda.org**20101230034422
 Ignore-this: d1f5166d72cc711f7e0d9981eac9105e
] 
[docs/webapi.rst: capitalization, formatting of section on URL character encoding, and a correction about Internet Explorer.
david-sarah@jacaranda.org**20101230034049
 Ignore-this: b3b9819d2fb264b4cdc5c8afd4e8c48d
] 
[docs: corrections and clarifications.
david-sarah@jacaranda.org**20101227051056
 Ignore-this: e33202858c7644c58f3f924b164294b6
] 
[docs: more formatting cleanups and corrections. Spell webapi and wapi as web-API.
david-sarah@jacaranda.org**20101227050533
 Ignore-this: 18b23cbfb780df585d8a722a1ec63e94
] 
[docs/debian.rst: bring description of building dependencies from source up-to-date, and change hostname from allmydata.com to tahoe-lafs.org.
david-sarah@jacaranda.org**20101212222912
 Ignore-this: f38462afc88b4475195610385a28391c
] 
[docs/architecture.rst: correct rst syntax.
david-sarah@jacaranda.org**20101212202003
 Ignore-this: 3fbe12feb28bec6f1c63aedbc79aad21
] 
[docs/architecture.rst: formatting.
david-sarah@jacaranda.org**20101212201719
 Ignore-this: 305fa5dfc2939355eaf6d0d2161eb1ff
] 
[docs: linkification, wording improvements.
david-sarah@jacaranda.org**20101212201234
 Ignore-this: 4e67287f527a8bc728cfbd93255d2aae
] 
[docs: formatting.
david-sarah@jacaranda.org**20101212201115
 Ignore-this: 2e0ed394ac7726651d3a4f2c4b0d3798
] 
[docs/configuration.rst: more formatting tweaks; which -> that.
david-sarah@jacaranda.org**20101212195522
 Ignore-this: a7becb7021854ca5a90edd892b36fdd7
] 
[docs/configuration.rst: more changes to formatting.
david-sarah@jacaranda.org**20101212194511
 Ignore-this: 491aac33e5f5268d224359f1447d10be
] 
[docs/configuration.rst: changes to formatting (mainly putting commands and filenames in monospace).
david-sarah@jacaranda.org**20101212181828
 Ignore-this: 8a1480e2d5f43bee678476424615b50f
] 
[scripts/backupdb.py: more accurate comment about path field.
david-sarah@jacaranda.org**20101212170320
 Ignore-this: 50e47a2228a85207bbcd188a78a0d4e6
] 
[scripts/cli.py: fix missing 'put' in usage example for 'tahoe put'.
david-sarah@jacaranda.org**20101212170207
 Ignore-this: 2cbadf066fff611fc03d3c0ff97ce6ec
] 
[docs/frontends/CLI.rst: changes to formatting (mainly putting commands and filenames in monospace), and to command syntax to reflect that DIRCAP/... is accepted. Clarify the syntax of 'tahoe put' and other minor corrections. Tahoe -> Tahoe-LAFS.
david-sarah@jacaranda.org**20101212165800
 Ignore-this: a123ef6b564aa8624d1e79c97068ea12
] 
[docs/frontends/CLI.rst: Unicode arguments to 'tahoe' work on Windows as of v1.7.1.
david-sarah@jacaranda.org**20101212063740
 Ignore-this: 3977a99dfa86ac33a44171deaf43aaab
] 
[docs/known_issues.rst: fix title and linkify another URL. refs #1225
david-sarah@jacaranda.org**20101212062817
 Ignore-this: cc91287f7fb51c23440b3d2fe79c449c
] 
[docs/known_issues.rst: fix an external link. refs #1225
david-sarah@jacaranda.org**20101212062435
 Ignore-this: b8cbf12f353131756c358965c48060ec
] 
[Fix a link from uri.rst to dirnodes.rst. refs #1225
david-sarah@jacaranda.org**20101212054502
 Ignore-this: af6205299f5c9a33229cab259c00f9d5
] 
[Fix a link from webapi.rst to FTP-and-SFTP.rst. refs #1225
david-sarah@jacaranda.org**20101212053435
 Ignore-this: 2b9f88678c3447ea860d6b61e8799858
] 
[More specific hyperlink to architecture.rst from helper.rst. refs #1225
david-sarah@jacaranda.org**20101212052607
 Ignore-this: 50424c768fca481252fabf58424852dc
] 
[Update hyperlinks between docs, and linkify some external references. refs #1225
david-sarah@jacaranda.org**20101212051459
 Ignore-this: cd43a4c3d3de1f832abfa88d5fc4ace1
] 
[docs/specifications/dirnodes.rst: fix references to mutable.rst. refs #1225
david-sarah@jacaranda.org**20101212012720
 Ignore-this: 6819b4b4e06e947ee48b365e840db37d
] 
[docs/specifications/mutable.rst: correct the magic string for v1 mutable containers. refs #1225
david-sarah@jacaranda.org**20101212011400
 Ignore-this: 99a5fcdd40cef83dbb08f323f6cdaaca
] 
[Move .txt files in docs/frontends and docs/specifications to .rst. refs #1225
david-sarah@jacaranda.org**20101212010251
 Ignore-this: 8796d35d928370f7dc6ad2dafdc1c0fe
] 
[Convert docs/frontends and docs/specifications to reStructuredText format (not including file moves).
david-sarah@jacaranda.org**20101212004632
 Ignore-this: e3ceb2d832d73875abe48624ddbb5622
] 
[scripts/cli.py: remove the disclaimer in the help for 'tahoe cp' that it does not handle non-ASCII filenames well. (At least, we intend to handle them.)
david-sarah@jacaranda.org**20101130002145
 Ignore-this: 94c003efaa20b9eb4a83503d79844ca
] 
[relnotes.txt: fifth -> sixth labor-of-love release
zooko@zooko.com**20101129045647
 Ignore-this: 21c245015268b38916e3a138d256c09d
] 
[Makefile: BB_BRANCH is set to the empty string for trunk, not the string 'trunk'.
david-sarah@jacaranda.org**20101128233512
 Ignore-this: 5a7ef8eb10475636d21b91e25b56c369
] 
[relnotes.txt: eleventh -> twelfth release.
david-sarah@jacaranda.org**20101128223321
 Ignore-this: 1e26410156a665271c1170803dea2c0d
] 
[relnotes.tst: point to known_issues.rst, not known_issues.txt.
david-sarah@jacaranda.org**20101128222918
 Ignore-this: 60194eb4544cac446fe4f60b3e34b887
] 
[quickstart.html: fix link to point to allmydata-tahoe-1.8.1.zip.
david-sarah@jacaranda.org**20101128221728
 Ignore-this: 7b3ee86f8256aa12f5d862f689f3ee29
] 
[TAG allmydata-tahoe-1.8.1
david-sarah@jacaranda.org**20101128212336
 Ignore-this: 9c18bdeaef4822f590d2a0d879e00621
] 
Patch bundle hash:
4d8ada6ea9d68920ae22f7d34a651554b45b3d07