Ticket #1225: docs-txt-to-rst.darcs.patch

File docs-txt-to-rst.darcs.patch, 138.0 KB (added by zooko, at 2010-10-15T05:36:34Z)

renamed the files from .txt to .rst, fixed a formatting issue in backdoors.rst, added NEWS entry, added darcs patch description (commit log entry)

Line 
11 patch for repository zooko@dev.allmydata.org:/home/darcs/tahoe-lafs/trunk:
2
3Thu Oct 14 23:08:22 MDT 2010  zooko@zooko.com
4  * docs: convert all .txt docs to .rst thanks to Ravi Pinjala
5  fixes #1225
6
7New patches:
8
9[docs: convert all .txt docs to .rst thanks to Ravi Pinjala
10zooko@zooko.com**20101015050822
11 Ignore-this: f2d4ec8a3205622a9332e584c54a8fd8
12 fixes #1225
13] {
14move ./docs/architecture.txt ./docs/architecture.rst
15move ./docs/backdoors.txt ./docs/backdoors.rst
16move ./docs/backupdb.txt ./docs/backupdb.rst
17move ./docs/configuration.txt ./docs/configuration.rst
18move ./docs/debian.txt ./docs/debian.rst
19move ./docs/filesystem-notes.txt ./docs/filesystem-notes.rst
20move ./docs/garbage-collection.txt ./docs/garbage-collection.rst
21move ./docs/helper.txt ./docs/helper.rst
22move ./docs/how_to_make_a_tahoe-lafs_release.txt ./docs/how_to_make_a_tahoe-lafs_release.rst
23move ./docs/known_issues.txt ./docs/known_issues.rst
24move ./docs/logging.txt ./docs/logging.rst
25move ./docs/performance.txt ./docs/performance.rst
26move ./docs/stats.txt ./docs/stats.rst
27hunk ./NEWS 3
28 User visible changes in Tahoe-LAFS.  -*- outline; coding: utf-8 -*-
29 
30+* Release 1.8.1 (coming)
31+
32+** Documentation
33+
34+ - All .txt documents have been converted to .rst format (#1225)
35+
36 * Release 1.8.0 (2010-09-23)
37 
38 ** New Features
39hunk ./docs/architecture.rst 1
40-= Tahoe-LAFS Architecture =
41+=======================
42+Tahoe-LAFS Architecture
43+=======================
44 
45hunk ./docs/architecture.rst 5
46-1.  Overview
47-2.  The Key-Value Store
48-3.  File Encoding
49-4.  Capabilities
50-5.  Server Selection
51-6.  Swarming Download, Trickling Upload
52-7.  The Filesystem Layer
53-8.  Leases, Refreshing, Garbage Collection
54-9.  File Repairer
55-10. Security
56-11. Reliability
57+1.  `Overview`_
58+2.  `The Key-Value Store`_
59+3.  `File Encoding`_
60+4.  `Capabilities`_
61+5.  `Server Selection`_
62+6.  `Swarming Download, Trickling Upload`_
63+7.  `The Filesystem Layer`_
64+8.  `Leases, Refreshing, Garbage Collection`_
65+9.  `File Repairer`_
66+10. `Security`_
67+11. `Reliability`_
68 
69 
70hunk ./docs/architecture.rst 18
71-== Overview  ==
72+Overview
73+========
74 
75 (See the docs/specifications directory for more details.)
76 
77hunk ./docs/architecture.rst 46
78 copies files from the local disk onto the decentralized filesystem. We later
79 provide read-only access to those files, allowing users to recover them.
80 There are several other applications built on top of the Tahoe-LAFS
81-filesystem (see the RelatedProjects page of the wiki for a list).
82+filesystem (see the `RelatedProjects
83+<http://tahoe-lafs.org/trac/tahoe-lafs/wiki/RelatedProjects>`_ page of the
84+wiki for a list).
85 
86 
87hunk ./docs/architecture.rst 51
88-== The Key-Value Store ==
89+The Key-Value Store
90+===================
91 
92 The key-value store is implemented by a grid of Tahoe-LAFS storage servers --
93 user-space processes. Tahoe-LAFS storage clients communicate with the storage
94hunk ./docs/architecture.rst 85
95 server to tell a new client about all the others.
96 
97 
98-== File Encoding ==
99+File Encoding
100+=============
101 
102 When a client stores a file on the grid, it first encrypts the file. It then
103 breaks the encrypted file into small segments, in order to reduce the memory
104hunk ./docs/architecture.rst 127
105 into plaintext, then emit the plaintext bytes to the output target.
106 
107 
108-== Capabilities ==
109+Capabilities
110+============
111 
112 Capabilities to immutable files represent a specific set of bytes. Think of
113 it like a hash function: you feed in a bunch of bytes, and you get out a
114hunk ./docs/architecture.rst 153
115 that these potential bytes are indeed the ones that you were looking for.
116 
117 The "key-value store" layer doesn't include human-meaningful names.
118-Capabilities sit on the "global+secure" edge of Zooko's Triangle[1]. They are
119+Capabilities sit on the "global+secure" edge of `Zooko's Triangle`_. They are
120 self-authenticating, meaning that nobody can trick you into accepting a file
121 that doesn't match the capability you used to refer to that file. The
122 filesystem layer (described below) adds human-meaningful names atop the
123hunk ./docs/architecture.rst 159
124 key-value layer.
125 
126+.. _`Zooko's Triangle`: http://en.wikipedia.org/wiki/Zooko%27s_triangle
127 
128hunk ./docs/architecture.rst 161
129-== Server Selection ==
130+
131+Server Selection
132+================
133 
134 When a file is uploaded, the encoded shares are sent to some servers. But to
135 which ones? The "server selection" algorithm is used to make this choice.
136hunk ./docs/architecture.rst 169
137 
138 The storage index is used to consistently-permute the set of all servers nodes
139-(by sorting them by HASH(storage_index+nodeid)). Each file gets a different
140+(by sorting them by ``HASH(storage_index+nodeid)``). Each file gets a different
141 permutation, which (on average) will evenly distribute shares among the grid
142 and avoid hotspots. Each server has announced its available space when it
143 connected to the introducer, and we use that available space information to
144hunk ./docs/architecture.rst 268
145   significantly hurt reliability (sometimes the permutation resulted in most
146   of the shares being dumped on a single node).
147 
148-  Another algorithm (known as "denver airport"[2]) uses the permuted hash to
149+  Another algorithm (known as "denver airport" [#naming]_) uses the permuted hash to
150   decide on an approximate target for each share, then sends lease requests
151   via Chord routing. The request includes the contact information of the
152   uploading node, and asks that the node which eventually accepts the lease
153hunk ./docs/architecture.rst 277
154   the same approach. This allows nodes to avoid maintaining a large number of
155   long-term connections, at the expense of complexity and latency.
156 
157+.. [#naming]  all of these names are derived from the location where they were
158+        concocted, in this case in a car ride from Boulder to DEN. To be
159+        precise, "Tahoe 1" was an unworkable scheme in which everyone who holds
160+        shares for a given file would form a sort of cabal which kept track of
161+        all the others, "Tahoe 2" is the first-100-nodes in the permuted hash
162+        described in this document, and "Tahoe 3" (or perhaps "Potrero hill 1")
163+        was the abandoned ring-with-many-hands approach.
164+
165 
166hunk ./docs/architecture.rst 286
167-== Swarming Download, Trickling Upload ==
168+Swarming Download, Trickling Upload
169+===================================
170 
171 Because the shares being downloaded are distributed across a large number of
172 nodes, the download process will pull from many of them at the same time. The
173hunk ./docs/architecture.rst 318
174 See "helper.txt" for details about the upload helper.
175 
176 
177-== The Filesystem Layer ==
178+The Filesystem Layer
179+====================
180 
181 The "filesystem" layer is responsible for mapping human-meaningful pathnames
182 (directories and filenames) to pieces of data. The actual bytes inside these
183hunk ./docs/architecture.rst 349
184 that are globally visible.
185 
186 
187-== Leases, Refreshing, Garbage Collection ==
188+Leases, Refreshing, Garbage Collection
189+======================================
190 
191 When a file or directory in the virtual filesystem is no longer referenced,
192 the space that its shares occupied on each storage server can be freed,
193hunk ./docs/architecture.rst 371
194 garbage collection.
195 
196 
197-== File Repairer ==
198+File Repairer
199+=============
200 
201 Shares may go away because the storage server hosting them has suffered a
202 failure: either temporary downtime (affecting availability of the file), or a
203hunk ./docs/architecture.rst 429
204   in client behavior.
205 
206 
207-== Security ==
208+Security
209+========
210 
211 The design goal for this project is that an attacker may be able to deny
212 service (i.e. prevent you from recovering a file that was uploaded earlier)
213hunk ./docs/architecture.rst 436
214 but can accomplish none of the following three attacks:
215 
216- 1) violate confidentiality: the attacker gets to view data to which you have
217-    not granted them access
218- 2) violate integrity: the attacker convinces you that the wrong data is
219-    actually the data you were intending to retrieve
220- 3) violate unforgeability: the attacker gets to modify a mutable file or
221-    directory (either the pathnames or the file contents) to which you have
222-    not given them write permission
223+1) violate confidentiality: the attacker gets to view data to which you have
224+   not granted them access
225+2) violate integrity: the attacker convinces you that the wrong data is
226+   actually the data you were intending to retrieve
227+3) violate unforgeability: the attacker gets to modify a mutable file or
228+   directory (either the pathnames or the file contents) to which you have
229+   not given them write permission
230 
231 Integrity (the promise that the downloaded data will match the uploaded data)
232 is provided by the hashes embedded in the capability (for immutable files) or
233hunk ./docs/architecture.rst 494
234 capabilities).
235 
236 
237-== Reliability ==
238+Reliability
239+===========
240 
241 File encoding and peer-node selection parameters can be adjusted to achieve
242 different goals. Each choice results in a number of properties; there are
243hunk ./docs/architecture.rst 560
244 view the disk consumption of each. It is also acquiring some sections with
245 availability/reliability numbers, as well as preliminary cost analysis data.
246 This tool will continue to evolve as our analysis improves.
247-
248-------------------------------
249-
250-[1]: http://en.wikipedia.org/wiki/Zooko%27s_triangle
251-
252-[2]: all of these names are derived from the location where they were
253-     concocted, in this case in a car ride from Boulder to DEN. To be
254-     precise, "Tahoe 1" was an unworkable scheme in which everyone who holds
255-     shares for a given file would form a sort of cabal which kept track of
256-     all the others, "Tahoe 2" is the first-100-nodes in the permuted hash
257-     described in this document, and "Tahoe 3" (or perhaps "Potrero hill 1")
258-     was the abandoned ring-with-many-hands approach.
259-
260hunk ./docs/backdoors.rst 1
261-Statement on Backdoors
262+======================
263+Statement on Backdoors
264+======================
265 
266 October 5, 2010
267 
268hunk ./docs/backdoors.rst 7
269-The New York Times has recently reported that the current U.S. administration is proposing a bill that would apparently, if passed, require communication systems to facilitate government wiretapping and access to encrypted data:
270+The New York Times has recently reported that the current U.S. administration
271+is proposing a bill that would apparently, if passed, require communication
272+systems to facilitate government wiretapping and access to encrypted data:
273 
274  http://www.nytimes.com/2010/09/27/us/27wiretap.html (login required; username/password pairs available at  http://www.bugmenot.com/view/nytimes.com).
275 
276hunk ./docs/backdoors.rst 13
277-Commentary by the  Electronic Frontier Foundation (https://www.eff.org/deeplinks/2010/09/government-seeks ),  Peter Suderman / Reason (http://reason.com/blog/2010/09/27/obama-administration-frustrate ),  Julian Sanchez / Cato Institute (http://www.cato-at-liberty.org/designing-an-insecure-internet/ ).
278+Commentary by the  Electronic Frontier Foundation
279+(https://www.eff.org/deeplinks/2010/09/government-seeks ),  Peter Suderman /
280+Reason (http://reason.com/blog/2010/09/27/obama-administration-frustrate ),
281+Julian Sanchez / Cato Institute
282+(http://www.cato-at-liberty.org/designing-an-insecure-internet/ ).
283 
284hunk ./docs/backdoors.rst 19
285-The core Tahoe developers promise never to change Tahoe-LAFS to facilitate government access to data stored or transmitted by it. Even if it were desirable to facilitate such access—which it is not—we believe it would not be technically feasible to do so without severely compromising Tahoe-LAFS' security against other attackers. There have been many examples in which backdoors intended for use by government have introduced vulnerabilities exploitable by other parties (a notable example being the Greek cellphone eavesdropping scandal in 2004/5). RFCs  1984 and  2804 elaborate on the security case against such backdoors.
286+The core Tahoe developers promise never to change Tahoe-LAFS to facilitate
287+government access to data stored or transmitted by it. Even if it were
288+desirable to facilitate such access—which it is not—we believe it would not be
289+technically feasible to do so without severely compromising Tahoe-LAFS'
290+security against other attackers. There have been many examples in which
291+backdoors intended for use by government have introduced vulnerabilities
292+exploitable by other parties (a notable example being the Greek cellphone
293+eavesdropping scandal in 2004/5). RFCs  1984 and  2804 elaborate on the
294+security case against such backdoors.
295 
296hunk ./docs/backdoors.rst 29
297-Note that since Tahoe-LAFS is open-source software, forks by people other than the current core developers are possible. In that event, we would try to persuade any such forks to adopt a similar policy.
298+Note that since Tahoe-LAFS is open-source software, forks by people other than
299+the current core developers are possible. In that event, we would try to
300+persuade any such forks to adopt a similar policy.
301 
302 The following Tahoe-LAFS developers agree with this statement:
303 
304hunk ./docs/backdoors.rst 36
305 David-Sarah Hopwood
306+
307 Zooko Wilcox-O'Hearn
308hunk ./docs/backdoors.rst 38
309+
310 Brian Warner
311hunk ./docs/backdoors.rst 40
312+
313 Kevan Carstensen
314hunk ./docs/backdoors.rst 42
315+
316 Frédéric Marti
317hunk ./docs/backdoors.rst 44
318+
319 Jack Lloyd
320hunk ./docs/backdoors.rst 46
321+
322 François Deppierraz
323hunk ./docs/backdoors.rst 48
324+
325 Yu Xue
326hunk ./docs/backdoors.rst 50
327+
328 Marc Tooley
329hunk ./docs/backdoors.rst 52
330+
331+Peter Secor
332+
333+Shawn Willden
334hunk ./docs/backupdb.rst 1
335-= The Tahoe BackupDB =
336+==================
337+The Tahoe BackupDB
338+==================
339 
340hunk ./docs/backupdb.rst 5
341-== Overview ==
342+1.  `Overview`_
343+2.  `Schema`_
344+3.  `Upload Operation`_
345+4.  `Directory Operations`_
346+
347+Overview
348+========
349 To speed up backup operations, Tahoe maintains a small database known as the
350 "backupdb". This is used to avoid re-uploading files which have already been
351 uploaded recently.
352hunk ./docs/backupdb.rst 44
353 as Debian etch (4.0 "oldstable") or Ubuntu Edgy (6.10) the "python-pysqlite2"
354 package won't work, but the "sqlite3-dev" package will.
355 
356-== Schema ==
357-
358-The database contains the following tables:
359-
360-CREATE TABLE version
361-(
362- version integer  # contains one row, set to 1
363-);
364-
365-CREATE TABLE local_files
366-(
367- path  varchar(1024),  PRIMARY KEY -- index, this is os.path.abspath(fn)
368- size  integer,         -- os.stat(fn)[stat.ST_SIZE]
369- mtime number,          -- os.stat(fn)[stat.ST_MTIME]
370- ctime number,          -- os.stat(fn)[stat.ST_CTIME]
371- fileid integer
372-);
373-
374-CREATE TABLE caps
375-(
376- fileid integer PRIMARY KEY AUTOINCREMENT,
377- filecap varchar(256) UNIQUE    -- URI:CHK:...
378-);
379+Schema
380+======
381 
382hunk ./docs/backupdb.rst 47
383-CREATE TABLE last_upload
384-(
385- fileid INTEGER PRIMARY KEY,
386- last_uploaded TIMESTAMP,
387- last_checked TIMESTAMP
388-);
389+The database contains the following tables::
390 
391hunk ./docs/backupdb.rst 49
392-CREATE TABLE directories
393-(
394- dirhash varchar(256) PRIMARY KEY,
395- dircap varchar(256),
396- last_uploaded TIMESTAMP,
397- last_checked TIMESTAMP
398-);
399+  CREATE TABLE version
400+  (
401+   version integer  # contains one row, set to 1
402+  );
403
404+  CREATE TABLE local_files
405+  (
406+   path  varchar(1024),  PRIMARY KEY -- index, this is os.path.abspath(fn)
407+   size  integer,         -- os.stat(fn)[stat.ST_SIZE]
408+   mtime number,          -- os.stat(fn)[stat.ST_MTIME]
409+   ctime number,          -- os.stat(fn)[stat.ST_CTIME]
410+   fileid integer
411+  );
412
413+  CREATE TABLE caps
414+  (
415+   fileid integer PRIMARY KEY AUTOINCREMENT,
416+   filecap varchar(256) UNIQUE    -- URI:CHK:...
417+  );
418
419+  CREATE TABLE last_upload
420+  (
421+   fileid INTEGER PRIMARY KEY,
422+   last_uploaded TIMESTAMP,
423+   last_checked TIMESTAMP
424+  );
425
426+  CREATE TABLE directories
427+  (
428+   dirhash varchar(256) PRIMARY KEY,
429+   dircap varchar(256),
430+   last_uploaded TIMESTAMP,
431+   last_checked TIMESTAMP
432+  );
433 
434hunk ./docs/backupdb.rst 84
435-== Upload Operation ==
436+Upload Operation
437+================
438 
439 The upload process starts with a pathname (like ~/.emacs) and wants to end up
440 with a file-cap (like URI:CHK:...).
441hunk ./docs/backupdb.rst 95
442 is not present in this table, the file must be uploaded. The upload process
443 is:
444 
445- 1. record the file's size, creation time, and modification time
446- 2. upload the file into the grid, obtaining an immutable file read-cap
447- 3. add an entry to the 'caps' table, with the read-cap, to get a fileid
448- 4. add an entry to the 'last_upload' table, with the current time
449- 5. add an entry to the 'local_files' table, with the fileid, the path,
450-    and the local file's size/ctime/mtime
451+1. record the file's size, creation time, and modification time
452+
453+2. upload the file into the grid, obtaining an immutable file read-cap
454+
455+3. add an entry to the 'caps' table, with the read-cap, to get a fileid
456+
457+4. add an entry to the 'last_upload' table, with the current time
458+
459+5. add an entry to the 'local_files' table, with the fileid, the path,
460+   and the local file's size/ctime/mtime
461 
462 If the path *is* present in 'local_files', the easy-to-compute identifying
463 information is compared: file size and ctime/mtime. If these differ, the file
464hunk ./docs/backupdb.rst 157
465 into the grid. The --no-timestamps can be used to disable this optimization,
466 forcing every byte of the file to be hashed and encoded.
467 
468-== Directory Operations ==
469+Directory Operations
470+====================
471 
472 Once the contents of a directory are known (a filecap for each file, and a
473 dircap for each directory), the backup process must find or create a tahoe
474hunk ./docs/configuration.rst 1
475+========================
476+Configuring a Tahoe node
477+========================
478 
479hunk ./docs/configuration.rst 5
480-= Configuring a Tahoe node =
481+1.  `Overall Node Configuration`_
482+2.  `Client Configuration`_
483+3.  `Storage Server Configuration`_
484+4.  `Running A Helper`_
485+5.  `Running An Introducer`_
486+6.  `Other Files in BASEDIR`_
487+7.  `Other files`_
488+8.  `Backwards Compatibility Files`_
489+9.  `Example`_
490 
491 A Tahoe node is configured by writing to files in its base directory. These
492 files are read by the node when it starts, so each time you change them, you
493hunk ./docs/configuration.rst 36
494 
495 The item descriptions below use the following types:
496 
497- boolean: one of (True, yes, on, 1, False, off, no, 0), case-insensitive
498- strports string: a Twisted listening-port specification string, like "tcp:80"
499-                  or "tcp:3456:interface=127.0.0.1". For a full description of
500-                  the format, see
501-                  http://twistedmatrix.com/documents/current/api/twisted.application.strports.html
502- FURL string: a Foolscap endpoint identifier, like
503-              pb://soklj4y7eok5c3xkmjeqpw@192.168.69.247:44801/eqpwqtzm
504+boolean
505+    one of (True, yes, on, 1, False, off, no, 0), case-insensitive
506 
507hunk ./docs/configuration.rst 39
508+strports string
509+    a Twisted listening-port specification string, like "tcp:80"
510+    or "tcp:3456:interface=127.0.0.1". For a full description of
511+    the format, see
512+    http://twistedmatrix.com/documents/current/api/twisted.application.strports.html
513 
514hunk ./docs/configuration.rst 45
515-== Overall Node Configuration ==
516+FURL string
517+    a Foolscap endpoint identifier, like
518+    pb://soklj4y7eok5c3xkmjeqpw@192.168.69.247:44801/eqpwqtzm
519+
520+
521+Overall Node Configuration
522+==========================
523 
524 This section controls the network behavior of the node overall: which ports
525 and IP addresses are used, when connections are timed out, etc. This
526hunk ./docs/configuration.rst 63
527 that port number in the tub.port option. If behind a NAT, you *may* need to
528 set the tub.location option described below.
529 
530+::
531 
532hunk ./docs/configuration.rst 65
533-[node]
534-
535-nickname = (UTF-8 string, optional)
536-
537- This value will be displayed in management tools as this node's "nickname".
538- If not provided, the nickname will be set to "<unspecified>". This string
539- shall be a UTF-8 encoded unicode string.
540-
541-web.port = (strports string, optional)
542-
543- This controls where the node's webserver should listen, providing filesystem
544- access and node status as defined in webapi.txt . This file contains a
545- Twisted "strports" specification such as "3456" or
546- "tcp:3456:interface=127.0.0.1". The 'tahoe create-node' or 'tahoe create-client'
547- commands set the web.port to "tcp:3456:interface=127.0.0.1" by default; this
548- is overridable by the "--webport" option. You can make it use SSL by writing
549- "ssl:3456:privateKey=mykey.pem:certKey=cert.pem" instead.
550-
551- If this is not provided, the node will not run a web server.
552-
553-web.static = (string, optional)
554-
555- This controls where the /static portion of the URL space is served. The
556- value is a directory name (~username is allowed, and non-absolute names are
557- interpreted relative to the node's basedir) which can contain HTML and other
558- files. This can be used to serve a javascript-based frontend to the Tahoe
559- node, or other services.
560-
561- The default value is "public_html", which will serve $BASEDIR/public_html .
562- With the default settings, http://127.0.0.1:3456/static/foo.html will serve
563- the contents of $BASEDIR/public_html/foo.html .
564-
565-tub.port = (integer, optional)
566-
567- This controls which port the node uses to accept Foolscap connections from
568- other nodes. If not provided, the node will ask the kernel for any available
569- port. The port will be written to a separate file (named client.port or
570- introducer.port), so that subsequent runs will re-use the same port.
571-
572-tub.location = (string, optional)
573-
574- In addition to running as a client, each Tahoe node also runs as a server,
575- listening for connections from other Tahoe clients. The node announces its
576- location by publishing a "FURL" (a string with some connection hints) to the
577- Introducer. The string it publishes can be found in
578- $BASEDIR/private/storage.furl . The "tub.location" configuration controls
579- what location is published in this announcement.
580-
581- If you don't provide tub.location, the node will try to figure out a useful
582- one by itself, by using tools like 'ifconfig' to determine the set of IP
583- addresses on which it can be reached from nodes both near and far. It will
584- also include the TCP port number on which it is listening (either the one
585- specified by tub.port, or whichever port was assigned by the kernel when
586- tub.port is left unspecified).
587-
588- You might want to override this value if your node lives behind a firewall
589- that is doing inbound port forwarding, or if you are using other proxies
590- such that the local IP address or port number is not the same one that
591- remote clients should use to connect. You might also want to control this
592- when using a Tor proxy to avoid revealing your actual IP address through the
593- Introducer announcement.
594-
595- The value is a comma-separated string of host:port location hints, like
596- this:
597-
598-  123.45.67.89:8098,tahoe.example.com:8098,127.0.0.1:8098
599-
600- A few examples:
601-
602-  Emulate default behavior, assuming your host has IP address 123.45.67.89
603-  and the kernel-allocated port number was 8098:
604-
605-   tub.port = 8098
606-   tub.location = 123.45.67.89:8098,127.0.0.1:8098
607-
608-  Use a DNS name so you can change the IP address more easily:
609-
610-   tub.port = 8098
611-   tub.location = tahoe.example.com:8098
612-
613-  Run a node behind a firewall (which has an external IP address) that has
614-  been configured to forward port 7912 to our internal node's port 8098:
615-
616-   tub.port = 8098
617-   tub.location = external-firewall.example.com:7912
618+  [node]
619 
620hunk ./docs/configuration.rst 67
621-  Run a node behind a Tor proxy (perhaps via torsocks), in client-only mode
622-  (i.e. we can make outbound connections, but other nodes will not be able to
623-  connect to us). The literal 'unreachable.example.org' will not resolve, but
624-  will serve as a reminder to human observers that this node cannot be
625-  reached. "Don't call us.. we'll call you":
626+  nickname = (UTF-8 string, optional)
627 
628hunk ./docs/configuration.rst 69
629-   tub.port = 8098
630-   tub.location = unreachable.example.org:0
631+    This value will be displayed in management tools as this node's "nickname".
632+    If not provided, the nickname will be set to "<unspecified>". This string
633+    shall be a UTF-8 encoded unicode string.
634 
635hunk ./docs/configuration.rst 73
636-  Run a node behind a Tor proxy, and make the server available as a Tor
637-  "hidden service". (this assumes that other clients are running their node
638-  with torsocks, such that they are prepared to connect to a .onion address).
639-  The hidden service must first be configured in Tor, by giving it a local
640-  port number and then obtaining a .onion name, using something in the torrc
641-  file like:
642+  web.port = (strports string, optional)
643 
644hunk ./docs/configuration.rst 75
645-    HiddenServiceDir /var/lib/tor/hidden_services/tahoe
646-    HiddenServicePort 29212 127.0.0.1:8098
647+    This controls where the node's webserver should listen, providing filesystem
648+    access and node status as defined in webapi.txt . This file contains a
649+    Twisted "strports" specification such as "3456" or
650+    "tcp:3456:interface=127.0.0.1". The 'tahoe create-node' or 'tahoe create-client'
651+    commands set the web.port to "tcp:3456:interface=127.0.0.1" by default; this
652+    is overridable by the "--webport" option. You can make it use SSL by writing
653+    "ssl:3456:privateKey=mykey.pem:certKey=cert.pem" instead.
654+   
655+    If this is not provided, the node will not run a web server.
656 
657hunk ./docs/configuration.rst 85
658-   once Tor is restarted, the .onion hostname will be in
659-   /var/lib/tor/hidden_services/tahoe/hostname . Then set up your tahoe.cfg
660-   like:
661+  web.static = (string, optional)
662 
663hunk ./docs/configuration.rst 87
664-    tub.port = 8098
665-    tub.location = ualhejtq2p7ohfbb.onion:29212
666+    This controls where the /static portion of the URL space is served. The
667+    value is a directory name (~username is allowed, and non-absolute names are
668+    interpreted relative to the node's basedir) which can contain HTML and other
669+    files. This can be used to serve a javascript-based frontend to the Tahoe
670+    node, or other services.
671+   
672+    The default value is "public_html", which will serve $BASEDIR/public_html .
673+    With the default settings, http://127.0.0.1:3456/static/foo.html will serve
674+    the contents of $BASEDIR/public_html/foo.html .
675 
676hunk ./docs/configuration.rst 97
677- Most users will not need to set tub.location .
678+  tub.port = (integer, optional)
679 
680hunk ./docs/configuration.rst 99
681- Note that the old 'advertised_ip_addresses' file from earlier releases is no
682- longer supported. Tahoe 1.3.0 and later will ignore this file.
683+    This controls which port the node uses to accept Foolscap connections from
684+    other nodes. If not provided, the node will ask the kernel for any available
685+    port. The port will be written to a separate file (named client.port or
686+    introducer.port), so that subsequent runs will re-use the same port.
687 
688hunk ./docs/configuration.rst 104
689-log_gatherer.furl = (FURL, optional)
690+  tub.location = (string, optional)
691 
692hunk ./docs/configuration.rst 106
693- If provided, this contains a single FURL string which is used to contact a
694- 'log gatherer', which will be granted access to the logport. This can be
695- used by centralized storage meshes to gather operational logs in a single
696- place. Note that when an old-style BASEDIR/log_gatherer.furl file exists
697- (see 'Backwards Compatibility Files', below), both are used. (for most other
698- items, the separate config file overrides the entry in tahoe.cfg)
699+    In addition to running as a client, each Tahoe node also runs as a server,
700+    listening for connections from other Tahoe clients. The node announces its
701+    location by publishing a "FURL" (a string with some connection hints) to the
702+    Introducer. The string it publishes can be found in
703+    $BASEDIR/private/storage.furl . The "tub.location" configuration controls
704+    what location is published in this announcement.
705+   
706+    If you don't provide tub.location, the node will try to figure out a useful
707+    one by itself, by using tools like 'ifconfig' to determine the set of IP
708+    addresses on which it can be reached from nodes both near and far. It will
709+    also include the TCP port number on which it is listening (either the one
710+    specified by tub.port, or whichever port was assigned by the kernel when
711+    tub.port is left unspecified).
712+   
713+    You might want to override this value if your node lives behind a firewall
714+    that is doing inbound port forwarding, or if you are using other proxies
715+    such that the local IP address or port number is not the same one that
716+    remote clients should use to connect. You might also want to control this
717+    when using a Tor proxy to avoid revealing your actual IP address through the
718+    Introducer announcement.
719+   
720+    The value is a comma-separated string of host:port location hints, like
721+    this:
722 
723hunk ./docs/configuration.rst 130
724-timeout.keepalive = (integer in seconds, optional)
725-timeout.disconnect = (integer in seconds, optional)
726+      123.45.67.89:8098,tahoe.example.com:8098,127.0.0.1:8098
727 
728hunk ./docs/configuration.rst 132
729- If timeout.keepalive is provided, it is treated as an integral number of
730- seconds, and sets the Foolscap "keepalive timer" to that value. For each
731- connection to another node, if nothing has been heard for a while, we will
732- attempt to provoke the other end into saying something. The duration of
733- silence that passes before sending the PING will be between KT and 2*KT.
734- This is mainly intended to keep NAT boxes from expiring idle TCP sessions,
735- but also gives TCP's long-duration keepalive/disconnect timers some traffic
736- to work with. The default value is 240 (i.e. 4 minutes).
737+    A few examples:
738 
739hunk ./docs/configuration.rst 134
740- If timeout.disconnect is provided, this is treated as an integral number of
741- seconds, and sets the Foolscap "disconnect timer" to that value. For each
742- connection to another node, if nothing has been heard for a while, we will
743- drop the connection. The duration of silence that passes before dropping the
744- connection will be between DT-2*KT and 2*DT+2*KT (please see ticket #521 for
745- more details). If we are sending a large amount of data to the other end
746- (which takes more than DT-2*KT to deliver), we might incorrectly drop the
747- connection. The default behavior (when this value is not provided) is to
748- disable the disconnect timer.
749+      Emulate default behavior, assuming your host has IP address 123.45.67.89
750+      and the kernel-allocated port number was 8098:
751+   
752+        tub.port = 8098
753+        tub.location = 123.45.67.89:8098,127.0.0.1:8098
754+   
755+      Use a DNS name so you can change the IP address more easily:
756+   
757+        tub.port = 8098
758+        tub.location = tahoe.example.com:8098
759+   
760+      Run a node behind a firewall (which has an external IP address) that has
761+      been configured to forward port 7912 to our internal node's port 8098:
762+   
763+        tub.port = 8098
764+        tub.location = external-firewall.example.com:7912
765+   
766+      Run a node behind a Tor proxy (perhaps via torsocks), in client-only mode
767+      (i.e. we can make outbound connections, but other nodes will not be able to
768+      connect to us). The literal 'unreachable.example.org' will not resolve, but
769+      will serve as a reminder to human observers that this node cannot be
770+      reached. "Don't call us.. we'll call you":
771+   
772+        tub.port = 8098
773+        tub.location = unreachable.example.org:0
774+   
775+      Run a node behind a Tor proxy, and make the server available as a Tor
776+      "hidden service". (this assumes that other clients are running their node
777+      with torsocks, such that they are prepared to connect to a .onion address).
778+      The hidden service must first be configured in Tor, by giving it a local
779+      port number and then obtaining a .onion name, using something in the torrc
780+      file like:
781+   
782+        HiddenServiceDir /var/lib/tor/hidden_services/tahoe
783+        HiddenServicePort 29212 127.0.0.1:8098
784+   
785+      once Tor is restarted, the .onion hostname will be in
786+      /var/lib/tor/hidden_services/tahoe/hostname . Then set up your tahoe.cfg
787+      like:
788+   
789+        tub.port = 8098
790+        tub.location = ualhejtq2p7ohfbb.onion:29212
791+   
792+    Most users will not need to set tub.location .
793+   
794+    Note that the old 'advertised_ip_addresses' file from earlier releases is no
795+    longer supported. Tahoe 1.3.0 and later will ignore this file.
796 
797hunk ./docs/configuration.rst 182
798- See ticket #521 for a discussion of how to pick these timeout values. Using
799- 30 minutes means we'll disconnect after 22 to 68 minutes of inactivity.
800- Receiving data will reset this timeout, however if we have more than 22min
801- of data in the outbound queue (such as 800kB in two pipelined segments of 10
802- shares each) and the far end has no need to contact us, our ping might be
803- delayed, so we may disconnect them by accident.
804+  log_gatherer.furl = (FURL, optional)
805 
806hunk ./docs/configuration.rst 184
807-ssh.port = (strports string, optional)
808-ssh.authorized_keys_file = (filename, optional)
809+    If provided, this contains a single FURL string which is used to contact a
810+    'log gatherer', which will be granted access to the logport. This can be
811+    used by centralized storage meshes to gather operational logs in a single
812+    place. Note that when an old-style BASEDIR/log_gatherer.furl file exists
813+    (see 'Backwards Compatibility Files', below), both are used. (for most other
814+    items, the separate config file overrides the entry in tahoe.cfg)
815 
816hunk ./docs/configuration.rst 191
817- This enables an SSH-based interactive Python shell, which can be used to
818- inspect the internal state of the node, for debugging. To cause the node to
819- accept SSH connections on port 8022 from the same keys as the rest of your
820- account, use:
821+  timeout.keepalive = (integer in seconds, optional)
822+  timeout.disconnect = (integer in seconds, optional)
823 
824hunk ./docs/configuration.rst 194
825-   [tub]
826-   ssh.port = 8022
827-   ssh.authorized_keys_file = ~/.ssh/authorized_keys
828+    If timeout.keepalive is provided, it is treated as an integral number of
829+    seconds, and sets the Foolscap "keepalive timer" to that value. For each
830+    connection to another node, if nothing has been heard for a while, we will
831+    attempt to provoke the other end into saying something. The duration of
832+    silence that passes before sending the PING will be between KT and 2*KT.
833+    This is mainly intended to keep NAT boxes from expiring idle TCP sessions,
834+    but also gives TCP's long-duration keepalive/disconnect timers some traffic
835+    to work with. The default value is 240 (i.e. 4 minutes).
836+   
837+    If timeout.disconnect is provided, this is treated as an integral number of
838+    seconds, and sets the Foolscap "disconnect timer" to that value. For each
839+    connection to another node, if nothing has been heard for a while, we will
840+    drop the connection. The duration of silence that passes before dropping the
841+    connection will be between DT-2*KT and 2*DT+2*KT (please see ticket #521 for
842+    more details). If we are sending a large amount of data to the other end
843+    (which takes more than DT-2*KT to deliver), we might incorrectly drop the
844+    connection. The default behavior (when this value is not provided) is to
845+    disable the disconnect timer.
846+   
847+    See ticket #521 for a discussion of how to pick these timeout values. Using
848+    30 minutes means we'll disconnect after 22 to 68 minutes of inactivity.
849+    Receiving data will reset this timeout, however if we have more than 22min
850+    of data in the outbound queue (such as 800kB in two pipelined segments of 10
851+    shares each) and the far end has no need to contact us, our ping might be
852+    delayed, so we may disconnect them by accident.
853 
854hunk ./docs/configuration.rst 220
855-tempdir = (string, optional)
856+  ssh.port = (strports string, optional)
857+  ssh.authorized_keys_file = (filename, optional)
858 
859hunk ./docs/configuration.rst 223
860- This specifies a temporary directory for the webapi server to use, for
861- holding large files while they are being uploaded. If a webapi client
862- attempts to upload a 10GB file, this tempdir will need to have at least 10GB
863- available for the upload to complete.
864+    This enables an SSH-based interactive Python shell, which can be used to
865+    inspect the internal state of the node, for debugging. To cause the node to
866+    accept SSH connections on port 8022 from the same keys as the rest of your
867+    account, use:
868+   
869+      [tub]
870+      ssh.port = 8022
871+      ssh.authorized_keys_file = ~/.ssh/authorized_keys
872 
873hunk ./docs/configuration.rst 232
874- The default value is the "tmp" directory in the node's base directory (i.e.
875- $NODEDIR/tmp), but it can be placed elsewhere. This directory is used for
876- files that usually (on a unix system) go into /tmp . The string will be
877- interpreted relative to the node's base directory.
878+  tempdir = (string, optional)
879 
880hunk ./docs/configuration.rst 234
881-== Client Configuration ==
882+    This specifies a temporary directory for the webapi server to use, for
883+    holding large files while they are being uploaded. If a webapi client
884+    attempts to upload a 10GB file, this tempdir will need to have at least 10GB
885+    available for the upload to complete.
886+   
887+    The default value is the "tmp" directory in the node's base directory (i.e.
888+    $NODEDIR/tmp), but it can be placed elsewhere. This directory is used for
889+    files that usually (on a unix system) go into /tmp . The string will be
890+    interpreted relative to the node's base directory.
891 
892hunk ./docs/configuration.rst 244
893-[client]
894-introducer.furl = (FURL string, mandatory)
895+Client Configuration
896+====================
897 
898hunk ./docs/configuration.rst 247
899- This FURL tells the client how to connect to the introducer. Each Tahoe grid
900- is defined by an introducer. The introducer's furl is created by the
901- introducer node and written into its base directory when it starts,
902- whereupon it should be published to everyone who wishes to attach a client
903- to that grid
904+::
905 
906hunk ./docs/configuration.rst 249
907-helper.furl = (FURL string, optional)
908+  [client]
909+  introducer.furl = (FURL string, mandatory)
910
911+    This FURL tells the client how to connect to the introducer. Each Tahoe grid
912+    is defined by an introducer. The introducer's furl is created by the
913+    introducer node and written into its base directory when it starts,
914+    whereupon it should be published to everyone who wishes to attach a client
915+    to that grid
916
917+  helper.furl = (FURL string, optional)
918
919+    If provided, the node will attempt to connect to and use the given helper
920+    for uploads. See docs/helper.txt for details.
921
922+  key_generator.furl = (FURL string, optional)
923
924+    If provided, the node will attempt to connect to and use the given
925+    key-generator service, using RSA keys from the external process rather than
926+    generating its own.
927
928+  stats_gatherer.furl = (FURL string, optional)
929
930+    If provided, the node will connect to the given stats gatherer and provide
931+    it with operational statistics.
932
933+  shares.needed = (int, optional) aka "k", default 3
934+  shares.total = (int, optional) aka "N", N >= k, default 10
935+  shares.happy = (int, optional) 1 <= happy <= N, default 7
936
937+    These three values set the default encoding parameters. Each time a new file
938+    is uploaded, erasure-coding is used to break the ciphertext into separate
939+    pieces. There will be "N" (i.e. shares.total) pieces created, and the file
940+    will be recoverable if any "k" (i.e. shares.needed) pieces are retrieved.
941+    The default values are 3-of-10 (i.e. shares.needed = 3, shares.total = 10).
942+    Setting k to 1 is equivalent to simple replication (uploading N copies of
943+    the file).
944
945+    These values control the tradeoff between storage overhead, performance, and
946+    reliability. To a first approximation, a 1MB file will use (1MB*N/k) of
947+    backend storage space (the actual value will be a bit more, because of other
948+    forms of overhead). Up to N-k shares can be lost before the file becomes
949+    unrecoverable, so assuming there are at least N servers, up to N-k servers
950+    can be offline without losing the file. So large N/k ratios are more
951+    reliable, and small N/k ratios use less disk space. Clearly, k must never be
952+    smaller than N.
953+   
954+    Large values of N will slow down upload operations slightly, since more
955+    servers must be involved, and will slightly increase storage overhead due to
956+    the hash trees that are created. Large values of k will cause downloads to
957+    be marginally slower, because more servers must be involved. N cannot be
958+    larger than 256, because of the 8-bit erasure-coding algorithm that Tahoe
959+    uses.
960+   
961+    shares.happy allows you control over the distribution of your immutable file.
962+    For a successful upload, shares are guaranteed to be initially placed on
963+    at least 'shares.happy' distinct servers, the correct functioning of any
964+    k of which is sufficient to guarantee the availability of the uploaded file.
965+    This value should not be larger than the number of servers on your grid.
966+   
967+    A value of shares.happy <= k is allowed, but does not provide any redundancy
968+    if some servers fail or lose shares.
969+   
970+    (Mutable files use a different share placement algorithm that does not
971+    consider this parameter.)
972 
973hunk ./docs/configuration.rst 314
974- If provided, the node will attempt to connect to and use the given helper
975- for uploads. See docs/helper.txt for details.
976 
977hunk ./docs/configuration.rst 315
978-key_generator.furl = (FURL string, optional)
979+Storage Server Configuration
980+============================
981 
982hunk ./docs/configuration.rst 318
983- If provided, the node will attempt to connect to and use the given
984- key-generator service, using RSA keys from the external process rather than
985- generating its own.
986+::
987 
988hunk ./docs/configuration.rst 320
989-stats_gatherer.furl = (FURL string, optional)
990+  [storage]
991+  enabled = (boolean, optional)
992
993+    If this is True, the node will run a storage server, offering space to other
994+    clients. If it is False, the node will not run a storage server, meaning
995+    that no shares will be stored on this node. Use False this for clients who
996+    do not wish to provide storage service. The default value is True.
997
998+  readonly = (boolean, optional)
999
1000+    If True, the node will run a storage server but will not accept any shares,
1001+    making it effectively read-only. Use this for storage servers which are
1002+    being decommissioned: the storage/ directory could be mounted read-only,
1003+    while shares are moved to other servers. Note that this currently only
1004+    affects immutable shares. Mutable shares (used for directories) will be
1005+    written and modified anyway. See ticket #390 for the current status of this
1006+    bug. The default value is False.
1007
1008+  reserved_space = (str, optional)
1009
1010+    If provided, this value defines how much disk space is reserved: the storage
1011+    server will not accept any share which causes the amount of free disk space
1012+    to drop below this value. (The free space is measured by a call to statvfs(2)
1013+    on Unix, or GetDiskFreeSpaceEx on Windows, and is the space available to the
1014+    user account under which the storage server runs.)
1015+   
1016+    This string contains a number, with an optional case-insensitive scale
1017+    suffix like "K" or "M" or "G", and an optional "B" or "iB" suffix. So
1018+    "100MB", "100M", "100000000B", "100000000", and "100000kb" all mean the same
1019+    thing. Likewise, "1MiB", "1024KiB", and "1048576B" all mean the same thing.
1020
1021+  expire.enabled =
1022+  expire.mode =
1023+  expire.override_lease_duration =
1024+  expire.cutoff_date =
1025+  expire.immutable =
1026+  expire.mutable =
1027
1028+    These settings control garbage-collection, in which the server will delete
1029+    shares that no longer have an up-to-date lease on them. Please see the
1030+    neighboring "garbage-collection.txt" document for full details.
1031 
1032hunk ./docs/configuration.rst 362
1033- If provided, the node will connect to the given stats gatherer and provide
1034- it with operational statistics.
1035 
1036hunk ./docs/configuration.rst 363
1037-shares.needed = (int, optional) aka "k", default 3
1038-shares.total = (int, optional) aka "N", N >= k, default 10
1039-shares.happy = (int, optional) 1 <= happy <= N, default 7
1040-
1041- These three values set the default encoding parameters. Each time a new file
1042- is uploaded, erasure-coding is used to break the ciphertext into separate
1043- pieces. There will be "N" (i.e. shares.total) pieces created, and the file
1044- will be recoverable if any "k" (i.e. shares.needed) pieces are retrieved.
1045- The default values are 3-of-10 (i.e. shares.needed = 3, shares.total = 10).
1046- Setting k to 1 is equivalent to simple replication (uploading N copies of
1047- the file).
1048-
1049- These values control the tradeoff between storage overhead, performance, and
1050- reliability. To a first approximation, a 1MB file will use (1MB*N/k) of
1051- backend storage space (the actual value will be a bit more, because of other
1052- forms of overhead). Up to N-k shares can be lost before the file becomes
1053- unrecoverable, so assuming there are at least N servers, up to N-k servers
1054- can be offline without losing the file. So large N/k ratios are more
1055- reliable, and small N/k ratios use less disk space. Clearly, k must never be
1056- smaller than N.
1057-
1058- Large values of N will slow down upload operations slightly, since more
1059- servers must be involved, and will slightly increase storage overhead due to
1060- the hash trees that are created. Large values of k will cause downloads to
1061- be marginally slower, because more servers must be involved. N cannot be
1062- larger than 256, because of the 8-bit erasure-coding algorithm that Tahoe
1063- uses.
1064-
1065- shares.happy allows you control over the distribution of your immutable file.
1066- For a successful upload, shares are guaranteed to be initially placed on
1067- at least 'shares.happy' distinct servers, the correct functioning of any
1068- k of which is sufficient to guarantee the availability of the uploaded file.
1069- This value should not be larger than the number of servers on your grid.
1070-
1071- A value of shares.happy <= k is allowed, but does not provide any redundancy
1072- if some servers fail or lose shares.
1073-
1074- (Mutable files use a different share placement algorithm that does not
1075-  consider this parameter.)
1076-
1077-
1078-== Storage Server Configuration ==
1079-
1080-[storage]
1081-enabled = (boolean, optional)
1082-
1083- If this is True, the node will run a storage server, offering space to other
1084- clients. If it is False, the node will not run a storage server, meaning
1085- that no shares will be stored on this node. Use False this for clients who
1086- do not wish to provide storage service. The default value is True.
1087-
1088-readonly = (boolean, optional)
1089-
1090- If True, the node will run a storage server but will not accept any shares,
1091- making it effectively read-only. Use this for storage servers which are
1092- being decommissioned: the storage/ directory could be mounted read-only,
1093- while shares are moved to other servers. Note that this currently only
1094- affects immutable shares. Mutable shares (used for directories) will be
1095- written and modified anyway. See ticket #390 for the current status of this
1096- bug. The default value is False.
1097-
1098-reserved_space = (str, optional)
1099-
1100- If provided, this value defines how much disk space is reserved: the storage
1101- server will not accept any share which causes the amount of free disk space
1102- to drop below this value. (The free space is measured by a call to statvfs(2)
1103- on Unix, or GetDiskFreeSpaceEx on Windows, and is the space available to the
1104- user account under which the storage server runs.)
1105-
1106- This string contains a number, with an optional case-insensitive scale
1107- suffix like "K" or "M" or "G", and an optional "B" or "iB" suffix. So
1108- "100MB", "100M", "100000000B", "100000000", and "100000kb" all mean the same
1109- thing. Likewise, "1MiB", "1024KiB", and "1048576B" all mean the same thing.
1110-
1111-expire.enabled =
1112-expire.mode =
1113-expire.override_lease_duration =
1114-expire.cutoff_date =
1115-expire.immutable =
1116-expire.mutable =
1117-
1118- These settings control garbage-collection, in which the server will delete
1119- shares that no longer have an up-to-date lease on them. Please see the
1120- neighboring "garbage-collection.txt" document for full details.
1121-
1122-
1123-== Running A Helper ==
1124+Running A Helper
1125+================
1126 
1127 A "helper" is a regular client node that also offers the "upload helper"
1128 service.
1129hunk ./docs/configuration.rst 369
1130 
1131-[helper]
1132-enabled = (boolean, optional)
1133+::
1134 
1135hunk ./docs/configuration.rst 371
1136- If True, the node will run a helper (see docs/helper.txt for details). The
1137- helper's contact FURL will be placed in private/helper.furl, from which it
1138- can be copied to any clients which wish to use it. Clearly nodes should not
1139- both run a helper and attempt to use one: do not create both helper.furl and
1140- run_helper in the same node. The default is False.
1141+  [helper]
1142+  enabled = (boolean, optional)
1143
1144+    If True, the node will run a helper (see docs/helper.txt for details). The
1145+    helper's contact FURL will be placed in private/helper.furl, from which it
1146+    can be copied to any clients which wish to use it. Clearly nodes should not
1147+    both run a helper and attempt to use one: do not create both helper.furl and
1148+    run_helper in the same node. The default is False.
1149 
1150 
1151hunk ./docs/configuration.rst 381
1152-== Running An Introducer ==
1153+Running An Introducer
1154+=====================
1155 
1156 The introducer node uses a different '.tac' file (named introducer.tac), and
1157 pays attention to the "[node]" section, but not the others.
1158hunk ./docs/configuration.rst 396
1159 copied into new client nodes before they are started for the first time.
1160 
1161 
1162-== Other Files in BASEDIR ==
1163+Other Files in BASEDIR
1164+======================
1165 
1166 Some configuration is not kept in tahoe.cfg, for the following reasons:
1167 
1168hunk ./docs/configuration.rst 401
1169- * it is generated by the node at startup, e.g. encryption keys. The node
1170-   never writes to tahoe.cfg
1171- * it is generated by user action, e.g. the 'tahoe create-alias' command
1172+* it is generated by the node at startup, e.g. encryption keys. The node
1173+  never writes to tahoe.cfg
1174+* it is generated by user action, e.g. the 'tahoe create-alias' command
1175 
1176 In addition, non-configuration persistent state is kept in the node's base
1177 directory, next to the configuration knobs.
1178hunk ./docs/configuration.rst 410
1179 
1180 This section describes these other files.
1181 
1182+private/node.pem
1183+  This contains an SSL private-key certificate. The node
1184+  generates this the first time it is started, and re-uses it on subsequent
1185+  runs. This certificate allows the node to have a cryptographically-strong
1186+  identifier (the Foolscap "TubID"), and to establish secure connections to
1187+  other nodes.
1188 
1189hunk ./docs/configuration.rst 417
1190-private/node.pem : This contains an SSL private-key certificate. The node
1191-generates this the first time it is started, and re-uses it on subsequent
1192-runs. This certificate allows the node to have a cryptographically-strong
1193-identifier (the Foolscap "TubID"), and to establish secure connections to
1194-other nodes.
1195+storage/
1196+  Nodes which host StorageServers will create this directory to hold
1197+  shares of files on behalf of other clients. There will be a directory
1198+  underneath it for each StorageIndex for which this node is holding shares.
1199+  There is also an "incoming" directory where partially-completed shares are
1200+  held while they are being received.
1201 
1202hunk ./docs/configuration.rst 424
1203-storage/ : Nodes which host StorageServers will create this directory to hold
1204-shares of files on behalf of other clients. There will be a directory
1205-underneath it for each StorageIndex for which this node is holding shares.
1206-There is also an "incoming" directory where partially-completed shares are
1207-held while they are being received.
1208+client.tac
1209+  this file defines the client, by constructing the actual Client
1210+  instance each time the node is started. It is used by the 'twistd'
1211+  daemonization program (in the "-y" mode), which is run internally by the
1212+  "tahoe start" command. This file is created by the "tahoe create-node" or
1213+  "tahoe create-client" commands.
1214 
1215hunk ./docs/configuration.rst 431
1216-client.tac : this file defines the client, by constructing the actual Client
1217-instance each time the node is started. It is used by the 'twistd'
1218-daemonization program (in the "-y" mode), which is run internally by the
1219-"tahoe start" command. This file is created by the "tahoe create-node" or
1220-"tahoe create-client" commands.
1221+private/control.furl
1222+  this file contains a FURL that provides access to a
1223+  control port on the client node, from which files can be uploaded and
1224+  downloaded. This file is created with permissions that prevent anyone else
1225+  from reading it (on operating systems that support such a concept), to insure
1226+  that only the owner of the client node can use this feature. This port is
1227+  intended for debugging and testing use.
1228 
1229hunk ./docs/configuration.rst 439
1230-private/control.furl : this file contains a FURL that provides access to a
1231-control port on the client node, from which files can be uploaded and
1232-downloaded. This file is created with permissions that prevent anyone else
1233-from reading it (on operating systems that support such a concept), to insure
1234-that only the owner of the client node can use this feature. This port is
1235-intended for debugging and testing use.
1236+private/logport.furl
1237+  this file contains a FURL that provides access to a
1238+  'log port' on the client node, from which operational logs can be retrieved.
1239+  Do not grant logport access to strangers, because occasionally secret
1240+  information may be placed in the logs.
1241 
1242hunk ./docs/configuration.rst 445
1243-private/logport.furl : this file contains a FURL that provides access to a
1244-'log port' on the client node, from which operational logs can be retrieved.
1245-Do not grant logport access to strangers, because occasionally secret
1246-information may be placed in the logs.
1247+private/helper.furl
1248+  if the node is running a helper (for use by other
1249+  clients), its contact FURL will be placed here. See docs/helper.txt for more
1250+  details.
1251 
1252hunk ./docs/configuration.rst 450
1253-private/helper.furl : if the node is running a helper (for use by other
1254-clients), its contact FURL will be placed here. See docs/helper.txt for more
1255-details.
1256+private/root_dir.cap (optional)
1257+  The command-line tools will read a directory
1258+  cap out of this file and use it, if you don't specify a '--dir-cap' option or
1259+  if you specify '--dir-cap=root'.
1260 
1261hunk ./docs/configuration.rst 455
1262-private/root_dir.cap (optional): The command-line tools will read a directory
1263-cap out of this file and use it, if you don't specify a '--dir-cap' option or
1264-if you specify '--dir-cap=root'.
1265-
1266-private/convergence (automatically generated): An added secret for encrypting
1267-immutable files. Everyone who has this same string in their
1268-private/convergence file encrypts their immutable files in the same way when
1269-uploading them. This causes identical files to "converge" -- to share the
1270-same storage space since they have identical ciphertext -- which conserves
1271-space and optimizes upload time, but it also exposes files to the possibility
1272-of a brute-force attack by people who know that string. In this attack, if
1273-the attacker can guess most of the contents of a file, then they can use
1274-brute-force to learn the remaining contents.
1275+private/convergence (automatically generated)
1276+  An added secret for encrypting
1277+  immutable files. Everyone who has this same string in their
1278+  private/convergence file encrypts their immutable files in the same way when
1279+  uploading them. This causes identical files to "converge" -- to share the
1280+  same storage space since they have identical ciphertext -- which conserves
1281+  space and optimizes upload time, but it also exposes files to the possibility
1282+  of a brute-force attack by people who know that string. In this attack, if
1283+  the attacker can guess most of the contents of a file, then they can use
1284+  brute-force to learn the remaining contents.
1285 
1286 So the set of people who know your private/convergence string is the set of
1287 people who converge their storage space with you when you and they upload
1288hunk ./docs/configuration.rst 478
1289 possible, put the empty string (so that private/convergence is a zero-length
1290 file).
1291 
1292+Other files
1293+===========
1294 
1295hunk ./docs/configuration.rst 481
1296-== Other files ==
1297-
1298-logs/ : Each Tahoe node creates a directory to hold the log messages produced
1299-as the node runs. These logfiles are created and rotated by the "twistd"
1300-daemonization program, so logs/twistd.log will contain the most recent
1301-messages, logs/twistd.log.1 will contain the previous ones, logs/twistd.log.2
1302-will be older still, and so on. twistd rotates logfiles after they grow
1303-beyond 1MB in size. If the space consumed by logfiles becomes troublesome,
1304-they should be pruned: a cron job to delete all files that were created more
1305-than a month ago in this logs/ directory should be sufficient.
1306-
1307-my_nodeid : this is written by all nodes after startup, and contains a
1308-base32-encoded (i.e. human-readable) NodeID that identifies this specific
1309-node. This NodeID is the same string that gets displayed on the web page (in
1310-the "which peers am I connected to" list), and the shortened form (the first
1311-characters) is recorded in various log messages.
1312+logs/
1313+  Each Tahoe node creates a directory to hold the log messages produced
1314+  as the node runs. These logfiles are created and rotated by the "twistd"
1315+  daemonization program, so logs/twistd.log will contain the most recent
1316+  messages, logs/twistd.log.1 will contain the previous ones, logs/twistd.log.2
1317+  will be older still, and so on. twistd rotates logfiles after they grow
1318+  beyond 1MB in size. If the space consumed by logfiles becomes troublesome,
1319+  they should be pruned: a cron job to delete all files that were created more
1320+  than a month ago in this logs/ directory should be sufficient.
1321 
1322hunk ./docs/configuration.rst 491
1323+my_nodeid
1324+  this is written by all nodes after startup, and contains a
1325+  base32-encoded (i.e. human-readable) NodeID that identifies this specific
1326+  node. This NodeID is the same string that gets displayed on the web page (in
1327+  the "which peers am I connected to" list), and the shortened form (the first
1328+  characters) is recorded in various log messages.
1329 
1330hunk ./docs/configuration.rst 498
1331-== Backwards Compatibility Files ==
1332+Backwards Compatibility Files
1333+=============================
1334 
1335 Tahoe releases before 1.3.0 had no 'tahoe.cfg' file, and used distinct files
1336 for each item listed below. For each configuration knob, if the distinct file
1337hunk ./docs/configuration.rst 503
1338-exists, it will take precedence over the corresponding item in tahoe.cfg .
1339+exists, it will take precedence over the corresponding item in tahoe.cfg.
1340 
1341hunk ./docs/configuration.rst 505
1342-
1343-[node]nickname : BASEDIR/nickname
1344-[node]web.port : BASEDIR/webport
1345-[node]tub.port : BASEDIR/client.port  (for Clients, not Introducers)
1346-[node]tub.port : BASEDIR/introducer.port  (for Introducers, not Clients)
1347-      (note that, unlike other keys, tahoe.cfg overrides the *.port file)
1348-[node]tub.location : replaces BASEDIR/advertised_ip_addresses
1349-[node]log_gatherer.furl : BASEDIR/log_gatherer.furl (one per line)
1350-[node]timeout.keepalive : BASEDIR/keepalive_timeout
1351-[node]timeout.disconnect : BASEDIR/disconnect_timeout
1352-[client]introducer.furl : BASEDIR/introducer.furl
1353-[client]helper.furl : BASEDIR/helper.furl
1354-[client]key_generator.furl : BASEDIR/key_generator.furl
1355-[client]stats_gatherer.furl : BASEDIR/stats_gatherer.furl
1356-[storage]enabled : BASEDIR/no_storage (False if no_storage exists)
1357-[storage]readonly : BASEDIR/readonly_storage (True if readonly_storage exists)
1358-[storage]sizelimit : BASEDIR/sizelimit
1359-[storage]debug_discard : BASEDIR/debug_discard_storage
1360-[helper]enabled : BASEDIR/run_helper (True if run_helper exists)
1361+===========================  ===============================  =================
1362+Config setting               File                             Comment
1363+===========================  ===============================  =================
1364+[node]nickname               BASEDIR/nickname
1365+[node]web.port               BASEDIR/webport
1366+[node]tub.port               BASEDIR/client.port              (for Clients, not Introducers)
1367+[node]tub.port               BASEDIR/introducer.port          (for Introducers, not Clients) (note that, unlike other keys, tahoe.cfg overrides this file)
1368+[node]tub.location           BASEDIR/advertised_ip_addresses
1369+[node]log_gatherer.furl      BASEDIR/log_gatherer.furl        (one per line)
1370+[node]timeout.keepalive      BASEDIR/keepalive_timeout
1371+[node]timeout.disconnect     BASEDIR/disconnect_timeout
1372+[client]introducer.furl      BASEDIR/introducer.furl
1373+[client]helper.furl          BASEDIR/helper.furl
1374+[client]key_generator.furl   BASEDIR/key_generator.furl
1375+[client]stats_gatherer.furl  BASEDIR/stats_gatherer.furl
1376+[storage]enabled             BASEDIR/no_storage               (False if no_storage exists)
1377+[storage]readonly            BASEDIR/readonly_storage         (True if readonly_storage exists)
1378+[storage]sizelimit           BASEDIR/sizelimit
1379+[storage]debug_discard       BASEDIR/debug_discard_storage
1380+[helper]enabled              BASEDIR/run_helper               (True if run_helper exists)
1381+===========================  ===============================  =================
1382 
1383 Note: the functionality of [node]ssh.port and [node]ssh.authorized_keys_file
1384 were previously combined, controlled by the presence of a
1385hunk ./docs/configuration.rst 533
1386 indicated which port the ssh server should listen on, and the contents of the
1387 file provided the ssh public keys to accept. Support for these files has been
1388 removed completely. To ssh into your Tahoe node, add [node]ssh.port and
1389-[node].ssh_authorized_keys_file statements to your tahoe.cfg .
1390+[node].ssh_authorized_keys_file statements to your tahoe.cfg.
1391 
1392 Likewise, the functionality of [node]tub.location is a variant of the
1393 now-unsupported BASEDIR/advertised_ip_addresses . The old file was additive
1394hunk ./docs/configuration.rst 542
1395 is not (tub.location is used verbatim).
1396 
1397 
1398-== Example ==
1399+Example
1400+=======
1401 
1402 The following is a sample tahoe.cfg file, containing values for all keys
1403 described above. Note that this is not a recommended configuration (most of
1404hunk ./docs/configuration.rst 549
1405 these are not the default values), merely a legal one.
1406 
1407-[node]
1408-nickname = Bob's Tahoe Node
1409-tub.port = 34912
1410-tub.location = 123.45.67.89:8098,44.55.66.77:8098
1411-web.port = 3456
1412-log_gatherer.furl = pb://soklj4y7eok5c3xkmjeqpw@192.168.69.247:44801/eqpwqtzm
1413-timeout.keepalive = 240
1414-timeout.disconnect = 1800
1415-ssh.port = 8022
1416-ssh.authorized_keys_file = ~/.ssh/authorized_keys
1417-
1418-[client]
1419-introducer.furl = pb://ok45ssoklj4y7eok5c3xkmj@tahoe.example:44801/ii3uumo
1420-helper.furl = pb://ggti5ssoklj4y7eok5c3xkmj@helper.tahoe.example:7054/kk8lhr
1421-
1422-[storage]
1423-enabled = True
1424-readonly_storage = True
1425-sizelimit = 10000000000
1426+::
1427 
1428hunk ./docs/configuration.rst 551
1429-[helper]
1430-run_helper = True
1431+  [node]
1432+  nickname = Bob's Tahoe Node
1433+  tub.port = 34912
1434+  tub.location = 123.45.67.89:8098,44.55.66.77:8098
1435+  web.port = 3456
1436+  log_gatherer.furl = pb://soklj4y7eok5c3xkmjeqpw@192.168.69.247:44801/eqpwqtzm
1437+  timeout.keepalive = 240
1438+  timeout.disconnect = 1800
1439+  ssh.port = 8022
1440+  ssh.authorized_keys_file = ~/.ssh/authorized_keys
1441
1442+  [client]
1443+  introducer.furl = pb://ok45ssoklj4y7eok5c3xkmj@tahoe.example:44801/ii3uumo
1444+  helper.furl = pb://ggti5ssoklj4y7eok5c3xkmj@helper.tahoe.example:7054/kk8lhr
1445
1446+  [storage]
1447+  enabled = True
1448+  readonly_storage = True
1449+  sizelimit = 10000000000
1450
1451+  [helper]
1452+  run_helper = True
1453hunk ./docs/debian.rst 1
1454-= Debian Support =
1455+==============
1456+Debian Support
1457+==============
1458 
1459hunk ./docs/debian.rst 5
1460-1.  Overview
1461-2.  TL;DR supporting package building instructions
1462-3.  TL;DR package building instructions for Tahoe
1463-4.  Building Debian Packages
1464-5.  Using Pre-Built Debian Packages
1465-6.  Building From Source on Debian Systems
1466+1.  `Overview`_
1467+2.  `TL;DR supporting package building instructions`_
1468+3.  `TL;DR package building instructions for Tahoe`_
1469+4.  `Building Debian Packages`_
1470+5.  `Using Pre-Built Debian Packages`_
1471+6.  `Building From Source on Debian Systems`_
1472 
1473hunk ./docs/debian.rst 12
1474-= Overview ==
1475+Overview
1476+========
1477 
1478 One convenient way to install Tahoe-LAFS is with debian packages.
1479 This document attempts to explain how to complete a desert island build for
1480hunk ./docs/debian.rst 20
1481 people in a hurry. It also attempts to explain more about our Debian packaging
1482 for those willing to read beyond the simple pragmatic packaging exercises.
1483 
1484-== TL;DR supporting package building instructions ==
1485+TL;DR supporting package building instructions
1486+==============================================
1487 
1488 There are only four supporting packages that are currently not available from
1489hunk ./docs/debian.rst 24
1490-the debian apt repositories in Debian Lenny:
1491+the debian apt repositories in Debian Lenny::
1492 
1493     python-foolscap python-zfec argparse zbase32
1494 
1495hunk ./docs/debian.rst 28
1496-First, we'll install some common packages for development:
1497+First, we'll install some common packages for development::
1498 
1499     sudo apt-get install -y build-essential debhelper cdbs python-central \
1500                     python-setuptools python python-dev python-twisted-core \
1501hunk ./docs/debian.rst 38
1502     sudo apt-file update
1503 
1504 
1505-To create packages for Lenny, we'll also install stdeb:   
1506+To create packages for Lenny, we'll also install stdeb:: 
1507 
1508     sudo apt-get install python-all-dev
1509     STDEB_VERSION="0.5.1"
1510hunk ./docs/debian.rst 48
1511     python setup.py --command-packages=stdeb.command bdist_deb
1512     sudo dpkg -i deb_dist/python-stdeb_$STDEB_VERSION-1_all.deb
1513 
1514-Now we're ready to build and install the zfec Debian package:
1515+Now we're ready to build and install the zfec Debian package::
1516 
1517     darcs get http://allmydata.org/source/zfec/trunk zfac
1518     cd zfac/zfec/
1519hunk ./docs/debian.rst 57
1520     dpkg-buildpackage -rfakeroot -uc -us
1521     sudo dpkg -i ../python-zfec_1.4.6-r333-1_amd64.deb
1522 
1523-We need to build a pyutil package:
1524+We need to build a pyutil package::
1525 
1526     wget http://pypi.python.org/packages/source/p/pyutil/pyutil-1.6.1.tar.gz
1527     tar -xvzf pyutil-1.6.1.tar.gz
1528hunk ./docs/debian.rst 67
1529     dpkg-buildpackage -rfakeroot -uc -us
1530     sudo dpkg -i ../python-pyutil_1.6.1-1_all.deb
1531 
1532-We also need to install argparse and zbase32:
1533+We also need to install argparse and zbase32::
1534 
1535     sudo easy_install argparse # argparse won't install with stdeb (!) :-(
1536     sudo easy_install zbase32 # XXX TODO: package with stdeb
1537hunk ./docs/debian.rst 72
1538 
1539-Finally, we'll fetch, unpack, build and install foolscap:
1540+Finally, we'll fetch, unpack, build and install foolscap::
1541 
1542     # You may not already have Brian's key:
1543     # gpg --recv-key 0x1514A7BD
1544hunk ./docs/debian.rst 86
1545     dpkg-buildpackage -rfakeroot -uc -us
1546     sudo dpkg -i ../python-foolscap_0.5.0-1_all.deb
1547 
1548-== TL;DR package building instructions for Tahoe ==
1549+TL;DR package building instructions for Tahoe
1550+=============================================
1551 
1552 If you want to build your own Debian packages from the darcs tree or from
1553hunk ./docs/debian.rst 90
1554-a source release, do the following:
1555+a source release, do the following::
1556 
1557     cd ~/
1558     mkdir src && cd src/
1559hunk ./docs/debian.rst 106
1560 /etc/defaults/allmydata-tahoe file to get Tahoe started. Data is by default
1561 stored in /var/lib/tahoelafsd/ and Tahoe runs as the 'tahoelafsd' user.
1562 
1563-== Building Debian Packages ==
1564+Building Debian Packages
1565+========================
1566 
1567 The Tahoe source tree comes with limited support for building debian packages
1568 on a variety of Debian and Ubuntu platforms. For each supported platform,
1569hunk ./docs/debian.rst 118
1570 
1571 To create debian packages from a Tahoe tree, you will need some additional
1572 tools installed. The canonical list of these packages is in the
1573-"Build-Depends" clause of misc/sid/debian/control , and includes:
1574+"Build-Depends" clause of misc/sid/debian/control , and includes::
1575 
1576  build-essential
1577  debhelper
1578hunk ./docs/debian.rst 139
1579 Note that we haven't tried to build source packages (.orig.tar.gz + dsc) yet,
1580 and there are no such source packages in our APT repository.
1581 
1582-== Using Pre-Built Debian Packages ==
1583+Using Pre-Built Debian Packages
1584+===============================
1585 
1586 The allmydata.org site hosts an APT repository with debian packages that are
1587hunk ./docs/debian.rst 143
1588-built after each checkin. The following wiki page describes this repository:
1589-
1590- http://allmydata.org/trac/tahoe/wiki/DownloadDebianPackages
1591+built after each checkin. `This wiki page
1592+<http://allmydata.org/trac/tahoe/wiki/DownloadDebianPackages>`_ describes this
1593+repository.
1594 
1595 The allmydata.org APT repository also includes debian packages of support
1596 libraries, like Foolscap, zfec, pycryptopp, and everything else you need that
1597hunk ./docs/debian.rst 151
1598 isn't already in debian.
1599 
1600-== Building From Source on Debian Systems ==
1601+Building From Source on Debian Systems
1602+======================================
1603 
1604 Many of Tahoe's build dependencies can be satisfied by first installing
1605 certain debian packages: simplejson is one of these. Some debian/ubuntu
1606hunk ./docs/filesystem-notes.rst 1
1607+=========================
1608+Filesystem-specific notes
1609+=========================
1610+
1611+1. ext3_
1612 
1613 Tahoe storage servers use a large number of subdirectories to store their
1614 shares on local disk. This format is simple and robust, but depends upon the
1615hunk ./docs/filesystem-notes.rst 11
1616 local filesystem to provide fast access to those directories.
1617 
1618-= ext3 =
1619+ext3
1620+====
1621 
1622 For moderate- or large-sized storage servers, you'll want to make sure the
1623 "directory index" feature is enabled on your ext3 directories, otherwise
1624hunk ./docs/filesystem-notes.rst 17
1625 share lookup may be very slow. Recent versions of ext3 enable this
1626-automatically, but older filesystems may not have it enabled.
1627+automatically, but older filesystems may not have it enabled::
1628 
1629hunk ./docs/filesystem-notes.rst 19
1630-$ sudo tune2fs -l /dev/sda1 |grep feature
1631-Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
1632+  $ sudo tune2fs -l /dev/sda1 |grep feature
1633+  Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
1634 
1635 If "dir_index" is present in the "features:" line, then you're all set. If
1636 not, you'll need to use tune2fs and e2fsck to enable and build the index. See
1637hunk ./docs/filesystem-notes.rst 24
1638-this page for some hints: http://wiki.dovecot.org/MailboxFormat/Maildir .
1639+<http://wiki.dovecot.org/MailboxFormat/Maildir> for some hints.
1640hunk ./docs/garbage-collection.rst 1
1641-= Garbage Collection in Tahoe =
1642+===========================
1643+Garbage Collection in Tahoe
1644+===========================
1645 
1646hunk ./docs/garbage-collection.rst 5
1647-1. Overview
1648-2. Client-side Renewal
1649-3. Server Side Expiration
1650-4. Expiration Progress
1651-5. Future Directions
1652+1. `Overview`_
1653+2. `Client-side Renewal`_
1654+3. `Server Side Expiration`_
1655+4. `Expiration Progress`_
1656+5. `Future Directions`_
1657 
1658hunk ./docs/garbage-collection.rst 11
1659-== Overview ==
1660+Overview
1661+========
1662 
1663 When a file or directory in the virtual filesystem is no longer referenced,
1664 the space that its shares occupied on each storage server can be freed,
1665hunk ./docs/garbage-collection.rst 46
1666 server can use the "expire.override_lease_duration" configuration setting to
1667 increase or decrease the effective duration to something other than 31 days).
1668 
1669-== Client-side Renewal ==
1670+Client-side Renewal
1671+===================
1672 
1673 If all of the files and directories which you care about are reachable from a
1674 single starting point (usually referred to as a "rootcap"), and you store
1675hunk ./docs/garbage-collection.rst 76
1676 appropriate for use by individual users as well, and may be incorporated
1677 directly into the client node.
1678 
1679-== Server Side Expiration ==
1680+Server Side Expiration
1681+======================
1682 
1683 Expiration must be explicitly enabled on each storage server, since the
1684 default behavior is to never expire shares. Expiration is enabled by adding
1685hunk ./docs/garbage-collection.rst 120
1686 expired whatever it is going to expire, the second and subsequent passes are
1687 not going to find any new leases to remove.
1688 
1689-The tahoe.cfg file uses the following keys to control lease expiration:
1690+The tahoe.cfg file uses the following keys to control lease expiration::
1691 
1692hunk ./docs/garbage-collection.rst 122
1693-[storage]
1694+  [storage]
1695 
1696hunk ./docs/garbage-collection.rst 124
1697-expire.enabled = (boolean, optional)
1698+  expire.enabled = (boolean, optional)
1699 
1700hunk ./docs/garbage-collection.rst 126
1701- If this is True, the storage server will delete shares on which all leases
1702- have expired. Other controls dictate when leases are considered to have
1703- expired. The default is False.
1704+    If this is True, the storage server will delete shares on which all leases
1705+    have expired. Other controls dictate when leases are considered to have
1706+    expired. The default is False.
1707 
1708hunk ./docs/garbage-collection.rst 130
1709-expire.mode = (string, "age" or "cutoff-date", required if expiration enabled)
1710+  expire.mode = (string, "age" or "cutoff-date", required if expiration enabled)
1711 
1712hunk ./docs/garbage-collection.rst 132
1713- If this string is "age", the age-based expiration scheme is used, and the
1714- "expire.override_lease_duration" setting can be provided to influence the
1715- lease ages. If it is "cutoff-date", the absolute-date-cutoff mode is used,
1716- and the "expire.cutoff_date" setting must be provided to specify the cutoff
1717- date. The mode setting currently has no default: you must provide a value.
1718+       If this string is "age", the age-based expiration scheme is used, and the
1719+       "expire.override_lease_duration" setting can be provided to influence the
1720+       lease ages. If it is "cutoff-date", the absolute-date-cutoff mode is used,
1721+       and the "expire.cutoff_date" setting must be provided to specify the cutoff
1722+       date. The mode setting currently has no default: you must provide a value.
1723 
1724hunk ./docs/garbage-collection.rst 138
1725- In a future release, this setting is likely to default to "age", but in this
1726- release it was deemed safer to require an explicit mode specification.
1727+       In a future release, this setting is likely to default to "age", but in this
1728+       release it was deemed safer to require an explicit mode specification.
1729 
1730hunk ./docs/garbage-collection.rst 141
1731-expire.override_lease_duration = (duration string, optional)
1732+  expire.override_lease_duration = (duration string, optional)
1733 
1734hunk ./docs/garbage-collection.rst 143
1735- When age-based expiration is in use, a lease will be expired if its
1736- "lease.create_renew" timestamp plus its "lease.duration" time is
1737- earlier/older than the current time. This key, if present, overrides the
1738- duration value for all leases, changing the algorithm from:
1739+       When age-based expiration is in use, a lease will be expired if its
1740+       "lease.create_renew" timestamp plus its "lease.duration" time is
1741+       earlier/older than the current time. This key, if present, overrides the
1742+       duration value for all leases, changing the algorithm from:
1743 
1744hunk ./docs/garbage-collection.rst 148
1745-   if (lease.create_renew_timestamp + lease.duration) < now:
1746-       expire_lease()
1747+    if (lease.create_renew_timestamp + lease.duration) < now:
1748+        expire_lease()
1749 
1750hunk ./docs/garbage-collection.rst 151
1751- to:
1752+    to:
1753 
1754hunk ./docs/garbage-collection.rst 153
1755-   if (lease.create_renew_timestamp + override_lease_duration) < now:
1756-       expire_lease()
1757+    if (lease.create_renew_timestamp + override_lease_duration) < now:
1758+        expire_lease()
1759 
1760hunk ./docs/garbage-collection.rst 156
1761- The value of this setting is a "duration string", which is a number of days,
1762- months, or years, followed by a units suffix, and optionally separated by a
1763- space, such as one of the following:
1764+       The value of this setting is a "duration string", which is a number of days,
1765+       months, or years, followed by a units suffix, and optionally separated by a
1766+       space, such as one of the following:
1767 
1768hunk ./docs/garbage-collection.rst 160
1769-  7days
1770-  31day
1771-  60 days
1772-  2mo
1773-  3 month
1774-  12 months
1775-  2years
1776+         7days
1777+         31day
1778+         60 days
1779+         2mo
1780+         3 month
1781+         12 months
1782+         2years
1783 
1784hunk ./docs/garbage-collection.rst 168
1785- This key is meant to compensate for the fact that clients do not yet have
1786- the ability to ask for leases that last longer than 31 days. A grid which
1787- wants to use faster or slower GC than a 31-day lease timer permits can use
1788- this parameter to implement it. The current fixed 31-day lease duration
1789- makes the server behave as if "lease.override_lease_duration = 31days" had
1790- been passed.
1791+       This key is meant to compensate for the fact that clients do not yet have
1792+       the ability to ask for leases that last longer than 31 days. A grid which
1793+       wants to use faster or slower GC than a 31-day lease timer permits can use
1794+       this parameter to implement it. The current fixed 31-day lease duration
1795+       makes the server behave as if "lease.override_lease_duration = 31days" had
1796+       been passed.
1797 
1798hunk ./docs/garbage-collection.rst 175
1799- This key is only valid when age-based expiration is in use (i.e. when
1800- "expire.mode = age" is used). It will be rejected if cutoff-date expiration
1801- is in use.
1802+       This key is only valid when age-based expiration is in use (i.e. when
1803+       "expire.mode = age" is used). It will be rejected if cutoff-date expiration
1804+       is in use.
1805 
1806hunk ./docs/garbage-collection.rst 179
1807-expire.cutoff_date = (date string, required if mode=cutoff-date)
1808+  expire.cutoff_date = (date string, required if mode=cutoff-date)
1809 
1810hunk ./docs/garbage-collection.rst 181
1811- When cutoff-date expiration is in use, a lease will be expired if its
1812- create/renew timestamp is older than the cutoff date. This string will be a
1813- date in the following format:
1814+       When cutoff-date expiration is in use, a lease will be expired if its
1815+       create/renew timestamp is older than the cutoff date. This string will be a
1816+       date in the following format:
1817 
1818hunk ./docs/garbage-collection.rst 185
1819-  2009-01-16   (January 16th, 2009)
1820-  2008-02-02
1821-  2007-12-25
1822+         2009-01-16   (January 16th, 2009)
1823+         2008-02-02
1824+         2007-12-25
1825 
1826hunk ./docs/garbage-collection.rst 189
1827- The actual cutoff time shall be midnight UTC at the beginning of the given
1828- day. Lease timers should naturally be generous enough to not depend upon
1829- differences in timezone: there should be at least a few days between the
1830- last renewal time and the cutoff date.
1831+       The actual cutoff time shall be midnight UTC at the beginning of the given
1832+       day. Lease timers should naturally be generous enough to not depend upon
1833+       differences in timezone: there should be at least a few days between the
1834+       last renewal time and the cutoff date.
1835 
1836hunk ./docs/garbage-collection.rst 194
1837- This key is only valid when cutoff-based expiration is in use (i.e. when
1838- "expire.mode = cutoff-date"). It will be rejected if age-based expiration is
1839- in use.
1840+       This key is only valid when cutoff-based expiration is in use (i.e. when
1841+       "expire.mode = cutoff-date"). It will be rejected if age-based expiration is
1842+       in use.
1843 
1844hunk ./docs/garbage-collection.rst 198
1845-expire.immutable = (boolean, optional)
1846+  expire.immutable = (boolean, optional)
1847 
1848hunk ./docs/garbage-collection.rst 200
1849- If this is False, then immutable shares will never be deleted, even if their
1850- leases have expired. This can be used in special situations to perform GC on
1851- mutable files but not immutable ones. The default is True.
1852+       If this is False, then immutable shares will never be deleted, even if their
1853+       leases have expired. This can be used in special situations to perform GC on
1854+       mutable files but not immutable ones. The default is True.
1855 
1856hunk ./docs/garbage-collection.rst 204
1857-expire.mutable = (boolean, optional)
1858+  expire.mutable = (boolean, optional)
1859 
1860hunk ./docs/garbage-collection.rst 206
1861- If this is False, then mutable shares will never be deleted, even if their
1862- leases have expired. This can be used in special situations to perform GC on
1863- immutable files but not mutable ones. The default is True.
1864+       If this is False, then mutable shares will never be deleted, even if their
1865+       leases have expired. This can be used in special situations to perform GC on
1866+       immutable files but not mutable ones. The default is True.
1867 
1868hunk ./docs/garbage-collection.rst 210
1869-== Expiration Progress ==
1870+Expiration Progress
1871+===================
1872 
1873 In the current release, leases are stored as metadata in each share file, and
1874 no separate database is maintained. As a result, checking and expiring leases
1875hunk ./docs/garbage-collection.rst 238
1876 crawler can be forcibly reset by stopping the node, deleting these two files,
1877 then restarting the node.
1878 
1879-== Future Directions ==
1880+Future Directions
1881+=================
1882 
1883 Tahoe's GC mechanism is undergoing significant changes. The global
1884 mark-and-sweep garbage-collection scheme can require considerable network
1885hunk ./docs/helper.rst 1
1886-= The Tahoe Upload Helper =
1887+=======================
1888+The Tahoe Upload Helper
1889+=======================
1890 
1891hunk ./docs/helper.rst 5
1892-1. Overview
1893-2. Setting Up A Helper
1894-3. Using a Helper
1895-4. Other Helper Modes
1896+1. `Overview`_
1897+2. `Setting Up A Helper`_
1898+3. `Using a Helper`_
1899+4. `Other Helper Modes`_
1900 
1901hunk ./docs/helper.rst 10
1902-== Overview ==
1903+Overview
1904+========
1905 
1906 As described in the "SWARMING DOWNLOAD, TRICKLING UPLOAD" section of
1907 architecture.txt, Tahoe uploads require more bandwidth than downloads: you
1908hunk ./docs/helper.rst 51
1909 other applications that are sharing the same uplink to compete more evenly
1910 for the limited bandwidth.
1911 
1912-
1913-
1914-== Setting Up A Helper ==
1915+Setting Up A Helper
1916+===================
1917 
1918 Who should consider running a helper?
1919 
1920hunk ./docs/helper.rst 56
1921- * Benevolent entities which wish to provide better upload speed for clients
1922-   that have slow uplinks
1923- * Folks which have machines with upload bandwidth to spare.
1924- * Server grid operators who want clients to connect to a small number of
1925-   helpers rather than a large number of storage servers (a "multi-tier"
1926-   architecture)
1927+* Benevolent entities which wish to provide better upload speed for clients
1928+  that have slow uplinks
1929+* Folks which have machines with upload bandwidth to spare.
1930+* Server grid operators who want clients to connect to a small number of
1931+  helpers rather than a large number of storage servers (a "multi-tier"
1932+  architecture)
1933 
1934 What sorts of machines are good candidates for running a helper?
1935 
1936hunk ./docs/helper.rst 65
1937- * The Helper needs to have good bandwidth to the storage servers. In
1938-   particular, it needs to have at least 3.3x better upload bandwidth than
1939-   the client does, or the client might as well upload directly to the
1940-   storage servers. In a commercial grid, the helper should be in the same
1941-   colo (and preferably in the same rack) as the storage servers.
1942- * The Helper will take on most of the CPU load involved in uploading a file.
1943-   So having a dedicated machine will give better results.
1944- * The Helper buffers ciphertext on disk, so the host will need at least as
1945-   much free disk space as there will be simultaneous uploads. When an upload
1946-   is interrupted, that space will be used for a longer period of time.
1947+* The Helper needs to have good bandwidth to the storage servers. In
1948+  particular, it needs to have at least 3.3x better upload bandwidth than
1949+  the client does, or the client might as well upload directly to the
1950+  storage servers. In a commercial grid, the helper should be in the same
1951+  colo (and preferably in the same rack) as the storage servers.
1952+* The Helper will take on most of the CPU load involved in uploading a file.
1953+  So having a dedicated machine will give better results.
1954+* The Helper buffers ciphertext on disk, so the host will need at least as
1955+  much free disk space as there will be simultaneous uploads. When an upload
1956+  is interrupted, that space will be used for a longer period of time.
1957 
1958 To turn a Tahoe-LAFS node into a helper (i.e. to run a helper service in
1959 addition to whatever else that node is doing), edit the tahoe.cfg file in your
1960hunk ./docs/helper.rst 87
1961 helper: you will need to give this FURL to any clients that wish to use your
1962 helper.
1963 
1964- cat $BASEDIR/private/helper.furl |mail -s "helper furl" friend@example.com
1965+::
1966+
1967+  cat $BASEDIR/private/helper.furl | mail -s "helper furl" friend@example.com
1968 
1969 You can tell if your node is running a helper by looking at its web status
1970 page. Assuming that you've set up the 'webport' to use port 3456, point your
1971hunk ./docs/helper.rst 112
1972 files in these directories that have not been modified for a week or two.
1973 Future versions of tahoe will try to self-manage these files a bit better.
1974 
1975-== Using a Helper ==
1976+Using a Helper
1977+==============
1978 
1979 Who should consider using a Helper?
1980 
1981hunk ./docs/helper.rst 117
1982- * clients with limited upstream bandwidth, such as a consumer ADSL line
1983- * clients who believe that the helper will give them faster uploads than
1984-   they could achieve with a direct upload
1985- * clients who experience problems with TCP connection fairness: if other
1986-   programs or machines in the same home are getting less than their fair
1987-   share of upload bandwidth. If the connection is being shared fairly, then
1988-   a Tahoe upload that is happening at the same time as a single FTP upload
1989-   should get half the bandwidth.
1990- * clients who have been given the helper.furl by someone who is running a
1991-   Helper and is willing to let them use it
1992+* clients with limited upstream bandwidth, such as a consumer ADSL line
1993+* clients who believe that the helper will give them faster uploads than
1994+  they could achieve with a direct upload
1995+* clients who experience problems with TCP connection fairness: if other
1996+  programs or machines in the same home are getting less than their fair
1997+  share of upload bandwidth. If the connection is being shared fairly, then
1998+  a Tahoe upload that is happening at the same time as a single FTP upload
1999+  should get half the bandwidth.
2000+* clients who have been given the helper.furl by someone who is running a
2001+  Helper and is willing to let them use it
2002 
2003 To take advantage of somebody else's Helper, take the helper.furl file that
2004 they give you, and copy it into your node's base directory, then restart the
2005hunk ./docs/helper.rst 132
2006 node:
2007 
2008- cat email >$BASEDIR/helper.furl
2009- tahoe restart $BASEDIR
2010+::
2011+
2012+  cat email >$BASEDIR/helper.furl
2013+  tahoe restart $BASEDIR
2014 
2015 This will signal the client to try and connect to the helper. Subsequent
2016 uploads will use the helper rather than using direct connections to the
2017hunk ./docs/helper.rst 156
2018 The upload/download status page (http://localhost:3456/status) will announce
2019 the using-helper-or-not state of each upload, in the "Helper?" column.
2020 
2021-== Other Helper Modes ==
2022+Other Helper Modes
2023+==================
2024 
2025 The Tahoe Helper only currently helps with one kind of operation: uploading
2026 immutable files. There are three other things it might be able to help with
2027hunk ./docs/helper.rst 163
2028 in the future:
2029 
2030- * downloading immutable files
2031- * uploading mutable files (such as directories)
2032- * downloading mutable files (like directories)
2033+* downloading immutable files
2034+* uploading mutable files (such as directories)
2035+* downloading mutable files (like directories)
2036 
2037 Since mutable files are currently limited in size, the ADSL upstream penalty
2038 is not so severe for them. There is no ADSL penalty to downloads, but there
2039hunk ./docs/known_issues.rst 1
2040-= known issues =
2041+============
2042+Known issues
2043+============
2044 
2045hunk ./docs/known_issues.rst 5
2046-*  overview
2047-*  issues in Tahoe-LAFS v1.8.0, released 2010-09-23
2048-  -  potential unauthorized access by JavaScript in unrelated files
2049-  -  potential disclosure of file through embedded hyperlinks or JavaScript in that file
2050-  -  command-line arguments are leaked to other local users
2051-  -  capabilities may be leaked to web browser phishing filter / "safe browsing" servers ===
2052-  -  known issues in the FTP and SFTP frontends ===
2053+* `Overview`_
2054+* `Issues in Tahoe-LAFS v1.8.0, released 2010-09-23`
2055 
2056hunk ./docs/known_issues.rst 8
2057-== overview ==
2058+  *  `Potential unauthorized access by JavaScript in unrelated files`_
2059+  *  `Potential disclosure of file through embedded hyperlinks or JavaScript in that file`_
2060+  *  `Command-line arguments are leaked to other local users`_
2061+  *  `Capabilities may be leaked to web browser phishing filter / "safe browsing" servers`_
2062+  *  `Known issues in the FTP and SFTP frontends`_
2063+
2064+Overview
2065+========
2066 
2067 Below is a list of known issues in recent releases of Tahoe-LAFS, and how to
2068 manage them.  The current version of this file can be found at
2069hunk ./docs/known_issues.rst 28
2070 
2071 http://tahoe-lafs.org/source/tahoe-lafs/trunk/docs/historical/historical_known_issues.txt
2072 
2073-== issues in Tahoe-LAFS v1.8.0, released 2010-09-18 ==
2074+Issues in Tahoe-LAFS v1.8.0, released 2010-09-23
2075+================================================
2076 
2077hunk ./docs/known_issues.rst 31
2078-=== potential unauthorized access by JavaScript in unrelated files ===
2079+Potential unauthorized access by JavaScript in unrelated files
2080+--------------------------------------------------------------
2081 
2082 If you view a file stored in Tahoe-LAFS through a web user interface,
2083 JavaScript embedded in that file might be able to access other files or
2084hunk ./docs/known_issues.rst 42
2085 have the ability to modify the contents of those files or directories,
2086 then that script could modify or delete those files or directories.
2087 
2088-==== how to manage it ====
2089+how to manage it
2090+~~~~~~~~~~~~~~~~
2091 
2092 For future versions of Tahoe-LAFS, we are considering ways to close off
2093 this leakage of authority while preserving ease of use -- the discussion
2094hunk ./docs/known_issues.rst 47
2095-of this issue is ticket #615.
2096+of this issue is ticket `#615 <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/615>`_.
2097 
2098 For the present, either do not view files stored in Tahoe-LAFS through a
2099 web user interface, or turn off JavaScript in your web browser before
2100hunk ./docs/known_issues.rst 55
2101 malicious JavaScript.
2102 
2103 
2104-=== potential disclosure of file through embedded hyperlinks or JavaScript in that file ===
2105+Potential disclosure of file through embedded hyperlinks or JavaScript in that file
2106+-----------------------------------------------------------------------------------
2107 
2108 If there is a file stored on a Tahoe-LAFS storage grid, and that file
2109 gets downloaded and displayed in a web browser, then JavaScript or
2110hunk ./docs/known_issues.rst 72
2111 browsers, so being careful which hyperlinks you click on is not
2112 sufficient to prevent this from happening.
2113 
2114-==== how to manage it ====
2115+how to manage it
2116+~~~~~~~~~~~~~~~~
2117 
2118 For future versions of Tahoe-LAFS, we are considering ways to close off
2119 this leakage of authority while preserving ease of use -- the discussion
2120hunk ./docs/known_issues.rst 77
2121-of this issue is ticket #127.
2122+of this issue is ticket `#127 <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/127>`_.
2123 
2124 For the present, a good work-around is that if you want to store and
2125 view a file on Tahoe-LAFS and you want that file to remain private, then
2126hunk ./docs/known_issues.rst 86
2127 written to maliciously leak access.
2128 
2129 
2130-=== command-line arguments are leaked to other local users ===
2131+Command-line arguments are leaked to other local users
2132+------------------------------------------------------
2133 
2134 Remember that command-line arguments are visible to other users (through
2135 the 'ps' command, or the windows Process Explorer tool), so if you are
2136hunk ./docs/known_issues.rst 96
2137 arguments.  This includes directory caps that you set up with the "tahoe
2138 add-alias" command.
2139 
2140-==== how to manage it ====
2141+how to manage it
2142+~~~~~~~~~~~~~~~~
2143 
2144 As of Tahoe-LAFS v1.3.0 there is a "tahoe create-alias" command that does
2145 the following technique for you.
2146hunk ./docs/known_issues.rst 105
2147 Bypass add-alias and edit the NODEDIR/private/aliases file directly, by
2148 adding a line like this:
2149 
2150-fun: URI:DIR2:ovjy4yhylqlfoqg2vcze36dhde:4d4f47qko2xm5g7osgo2yyidi5m4muyo2vjjy53q4vjju2u55mfa
2151+  fun: URI:DIR2:ovjy4yhylqlfoqg2vcze36dhde:4d4f47qko2xm5g7osgo2yyidi5m4muyo2vjjy53q4vjju2u55mfa
2152 
2153 By entering the dircap through the editor, the command-line arguments
2154 are bypassed, and other users will not be able to see them. Once you've
2155hunk ./docs/known_issues.rst 116
2156 access to your files and directories.
2157 
2158 
2159-=== capabilities may be leaked to web browser phishing filter / "safe browsing" servers ===
2160+Capabilities may be leaked to web browser phishing filter / "safe browsing" servers
2161+-----------------------------------------------------------------------------------
2162 
2163 Firefox, Internet Explorer, and Chrome include a "phishing filter" or
2164 "safe browing" component, which is turned on by default, and which sends
2165hunk ./docs/known_issues.rst 149
2166 version of this file stated that Firefox had abandoned their phishing
2167 filter; this was incorrect.
2168 
2169-==== how to manage it ====
2170+how to manage it
2171+~~~~~~~~~~~~~~~~
2172 
2173 If you use any phishing filter or "safe browsing" feature, consider either
2174 disabling it, or not using the WUI via that browser. Phishing filters have
2175hunk ./docs/known_issues.rst 159
2176 or malware attackers have learnt how to bypass them.
2177 
2178 To disable the filter in IE7 or IE8:
2179- - Click Internet Options from the Tools menu.
2180- - Click the Advanced tab.
2181- - If an "Enable SmartScreen Filter" option is present, uncheck it.
2182-   If a "Use Phishing Filter" or "Phishing Filter" option is present,
2183-   set it to Disable.
2184- - Confirm (click OK or Yes) out of all dialogs.
2185+````````````````````````````````````
2186+
2187+- Click Internet Options from the Tools menu.
2188+
2189+- Click the Advanced tab.
2190+
2191+- If an "Enable SmartScreen Filter" option is present, uncheck it.
2192+  If a "Use Phishing Filter" or "Phishing Filter" option is present,
2193+  set it to Disable.
2194+
2195+- Confirm (click OK or Yes) out of all dialogs.
2196 
2197 If you have a version of IE that splits the settings between security
2198 zones, do this for all zones.
2199hunk ./docs/known_issues.rst 175
2200 
2201 To disable the filter in Firefox:
2202- - Click Options from the Tools menu.
2203- - Click the Security tab.
2204- - Uncheck both the "Block reported attack sites" and "Block reported
2205-   web forgeries" options.
2206- - Click OK.
2207+`````````````````````````````````
2208+
2209+- Click Options from the Tools menu.
2210+
2211+- Click the Security tab.
2212+
2213+- Uncheck both the "Block reported attack sites" and "Block reported
2214+  web forgeries" options.
2215+
2216+- Click OK.
2217 
2218 To disable the filter in Chrome:
2219hunk ./docs/known_issues.rst 187
2220- - Click Options from the Tools menu.
2221- - Click the "Under the Hood" tab and find the "Privacy" section.
2222- - Uncheck the "Enable phishing and malware protection" option.
2223- - Click Close.
2224+````````````````````````````````
2225+
2226+- Click Options from the Tools menu.
2227+
2228+- Click the "Under the Hood" tab and find the "Privacy" section.
2229+
2230+- Uncheck the "Enable phishing and malware protection" option.
2231+
2232+- Click Close.
2233 
2234 
2235hunk ./docs/known_issues.rst 198
2236-=== known issues in the FTP and SFTP frontends ===
2237+Known issues in the FTP and SFTP frontends
2238+------------------------------------------
2239 
2240 These are documented in docs/frontends/FTP-and-SFTP.txt and at
2241 <http://tahoe-lafs.org/trac/tahoe-lafs/wiki/SftpFrontend>.
2242hunk ./docs/logging.rst 1
2243-= Tahoe Logging =
2244+=============
2245+Tahoe Logging
2246+=============
2247 
2248hunk ./docs/logging.rst 5
2249-1.  Overview
2250-2.  Realtime Logging
2251-3.  Incidents
2252-4.  Working with flogfiles
2253-5.  Gatherers
2254-  5.1.  Incident Gatherer
2255-  5.2.  Log Gatherer
2256-6.  Local twistd.log files
2257-7.  Adding log messages
2258-8.  Log Messages During Unit Tests
2259+1.  `Overview`_
2260+2.  `Realtime Logging`_
2261+3.  `Incidents`_
2262+4.  `Working with flogfiles`_
2263+5.  `Gatherers`_
2264 
2265hunk ./docs/logging.rst 11
2266-== Overview ==
2267+    1.  `Incident Gatherer`_
2268+    2.  `Log Gatherer`_
2269+
2270+6.  `Local twistd.log files`_
2271+7.  `Adding log messages`_
2272+8.  `Log Messages During Unit Tests`_
2273+
2274+Overview
2275+========
2276 
2277 Tahoe uses the Foolscap logging mechanism (known as the "flog" subsystem) to
2278 record information about what is happening inside the Tahoe node. This is
2279hunk ./docs/logging.rst 34
2280 /usr/bin/flogtool) which is used to get access to many foolscap logging
2281 features.
2282 
2283-== Realtime Logging ==
2284+Realtime Logging
2285+================
2286 
2287 When you are working on Tahoe code, and want to see what the node is doing,
2288 the easiest tool to use is "flogtool tail". This connects to the tahoe node
2289hunk ./docs/logging.rst 46
2290 BASEDIR/private/logport.furl . The following command will connect to this
2291 port and start emitting log information:
2292 
2293- flogtool tail BASEDIR/private/logport.furl
2294+  flogtool tail BASEDIR/private/logport.furl
2295 
2296 The "--save-to FILENAME" option will save all received events to a file,
2297 where then can be examined later with "flogtool dump" or "flogtool
2298hunk ./docs/logging.rst 54
2299 before subscribing to new ones (without --catch-up, you will only hear about
2300 events that occur after the tool has connected and subscribed).
2301 
2302-== Incidents ==
2303+Incidents
2304+=========
2305 
2306 Foolscap keeps a short list of recent events in memory. When something goes
2307 wrong, it writes all the history it has (and everything that gets logged in
2308hunk ./docs/logging.rst 82
2309 parent/child relationships of log events is displayed in a nested format.
2310 "flogtool web-viewer" is still fairly immature.
2311 
2312-== Working with flogfiles ==
2313+Working with flogfiles
2314+======================
2315 
2316 The "flogtool filter" command can be used to take a large flogfile (perhaps
2317 one created by the log-gatherer, see below) and copy a subset of events into
2318hunk ./docs/logging.rst 96
2319 were emitted with a given facility (like foolscap.negotiation or
2320 tahoe.upload).
2321 
2322-== Gatherers ==
2323+Gatherers
2324+=========
2325 
2326 In a deployed Tahoe grid, it is useful to get log information automatically
2327 transferred to a central log-gatherer host. This offloads the (admittedly
2328hunk ./docs/logging.rst 113
2329 The gatherer will write to files in its working directory, which can then be
2330 examined with tools like "flogtool dump" as described above.
2331 
2332-=== Incident Gatherer ===
2333+Incident Gatherer
2334+-----------------
2335 
2336 The "incident gatherer" only collects Incidents: records of the log events
2337 that occurred just before and slightly after some high-level "trigger event"
2338hunk ./docs/logging.rst 133
2339 "gatherer.tac" file should be modified to add classifier functions.
2340 
2341 The incident gatherer writes incident names (which are simply the relative
2342-pathname of the incident-*.flog.bz2 file) into classified/CATEGORY. For
2343+pathname of the incident-\*.flog.bz2 file) into classified/CATEGORY. For
2344 example, the classified/mutable-retrieve-uncoordinated-write-error file
2345 contains a list of all incidents which were triggered by an uncoordinated
2346 write that was detected during mutable file retrieval (caused when somebody
2347hunk ./docs/logging.rst 158
2348 node which generated it to the gatherer. The gatherer will automatically
2349 catch up to any incidents which occurred while it is offline.
2350 
2351-=== Log Gatherer ===
2352+Log Gatherer
2353+------------
2354 
2355 The "Log Gatherer" subscribes to hear about every single event published by
2356 the connected nodes, regardless of severity. This server writes these log
2357hunk ./docs/logging.rst 186
2358 the outbound queue grows too large. When this occurs, there will be gaps
2359 (non-sequential event numbers) in the log-gatherer's flogfiles.
2360 
2361-== Local twistd.log files ==
2362+Local twistd.log files
2363+======================
2364 
2365 [TODO: not yet true, requires foolscap-0.3.1 and a change to allmydata.node]
2366 
2367hunk ./docs/logging.rst 203
2368 (i.e. not the log.NOISY debugging events). In addition, foolscap internal
2369 events (like connection negotiation messages) are not bridged to twistd.log .
2370 
2371-== Adding log messages ==
2372+Adding log messages
2373+===================
2374 
2375 When adding new code, the Tahoe developer should add a reasonable number of
2376 new log events. For details, please see the Foolscap logging documentation,
2377hunk ./docs/logging.rst 210
2378 but a few notes are worth stating here:
2379 
2380- * use a facility prefix of "tahoe.", like "tahoe.mutable.publish"
2381+* use a facility prefix of "tahoe.", like "tahoe.mutable.publish"
2382 
2383hunk ./docs/logging.rst 212
2384- * assign each severe (log.WEIRD or higher) event a unique message
2385-   identifier, as the umid= argument to the log.msg() call. The
2386-   misc/coding_tools/make_umid script may be useful for this purpose. This will make it
2387-   easier to write a classification function for these messages.
2388+* assign each severe (log.WEIRD or higher) event a unique message
2389+  identifier, as the umid= argument to the log.msg() call. The
2390+  misc/coding_tools/make_umid script may be useful for this purpose. This will make it
2391+  easier to write a classification function for these messages.
2392 
2393hunk ./docs/logging.rst 217
2394- * use the parent= argument whenever the event is causally/temporally
2395-   clustered with its parent. For example, a download process that involves
2396-   three sequential hash fetches could announce the send and receipt of those
2397-   hash-fetch messages with a parent= argument that ties them to the overall
2398-   download process. However, each new wapi download request should be
2399-   unparented.
2400+* use the parent= argument whenever the event is causally/temporally
2401+  clustered with its parent. For example, a download process that involves
2402+  three sequential hash fetches could announce the send and receipt of those
2403+  hash-fetch messages with a parent= argument that ties them to the overall
2404+  download process. However, each new wapi download request should be
2405+  unparented.
2406 
2407hunk ./docs/logging.rst 224
2408- * use the format= argument in preference to the message= argument. E.g.
2409-   use log.msg(format="got %(n)d shares, need %(k)d", n=n, k=k) instead of
2410-   log.msg("got %d shares, need %d" % (n,k)). This will allow later tools to
2411-   analyze the event without needing to scrape/reconstruct the structured
2412-   data out of the formatted string.
2413+* use the format= argument in preference to the message= argument. E.g.
2414+  use log.msg(format="got %(n)d shares, need %(k)d", n=n, k=k) instead of
2415+  log.msg("got %d shares, need %d" % (n,k)). This will allow later tools to
2416+  analyze the event without needing to scrape/reconstruct the structured
2417+  data out of the formatted string.
2418 
2419hunk ./docs/logging.rst 230
2420- * Pass extra information as extra keyword arguments, even if they aren't
2421-   included in the format= string. This information will be displayed in the
2422-   "flogtool dump --verbose" output, as well as being available to other
2423-   tools. The umid= argument should be passed this way.
2424+* Pass extra information as extra keyword arguments, even if they aren't
2425+  included in the format= string. This information will be displayed in the
2426+  "flogtool dump --verbose" output, as well as being available to other
2427+  tools. The umid= argument should be passed this way.
2428 
2429hunk ./docs/logging.rst 235
2430- * use log.err for the catch-all addErrback that gets attached to the end of
2431-   any given Deferred chain. When used in conjunction with LOGTOTWISTED=1,
2432-   log.err() will tell Twisted about the error-nature of the log message,
2433-   causing Trial to flunk the test (with an "ERROR" indication that prints a
2434-   copy of the Failure, including a traceback). Don't use log.err for events
2435-   that are BAD but handled (like hash failures: since these are often
2436-   deliberately provoked by test code, they should not cause test failures):
2437-   use log.msg(level=BAD) for those instead.
2438+* use log.err for the catch-all addErrback that gets attached to the end of
2439+  any given Deferred chain. When used in conjunction with LOGTOTWISTED=1,
2440+  log.err() will tell Twisted about the error-nature of the log message,
2441+  causing Trial to flunk the test (with an "ERROR" indication that prints a
2442+  copy of the Failure, including a traceback). Don't use log.err for events
2443+  that are BAD but handled (like hash failures: since these are often
2444+  deliberately provoked by test code, they should not cause test failures):
2445+  use log.msg(level=BAD) for those instead.
2446 
2447 
2448hunk ./docs/logging.rst 245
2449-== Log Messages During Unit Tests ==
2450+Log Messages During Unit Tests
2451+==============================
2452 
2453 If a test is failing and you aren't sure why, start by enabling
2454 FLOGTOTWISTED=1 like this:
2455hunk ./docs/logging.rst 251
2456 
2457- make test FLOGTOTWISTED=1
2458+  make test FLOGTOTWISTED=1
2459 
2460 With FLOGTOTWISTED=1, sufficiently-important log events will be written into
2461 _trial_temp/test.log, which may give you more ideas about why the test is
2462hunk ./docs/logging.rst 263
2463 If that isn't enough, look at the detailed foolscap logging messages instead,
2464 by running the tests like this:
2465 
2466- make test FLOGFILE=flog.out.bz2 FLOGLEVEL=1 FLOGTOTWISTED=1
2467+  make test FLOGFILE=flog.out.bz2 FLOGLEVEL=1 FLOGTOTWISTED=1
2468 
2469 The first environment variable will cause foolscap log events to be written
2470 to ./flog.out.bz2 (instead of merely being recorded in the circular buffers
2471hunk ./docs/performance.rst 1
2472-= Performance costs for some common operations =
2473+============================================
2474+Performance costs for some common operations
2475+============================================
2476 
2477hunk ./docs/performance.rst 5
2478-1.  Publishing an A-byte immutable file
2479-2.  Publishing an A-byte mutable file
2480-3.  Downloading B bytes of an A-byte immutable file
2481-4.  Downloading B bytes of an A-byte mutable file
2482-5.  Modifying B bytes of an A-byte mutable file
2483-6.  Inserting/Removing B bytes in an A-byte mutable file
2484-7.  Adding an entry to an A-entry directory
2485-8.  Listing an A entry directory
2486-9.  Performing a file-check on an A-byte file
2487-10. Performing a file-verify on an A-byte file
2488-11. Repairing an A-byte file (mutable or immutable)
2489+1.  `Publishing an A-byte immutable file`_
2490+2.  `Publishing an A-byte mutable file`_
2491+3.  `Downloading B bytes of an A-byte immutable file`_
2492+4.  `Downloading B bytes of an A-byte mutable file`_
2493+5.  `Modifying B bytes of an A-byte mutable file`_
2494+6.  `Inserting/Removing B bytes in an A-byte mutable file`_
2495+7.  `Adding an entry to an A-entry directory`_
2496+8.  `Listing an A entry directory`_
2497+9.  `Performing a file-check on an A-byte file`_
2498+10. `Performing a file-verify on an A-byte file`_
2499+11. `Repairing an A-byte file (mutable or immutable)`_
2500 
2501hunk ./docs/performance.rst 17
2502-== Publishing an A-byte immutable file ==
2503+Publishing an ``A``-byte immutable file
2504+=======================================
2505 
2506 network: A
2507hunk ./docs/performance.rst 21
2508+
2509 memory footprint: N/k*128KiB
2510 
2511 notes: An immutable file upload requires an additional I/O pass over the entire
2512hunk ./docs/performance.rst 25
2513-       source file before the upload process can start, since convergent
2514-       encryption derives the encryption key in part from the contents of the
2515-       source file.
2516+source file before the upload process can start, since convergent
2517+encryption derives the encryption key in part from the contents of the
2518+source file.
2519 
2520hunk ./docs/performance.rst 29
2521-== Publishing an A-byte mutable file ==
2522+Publishing an ``A``-byte mutable file
2523+=====================================
2524 
2525 network: A
2526hunk ./docs/performance.rst 33
2527+
2528 memory footprint: N/k*A
2529hunk ./docs/performance.rst 35
2530+
2531 cpu: O(A) + a large constant for RSA keypair generation
2532 
2533hunk ./docs/performance.rst 38
2534-notes: Tahoe-LAFS generates a new RSA keypair for each mutable file that
2535-       it publishes to a grid. This takes up to 1 or 2 seconds on a
2536-       typical desktop PC.
2537+notes: Tahoe-LAFS generates a new RSA keypair for each mutable file that it
2538+publishes to a grid. This takes up to 1 or 2 seconds on a typical desktop PC.
2539 
2540hunk ./docs/performance.rst 41
2541-       Part of the process of encrypting, encoding, and uploading a
2542-       mutable file to a Tahoe-LAFS grid requires that the entire file
2543-       be in memory at once. For larger files, this may cause
2544-       Tahoe-LAFS to have an unacceptably large memory footprint (at
2545-       least when uploading a mutable file).
2546+Part of the process of encrypting, encoding, and uploading a mutable file to a
2547+Tahoe-LAFS grid requires that the entire file be in memory at once. For larger
2548+files, this may cause Tahoe-LAFS to have an unacceptably large memory footprint
2549+(at least when uploading a mutable file).
2550 
2551hunk ./docs/performance.rst 46
2552-== Downloading B bytes of an A-byte immutable file ==
2553+Downloading ``B`` bytes of an ``A``-byte immutable file
2554+=======================================================
2555 
2556 network: B
2557hunk ./docs/performance.rst 50
2558+
2559 memory footprint: 128KiB
2560 
2561 notes: When Tahoe-LAFS 1.8.0 or later is asked to read an arbitrary range
2562hunk ./docs/performance.rst 54
2563-       of an immutable file, only the 128-KiB segments that overlap the
2564-       requested range will be downloaded.
2565+of an immutable file, only the 128-KiB segments that overlap the
2566+requested range will be downloaded.
2567 
2568hunk ./docs/performance.rst 57
2569-       (Earlier versions would download from the beginning of the file up
2570-       until the end of the requested range, and then continue to download
2571-       the rest of the file even after the request was satisfied.)
2572+(Earlier versions would download from the beginning of the file up
2573+until the end of the requested range, and then continue to download
2574+the rest of the file even after the request was satisfied.)
2575 
2576hunk ./docs/performance.rst 61
2577-== Downloading B bytes of an A-byte mutable file ==
2578+Downloading ``B`` bytes of an ``A``-byte mutable file
2579+=====================================================
2580 
2581 network: A
2582hunk ./docs/performance.rst 65
2583+
2584 memory footprint: A
2585 
2586 notes: As currently implemented, mutable files must be downloaded in
2587hunk ./docs/performance.rst 69
2588-       their entirety before any part of them can be read. We are
2589-       exploring fixes for this; see ticket #393 for more information.
2590+their entirety before any part of them can be read. We are
2591+exploring fixes for this; see ticket #393 for more information.
2592 
2593hunk ./docs/performance.rst 72
2594-== Modifying B bytes of an A-byte mutable file ==
2595+Modifying ``B`` bytes of an ``A``-byte mutable file
2596+===================================================
2597 
2598 network: A
2599hunk ./docs/performance.rst 76
2600+
2601 memory footprint: N/k*A
2602 
2603 notes: If you upload a changed version of a mutable file that you
2604hunk ./docs/performance.rst 80
2605-       earlier put onto your grid with, say, 'tahoe put --mutable',
2606-       Tahoe-LAFS will replace the old file with the new file on the
2607-       grid, rather than attempting to modify only those portions of the
2608-       file that have changed. Modifying a file in this manner is
2609-       essentially uploading the file over again, except that it re-uses
2610-       the existing RSA keypair instead of generating a new one.
2611+earlier put onto your grid with, say, 'tahoe put --mutable',
2612+Tahoe-LAFS will replace the old file with the new file on the
2613+grid, rather than attempting to modify only those portions of the
2614+file that have changed. Modifying a file in this manner is
2615+essentially uploading the file over again, except that it re-uses
2616+the existing RSA keypair instead of generating a new one.
2617 
2618hunk ./docs/performance.rst 87
2619-== Inserting/Removing B bytes in an A-byte mutable file ==
2620+Inserting/Removing ``B`` bytes in an ``A``-byte mutable file
2621+============================================================
2622 
2623 network: A
2624hunk ./docs/performance.rst 91
2625+
2626 memory footprint: N/k*A
2627 
2628 notes: Modifying any part of a mutable file in Tahoe-LAFS requires that
2629hunk ./docs/performance.rst 95
2630-       the entire file be downloaded, modified, held in memory while it is
2631-       encrypted and encoded, and then re-uploaded. A future version of the
2632-       mutable file layout ("LDMF") may provide efficient inserts and
2633-       deletes. Note that this sort of modification is mostly used internally
2634-       for directories, and isn't something that the WUI, CLI, or other
2635-       interfaces will do -- instead, they will simply overwrite the file to
2636-       be modified, as described in "Modifying B bytes of an A-byte mutable
2637-       file".
2638+the entire file be downloaded, modified, held in memory while it is
2639+encrypted and encoded, and then re-uploaded. A future version of the
2640+mutable file layout ("LDMF") may provide efficient inserts and
2641+deletes. Note that this sort of modification is mostly used internally
2642+for directories, and isn't something that the WUI, CLI, or other
2643+interfaces will do -- instead, they will simply overwrite the file to
2644+be modified, as described in "Modifying B bytes of an A-byte mutable
2645+file".
2646 
2647hunk ./docs/performance.rst 104
2648-== Adding an entry to an A-entry directory ==
2649+Adding an entry to an ``A``-entry directory
2650+===========================================
2651 
2652 network: O(A)
2653hunk ./docs/performance.rst 108
2654+
2655 memory footprint: N/k*A
2656 
2657 notes: In Tahoe-LAFS, directories are implemented as specialized mutable
2658hunk ./docs/performance.rst 112
2659-       files. So adding an entry to a directory is essentially adding B
2660-       (actually, 300-330) bytes somewhere in an existing mutable file.
2661+files. So adding an entry to a directory is essentially adding B
2662+(actually, 300-330) bytes somewhere in an existing mutable file.
2663 
2664hunk ./docs/performance.rst 115
2665-== Listing an A entry directory ==
2666+Listing an ``A`` entry directory
2667+================================
2668 
2669 network: O(A)
2670hunk ./docs/performance.rst 119
2671+
2672 memory footprint: N/k*A
2673 
2674 notes: Listing a directory requires that the mutable file storing the
2675hunk ./docs/performance.rst 123
2676-       directory be downloaded from the grid. So listing an A entry
2677-       directory requires downloading a (roughly) 330 * A byte mutable
2678-       file, since each directory entry is about 300-330 bytes in size.
2679+directory be downloaded from the grid. So listing an A entry
2680+directory requires downloading a (roughly) 330 * A byte mutable
2681+file, since each directory entry is about 300-330 bytes in size.
2682 
2683hunk ./docs/performance.rst 127
2684-== Performing a file-check on an A-byte file ==
2685+Performing a file-check on an ``A``-byte file
2686+=============================================
2687 
2688 network: O(S), where S is the number of servers on your grid
2689hunk ./docs/performance.rst 131
2690+
2691 memory footprint: negligible
2692 
2693 notes: To check a file, Tahoe-LAFS queries all the servers that it knows
2694hunk ./docs/performance.rst 135
2695-       about. Note that neither of these values directly depend on the size
2696-       of the file. This is relatively inexpensive, compared to the verify
2697-       and repair operations.
2698+about. Note that neither of these values directly depend on the size
2699+of the file. This is relatively inexpensive, compared to the verify
2700+and repair operations.
2701 
2702hunk ./docs/performance.rst 139
2703-== Performing a file-verify on an A-byte file ==
2704+Performing a file-verify on an ``A``-byte file
2705+==============================================
2706 
2707 network: N/k*A
2708hunk ./docs/performance.rst 143
2709+
2710 memory footprint: N/k*128KiB
2711 
2712 notes: To verify a file, Tahoe-LAFS downloads all of the ciphertext
2713hunk ./docs/performance.rst 147
2714-       shares that were originally uploaded to the grid and integrity
2715-       checks them. This is, for well-behaved grids, likely to be more
2716-       expensive than downloading an A-byte file, since only a fraction
2717-       of these shares are necessary to recover the file.
2718+shares that were originally uploaded to the grid and integrity
2719+checks them. This is, for well-behaved grids, likely to be more
2720+expensive than downloading an A-byte file, since only a fraction
2721+of these shares are necessary to recover the file.
2722 
2723hunk ./docs/performance.rst 152
2724-== Repairing an A-byte file (mutable or immutable) ==
2725+Repairing an ``A``-byte file (mutable or immutable)
2726+===================================================
2727 
2728 network: variable; up to around O(A)
2729hunk ./docs/performance.rst 156
2730+
2731 memory footprint: from 128KiB to (1+N/k)*128KiB
2732 
2733 notes: To repair a file, Tahoe-LAFS downloads the file, and generates/uploads
2734hunk ./docs/performance.rst 160
2735-       missing shares in the same way as when it initially uploads the file.
2736-       So, depending on how many shares are missing, this can be about as
2737-       expensive as initially uploading the file in the first place.
2738+missing shares in the same way as when it initially uploads the file.
2739+So, depending on how many shares are missing, this can be about as
2740+expensive as initially uploading the file in the first place.
2741hunk ./docs/stats.rst 1
2742-= Tahoe Statistics =
2743+================
2744+Tahoe Statistics
2745+================
2746 
2747hunk ./docs/stats.rst 5
2748-1. Overview
2749-2. Statistics Categories
2750-3. Running a Tahoe Stats-Gatherer Service
2751-4. Using Munin To Graph Stats Values
2752+1. `Overview`_
2753+2. `Statistics Categories`_
2754+3. `Running a Tahoe Stats-Gatherer Service`_
2755+4. `Using Munin To Graph Stats Values`_
2756 
2757hunk ./docs/stats.rst 10
2758-== Overview ==
2759+Overview
2760+========
2761 
2762 Each Tahoe node collects and publishes statistics about its operations as it
2763 runs. These include counters of how many files have been uploaded and
2764hunk ./docs/stats.rst 26
2765 block, along with a copy of the raw counters. To obtain just the raw counters
2766 (in JSON format), use /statistics?t=json instead.
2767 
2768-== Statistics Categories ==
2769+Statistics Categories
2770+=====================
2771 
2772 The stats dictionary contains two keys: 'counters' and 'stats'. 'counters'
2773 are strictly counters: they are reset to zero when the node is started, and
2774hunk ./docs/stats.rst 42
2775 
2776 The currently available stats (as of release 1.6.0 or so) are described here:
2777 
2778-counters.storage_server.*: this group counts inbound storage-server
2779-                           operations. They are not provided by client-only
2780-                           nodes which have been configured to not run a
2781-                           storage server (with [storage]enabled=false in
2782-                           tahoe.cfg)
2783-  allocate, write, close, abort: these are for immutable file uploads.
2784-                                 'allocate' is incremented when a client asks
2785-                                 if it can upload a share to the server.
2786-                                 'write' is incremented for each chunk of
2787-                                 data written. 'close' is incremented when
2788-                                 the share is finished. 'abort' is
2789-                                 incremented if the client abandons the
2790-                                 uploaed.
2791-  get, read: these are for immutable file downloads. 'get' is incremented
2792-             when a client asks if the server has a specific share. 'read' is
2793-             incremented for each chunk of data read.
2794-  readv, writev: these are for immutable file creation, publish, and
2795-                 retrieve. 'readv' is incremented each time a client reads
2796-                 part of a mutable share. 'writev' is incremented each time a
2797-                 client sends a modification request.
2798-  add-lease, renew, cancel: these are for share lease modifications.
2799-                            'add-lease' is incremented when an 'add-lease'
2800-                            operation is performed (which either adds a new
2801-                            lease or renews an existing lease). 'renew' is
2802-                            for the 'renew-lease' operation (which can only
2803-                            be used to renew an existing one). 'cancel' is
2804-                            used for the 'cancel-lease' operation.
2805-  bytes_freed: this counts how many bytes were freed when a 'cancel-lease'
2806-               operation removed the last lease from a share and the share
2807-               was thus deleted.
2808-  bytes_added: this counts how many bytes were consumed by immutable share
2809-               uploads. It is incremented at the same time as the 'close'
2810-               counter.
2811+**counters.storage_server.\***
2812 
2813hunk ./docs/stats.rst 44
2814-stats.storage_server.*:
2815- allocated: this counts how many bytes are currently 'allocated', which
2816-            tracks the space that will eventually be consumed by immutable
2817-            share upload operations. The stat is increased as soon as the
2818-            upload begins (at the same time the 'allocated' counter is
2819-            incremented), and goes back to zero when the 'close' or 'abort'
2820-            message is received (at which point the 'disk_used' stat should
2821-            incremented by the same amount).
2822- disk_total
2823- disk_used
2824- disk_free_for_root
2825- disk_free_for_nonroot
2826- disk_avail
2827- reserved_space: these all reflect disk-space usage policies and status.
2828-                 'disk_total' is the total size of disk where the storage
2829-                 server's BASEDIR/storage/shares directory lives, as reported
2830-                 by /bin/df or equivalent. 'disk_used', 'disk_free_for_root',
2831-                 and 'disk_free_for_nonroot' show related information.
2832-                 'reserved_space' reports the reservation configured by the
2833-                 tahoe.cfg [storage]reserved_space value. 'disk_avail'
2834-                 reports the remaining disk space available for the Tahoe
2835-                 server after subtracting reserved_space from disk_avail. All
2836-                 values are in bytes.
2837- accepting_immutable_shares: this is '1' if the storage server is currently
2838-                             accepting uploads of immutable shares. It may be
2839-                             '0' if a server is disabled by configuration, or
2840-                             if the disk is full (i.e. disk_avail is less
2841-                             than reserved_space).
2842- total_bucket_count: this counts the number of 'buckets' (i.e. unique
2843-                     storage-index values) currently managed by the storage
2844-                     server. It indicates roughly how many files are managed
2845-                     by the server.
2846- latencies.*.*: these stats keep track of local disk latencies for
2847-                storage-server operations. A number of percentile values are
2848-                tracked for many operations. For example,
2849-                'storage_server.latencies.readv.50_0_percentile' records the
2850-                median response time for a 'readv' request. All values are in
2851-                seconds. These are recorded by the storage server, starting
2852-                from the time the request arrives (post-deserialization) and
2853-                ending when the response begins serialization. As such, they
2854-                are mostly useful for measuring disk speeds. The operations
2855-                tracked are the same as the counters.storage_server.* counter
2856-                values (allocate, write, close, get, read, add-lease, renew,
2857-                cancel, readv, writev). The percentile values tracked are:
2858-                mean, 01_0_percentile, 10_0_percentile, 50_0_percentile,
2859-                90_0_percentile, 95_0_percentile, 99_0_percentile,
2860-                99_9_percentile. (the last value, 99.9 percentile, means that
2861-                999 out of the last 1000 operations were faster than the
2862-                given number, and is the same threshold used by Amazon's
2863-                internal SLA, according to the Dynamo paper).
2864+    this group counts inbound storage-server operations. They are not provided
2865+    by client-only nodes which have been configured to not run a storage server
2866+    (with [storage]enabled=false in tahoe.cfg)
2867+                           
2868+    allocate, write, close, abort
2869+        these are for immutable file uploads. 'allocate' is incremented when a
2870+        client asks if it can upload a share to the server. 'write' is
2871+        incremented for each chunk of data written. 'close' is incremented when
2872+        the share is finished. 'abort' is incremented if the client abandons
2873+        the upload.
2874 
2875hunk ./docs/stats.rst 55
2876-counters.uploader.files_uploaded
2877-counters.uploader.bytes_uploaded
2878-counters.downloader.files_downloaded
2879-counters.downloader.bytes_downloaded
2880+    get, read
2881+        these are for immutable file downloads. 'get' is incremented
2882+        when a client asks if the server has a specific share. 'read' is
2883+        incremented for each chunk of data read.
2884 
2885hunk ./docs/stats.rst 60
2886- These count client activity: a Tahoe client will increment these when it
2887- uploads or downloads an immutable file. 'files_uploaded' is incremented by
2888- one for each operation, while 'bytes_uploaded' is incremented by the size of
2889- the file.
2890+    readv, writev
2891+        these are for immutable file creation, publish, and retrieve. 'readv'
2892+        is incremented each time a client reads part of a mutable share.
2893+        'writev' is incremented each time a client sends a modification
2894+        request.
2895 
2896hunk ./docs/stats.rst 66
2897-counters.mutable.files_published
2898-counters.mutable.bytes_published
2899-counters.mutable.files_retrieved
2900-counters.mutable.bytes_retrieved
2901+    add-lease, renew, cancel
2902+        these are for share lease modifications. 'add-lease' is incremented
2903+        when an 'add-lease' operation is performed (which either adds a new
2904+        lease or renews an existing lease). 'renew' is for the 'renew-lease'
2905+        operation (which can only be used to renew an existing one). 'cancel'
2906+        is used for the 'cancel-lease' operation.
2907+
2908+    bytes_freed
2909+        this counts how many bytes were freed when a 'cancel-lease'
2910+        operation removed the last lease from a share and the share
2911+        was thus deleted.
2912+
2913+    bytes_added
2914+        this counts how many bytes were consumed by immutable share
2915+        uploads. It is incremented at the same time as the 'close'
2916+        counter.
2917+
2918+**stats.storage_server.\***
2919+
2920+    allocated
2921+        this counts how many bytes are currently 'allocated', which
2922+        tracks the space that will eventually be consumed by immutable
2923+        share upload operations. The stat is increased as soon as the
2924+        upload begins (at the same time the 'allocated' counter is
2925+        incremented), and goes back to zero when the 'close' or 'abort'
2926+        message is received (at which point the 'disk_used' stat should
2927+        incremented by the same amount).
2928+
2929+    disk_total, disk_used, disk_free_for_root, disk_free_for_nonroot, disk_avail, reserved_space
2930+        these all reflect disk-space usage policies and status.
2931+        'disk_total' is the total size of disk where the storage
2932+        server's BASEDIR/storage/shares directory lives, as reported
2933+        by /bin/df or equivalent. 'disk_used', 'disk_free_for_root',
2934+        and 'disk_free_for_nonroot' show related information.
2935+        'reserved_space' reports the reservation configured by the
2936+        tahoe.cfg [storage]reserved_space value. 'disk_avail'
2937+        reports the remaining disk space available for the Tahoe
2938+        server after subtracting reserved_space from disk_avail. All
2939+        values are in bytes.
2940+
2941+    accepting_immutable_shares
2942+        this is '1' if the storage server is currently accepting uploads of
2943+        immutable shares. It may be '0' if a server is disabled by
2944+        configuration, or if the disk is full (i.e. disk_avail is less than
2945+        reserved_space).
2946+
2947+    total_bucket_count
2948+        this counts the number of 'buckets' (i.e. unique
2949+        storage-index values) currently managed by the storage
2950+        server. It indicates roughly how many files are managed
2951+        by the server.
2952+
2953+    latencies.*.*
2954+        these stats keep track of local disk latencies for
2955+        storage-server operations. A number of percentile values are
2956+        tracked for many operations. For example,
2957+        'storage_server.latencies.readv.50_0_percentile' records the
2958+        median response time for a 'readv' request. All values are in
2959+        seconds. These are recorded by the storage server, starting
2960+        from the time the request arrives (post-deserialization) and
2961+        ending when the response begins serialization. As such, they
2962+        are mostly useful for measuring disk speeds. The operations
2963+        tracked are the same as the counters.storage_server.* counter
2964+        values (allocate, write, close, get, read, add-lease, renew,
2965+        cancel, readv, writev). The percentile values tracked are:
2966+        mean, 01_0_percentile, 10_0_percentile, 50_0_percentile,
2967+        90_0_percentile, 95_0_percentile, 99_0_percentile,
2968+        99_9_percentile. (the last value, 99.9 percentile, means that
2969+        999 out of the last 1000 operations were faster than the
2970+        given number, and is the same threshold used by Amazon's
2971+        internal SLA, according to the Dynamo paper).
2972+
2973+**counters.uploader.files_uploaded**
2974+
2975+**counters.uploader.bytes_uploaded**
2976+
2977+**counters.downloader.files_downloaded**
2978+
2979+**counters.downloader.bytes_downloaded**
2980+
2981+    These count client activity: a Tahoe client will increment these when it
2982+    uploads or downloads an immutable file. 'files_uploaded' is incremented by
2983+    one for each operation, while 'bytes_uploaded' is incremented by the size of
2984+    the file.
2985+
2986+**counters.mutable.files_published**
2987+
2988+**counters.mutable.bytes_published**
2989+
2990+**counters.mutable.files_retrieved**
2991+
2992+**counters.mutable.bytes_retrieved**
2993 
2994  These count client activity for mutable files. 'published' is the act of
2995  changing an existing mutable file (or creating a brand-new mutable file).
2996hunk ./docs/stats.rst 163
2997  'retrieved' is the act of reading its current contents.
2998 
2999-counters.chk_upload_helper.*
3000+**counters.chk_upload_helper.\***
3001 
3002hunk ./docs/stats.rst 165
3003- These count activity of the "Helper", which receives ciphertext from clients
3004- and performs erasure-coding and share upload for files that are not already
3005- in the grid. The code which implements these counters is in
3006- src/allmydata/immutable/offloaded.py .
3007+    These count activity of the "Helper", which receives ciphertext from clients
3008+    and performs erasure-coding and share upload for files that are not already
3009+    in the grid. The code which implements these counters is in
3010+    src/allmydata/immutable/offloaded.py .
3011 
3012hunk ./docs/stats.rst 170
3013-  upload_requests: incremented each time a client asks to upload a file
3014-  upload_already_present: incremented when the file is already in the grid
3015-  upload_need_upload: incremented when the file is not already in the grid
3016-  resumes: incremented when the helper already has partial ciphertext for
3017-           the requested upload, indicating that the client is resuming an
3018-           earlier upload
3019-  fetched_bytes: this counts how many bytes of ciphertext have been fetched
3020-                 from uploading clients
3021-  encoded_bytes: this counts how many bytes of ciphertext have been
3022-                 encoded and turned into successfully-uploaded shares. If no
3023-                 uploads have failed or been abandoned, encoded_bytes should
3024-                 eventually equal fetched_bytes.
3025+    upload_requests
3026+        incremented each time a client asks to upload a file
3027+        upload_already_present: incremented when the file is already in the grid
3028 
3029hunk ./docs/stats.rst 174
3030-stats.chk_upload_helper.*
3031+    upload_need_upload
3032+        incremented when the file is not already in the grid
3033 
3034hunk ./docs/stats.rst 177
3035- These also track Helper activity:
3036+    resumes
3037+        incremented when the helper already has partial ciphertext for
3038+        the requested upload, indicating that the client is resuming an
3039+        earlier upload
3040 
3041hunk ./docs/stats.rst 182
3042-  active_uploads: how many files are currently being uploaded. 0 when idle.
3043-  incoming_count: how many cache files are present in the incoming/ directory,
3044-                  which holds ciphertext files that are still being fetched
3045-                  from the client
3046-  incoming_size: total size of cache files in the incoming/ directory
3047-  incoming_size_old: total size of 'old' cache files (more than 48 hours)
3048-  encoding_count: how many cache files are present in the encoding/ directory,
3049-                  which holds ciphertext files that are being encoded and
3050-                  uploaded
3051-  encoding_size: total size of cache files in the encoding/ directory
3052-  encoding_size_old: total size of 'old' cache files (more than 48 hours)
3053+    fetched_bytes
3054+        this counts how many bytes of ciphertext have been fetched
3055+        from uploading clients
3056 
3057hunk ./docs/stats.rst 186
3058-stats.node.uptime: how many seconds since the node process was started
3059+    encoded_bytes
3060+        this counts how many bytes of ciphertext have been
3061+        encoded and turned into successfully-uploaded shares. If no
3062+        uploads have failed or been abandoned, encoded_bytes should
3063+        eventually equal fetched_bytes.
3064 
3065hunk ./docs/stats.rst 192
3066-stats.cpu_monitor.*:
3067-  .1min_avg, 5min_avg, 15min_avg: estimate of what percentage of system CPU
3068-                                  time was consumed by the node process, over
3069-                                  the given time interval. Expressed as a
3070-                                  float, 0.0 for 0%, 1.0 for 100%
3071-  .total: estimate of total number of CPU seconds consumed by node since
3072-          the process was started. Ticket #472 indicates that .total may
3073-          sometimes be negative due to wraparound of the kernel's counter.
3074+**stats.chk_upload_helper.\***
3075 
3076hunk ./docs/stats.rst 194
3077-stats.load_monitor.*:
3078- When enabled, the "load monitor" continually schedules a one-second
3079- callback, and measures how late the response is. This estimates system load
3080- (if the system is idle, the response should be on time). This is only
3081- enabled if a stats-gatherer is configured.
3082+    These also track Helper activity:
3083 
3084hunk ./docs/stats.rst 196
3085- .avg_load: average "load" value (seconds late) over the last minute
3086- .max_load: maximum "load" value over the last minute
3087+    active_uploads
3088+        how many files are currently being uploaded. 0 when idle.
3089+   
3090+    incoming_count
3091+        how many cache files are present in the incoming/ directory,
3092+        which holds ciphertext files that are still being fetched
3093+        from the client
3094 
3095hunk ./docs/stats.rst 204
3096+    incoming_size
3097+        total size of cache files in the incoming/ directory
3098 
3099hunk ./docs/stats.rst 207
3100-== Running a Tahoe Stats-Gatherer Service ==
3101+    incoming_size_old
3102+        total size of 'old' cache files (more than 48 hours)
3103+
3104+    encoding_count
3105+        how many cache files are present in the encoding/ directory,
3106+        which holds ciphertext files that are being encoded and
3107+        uploaded
3108+
3109+    encoding_size
3110+        total size of cache files in the encoding/ directory
3111+
3112+    encoding_size_old
3113+        total size of 'old' cache files (more than 48 hours)
3114+
3115+**stats.node.uptime**
3116+    how many seconds since the node process was started
3117+
3118+**stats.cpu_monitor.\***
3119+
3120+    1min_avg, 5min_avg, 15min_avg
3121+        estimate of what percentage of system CPU time was consumed by the
3122+        node process, over the given time interval. Expressed as a float, 0.0
3123+        for 0%, 1.0 for 100%
3124+
3125+    total
3126+        estimate of total number of CPU seconds consumed by node since
3127+        the process was started. Ticket #472 indicates that .total may
3128+        sometimes be negative due to wraparound of the kernel's counter.
3129+
3130+**stats.load_monitor.\***
3131+
3132+    When enabled, the "load monitor" continually schedules a one-second
3133+    callback, and measures how late the response is. This estimates system load
3134+    (if the system is idle, the response should be on time). This is only
3135+    enabled if a stats-gatherer is configured.
3136+
3137+    avg_load
3138+        average "load" value (seconds late) over the last minute
3139+
3140+    max_load
3141+        maximum "load" value over the last minute
3142+
3143+
3144+Running a Tahoe Stats-Gatherer Service
3145+======================================
3146 
3147 The "stats-gatherer" is a simple daemon that periodically collects stats from
3148 several tahoe nodes. It could be useful, e.g., in a production environment,
3149hunk ./docs/stats.rst 259
3150 host. It merely gatherers statistics from many nodes into a single place: it
3151 does not do any actual analysis.
3152 
3153-The stats gatherer listens on a network port using the same Foolscap
3154+The stats gatherer listens on a network port using the same Foolscap_
3155 connection library that Tahoe clients use to connect to storage servers.
3156 Tahoe nodes can be configured to connect to the stats gatherer and publish
3157hunk ./docs/stats.rst 262
3158-their stats on a periodic basis. (in fact, what happens is that nodes connect
3159+their stats on a periodic basis. (In fact, what happens is that nodes connect
3160 to the gatherer and offer it a second FURL which points back to the node's
3161 "stats port", which the gatherer then uses to pull stats on a periodic basis.
3162 The initial connection is flipped to allow the nodes to live behind NAT
3163hunk ./docs/stats.rst 266
3164-boxes, as long as the stats-gatherer has a reachable IP address)
3165+boxes, as long as the stats-gatherer has a reachable IP address.)
3166+
3167+.. _Foolscap: http://foolscap.lothar.com/trac
3168 
3169 The stats-gatherer is created in the same fashion as regular tahoe client
3170 nodes and introducer nodes. Choose a base directory for the gatherer to live
3171hunk ./docs/stats.rst 274
3172 in (but do not create the directory). Then run:
3173 
3174- tahoe create-stats-gatherer $BASEDIR
3175+::
3176+
3177+   tahoe create-stats-gatherer $BASEDIR
3178 
3179 and start it with "tahoe start $BASEDIR". Once running, the gatherer will
3180 write a FURL into $BASEDIR/stats_gatherer.furl .
3181hunk ./docs/stats.rst 285
3182 this FURL into the node's tahoe.cfg file, in a section named "[client]",
3183 under a key named "stats_gatherer.furl", like so:
3184 
3185- [client]
3186- stats_gatherer.furl = pb://qbo4ktl667zmtiuou6lwbjryli2brv6t@192.168.0.8:49997/wxycb4kaexzskubjnauxeoptympyf45y
3187+::
3188+
3189+    [client]
3190+    stats_gatherer.furl = pb://qbo4ktl667zmtiuou6lwbjryli2brv6t@192.168.0.8:49997/wxycb4kaexzskubjnauxeoptympyf45y
3191 
3192 or simply copy the stats_gatherer.furl file into the node's base directory
3193 (next to the tahoe.cfg file): it will be interpreted in the same way.
3194hunk ./docs/stats.rst 317
3195 total-disk-available number for the entire grid (however, the "disk watcher"
3196 daemon, in misc/operations_helpers/spacetime/, is better suited for this specific task).
3197 
3198-== Using Munin To Graph Stats Values ==
3199+Using Munin To Graph Stats Values
3200+=================================
3201 
3202 The misc/munin/ directory contains various plugins to graph stats for Tahoe
3203hunk ./docs/stats.rst 321
3204-nodes. They are intended for use with the Munin system-management tool, which
3205+nodes. They are intended for use with the Munin_ system-management tool, which
3206 typically polls target systems every 5 minutes and produces a web page with
3207 graphs of various things over multiple time scales (last hour, last month,
3208 last year).
3209hunk ./docs/stats.rst 326
3210 
3211+.. _Munin: http://munin-monitoring.org/
3212+
3213 Most of the plugins are designed to pull stats from a single Tahoe node, and
3214 are configured with the e.g. http://localhost:3456/statistics?t=json URL. The
3215 "tahoe_stats" plugin is designed to read from the pickle file created by the
3216}
3217
3218Context:
3219
3220[docs: add statement on our refusal to insert backdoors
3221zooko@zooko.com**20101006051147
3222 Ignore-this: 644d308319a7b80c4434bdff9760404a
3223]
3224[setup: add --multi-version to the "setup.py develop" command-line
3225zooko@zooko.com**20101005182350
3226 Ignore-this: 709155cc21caff29826b8d41a8c8d63d
3227 fixes #530. I earlier tried this twice (see #530 for history) and then twice rolled it back due to some problems that arose. However, I didn't write down what the problems were in enough detail on the ticket that I can tell today whether those problems are still issues, so here goes the third attempt. (I did write down on the ticket that it would not create site.py or .pth files in the target directory with --multi-version mode, but I didn't explain why *that* was a problem.)
3228]
3229[setup: use execfile to access _auto_deps.py in its proper location of src/allmydata/ instead of copying it into place when setup.py is executed
3230zooko@zooko.com**20100906055714
3231 Ignore-this: c179b42672d775580afad40121f86812
3232]
3233[trivial: M-x whitespace-cleanup
3234zooko@zooko.com**20100903144712
3235 Ignore-this: 1bb764d11ac69b4a35ea091cfb13158a
3236]
3237[minor: remove unused interface declaration, change allmydata.org to tahoe-lafs.org in email address, fix wording in relnotes.txt
3238zooko@zooko.com**20100930153708
3239 Ignore-this: a452969228afed2774de375e29fa3048
3240]
3241[immutable/repairer.py: don't use the default happiness setting when repairing
3242Kevan Carstensen <kevan@isnotajoke.com>**20100927200102
3243 Ignore-this: bd704d9744b970849da8d46a16b8089a
3244]
3245[NEWS: note dependency updates to pycryptopp and pycrypto.
3246david-sarah@jacaranda.org**20100924191207
3247 Ignore-this: eeaf5c9c9104f24c450c2ec4482ac1ee
3248]
3249[TAG allmydata-tahoe-1.8.0
3250zooko@zooko.com**20100924021631
3251 Ignore-this: 494ca0a885c5e20c883845fc53e7ab5d
3252]
3253Patch bundle hash:
3254d604191dabdc776d1b4bdb4400d7da625e13603e