Ticket #928: new-tests-for-928-darcspatch.txt

File new-tests-for-928-darcspatch.txt, 65.8 KB (added by davidsarah, at 2010-01-29T12:47:54Z)

New tests for #928 (patch bundle includes dontwait6). Problem with errback instead of callback fixed; all existing tests run green. Known bug in _copy_all_shares_from.

Line 
1Wed Jan 27 23:34:17 GMT Standard Time 2010  zooko@zooko.com
2  * immutable: download from the first servers which provide at least K buckets instead of waiting for all servers to reply
3  This should put an end to the phenomenon I've been seeing that a single hung server can cause all downloads on a grid to hang.  Also it should speed up all downloads by (a) not-waiting for responses to queries that it doesn't need, and (b) downloading shares from the servers which answered the initial query the fastest.
4  Also, do not count how many buckets you've gotten when deciding whether the download has enough shares or not -- instead count how many buckets to *unique* shares that you've gotten.  This appears to improve a slightly weird behavior in the current download code in which receiving >= K different buckets all to the same sharenumber would make it think it had enough to download the file when in fact it hadn't.
5  This patch needs tests before it is actually ready for trunk.
6
7Fri Jan 29 12:38:45 GMT Standard Time 2010  david-sarah@jacaranda.org
8  * New tests for #928
9
10New patches:
11
12[immutable: download from the first servers which provide at least K buckets instead of waiting for all servers to reply
13zooko@zooko.com**20100127233417
14 Ignore-this: c855355a40d96827e1d0c469a8d8ab3f
15 This should put an end to the phenomenon I've been seeing that a single hung server can cause all downloads on a grid to hang.  Also it should speed up all downloads by (a) not-waiting for responses to queries that it doesn't need, and (b) downloading shares from the servers which answered the initial query the fastest.
16 Also, do not count how many buckets you've gotten when deciding whether the download has enough shares or not -- instead count how many buckets to *unique* shares that you've gotten.  This appears to improve a slightly weird behavior in the current download code in which receiving >= K different buckets all to the same sharenumber would make it think it had enough to download the file when in fact it hadn't.
17 This patch needs tests before it is actually ready for trunk.
18] {
19hunk ./src/allmydata/immutable/download.py 791
20         self._opened = False
21 
22         self.active_buckets = {} # k: shnum, v: bucket
23-        self._share_buckets = [] # list of (sharenum, bucket) tuples
24+        self._share_buckets = {} # k: sharenum, v: list of buckets
25         self._share_vbuckets = {} # k: shnum, v: set of ValidatedBuckets
26 
27         self._fetch_failures = {"uri_extension": 0, "crypttext_hash_tree": 0, }
28hunk ./src/allmydata/immutable/download.py 872
29         return d
30 
31     def _get_all_shareholders(self):
32-        dl = []
33+        """ Once the number of buckets that I know about is >= K then I
34+        callback the Deferred that I return.
35+
36+        If all of the get_buckets deferreds have fired (whether callback or
37+        errback) and I still don't have enough buckets then I'll callback the
38+        Deferred that I return.
39+        """
40+        self._wait_for_enough_buckets_d = defer.Deferred()
41+
42+        self._queries_sent = 0
43+        self._responses_received = 0
44+        self._queries_failed = 0
45         sb = self._storage_broker
46         servers = sb.get_servers_for_index(self._storage_index)
47         if not servers:
48hunk ./src/allmydata/immutable/download.py 892
49             self.log(format="sending DYHB to [%(peerid)s]",
50                      peerid=idlib.shortnodeid_b2a(peerid),
51                      level=log.NOISY, umid="rT03hg")
52+            self._queries_sent += 1
53             d = ss.callRemote("get_buckets", self._storage_index)
54             d.addCallbacks(self._got_response, self._got_error,
55                            callbackArgs=(peerid,))
56hunk ./src/allmydata/immutable/download.py 896
57-            dl.append(d)
58-        self._responses_received = 0
59-        self._queries_sent = len(dl)
60         if self._status:
61             self._status.set_status("Locating Shares (%d/%d)" %
62                                     (self._responses_received,
63hunk ./src/allmydata/immutable/download.py 900
64                                      self._queries_sent))
65-        return defer.DeferredList(dl)
66+        return self._wait_for_enough_buckets_d
67 
68     def _got_response(self, buckets, peerid):
69         self.log(format="got results from [%(peerid)s]: shnums %(shnums)s",
70hunk ./src/allmydata/immutable/download.py 918
71         for sharenum, bucket in buckets.iteritems():
72             b = layout.ReadBucketProxy(bucket, peerid, self._storage_index)
73             self.add_share_bucket(sharenum, b)
74+            # If we just got enough buckets for the first time, then fire the
75+            # deferred. Then remove it from self so that we don't fire it
76+            # again.
77+            if self._wait_for_enough_buckets_d and len(self._share_buckets) >= self._verifycap.needed_shares:
78+                self._wait_for_enough_buckets_d.callback(True)
79+                self._wait_for_enough_buckets_d = None
80+
81+            # Else, if we ran out of outstanding requests then fire it and
82+            # remove it from self.
83+            assert (self._responses_received+self._queries_failed) <= self._queries_sent
84+            if self._wait_for_enough_buckets_d and (self._responses_received+self._queries_failed) == self._queries_sent:
85+                self._wait_for_enough_buckets_d.callback(False)
86+                self._wait_for_enough_buckets_d = None
87 
88             if self._results:
89                 if peerid not in self._results.servermap:
90hunk ./src/allmydata/immutable/download.py 939
91 
92     def add_share_bucket(self, sharenum, bucket):
93         # this is split out for the benefit of test_encode.py
94-        self._share_buckets.append( (sharenum, bucket) )
95+        self._share_buckets.setdefault(sharenum, []).append(bucket)
96 
97     def _got_error(self, f):
98         level = log.WEIRD
99hunk ./src/allmydata/immutable/download.py 947
100             level = log.UNUSUAL
101         self.log("Error during get_buckets", failure=f, level=level,
102                          umid="3uuBUQ")
103+        # If we ran out of outstanding requests then errback it and remove it
104+        # from self.
105+        self._queries_failed += 1
106+        assert (self._responses_received+self._queries_failed) <= self._queries_sent
107+        if self._wait_for_enough_buckets_d and self._responses_received == self._queries_sent:
108+            self._wait_for_enough_buckets_d.errback()
109+            self._wait_for_enough_buckets_d = None
110 
111     def bucket_failed(self, vbucket):
112         shnum = vbucket.sharenum
113hunk ./src/allmydata/immutable/download.py 996
114         uri_extension_fetch_started = time.time()
115 
116         vups = []
117-        for sharenum, bucket in self._share_buckets:
118-            vups.append(ValidatedExtendedURIProxy(bucket, self._verifycap, self._fetch_failures))
119+        for sharenum, buckets in self._share_buckets.iteritems():
120+            for bucket in buckets:
121+                vups.append(ValidatedExtendedURIProxy(bucket, self._verifycap, self._fetch_failures))
122         vto = ValidatedThingObtainer(vups, debugname="vups", log_id=self._parentmsgid)
123         d = vto.start()
124 
125hunk ./src/allmydata/immutable/download.py 1034
126 
127     def _get_crypttext_hash_tree(self, res):
128         vchtps = []
129-        for sharenum, bucket in self._share_buckets:
130-            vchtp = ValidatedCrypttextHashTreeProxy(bucket, self._crypttext_hash_tree, self._vup.num_segments, self._fetch_failures)
131-            vchtps.append(vchtp)
132+        for sharenum, buckets in self._share_buckets.iteritems():
133+            for bucket in buckets:
134+                vchtp = ValidatedCrypttextHashTreeProxy(bucket, self._crypttext_hash_tree, self._vup.num_segments, self._fetch_failures)
135+                vchtps.append(vchtp)
136 
137         _get_crypttext_hash_tree_started = time.time()
138         if self._status:
139hunk ./src/allmydata/immutable/download.py 1088
140 
141 
142     def _download_all_segments(self, res):
143-        for sharenum, bucket in self._share_buckets:
144-            vbucket = ValidatedReadBucketProxy(sharenum, bucket, self._share_hash_tree, self._vup.num_segments, self._vup.block_size, self._vup.share_size)
145-            self._share_vbuckets.setdefault(sharenum, set()).add(vbucket)
146+        for sharenum, buckets in self._share_buckets.iteritems():
147+            for bucket in buckets:
148+                vbucket = ValidatedReadBucketProxy(sharenum, bucket, self._share_hash_tree, self._vup.num_segments, self._vup.block_size, self._vup.share_size)
149+                self._share_vbuckets.setdefault(sharenum, set()).add(vbucket)
150 
151         # after the above code, self._share_vbuckets contains enough
152         # buckets to complete the download, and some extra ones to
153}
154[New tests for #928
155david-sarah@jacaranda.org**20100129123845
156 Ignore-this: 5c520f40141f0d9c000ffb05a4698995
157] {
158hunk ./src/allmydata/immutable/download.py 3
159 import random, weakref, itertools, time
160 from zope.interface import implements
161-from twisted.internet import defer
162+from twisted.internet import defer, reactor
163 from twisted.internet.interfaces import IPushProducer, IConsumer
164 from foolscap.api import DeadReferenceError, RemoteException, eventually
165 
166hunk ./src/allmydata/immutable/download.py 838
167 
168         # first step: who should we download from?
169         d = defer.maybeDeferred(self._get_all_shareholders)
170-        d.addCallback(self._got_all_shareholders)
171+        d.addBoth(self._got_all_shareholders)
172         # now get the uri_extension block from somebody and integrity check
173         # it and parse and validate its contents
174         d.addCallback(self._obtain_uri_extension)
175hunk ./src/allmydata/immutable/download.py 875
176         """ Once the number of buckets that I know about is >= K then I
177         callback the Deferred that I return.
178 
179-        If all of the get_buckets deferreds have fired (whether callback or
180-        errback) and I still don't have enough buckets then I'll callback the
181-        Deferred that I return.
182+        If all of the get_buckets deferreds have fired (whether callback
183+        or errback) and I still don't have enough buckets then I'll also
184+        callback -- not errback -- the Deferred that I return.
185         """
186hunk ./src/allmydata/immutable/download.py 879
187-        self._wait_for_enough_buckets_d = defer.Deferred()
188+        wait_for_enough_buckets_d = defer.Deferred()
189+        self._wait_for_enough_buckets_d = wait_for_enough_buckets_d
190 
191hunk ./src/allmydata/immutable/download.py 882
192-        self._queries_sent = 0
193-        self._responses_received = 0
194-        self._queries_failed = 0
195         sb = self._storage_broker
196         servers = sb.get_servers_for_index(self._storage_index)
197         if not servers:
198hunk ./src/allmydata/immutable/download.py 886
199             raise NoServersError("broker gave us no servers!")
200+
201+        self._total_queries = len(servers)
202+        self._responses_received = 0
203+        self._queries_failed = 0
204         for (peerid,ss) in servers:
205             self.log(format="sending DYHB to [%(peerid)s]",
206                      peerid=idlib.shortnodeid_b2a(peerid),
207hunk ./src/allmydata/immutable/download.py 894
208                      level=log.NOISY, umid="rT03hg")
209-            self._queries_sent += 1
210             d = ss.callRemote("get_buckets", self._storage_index)
211             d.addCallbacks(self._got_response, self._got_error,
212                            callbackArgs=(peerid,))
213hunk ./src/allmydata/immutable/download.py 897
214+            d.addBoth(self._check_got_all_responses)
215+
216         if self._status:
217             self._status.set_status("Locating Shares (%d/%d)" %
218hunk ./src/allmydata/immutable/download.py 901
219-                                    (self._responses_received,
220-                                     self._queries_sent))
221-        return self._wait_for_enough_buckets_d
222+                                    (len(self._share_buckets),
223+                                     self._verifycap.needed_shares))
224+        return wait_for_enough_buckets_d
225+
226+    def _check_got_all_responses(self, ignored=None):
227+        assert (self._responses_received+self._queries_failed) <= self._total_queries
228+        if self._wait_for_enough_buckets_d and (self._responses_received+self._queries_failed) == self._total_queries:
229+            reactor.callLater(0, self._wait_for_enough_buckets_d.callback, False)
230+            self._wait_for_enough_buckets_d = None
231 
232     def _got_response(self, buckets, peerid):
233hunk ./src/allmydata/immutable/download.py 912
234+        self._responses_received += 1
235         self.log(format="got results from [%(peerid)s]: shnums %(shnums)s",
236                  peerid=idlib.shortnodeid_b2a(peerid),
237                  shnums=sorted(buckets.keys()),
238hunk ./src/allmydata/immutable/download.py 917
239                  level=log.NOISY, umid="o4uwFg")
240-        self._responses_received += 1
241         if self._results:
242             elapsed = time.time() - self._started
243             self._results.timings["servers_peer_selection"][peerid] = elapsed
244hunk ./src/allmydata/immutable/download.py 923
245         if self._status:
246             self._status.set_status("Locating Shares (%d/%d)" %
247                                     (self._responses_received,
248-                                     self._queries_sent))
249+                                     self._total_queries))
250         for sharenum, bucket in buckets.iteritems():
251             b = layout.ReadBucketProxy(bucket, peerid, self._storage_index)
252             self.add_share_bucket(sharenum, b)
253hunk ./src/allmydata/immutable/download.py 931
254             # deferred. Then remove it from self so that we don't fire it
255             # again.
256             if self._wait_for_enough_buckets_d and len(self._share_buckets) >= self._verifycap.needed_shares:
257-                self._wait_for_enough_buckets_d.callback(True)
258-                self._wait_for_enough_buckets_d = None
259-
260-            # Else, if we ran out of outstanding requests then fire it and
261-            # remove it from self.
262-            assert (self._responses_received+self._queries_failed) <= self._queries_sent
263-            if self._wait_for_enough_buckets_d and (self._responses_received+self._queries_failed) == self._queries_sent:
264-                self._wait_for_enough_buckets_d.callback(False)
265+                reactor.callLater(0, self._wait_for_enough_buckets_d.callback, True)
266                 self._wait_for_enough_buckets_d = None
267 
268             if self._results:
269hunk ./src/allmydata/immutable/download.py 944
270         self._share_buckets.setdefault(sharenum, []).append(bucket)
271 
272     def _got_error(self, f):
273+        self._queries_failed += 1
274         level = log.WEIRD
275         if f.check(DeadReferenceError):
276             level = log.UNUSUAL
277hunk ./src/allmydata/immutable/download.py 950
278         self.log("Error during get_buckets", failure=f, level=level,
279                          umid="3uuBUQ")
280-        # If we ran out of outstanding requests then errback it and remove it
281-        # from self.
282-        self._queries_failed += 1
283-        assert (self._responses_received+self._queries_failed) <= self._queries_sent
284-        if self._wait_for_enough_buckets_d and self._responses_received == self._queries_sent:
285-            self._wait_for_enough_buckets_d.errback()
286-            self._wait_for_enough_buckets_d = None
287 
288     def bucket_failed(self, vbucket):
289         shnum = vbucket.sharenum
290hunk ./src/allmydata/test/no_network.py 19
291 import os.path
292 from zope.interface import implements
293 from twisted.application import service
294-from twisted.internet import reactor
295+from twisted.internet import defer, reactor
296 from twisted.python.failure import Failure
297 from foolscap.api import Referenceable, fireEventually, RemoteException
298 from base64 import b32encode
299hunk ./src/allmydata/test/no_network.py 41
300     def __init__(self, original):
301         self.original = original
302         self.broken = False
303+        self.hung_until = None
304         self.post_call_notifier = None
305         self.disconnectors = {}
306 
307hunk ./src/allmydata/test/no_network.py 61
308                 return a
309         args = tuple([wrap(a) for a in args])
310         kwargs = dict([(k,wrap(kwargs[k])) for k in kwargs])
311+
312+        def _really_call():
313+            meth = getattr(self.original, "remote_" + methname)
314+            return meth(*args, **kwargs)
315+
316         def _call():
317             if self.broken:
318                 raise IntentionalError("I was asked to break")
319hunk ./src/allmydata/test/no_network.py 69
320-            meth = getattr(self.original, "remote_" + methname)
321-            return meth(*args, **kwargs)
322+            if self.hung_until:
323+                d2 = defer.Deferred()
324+                self.hung_until.addCallback(lambda ign: _really_call())
325+                self.hung_until.addCallback(lambda res: d2.callback(res))
326+                def _err(res):
327+                    d2.errback(res)
328+                    return res
329+                self.hung_until.addErrback(_err)
330+                return d2
331+            return _really_call()
332+
333         d = fireEventually()
334         d.addCallback(lambda res: _call())
335         def _wrap_exception(f):
336hunk ./src/allmydata/test/no_network.py 258
337         # asked to hold a share
338         self.servers_by_id[serverid].broken = True
339 
340+    def hang_server(self, serverid, until=defer.Deferred()):
341+        # hang the given server until 'until' fires
342+        self.servers_by_id[serverid].hung_until = until
343+
344+
345 class GridTestMixin:
346     def setUp(self):
347         self.s = service.MultiService()
348addfile ./src/allmydata/test/test_hung_server.py
349hunk ./src/allmydata/test/test_hung_server.py 1
350+
351+import os, shutil
352+from twisted.trial import unittest
353+from twisted.internet import defer, reactor
354+from twisted.python import failure
355+from allmydata import uri
356+from allmydata.util.consumer import download_to_data
357+from allmydata.immutable import upload
358+from allmydata.storage.common import storage_index_to_dir
359+from allmydata.test.no_network import GridTestMixin
360+from allmydata.test.common import ShouldFailMixin
361+from allmydata.interfaces import NotEnoughSharesError
362+
363+immutable_plaintext = "data" * 10000
364+mutable_plaintext = "muta" * 10000
365+
366+class HungServerDownloadTest(GridTestMixin, ShouldFailMixin, unittest.TestCase):
367+    timeout = 30
368+
369+    def _break(self, servers):
370+        for (id, ss) in servers:
371+            self.g.break_server(id)
372+
373+    def _hang(self, servers, **kwargs):
374+        for (id, ss) in servers:
375+            self.g.hang_server(id, **kwargs)
376+
377+    def _delete_all_shares_from(self, servers):
378+        serverids = [id for (id, ss) in servers]
379+        for (i_shnum, i_serverid, i_sharefile) in self.shares:
380+            if i_serverid in serverids:
381+                os.unlink(i_sharefile)
382+
383+    # untested
384+    def _pick_a_share_from(self, server):
385+        (id, ss) = server
386+        for (i_shnum, i_serverid, i_sharefile) in self.shares:
387+            if i_serverid == id:
388+                return (i_shnum, i_sharefile)
389+        raise AssertionError("server %r had no shares" % server)
390+
391+    # untested
392+    def _copy_all_shares_from(self, from_servers, to_server):
393+        serverids = [id for (id, ss) in from_servers]
394+        for (i_shnum, i_serverid, i_sharefile) in self.shares:
395+            if i_serverid in serverids:
396+                self._copy_share((i_shnum, i_sharefile), to_server)
397+
398+    # untested
399+    def _copy_share(self, share, to_server):
400+         (sharenum, sharefile) = share
401+         (id, ss) = to_server
402+         # FIXME: this doesn't work because we only have a LocalWrapper
403+         shares_dir = os.path.join(ss.storedir, "shares")
404+         si = uri.from_string(self.uri).get_storage_index()
405+         si_dir = os.path.join(shares_dir, storage_index_to_dir(si))
406+         if not os.path.exists(si_dir):
407+             os.makedirs(si_dir)
408+         new_sharefile = os.path.join(si_dir, str(sharenum))
409+         shutil.copy(sharefile, new_sharefile)
410+         self.shares = self.find_shares(self.uri)
411+         # Make sure that the storage server has the share.
412+         self.failUnless((sharenum, ss.my_nodeid, new_sharefile)
413+                         in self.shares)
414+
415+    # untested
416+    def _add_server(self, server_number, readonly=False):
417+        ss = self.g.make_server(server_number, readonly)
418+        self.g.add_server(server_number, ss)
419+        self.shares = self.find_shares(self.uri)
420+
421+    def _set_up(self, testdir, num_clients=1, num_servers=10):
422+        self.basedir = "download/" + testdir
423+        self.set_up_grid(num_clients=num_clients, num_servers=num_servers)
424+
425+        self.c0 = self.g.clients[0]
426+        sb = self.c0.nodemaker.storage_broker
427+        self.servers = [(id, ss) for (id, ss) in sb.get_all_servers()]
428+
429+        data = upload.Data(immutable_plaintext, convergence="")
430+        d = self.c0.upload(data)
431+        def _uploaded(ur):
432+            self.uri = ur.uri
433+            self.shares = self.find_shares(self.uri)
434+        d.addCallback(_uploaded)
435+        return d
436+
437+    def test_10_good_sanity_check(self):
438+        d = self._set_up("test_10_good_sanity_check")
439+        d.addCallback(lambda ign: self.download_immutable())
440+        return d
441+
442+    def test_3_good_7_hung(self):
443+        d = self._set_up("test_3_good_7_hung")
444+        d.addCallback(lambda ign: self._hang(self.servers[3:]))
445+        d.addCallback(lambda ign: self.download_immutable())
446+        return d
447+
448+    def test_3_good_7_noshares(self):
449+        d = self._set_up("test_3_good_7_noshares")
450+        d.addCallback(lambda ign: self._delete_all_shares_from(self.servers[3:]))
451+        d.addCallback(lambda ign: self.download_immutable())
452+        return d
453+
454+    def test_2_good_8_broken_fail(self):
455+        d = self._set_up("test_2_good_8_broken_fail")
456+        d.addCallback(lambda ign: self._break(self.servers[2:]))
457+        d.addCallback(lambda ign:
458+                      self.shouldFail(NotEnoughSharesError, "test_2_good_8_broken_fail",
459+                                      "Failed to get enough shareholders: have 2, need 3",
460+                                      self.download_immutable))
461+        return d
462+
463+    def test_2_good_8_noshares_fail(self):
464+        d = self._set_up("test_2_good_8_noshares_fail")
465+        d.addCallback(lambda ign: self._delete_all_shares_from(self.servers[2:]))
466+        d.addCallback(lambda ign:
467+                      self.shouldFail(NotEnoughSharesError, "test_2_good_8_noshares_fail",
468+                                      "Failed to get enough shareholders: have 2, need 3",
469+                                      self.download_immutable))
470+        return d
471+
472+    def test_2_good_8_hung_then_1_recovers(self):
473+        recovered = defer.Deferred()
474+        d = self._set_up("test_2_good_8_hung_then_1_recovers")
475+        d.addCallback(lambda ign: self._hang(self.servers[2:3], until=recovered))
476+        d.addCallback(lambda ign: self._hang(self.servers[3:]))
477+        d.addCallback(lambda ign: self.download_immutable())
478+        reactor.callLater(5, recovered.callback, None)
479+        return d
480+
481+    def test_2_good_8_hung_then_1_recovers_with_2_shares(self):
482+        recovered = defer.Deferred()
483+        d = self._set_up("test_2_good_8_hung_then_1_recovers_with_2_shares")
484+        d.addCallback(lambda ign: self._copy_all_shares_from(self.servers[0:1], self.servers[2]))
485+        d.addCallback(lambda ign: self._hang(self.servers[2:3], until=recovered))
486+        d.addCallback(lambda ign: self._hang(self.servers[3:]))
487+        d.addCallback(lambda ign: self.download_immutable())
488+        reactor.callLater(5, recovered.callback, None)
489+        return d
490+
491+    def download_immutable(self):
492+        n = self.c0.create_node_from_uri(self.uri)
493+        d = download_to_data(n)
494+        def _got_data(data):
495+            self.failUnlessEqual(data, immutable_plaintext)
496+        d.addCallback(_got_data)
497+        return d
498+
499+    # unused
500+    def download_mutable(self):
501+        n = self.c0.create_node_from_uri(self.uri)
502+        d = n.download_best_version()
503+        def _got_data(data):
504+            self.failUnlessEqual(data, mutable_plaintext)
505+        d.addCallback(_got_data)
506+        return d
507}
508
509Context:
510
511[test_runner: cleanup, refactor common code into a non-executable method
512Brian Warner <warner@lothar.com>**20100127224040
513 Ignore-this: 4cb4aada87777771f688edfd8129ffca
514 
515 Having both test_node() and test_client() (one of which calls the other) felt
516 confusing to me, so I changed it to have test_node(), test_client(), and a
517 common do_create() helper method.
518]
519[scripts/runner.py: simplify David-Sarah's clever grouped-commands usage trick
520Brian Warner <warner@lothar.com>**20100127223758
521 Ignore-this: 70877ebf06ae59f32960b0aa4ce1d1ae
522]
523[tahoe backup: skip all symlinks, with warning. Fixes #850, addresses #641.
524Brian Warner <warner@lothar.com>**20100127223517
525 Ignore-this: ab5cf05158d32a575ca8efc0f650033f
526]
527[NEWS: update with all recent user-visible changes
528Brian Warner <warner@lothar.com>**20100127222209
529 Ignore-this: 277d24568018bf4f3fb7736fda64eceb
530]
531["tahoe backup": fix --exclude-vcs docs to include Git
532Brian Warner <warner@lothar.com>**20100127201044
533 Ignore-this: 756a58dde21bdc65aa62b81803605b5
534]
535[docs: fix references to --no-storage, explanation of [storage] section
536Brian Warner <warner@lothar.com>**20100127200956
537 Ignore-this: f4be1763a585e1ac6299a4f1b94a59e0
538]
539[cli: merge the better version of David-Sarah's split-usage-and-help patch with the earlier version that I mistakenly committed
540zooko@zooko.com**20100126044559
541 Ignore-this: 284d188e13b7901013cbb650168e6447
542]
543[Split tahoe --help options into groups.
544david-sarah@jacaranda.org**20100112043935
545 Ignore-this: 610f9c41b00e6863e3cd047379733e3a
546]
547[Miscellaneous documentation, test, and code formatting tweaks.
548david-sarah@jacaranda.org**20100127070309
549 Ignore-this: 84ca7e4bb7c64221ae2c61144ef5edef
550]
551[Prevent mutable objects from being retrieved from an immutable directory, and associated forward-compatibility improvements.
552david-sarah@jacaranda.org**20100127064430
553 Ignore-this: 5ef6a3554cf6bef0bf0712cc7d6c0252
554]
555[docs: further CREDITS level-ups for Nils, Kevan, David-Sarah
556zooko@zooko.com**20100126170021
557 Ignore-this: 1e513e85cf7b7abf57f056e6d7544b38
558]
559[ftpd: clearer error message if Twisted needs a patch (by Nils Durner)
560zooko@zooko.com**20100126143411
561 Ignore-this: 440e6831ae6da5135c1edd081c93871f
562]
563[Add 'docs/performance.txt', which (for the moment) describes mutable file performance issues
564Kevan Carstensen <kevan@isnotajoke.com>**20100115204500
565 Ignore-this: ade4e500217db2509aee35aacc8c5dbf
566]
567[docs: more CREDITS for François, Kevan, and David-Sarah
568zooko@zooko.com**20100126132133
569 Ignore-this: f37d4977c13066fcac088ba98a31b02e
570]
571[tahoe_backup.py: display warnings on errors instead of stopping the whole backup. Fix #729.
572francois@ctrlaltdel.ch**20100120094249
573 Ignore-this: 7006ea4b0910b6d29af6ab4a3997a8f9
574 
575 This patch displays a warning to the user in two cases:
576   
577   1. When special files like symlinks, fifos, devices, etc. are found in the
578      local source.
579   
580   2. If files or directories are not readables by the user running the 'tahoe
581      backup' command.
582 
583 In verbose mode, the number of skipped files and directories is printed at the
584 end of the backup.
585 
586 Exit status returned by 'tahoe backup':
587 
588   - 0 everything went fine
589   - 1 the backup failed
590   - 2 files were skipped during the backup
591 
592]
593[Message saying that we couldn't find bin/tahoe should say where we looked
594david-sarah@jacaranda.org**20100116204556
595 Ignore-this: 1068576fd59ea470f1e19196315d1bb
596]
597[Change running.html to describe 'tahoe run'
598david-sarah@jacaranda.org**20100112044409
599 Ignore-this: 23ad0114643ce31b56e19bb14e011e4f
600]
601[cli: split usage strings into groups (patch by David-Sarah Hopwood)
602zooko@zooko.com**20100126043921
603 Ignore-this: 51928d266a7292b873f87f7d53c9a01e
604]
605[Add create-node CLI command, and make create-client equivalent to create-node --no-storage (fixes #760)
606david-sarah@jacaranda.org**20100116052055
607 Ignore-this: 47d08b18c69738685e13ff365738d5a
608]
609[contrib/fuse/runtests.py: Fix #888, configure settings in tahoe.cfg and don't treat warnings as failure
610francois@ctrlaltdel.ch**20100109123010
611 Ignore-this: 2590d44044acd7dfa3690c416cae945c
612 
613 Fix a few bitrotten pieces in the FUSE test script.  It now configures tahoe
614 node settings by editing tahoe.cfg which is the new supported method.
615 
616 It alos tolerate warnings issued by the mount command, the cause of these
617 warnings is the same as in #876 (contrib/fuse/runtests.py doesn't tolerate
618 deprecations warnings).
619 
620]
621[Fix webapi t=mkdir with multpart/form-data, as on the Welcome page. Closes #919.
622Brian Warner <warner@lothar.com>**20100121065052
623 Ignore-this: 1f20ea0a0f1f6d6c1e8e14f193a92c87
624]
625[Fix boodlegrid use of set_children
626david-sarah@jacaranda.org**20100126063414
627 Ignore-this: 3aa2d4836f76303b2bacecd09611f999
628]
629[Remove replace= parameter to mkdir-immutable and mkdir-with-children
630david-sarah@jacaranda.org**20100124224325
631 Ignore-this: 25207bcc946c0c43d9528718e76ba7b
632]
633[Warn about test failures due to setting FLOG* env vars
634david-sarah@jacaranda.org**20100124220629
635 Ignore-this: 1c25247ca0f0840390a1b7259a9f4a3c
636]
637[Patch to accept t=set-children as well as t=set_children
638david-sarah@jacaranda.org**20100124030020
639 Ignore-this: 2c061f12af817cdf77feeeb64098ec3a
640]
641[tahoe_add_alias.py: minor refactoring
642Brian Warner <warner@lothar.com>**20100115064220
643 Ignore-this: 29910e81ad11209c9e493d65fd2dab9b
644]
645[test_dirnode.py: reduce scope of a Client instance, suggested by Kevan.
646Brian Warner <warner@lothar.com>**20100115062713
647 Ignore-this: b35efd9e6027e43de6c6f509bfb4ccaa
648]
649[test_provisioning: STAN is not always a list. Fix by David-Sarah Hopwood.
650Brian Warner <warner@lothar.com>**20100115014632
651 Ignore-this: 9989de7f1e00907706d2b63153138219
652]
653[web/directory.py mkdir-immutable: hush pyflakes, add TODO for #903 behavior
654Brian Warner <warner@lothar.com>**20100114222804
655 Ignore-this: 717cd3b9a1c8aeee76938c9641db7356
656]
657[hush pyflakes-0.4.0 warnings: slightly less-trivial fixes. Closes #900.
658Brian Warner <warner@lothar.com>**20100114221719
659 Ignore-this: f774f4637e256ad55502659413a811a8
660 
661 This includes one fix (in test_web) which was testing the wrong thing.
662]
663[hush pyflakes-0.4.0 warnings: remove trivial unused variables. For #900.
664Brian Warner <warner@lothar.com>**20100114221529
665 Ignore-this: e96106c8f1a99fbf93306fbfe9a294cf
666]
667[tahoe add-alias/create-alias: don't corrupt non-newline-terminated alias
668Brian Warner <warner@lothar.com>**20100114210246
669 Ignore-this: 9c994792e53a85159d708760a9b1b000
670 file. Closes #741.
671]
672[change docs and --help to use "grid" instead of "virtual drive": closes #892.
673Brian Warner <warner@lothar.com>**20100114201119
674 Ignore-this: a20d4a4dcc4de4e3b404ff72d40fc29b
675 
676 Thanks to David-Sarah Hopwood for the patch.
677]
678[backupdb.txt: fix ST_CTIME reference
679Brian Warner <warner@lothar.com>**20100114194052
680 Ignore-this: 5a189c7a1181b07dd87f0a08ea31b6d3
681]
682[client.py: fix/update comments on KeyGenerator
683Brian Warner <warner@lothar.com>**20100113004226
684 Ignore-this: 2208adbb3fd6a911c9f44e814583cabd
685]
686[Clean up log.err calls, for one of the issues in #889.
687Brian Warner <warner@lothar.com>**20100112013343
688 Ignore-this: f58455ce15f1fda647c5fb25d234d2db
689 
690 allmydata.util.log.err() either takes a Failure as the first positional
691 argument, or takes no positional arguments and must be invoked in an
692 exception handler. Fixed its signature to match both foolscap.logging.log.err
693 and twisted.python.log.err . Included a brief unit test.
694]
695[tidy up DeadReferenceError handling, ignore them in add_lease calls
696Brian Warner <warner@lothar.com>**20100112000723
697 Ignore-this: 72f1444e826fd0b9db6d318f89603c38
698 
699 Stop checking separately for ConnectionDone/ConnectionLost, since those have
700 been folded into DeadReferenceError since foolscap-0.3.1 . Write
701 rrefutil.trap_deadref() in terms of rrefutil.trap_and_discard() to improve
702 code coverage.
703]
704[NEWS: improve "tahoe backup" notes, mention first-backup-after-upgrade duration
705Brian Warner <warner@lothar.com>**20100111190132
706 Ignore-this: 10347c590b3375964579ba6c2b0edb4f
707 
708 Thanks to Francois Deppierraz for the suggestion.
709]
710[test_repairer: add (commented-out) test_each_byte, to see exactly what the
711Brian Warner <warner@lothar.com>**20100110203552
712 Ignore-this: 8e84277d5304752edeff052b97821815
713 Verifier misses
714 
715 The results (described in #819) match our expectations: it misses corruption
716 in unused share fields and in most container fields (which are only visible
717 to the storage server, not the client). 1265 bytes of a 2753 byte
718 share (hosting a 56-byte file with an artifically small segment size) are
719 unused, mostly in the unused tail of the overallocated UEB space (765 bytes),
720 and the allocated-but-unwritten plaintext_hash_tree (480 bytes).
721]
722[repairer: fix some wrong offsets in the randomized verifier tests, debugged by Brian
723zooko@zooko.com**20100110203721
724 Ignore-this: 20604a609db8706555578612c1c12feb
725 fixes #819
726]
727[test_repairer: fix colliding basedir names, which caused test inconsistencies
728Brian Warner <warner@lothar.com>**20100110084619
729 Ignore-this: b1d56dd27e6ab99a7730f74ba10abd23
730]
731[repairer: add deterministic test for #819, mark as TODO
732zooko@zooko.com**20100110013619
733 Ignore-this: 4cb8bb30b25246de58ed2b96fa447d68
734]
735[contrib/fuse/runtests.py: Tolerate the tahoe CLI returning deprecation warnings
736francois@ctrlaltdel.ch**20100109175946
737 Ignore-this: 419c354d9f2f6eaec03deb9b83752aee
738 
739 Depending on the versions of external libraries such as Twisted of Foolscap,
740 the tahoe CLI can display deprecation warnings on stdout.  The tests should
741 not interpret those warnings as a failure if the node is in fact correctly
742 started.
743   
744 See http://allmydata.org/trac/tahoe/ticket/859 for an example of deprecation
745 warnings.
746 
747 fixes #876
748]
749[contrib: fix fuse_impl_c to use new Python API
750zooko@zooko.com**20100109174956
751 Ignore-this: 51ca1ec7c2a92a0862e9b99e52542179
752 original patch by Thomas Delaet, fixed by François, reviewed by Brian, committed by me
753]
754[docs: CREDITS: add David-Sarah to the CREDITS file
755zooko@zooko.com**20100109060435
756 Ignore-this: 896062396ad85f9d2d4806762632f25a
757]
758[mutable/publish: don't loop() right away upon DeadReferenceError. Closes #877
759Brian Warner <warner@lothar.com>**20100102220841
760 Ignore-this: b200e707b3f13aa8251981362b8a3e61
761 
762 The bug was that a disconnected server could cause us to re-enter the initial
763 loop() call, sending multiple queries to a single server, provoking an
764 incorrect UCWE. To fix it, stall the loop() with an eventual.fireEventually()
765]
766[immutable/checker.py: oops, forgot some imports. Also hush pyflakes.
767Brian Warner <warner@lothar.com>**20091229233909
768 Ignore-this: 4d61bd3f8113015a4773fd4768176e51
769]
770[mutable repair: return successful=False when numshares<k (thus repair fails),
771Brian Warner <warner@lothar.com>**20091229233746
772 Ignore-this: d881c3275ff8c8bee42f6a80ca48441e
773 instead of weird errors. Closes #874 and #786.
774 
775 Previously, if the file had 0 shares, this would raise TypeError as it tried
776 to call download_version(None). If the file had some shares but fewer than
777 'k', it would incorrectly raise MustForceRepairError.
778 
779 Added get_successful() to the IRepairResults API, to give repair() a place to
780 report non-code-bug problems like this.
781]
782[node.py/interfaces.py: minor docs fixes
783Brian Warner <warner@lothar.com>**20091229230409
784 Ignore-this: c86ad6342ef0f95d50639b4f99cd4ddf
785]
786[NEWS: fix 1.4.1 announcement w.r.t. add-lease behavior in older releases
787Brian Warner <warner@lothar.com>**20091229230310
788 Ignore-this: bbbbb9c961f3bbcc6e5dbe0b1594822
789]
790[checker: don't let failures in add-lease affect checker results. Closes #875.
791Brian Warner <warner@lothar.com>**20091229230108
792 Ignore-this: ef1a367b93e4d01298c2b1e6ca59c492
793 
794 Mutable servermap updates and the immutable checker, when run with
795 add_lease=True, send both the do-you-have-block and add-lease commands in
796 parallel, to avoid an extra round trip time. Many older servers have problems
797 with add-lease and raise various exceptions, which don't generally matter.
798 The client-side code was catching+ignoring some of them, but unrecognized
799 exceptions were passed through to the DYHB code, concealing the DYHB results
800 from the checker, making it think the server had no shares.
801 
802 The fix is to separate the code paths. Both commands are sent at the same
803 time, but the errback path from add-lease is handled separately. Known
804 exceptions are ignored, the others (both unknown-remote and all-local) are
805 logged (log.WEIRD, which will trigger an Incident), but neither will affect
806 the DYHB results.
807 
808 The add-lease message is sent first, and we know that the server handles them
809 synchronously. So when the checker is done, we can be sure that all the
810 add-lease messages have been retired. This makes life easier for unit tests.
811]
812[test_cli: verify fix for "tahoe get" not creating empty file on error (#121)
813Brian Warner <warner@lothar.com>**20091227235444
814 Ignore-this: 6444d52413b68eb7c11bc3dfdc69c55f
815]
816[addendum to "Fix 'tahoe ls' on files (#771)"
817Brian Warner <warner@lothar.com>**20091227232149
818 Ignore-this: 6dd5e25f8072a3153ba200b7fdd49491
819 
820 tahoe_ls.py: tolerate missing metadata
821 web/filenode.py: minor cleanups
822 test_cli.py: test 'tahoe ls FILECAP'
823]
824[Fix 'tahoe ls' on files (#771). Patch adapted from Kevan Carstensen.
825Brian Warner <warner@lothar.com>**20091227225443
826 Ignore-this: 8bf8c7b1cd14ea4b0ebd453434f4fe07
827 
828 web/filenode.py: also serve edge metadata when using t=json on a
829                  DIRCAP/childname object.
830 tahoe_ls.py: list file objects as if we were listing one-entry directories.
831              Show edge metadata if we have it, which will be true when doing
832              'tahoe ls DIRCAP/filename' and false when doing 'tahoe ls
833              FILECAP'
834]
835[tahoe_get: don't create the output file on error. Closes #121.
836Brian Warner <warner@lothar.com>**20091227220404
837 Ignore-this: 58d5e793a77ec6e87d9394ade074b926
838]
839[webapi: don't accept zero-length childnames during traversal. Closes #358, #676.
840Brian Warner <warner@lothar.com>**20091227201043
841 Ignore-this: a9119dec89e1c7741f2289b0cad6497b
842 
843 This forbids operations that would implicitly create a directory with a
844 zero-length (empty string) name, like what you'd get if you did "tahoe put
845 local /oops/blah" (#358) or "POST /uri/CAP//?t=mkdir" (#676). The error
846 message is fairly friendly too.
847 
848 Also added code to "tahoe put" to catch this error beforehand and suggest the
849 correct syntax (i.e. without the leading slash).
850]
851[CLI: send 'Accept:' header to ask for text/plain tracebacks. Closes #646.
852Brian Warner <warner@lothar.com>**20091227195828
853 Ignore-this: 44c258d4d4c7dac0ed58adb22f73331
854 
855 The webapi has been looking for an Accept header since 1.4.0, but it treats a
856 missing header as equal to */* (to honor RFC2616). This change finally
857 modifies our CLI tools to ask for "text/plain, application/octet-stream",
858 which seems roughly correct (we either want a plain-text traceback or error
859 message, or an uninterpreted chunk of binary data to save to disk). Some day
860 we'll figure out how JSON fits into this scheme.
861]
862[Makefile: upload-tarballs: switch from xfer-client to flappclient, closes #350
863Brian Warner <warner@lothar.com>**20091227163703
864 Ignore-this: 3beeecdf2ad9c2438ab57f0e33dcb357
865 
866 I've also set up a new flappserver on source@allmydata.org to receive the
867 tarballs. We still need to replace the gutsy buildslave (which is where the
868 tarballs used to be generated+uploaded) and give it the new FURL.
869]
870[misc/ringsim.py: make it deterministic, more detail about grid-is-full behavior
871Brian Warner <warner@lothar.com>**20091227024832
872 Ignore-this: a691cc763fb2e98a4ce1767c36e8e73f
873]
874[misc/ringsim.py: tool to discuss #302
875Brian Warner <warner@lothar.com>**20091226060339
876 Ignore-this: fc171369b8f0d97afeeb8213e29d10ed
877]
878[docs: fix helper.txt to describe new config style
879zooko@zooko.com**20091224223522
880 Ignore-this: 102e7692dc414a4b466307f7d78601fe
881]
882[docs/stats.txt: add TOC, notes about controlling gatherer's listening port
883Brian Warner <warner@lothar.com>**20091224202133
884 Ignore-this: 8eef63b0e18db5aa8249c2eafde02c05
885 
886 Thanks to Jody Harris for the suggestions.
887]
888[Add docs/stats.py, explaining Tahoe stats, the gatherer, and the munin plugins.
889Brian Warner <warner@lothar.com>**20091223052400
890 Ignore-this: 7c9eeb6e5644eceda98b59a67730ccd5
891]
892[more #859: avoid deprecation warning for unit tests too, hush pyflakes
893Brian Warner <warner@lothar.com>**20091215000147
894 Ignore-this: 193622e24d31077da825a11ed2325fd3
895 
896 * factor maybe-import-sha logic into util.hashutil
897]
898[use hashlib module if available, thus avoiding a DeprecationWarning for importing the old sha module; fixes #859
899zooko@zooko.com**20091214212703
900 Ignore-this: 8d0f230a4bf8581dbc1b07389d76029c
901]
902[docs: reflow architecture.txt to 78-char lines
903zooko@zooko.com**20091208232943
904 Ignore-this: 88f55166415f15192e39407815141f77
905]
906[docs: update the about.html a little
907zooko@zooko.com**20091208212737
908 Ignore-this: 3fe2d9653c6de0727d3e82bd70f2a8ed
909]
910[docs: remove obsolete doc file "codemap.txt"
911zooko@zooko.com**20091113163033
912 Ignore-this: 16bc21a1835546e71d1b344c06c61ebb
913 I started to update this to reflect the current codebase, but then I thought (a) nobody seemed to notice that it hasn't been updated since December 2007, and (b) it will just bit-rot again, so I'm removing it.
914]
915[mutable/retrieve.py: stop reaching into private MutableFileNode attributes
916Brian Warner <warner@lothar.com>**20091208172921
917 Ignore-this: 61e548798c1105aed66a792bf26ceef7
918]
919[mutable/servermap.py: stop reaching into private MutableFileNode attributes
920Brian Warner <warner@lothar.com>**20091208172608
921 Ignore-this: b40a6b62f623f9285ad96fda139c2ef2
922]
923[mutable/servermap.py: oops, query N+e servers in MODE_WRITE, not k+e
924Brian Warner <warner@lothar.com>**20091208171156
925 Ignore-this: 3497f4ab70dae906759007c3cfa43bc
926 
927 under normal conditions, this wouldn't cause any problems, but if the shares
928 are really sparse (perhaps because new servers were added), then
929 file-modifies might stop looking too early and leave old shares in place
930]
931[control.py: fix speedtest: use download_best_version (not read) on mutable nodes
932Brian Warner <warner@lothar.com>**20091207060512
933 Ignore-this: 7125eabfe74837e05f9291dd6414f917
934]
935[FTP-and-SFTP.txt: fix ssh-keygen pointer
936Brian Warner <warner@lothar.com>**20091207052803
937 Ignore-this: bc2a70ee8c58ec314e79c1262ccb22f7
938]
939[setup: ignore _darcs in the "test-clean" test and make the "clean" step remove all .egg's in the root dir
940zooko@zooko.com**20091206184835
941 Ignore-this: 6066bd160f0db36d7bf60aba405558d2
942]
943[remove MutableFileNode.download(), prefer download_best_version() instead
944Brian Warner <warner@lothar.com>**20091201225438
945 Ignore-this: 5733eb373a902063e09fd52cc858dec0
946]
947[Simplify immutable download API: use just filenode.read(consumer, offset, size)
948Brian Warner <warner@lothar.com>**20091201225330
949 Ignore-this: bdedfb488ac23738bf52ae6d4ab3a3fb
950 
951 * remove Downloader.download_to_data/download_to_filename/download_to_filehandle
952 * remove download.Data/FileName/FileHandle targets
953 * remove filenode.download/download_to_data/download_to_filename methods
954 * leave Downloader.download (the whole Downloader will go away eventually)
955 * add util.consumer.MemoryConsumer/download_to_data, for convenience
956   (this is mostly used by unit tests, but it gets used by enough non-test
957    code to warrant putting it in allmydata.util)
958 * update tests
959 * removes about 180 lines of code. Yay negative code days!
960 
961 Overall plan is to rewrite immutable/download.py and leave filenode.read() as
962 the sole read-side API.
963]
964[server.py: undo my bogus 'correction' of David-Sarah's comment fix
965Brian Warner <warner@lothar.com>**20091201024607
966 Ignore-this: ff4bb58f6a9e045b900ac3a89d6f506a
967 
968 and move it to a better line
969]
970[Implement more coherent behavior when copying with dircaps/filecaps (closes #761). Patch by Kevan Carstensen.
971"Brian Warner <warner@lothar.com>"**20091130211009]
972[storage.py: update comment
973"Brian Warner <warner@lothar.com>"**20091130195913]
974[storage server: detect disk space usage on Windows too (fixes #637)
975david-sarah@jacaranda.org**20091121055644
976 Ignore-this: 20fb30498174ce997befac7701fab056
977]
978[make status of finished operations consistently "Finished"
979david-sarah@jacaranda.org**20091121061543
980 Ignore-this: 97d483e8536ccfc2934549ceff7055a3
981]
982[NEWS: update with all user-visible changes since the last release
983Brian Warner <warner@lothar.com>**20091127224217
984 Ignore-this: 741da6cd928e939fb6d21a61ea3daf0b
985]
986[update "tahoe backup" docs, and webapi.txt's mkdir-with-children
987Brian Warner <warner@lothar.com>**20091127055900
988 Ignore-this: defac1fb9a2335b0af3ef9dbbcc67b7e
989]
990[Add dirnodes to backupdb and "tahoe backup", closes #606.
991Brian Warner <warner@lothar.com>**20091126234257
992 Ignore-this: fa88796fcad1763c6a2bf81f56103223
993 
994 * backups now share dirnodes with any previous backup, in any location,
995   so renames and moves are handled very efficiently
996 * "tahoe backup" no longer bothers reading the previous snapshot
997 * if you switch grids, you should delete ~/.tahoe/private/backupdb.sqlite,
998   to force new uploads of all files and directories
999]
1000[webapi: fix t=check for DIR2-LIT (i.e. empty immutable directories)
1001Brian Warner <warner@lothar.com>**20091126232731
1002 Ignore-this: 8513c890525c69c1eca0e80d53a231f8
1003]
1004[PipelineError: fix str() on python2.4 . Closes #842.
1005Brian Warner <warner@lothar.com>**20091124212512
1006 Ignore-this: e62c92ea9ede2ab7d11fe63f43b9c942
1007]
1008[test_uri.py: s/NewDirnode/Dirnode/ , now that they aren't "new" anymore
1009Brian Warner <warner@lothar.com>**20091120075553
1010 Ignore-this: 61c8ef5e45a9d966873a610d8349b830
1011]
1012[interface name cleanups: IFileNode, IImmutableFileNode, IMutableFileNode
1013Brian Warner <warner@lothar.com>**20091120075255
1014 Ignore-this: e3d193c229e2463e1d0b0c92306de27f
1015 
1016 The proper hierarchy is:
1017  IFilesystemNode
1018  +IFileNode
1019  ++IMutableFileNode
1020  ++IImmutableFileNode
1021  +IDirectoryNode
1022 
1023 Also expand test_client.py (NodeMaker) to hit all IFilesystemNode types.
1024]
1025[class name cleanups: s/FileNode/ImmutableFileNode/
1026Brian Warner <warner@lothar.com>**20091120072239
1027 Ignore-this: 4b3218f2d0e585c62827e14ad8ed8ac1
1028 
1029 also fix test/bench_dirnode.py for recent dirnode changes
1030]
1031[Use DIR-IMM and t=mkdir-immutable for "tahoe backup", for #828
1032Brian Warner <warner@lothar.com>**20091118192813
1033 Ignore-this: a4720529c9bc6bc8b22a3d3265925491
1034]
1035[web/directory.py: use "DIR-IMM" to describe immutable directories, not DIR-RO
1036Brian Warner <warner@lothar.com>**20091118191832
1037 Ignore-this: aceafd6ab4bf1cc0c2a719ef7319ac03
1038]
1039[web/info.py: hush pyflakes
1040Brian Warner <warner@lothar.com>**20091118191736
1041 Ignore-this: edc5f128a2b8095fb20686a75747c8
1042]
1043[make get_size/get_current_size consistent for all IFilesystemNode classes
1044Brian Warner <warner@lothar.com>**20091118191624
1045 Ignore-this: bd3449cf96e4827abaaf962672c1665a
1046 
1047 * stop caching most_recent_size in dirnode, rely upon backing filenode for it
1048 * start caching most_recent_size in MutableFileNode
1049 * return None when you don't know, not "?"
1050 * only render None as "?" in the web "more info" page
1051 * add get_size/get_current_size to UnknownNode
1052]
1053[ImmutableDirectoryURIVerifier: fix verifycap handling
1054Brian Warner <warner@lothar.com>**20091118164238
1055 Ignore-this: 6bba5c717b54352262eabca6e805d590
1056]
1057[Add t=mkdir-immutable to the webapi. Closes #607.
1058Brian Warner <warner@lothar.com>**20091118070900
1059 Ignore-this: 311e5fab9a5f28b9e8a28d3d08f3c0d
1060 
1061 * change t=mkdir-with-children to not use multipart/form encoding. Instead,
1062   the request body is all JSON. t=mkdir-immutable uses this format too.
1063 * make nodemaker.create_immutable_dirnode() get convergence from SecretHolder,
1064   but let callers override it
1065 * raise NotDeepImmutableError instead of using assert()
1066 * add mutable= argument to DirectoryNode.create_subdirectory(), default True
1067]
1068[move convergence secret into SecretHolder, next to lease secret
1069Brian Warner <warner@lothar.com>**20091118015444
1070 Ignore-this: 312f85978a339f2d04deb5bcb8f511bc
1071]
1072[nodemaker: implement immutable directories (internal interface), for #607
1073Brian Warner <warner@lothar.com>**20091112002233
1074 Ignore-this: d09fccf41813fdf7e0db177ed9e5e130
1075 
1076 * nodemaker.create_from_cap() now handles DIR2-CHK and DIR2-LIT
1077 * client.create_immutable_dirnode() is used to create them
1078 * no webapi yet
1079]
1080[stop using IURI()/etc as an adapter
1081Brian Warner <warner@lothar.com>**20091111224542
1082 Ignore-this: 9611da7ea6a4696de2a3b8c08776e6e0
1083]
1084[clean up uri-vs-cap terminology, emphasize cap instances instead of URI strings
1085Brian Warner <warner@lothar.com>**20091111222619
1086 Ignore-this: 93626385f6e7f039ada71f54feefe267
1087 
1088  * "cap" means a python instance which encapsulates a filecap/dircap (uri.py)
1089  * "uri" means a string with a "URI:" prefix
1090  * FileNode instances are created with (and retain) a cap instance, and
1091    generate uri strings on demand
1092  * .get_cap/get_readcap/get_verifycap/get_repaircap return cap instances
1093  * .get_uri/get_readonly_uri return uri strings
1094 
1095 * add filenode.download_to_filename() for control.py, should find a better way
1096 * use MutableFileNode.init_from_cap, not .init_from_uri
1097 * directory URI instances: use get_filenode_cap, not get_filenode_uri
1098 * update/cleanup bench_dirnode.py to match, add Makefile target to run it
1099]
1100[add parser for immutable directory caps: DIR2-CHK, DIR2-LIT, DIR2-CHK-Verifier
1101Brian Warner <warner@lothar.com>**20091104181351
1102 Ignore-this: 854398cc7a75bada57fa97c367b67518
1103]
1104[wui: s/TahoeLAFS/Tahoe-LAFS/
1105zooko@zooko.com**20091029035050
1106 Ignore-this: 901e64cd862e492ed3132bd298583c26
1107]
1108[tests: bump up the timeout on test_repairer to see if 120 seconds was too short for François's ARM box to do the test even when it was doing it right.
1109zooko@zooko.com**20091027224800
1110 Ignore-this: 95e93dc2e018b9948253c2045d506f56
1111]
1112[dirnode.pack_children(): add deep_immutable= argument
1113Brian Warner <warner@lothar.com>**20091026162809
1114 Ignore-this: d5a2371e47662c4bc6eff273e8181b00
1115 
1116 This will be used by DIR2:CHK to enforce the deep-immutability requirement.
1117]
1118[webapi: use t=mkdir-with-children instead of a children= arg to t=mkdir .
1119Brian Warner <warner@lothar.com>**20091026011321
1120 Ignore-this: 769cab30b6ab50db95000b6c5a524916
1121 
1122 This is safer: in the earlier API, an old webapi server would silently ignore
1123 the initial children, and clients trying to set them would have to fetch the
1124 newly-created directory to discover the incompatibility. In the new API,
1125 clients using t=mkdir-with-children against an old webapi server will get a
1126 clear error.
1127]
1128[nodemaker.create_new_mutable_directory: pack_children() in initial_contents=
1129Brian Warner <warner@lothar.com>**20091020005118
1130 Ignore-this: bd43c4eefe06fd32b7492bcb0a55d07e
1131 instead of creating an empty file and then adding the children later.
1132 
1133 This should speed up mkdir(initial_children) considerably, removing two
1134 roundtrips and an entire read-modify-write cycle, probably bringing it down
1135 to a single roundtrip. A quick test (against the volunteergrid) suggests a
1136 30% speedup.
1137 
1138 test_dirnode: add new tests to enforce the restrictions that interfaces.py
1139 claims for create_new_mutable_directory(): no UnknownNodes, metadata dicts
1140]
1141[test_dirnode.py: add tests of initial_children= args to client.create_dirnode
1142Brian Warner <warner@lothar.com>**20091017194159
1143 Ignore-this: 2e2da28323a4d5d815466387914abc1b
1144 and nodemaker.create_new_mutable_directory
1145]
1146[update many dirnode interfaces to accept dict-of-nodes instead of dict-of-caps
1147Brian Warner <warner@lothar.com>**20091017192829
1148 Ignore-this: b35472285143862a856bf4b361d692f0
1149 
1150 interfaces.py: define INodeMaker, document argument values, change
1151                create_new_mutable_directory() to take dict-of-nodes. Change
1152                dirnode.set_nodes() and dirnode.create_subdirectory() too.
1153 nodemaker.py: use INodeMaker, update create_new_mutable_directory()
1154 client.py: have create_dirnode() delegate initial_children= to nodemaker
1155 dirnode.py (Adder): take dict-of-nodes instead of list-of-nodes, which
1156                     updates set_nodes() and create_subdirectory()
1157 web/common.py (convert_initial_children_json): create dict-of-nodes
1158 web/directory.py: same
1159 web/unlinked.py: same
1160 test_dirnode.py: update tests to match
1161]
1162[dirnode.py: move pack_children() out to a function, for eventual use by others
1163Brian Warner <warner@lothar.com>**20091017180707
1164 Ignore-this: 6a823fb61f2c180fd38d6742d3196a7a
1165]
1166[move dirnode.CachingDict to dictutil.AuxValueDict, generalize method names,
1167Brian Warner <warner@lothar.com>**20091017180005
1168 Ignore-this: b086933cf429df0fcea16a308d2640dd
1169 improve tests. Let dirnode _pack_children accept either dict or AuxValueDict.
1170]
1171[test/common.py: update FakeMutableFileNode to new contents= callable scheme
1172Brian Warner <warner@lothar.com>**20091013052154
1173 Ignore-this: 62f00a76454a2190d1c8641c5993632f
1174]
1175[The initial_children= argument to nodemaker.create_new_mutable_directory is
1176Brian Warner <warner@lothar.com>**20091013031922
1177 Ignore-this: 72e45317c21f9eb9ec3bd79bd4311f48
1178 now enabled.
1179]
1180[client.create_mutable_file(contents=) now accepts a callable, which is
1181Brian Warner <warner@lothar.com>**20091013031232
1182 Ignore-this: 3c89d2f50c1e652b83f20bd3f4f27c4b
1183 invoked with the new MutableFileNode and is supposed to return the initial
1184 contents. This can be used by e.g. a new dirnode which needs the filenode's
1185 writekey to encrypt its initial children.
1186 
1187 create_mutable_file() still accepts a bytestring too, or None for an empty
1188 file.
1189]
1190[webapi: t=mkdir now accepts initial children, using the same JSON that t=json
1191Brian Warner <warner@lothar.com>**20091013023444
1192 Ignore-this: 574a46ed46af4251abf8c9580fd31ef7
1193 emits.
1194 
1195 client.create_dirnode(initial_children=) now works.
1196]
1197[replace dirnode.create_empty_directory() with create_subdirectory(), which
1198Brian Warner <warner@lothar.com>**20091013021520
1199 Ignore-this: 6b57cb51bcfcc6058d0df569fdc8a9cf
1200 takes an initial_children= argument
1201]
1202[dirnode.set_children: change return value: fire with self instead of None
1203Brian Warner <warner@lothar.com>**20091013015026
1204 Ignore-this: f1d14e67e084e4b2a4e25fa849b0e753
1205]
1206[dirnode.set_nodes: change return value: fire with self instead of None
1207Brian Warner <warner@lothar.com>**20091013014546
1208 Ignore-this: b75b3829fb53f7399693f1c1a39aacae
1209]
1210[dirnode.set_children: take a dict, not a list
1211Brian Warner <warner@lothar.com>**20091013002440
1212 Ignore-this: 540ce72ce2727ee053afaae1ff124e21
1213]
1214[dirnode.set_uri/set_children: change signature to take writecap+readcap
1215Brian Warner <warner@lothar.com>**20091012235126
1216 Ignore-this: 5df617b2d379a51c79148a857e6026b1
1217 instead of a single cap. The webapi t=set_children call benefits too.
1218]
1219[replace Client.create_empty_dirnode() with create_dirnode(), in anticipation
1220Brian Warner <warner@lothar.com>**20091012224506
1221 Ignore-this: cbdaa4266ecb3c6496ffceab4f95709d
1222 of adding initial_children= argument.
1223 
1224 Includes stubbed-out initial_children= support.
1225]
1226[test_web.py: use a less-fake client, making test harness smaller
1227Brian Warner <warner@lothar.com>**20091012222808
1228 Ignore-this: 29e95147f8c94282885c65b411d100bb
1229]
1230[webapi.txt: document t=set_children, other small edits
1231Brian Warner <warner@lothar.com>**20091009200446
1232 Ignore-this: 4d7e76b04a7b8eaa0a981879f778ea5d
1233]
1234[Verifier: check the full cryptext-hash tree on each share. Removed .todos
1235Brian Warner <warner@lothar.com>**20091005221849
1236 Ignore-this: 6fb039c5584812017d91725e687323a5
1237 from the last few test_repairer tests that were waiting on this.
1238]
1239[Verifier: check the full block-hash-tree on each share
1240Brian Warner <warner@lothar.com>**20091005214844
1241 Ignore-this: 3f7ccf6d253f32340f1bf1da27803eee
1242 
1243 Removed the .todo from two test_repairer tests that check this. The only
1244 remaining .todos are on the three crypttext-hash-tree tests.
1245]
1246[Verifier: check the full share-hash chain on each share
1247Brian Warner <warner@lothar.com>**20091005213443
1248 Ignore-this: 3d30111904158bec06a4eac22fd39d17
1249 
1250 Removed the .todo from two test_repairer tests that check this.
1251]
1252[test_repairer: rename Verifier test cases to be more precise and less verbose
1253Brian Warner <warner@lothar.com>**20091005201115
1254 Ignore-this: 64be7094e33338c7c2aea9387e138771
1255]
1256[immutable/checker.py: rearrange code a little bit, make it easier to follow
1257Brian Warner <warner@lothar.com>**20091005200252
1258 Ignore-this: 91cc303fab66faf717433a709f785fb5
1259]
1260[test/common.py: wrap docstrings to 80cols so I can read them more easily
1261Brian Warner <warner@lothar.com>**20091005200143
1262 Ignore-this: b180a3a0235cbe309c87bd5e873cbbb3
1263]
1264[immutable/download.py: wrap to 80cols, no functional changes
1265Brian Warner <warner@lothar.com>**20091005192542
1266 Ignore-this: 6b05fe3dc6d78832323e708b9e6a1fe
1267]
1268[CHK-hashes.svg: cross out plaintext hashes, since we don't include
1269Brian Warner <warner@lothar.com>**20091005010803
1270 Ignore-this: bea2e953b65ec7359363aa20de8cb603
1271 them (until we finish #453)
1272]
1273[docs: a few licensing clarifications requested by Ubuntu
1274zooko@zooko.com**20090927033226
1275 Ignore-this: 749fc8c9aeb6dc643669854a3e81baa7
1276]
1277[setup: remove binary WinFUSE modules
1278zooko@zooko.com**20090924211436
1279 Ignore-this: 8aefc571d2ae22b9405fc650f2c2062
1280 I would prefer to have just source code, or indications of what 3rd-party packages are required, under revision control, and have the build process generate o
1281 r acquire the binaries as needed.  Also, having these in our release tarballs is interfering with getting Tahoe-LAFS uploaded into Ubuntu Karmic.  (Technicall
1282 y, they would accept binary modules as long as they came with the accompanying source so that they could satisfy their obligations under GPL2+ and TGPPL1+, bu
1283 t it is easier for now to remove the binaries from the source tree.)
1284 In this case, the binaries are from the tahoe-w32-client project: http://allmydata.org/trac/tahoe-w32-client , from which you can also get the source.
1285]
1286[setup: remove binary _fusemodule.so 's
1287zooko@zooko.com**20090924211130
1288 Ignore-this: 74487bbe27d280762ac5dd5f51e24186
1289 I would prefer to have just source code, or indications of what 3rd-party packages are required, under revision control, and have the build process generate or acquire the binaries as needed.  Also, having these in our release tarballs is interfering with getting Tahoe-LAFS uploaded into Ubuntu Karmic.  (Technically, they would accept binary modules as long as they came with the accompanying source so that they could satisfy their obligations under GPL2+ and TGPPL1+, but it is easier for now to remove the binaries from the source tree.)
1290 In this case, these modules come from the MacFUSE project: http://code.google.com/p/macfuse/
1291]
1292[doc: add a copy of LGPL2 for documentation purposes for ubuntu
1293zooko@zooko.com**20090924054218
1294 Ignore-this: 6a073b48678a7c84dc4fbcef9292ab5b
1295]
1296[setup: remove a convenience copy of figleaf, to ease inclusion into Ubuntu Karmic Koala
1297zooko@zooko.com**20090924053215
1298 Ignore-this: a0b0c990d6e2ee65c53a24391365ac8d
1299 We need to carefully document the licence of figleaf in order to get Tahoe-LAFS into Ubuntu Karmic Koala.  However, figleaf isn't really a part of Tahoe-LAFS per se -- this is just a "convenience copy" of a development tool.  The quickest way to make Tahoe-LAFS acceptable for Karmic then, is to remove figleaf from the Tahoe-LAFS tarball itself.  People who want to run figleaf on Tahoe-LAFS (as everyone should want) can install figleaf themselves.  I haven't tested this -- there may be incompatibilities between upstream figleaf and the copy that we had here...
1300]
1301[setup: shebang for misc/build-deb.py to fail quickly
1302zooko@zooko.com**20090819135626
1303 Ignore-this: 5a1b893234d2d0bb7b7346e84b0a6b4d
1304 Without this patch, when I ran "chmod +x ./misc/build-deb.py && ./misc/build-deb.py" then it hung indefinitely.  (I wonder what it was doing.)
1305]
1306[docs: Shawn Willden grants permission for his contributions under GPL2+|TGPPL1+
1307zooko@zooko.com**20090921164651
1308 Ignore-this: ef1912010d07ff2ffd9678e7abfd0d57
1309]
1310[docs: Csaba Henk granted permission to license fuse.py under the same terms as Tahoe-LAFS itself
1311zooko@zooko.com**20090921154659
1312 Ignore-this: c61ba48dcb7206a89a57ca18a0450c53
1313]
1314[setup: mark setup.py as having utf-8 encoding in it
1315zooko@zooko.com**20090920180343
1316 Ignore-this: 9d3850733700a44ba7291e9c5e36bb91
1317]
1318[doc: licensing cleanups
1319zooko@zooko.com**20090920171631
1320 Ignore-this: 7654f2854bf3c13e6f4d4597633a6630
1321 Use nice utf-8 © instead of "(c)". Remove licensing statements on utility modules that have been assigned to allmydata.com by their original authors. (Nattraverso was not assigned to allmydata.com -- it was LGPL'ed -- but I checked and src/allmydata/util/iputil.py was completely rewritten and doesn't contain any line of code from nattraverso.)  Add notes to misc/debian/copyright about licensing on files that aren't just allmydata.com-licensed.
1322]
1323[build-deb.py: run darcsver early, otherwise we get the wrong version later on
1324Brian Warner <warner@lothar.com>**20090918033620
1325 Ignore-this: 6635c5b85e84f8aed0d8390490c5392a
1326]
1327[new approach for debian packaging, sharing pieces across distributions. Still experimental, still only works for sid.
1328warner@lothar.com**20090818190527
1329 Ignore-this: a75eb63db9106b3269badbfcdd7f5ce1
1330]
1331[new experimental deb-packaging rules. Only works for sid so far.
1332Brian Warner <warner@lothar.com>**20090818014052
1333 Ignore-this: 3a26ad188668098f8f3cc10a7c0c2f27
1334]
1335[setup.py: read _version.py and pass to setup(version=), so more commands work
1336Brian Warner <warner@lothar.com>**20090818010057
1337 Ignore-this: b290eb50216938e19f72db211f82147e
1338 like "setup.py --version" and "setup.py --fullname"
1339]
1340[test/check_speed.py: fix shbang line
1341Brian Warner <warner@lothar.com>**20090818005948
1342 Ignore-this: 7f3a37caf349c4c4de704d0feb561f8d
1343]
1344[setup: remove bundled version of darcsver-1.2.1
1345zooko@zooko.com**20090816233432
1346 Ignore-this: 5357f26d2803db2d39159125dddb963a
1347 That version of darcsver emits a scary error message when the darcs executable or the _darcs subdirectory is not found.
1348 This error is hidden (unless the --loud option is passed) in darcsver >= 1.3.1.
1349 Fixes #788.
1350]
1351[de-Service-ify Helper, pass in storage_broker and secret_holder directly.
1352Brian Warner <warner@lothar.com>**20090815201737
1353 Ignore-this: 86b8ac0f90f77a1036cd604dd1304d8b
1354 This makes it more obvious that the Helper currently generates leases with
1355 the Helper's own secrets, rather than getting values from the client, which
1356 is arguably a bug that will likely be resolved with the Accounting project.
1357]
1358[immutable.Downloader: pass StorageBroker to constructor, stop being a Service
1359Brian Warner <warner@lothar.com>**20090815192543
1360 Ignore-this: af5ab12dbf75377640a670c689838479
1361 child of the client, access with client.downloader instead of
1362 client.getServiceNamed("downloader"). The single "Downloader" instance is
1363 scheduled for demolition anyways, to be replaced by individual
1364 filenode.download calls.
1365]
1366[tests: double the timeout on test_runner.RunNode.test_introducer since feisty hit a timeout
1367zooko@zooko.com**20090815160512
1368 Ignore-this: ca7358bce4bdabe8eea75dedc39c0e67
1369 I'm not sure if this is an actual timing issue (feisty is running on an overloaded VM if I recall correctly), or it there is a deeper bug.
1370]
1371[stop making History be a Service, it wasn't necessary
1372Brian Warner <warner@lothar.com>**20090815114415
1373 Ignore-this: b60449231557f1934a751c7effa93cfe
1374]
1375[Overhaul IFilesystemNode handling, to simplify tests and use POLA internally.
1376Brian Warner <warner@lothar.com>**20090815112846
1377 Ignore-this: 1db1b9c149a60a310228aba04c5c8e5f
1378 
1379 * stop using IURI as an adapter
1380 * pass cap strings around instead of URI instances
1381 * move filenode/dirnode creation duties from Client to new NodeMaker class
1382 * move other Client duties to KeyGenerator, SecretHolder, History classes
1383 * stop passing Client reference to dirnode/filenode constructors
1384   - pass less-powerful references instead, like StorageBroker or Uploader
1385 * always create DirectoryNodes by wrapping a filenode (mutable for now)
1386 * remove some specialized mock classes from unit tests
1387 
1388 Detailed list of changes (done one at a time, then merged together)
1389 
1390 always pass a string to create_node_from_uri(), not an IURI instance
1391 always pass a string to IFilesystemNode constructors, not an IURI instance
1392 stop using IURI() as an adapter, switch on cap prefix in create_node_from_uri()
1393 client.py: move SecretHolder code out to a separate class
1394 test_web.py: hush pyflakes
1395 client.py: move NodeMaker functionality out into a separate object
1396 LiteralFileNode: stop storing a Client reference
1397 immutable Checker: remove Client reference, it only needs a SecretHolder
1398 immutable Upload: remove Client reference, leave SecretHolder and StorageBroker
1399 immutable Repairer: replace Client reference with StorageBroker and SecretHolder
1400 immutable FileNode: remove Client reference
1401 mutable.Publish: stop passing Client
1402 mutable.ServermapUpdater: get StorageBroker in constructor, not by peeking into Client reference
1403 MutableChecker: reference StorageBroker and History directly, not through Client
1404 mutable.FileNode: removed unused indirection to checker classes
1405 mutable.FileNode: remove Client reference
1406 client.py: move RSA key generation into a separate class, so it can be passed to the nodemaker
1407 move create_mutable_file() into NodeMaker
1408 test_dirnode.py: stop using FakeClient mockups, use NoNetworkGrid instead. This simplifies the code, but takes longer to run (17s instead of 6s). This should come down later when other cleanups make it possible to use simpler (non-RSA) fake mutable files for dirnode tests.
1409 test_mutable.py: clean up basedir names
1410 client.py: move create_empty_dirnode() into NodeMaker
1411 dirnode.py: get rid of DirectoryNode.create
1412 remove DirectoryNode.init_from_uri, refactor NodeMaker for customization, simplify test_web's mock Client to match
1413 stop passing Client to DirectoryNode, make DirectoryNode.create_with_mutablefile the normal DirectoryNode constructor, start removing client from NodeMaker
1414 remove Client from NodeMaker
1415 move helper status into History, pass History to web.Status instead of Client
1416 test_mutable.py: fix minor typo
1417]
1418[docs: edits for docs/running.html from Sam Mason
1419zooko@zooko.com**20090809201416
1420 Ignore-this: 2207e80449943ebd4ed50cea57c43143
1421]
1422[docs: install.html: instruct Debian users to use this document and not to go find the DownloadDebianPackages page, ignore the warning at the top of it, and try it
1423zooko@zooko.com**20090804123840
1424 Ignore-this: 49da654f19d377ffc5a1eff0c820e026
1425 http://allmydata.org/pipermail/tahoe-dev/2009-August/002507.html
1426]
1427[docs: relnotes.txt: reflow to 63 chars wide because google groups and some web forms seem to wrap to that
1428zooko@zooko.com**20090802135016
1429 Ignore-this: 53b1493a0491bc30fb2935fad283caeb
1430]
1431[docs: about.html: fix English usage noticed by Amber
1432zooko@zooko.com**20090802050533
1433 Ignore-this: 89965c4650f9bd100a615c401181a956
1434]
1435[docs: fix mis-spelled word in about.html
1436zooko@zooko.com**20090802050320
1437 Ignore-this: fdfd0397bc7cef9edfde425dddeb67e5
1438]
1439[TAG allmydata-tahoe-1.5.0
1440zooko@zooko.com**20090802031303
1441 Ignore-this: 94e5558e7225c39a86aae666ea00f166
1442]
1443Patch bundle hash:
14441511b7b5919befc80a858a67595b78a3771973ef