Ticket #393: 393status28.dpatch

File 393status28.dpatch, 685.9 KB (added by kevan, at 2010-08-07T00:14:55Z)
Line 
1Thu Jun 24 16:46:37 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
2  * Misc. changes to support the work I'm doing
3 
4      - Add a notion of file version number to interfaces.py
5      - Alter mutable file node interfaces to have a notion of version,
6        though this may be changed later.
7      - Alter mutable/filenode.py to conform to these changes.
8      - Add a salt hasher to util/hashutil.py
9
10Thu Jun 24 16:48:33 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
11  * nodemaker.py: create MDMF files when asked to
12
13Thu Jun 24 16:49:05 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
14  * storage/server.py: minor code cleanup
15
16Thu Jun 24 16:49:24 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
17  * test/test_mutable.py: alter some tests that were failing due to MDMF; minor code cleanup.
18
19Fri Jun 25 17:35:20 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
20  * test/test_mutable.py: change the definition of corrupt() to work with MDMF as well as SDMF files, change users of corrupt to use the new definition
21
22Sat Jun 26 16:41:18 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
23  * Alter the ServermapUpdater to find MDMF files
24 
25  The servermapupdater should find MDMF files on a grid in the same way
26  that it finds SDMF files. This patch makes it do that.
27
28Sat Jun 26 16:42:04 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
29  * Make a segmented mutable uploader
30 
31  The mutable file uploader should be able to publish files with one
32  segment and files with multiple segments. This patch makes it do that.
33  This is still incomplete, and rather ugly -- I need to flesh out error
34  handling, I need to write tests, and I need to remove some of the uglier
35  kludges in the process before I can call this done.
36
37Sat Jun 26 16:43:14 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
38  * Write a segmented mutable downloader
39 
40  The segmented mutable downloader can deal with MDMF files (files with
41  one or more segments in MDMF format) and SDMF files (files with one
42  segment in SDMF format). It is backwards compatible with the old
43  file format.
44 
45  This patch also contains tests for the segmented mutable downloader.
46
47Mon Jun 28 15:50:48 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
48  * mutable/checker.py: check MDMF files
49 
50  This patch adapts the mutable file checker and verifier to check and
51  verify MDMF files. It does this by using the new segmented downloader,
52  which is trained to perform verification operations on request. This
53  removes some code duplication.
54
55Mon Jun 28 15:52:01 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
56  * mutable/retrieve.py: learn how to verify mutable files
57
58Wed Jun 30 11:33:05 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
59  * interfaces.py: add IMutableSlotWriter
60
61Thu Jul  1 16:28:06 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
62  * test/test_mutable.py: temporarily disable two tests that are now irrelevant
63
64Fri Jul  2 15:55:31 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
65  * Add MDMF reader and writer, and SDMF writer
66 
67  The MDMF/SDMF reader MDMF writer, and SDMF writer are similar to the
68  object proxies that exist for immutable files. They abstract away
69  details of connection, state, and caching from their callers (in this
70  case, the download, servermap updater, and uploader), and expose methods
71  to get and set information on the remote server.
72 
73  MDMFSlotReadProxy reads a mutable file from the server, doing the right
74  thing (in most cases) regardless of whether the file is MDMF or SDMF. It
75  allows callers to tell it how to batch and flush reads.
76 
77  MDMFSlotWriteProxy writes an MDMF mutable file to a server.
78 
79  SDMFSlotWriteProxy writes an SDMF mutable file to a server.
80 
81  This patch also includes tests for MDMFSlotReadProxy,
82  SDMFSlotWriteProxy, and MDMFSlotWriteProxy.
83
84Fri Jul  2 15:55:54 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
85  * mutable/publish.py: cleanup + simplification
86
87Fri Jul  2 15:57:10 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
88  * test/test_mutable.py: remove tests that are no longer relevant
89
90Tue Jul  6 14:52:17 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
91  * interfaces.py: create IMutableUploadable
92
93Tue Jul  6 14:52:57 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
94  * mutable/publish.py: add MutableDataHandle and MutableFileHandle
95
96Tue Jul  6 14:55:41 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
97  * mutable/publish.py: reorganize in preparation of file-like uploadables
98
99Tue Jul  6 14:56:49 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
100  * test/test_mutable.py: write tests for MutableFileHandle and MutableDataHandle
101
102Wed Jul  7 17:00:31 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
103  * Alter tests to work with the new APIs
104
105Wed Jul  7 17:07:32 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
106  * Alter mutable files to use file-like objects for publishing instead of strings.
107
108Thu Jul  8 12:35:22 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
109  * test/test_sftp.py: alter a setup routine to work with new mutable file APIs.
110
111Thu Jul  8 12:36:00 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
112  * mutable/publish.py: make MutableFileHandle seek to the beginning of its file handle before reading.
113
114Fri Jul  9 16:29:12 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
115  * Refactor download interfaces to be more uniform, per #993
116
117Fri Jul 16 18:44:46 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
118  * frontends/sftpd.py: alter a mutable file overwrite to work with the new API
119
120Fri Jul 16 18:45:16 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
121  * mutable/filenode.py: implement most of IVersion, per #993
122
123Fri Jul 16 18:45:49 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
124  * mutable/publish.py: enable segmented uploading for big files, change constants and wording, change MutableDataHandle to MutableData
125
126Fri Jul 16 18:50:49 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
127  * immutable/filenode.py: fix broken implementation of #993 interfaces
128
129Fri Jul 16 18:51:23 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
130  * mutable/retrieve.py: alter Retrieve so that it can download parts of mutable files
131
132Fri Jul 16 18:52:10 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
133  * change MutableDataHandle to MutableData in code.
134
135Fri Jul 16 18:52:30 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
136  * tests: fix tests that were broken by #993
137
138Fri Jul 16 18:54:02 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
139  * test/test_immutable.py: add tests for #993-related modifications
140
141Fri Jul 16 18:54:26 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
142  * web/filenode.py: alter download code to use the new #993 interface.
143
144Fri Jul 16 18:55:01 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
145  * test/common.py: remove FileTooLargeErrors that tested for an SDMF limitation that no longer exists
146
147Tue Jul 20 14:31:09 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
148  * nodemaker.py: resolve a conflict introduced in one of the 1.7.1 patches
149
150Tue Jul 27 15:46:51 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
151  * frontends/sftpd.py: fix conflicts with trunk
152
153Tue Jul 27 15:47:03 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
154  * interfaces.py: Create an IWritable interface
155
156Tue Jul 27 15:47:25 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
157  * mutable/layout.py: Alter MDMFSlotWriteProxy to perform all write operations in one actual write operation
158
159Tue Jul 27 15:48:17 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
160  * test/test_mutable.py: test that write operations occur all at once
161
162Tue Jul 27 15:48:53 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
163  * test/test_storage.py: modify proxy tests to work with the new writing semantics
164
165Tue Jul 27 15:50:01 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
166  * mutable/publish.py: alter mutable publisher to work with new writing semantics
167
168Wed Jul 28 16:24:34 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
169  * test/test_mutable.py: Add tests for new servermap behavior
170
171Fri Jul 30 16:40:29 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
172  * mutable/filenode.py: add an update method.
173
174Fri Jul 30 16:40:56 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
175  * mutable/publish.py: learn how to update as well as publish files.
176
177Fri Jul 30 16:41:41 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
178  * mutable/retrieve.py: expose decoding and decrypting methods to callers
179
180Fri Jul 30 16:42:29 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
181  * mutable/servermap.py: lay some groundwork for IWritable
182
183Mon Aug  2 15:48:21 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
184  * mutable: fix bugs that prevented the update tests from working
185
186Thu Aug  5 17:03:38 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
187  * mutable: fix some bugs in update logic.
188
189Thu Aug  5 17:04:08 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
190  * test/test_mutable.py: add tests for updating behavior
191
192Fri Aug  6 16:48:49 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
193  * web: add a handler to PUT that can update MDMF files
194
195Fri Aug  6 16:49:36 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
196  * test: add tests for new web update behavior
197
198New patches:
199
200[Misc. changes to support the work I'm doing
201Kevan Carstensen <kevan@isnotajoke.com>**20100624234637
202 Ignore-this: fdd18fa8cc05f4b4b15ff53ee24a1819
203 
204     - Add a notion of file version number to interfaces.py
205     - Alter mutable file node interfaces to have a notion of version,
206       though this may be changed later.
207     - Alter mutable/filenode.py to conform to these changes.
208     - Add a salt hasher to util/hashutil.py
209] {
210hunk ./src/allmydata/interfaces.py 7
211      ChoiceOf, IntegerConstraint, Any, RemoteInterface, Referenceable
212 
213 HASH_SIZE=32
214+SALT_SIZE=16
215+
216+SDMF_VERSION=0
217+MDMF_VERSION=1
218 
219 Hash = StringConstraint(maxLength=HASH_SIZE,
220                         minLength=HASH_SIZE)# binary format 32-byte SHA256 hash
221hunk ./src/allmydata/interfaces.py 811
222         writer-visible data using this writekey.
223         """
224 
225+    def set_version(version):
226+        """Tahoe-LAFS supports SDMF and MDMF mutable files. By default,
227+        we upload in SDMF for reasons of compatibility. If you want to
228+        change this, set_version will let you do that.
229+
230+        To say that this file should be uploaded in SDMF, pass in a 0. To
231+        say that the file should be uploaded as MDMF, pass in a 1.
232+        """
233+
234+    def get_version():
235+        """Returns the mutable file protocol version."""
236+
237 class NotEnoughSharesError(Exception):
238     """Download was unable to get enough shares"""
239 
240hunk ./src/allmydata/mutable/filenode.py 8
241 from twisted.internet import defer, reactor
242 from foolscap.api import eventually
243 from allmydata.interfaces import IMutableFileNode, \
244-     ICheckable, ICheckResults, NotEnoughSharesError
245+     ICheckable, ICheckResults, NotEnoughSharesError, MDMF_VERSION, SDMF_VERSION
246 from allmydata.util import hashutil, log
247 from allmydata.util.assertutil import precondition
248 from allmydata.uri import WriteableSSKFileURI, ReadonlySSKFileURI
249hunk ./src/allmydata/mutable/filenode.py 67
250         self._sharemap = {} # known shares, shnum-to-[nodeids]
251         self._cache = ResponseCache()
252         self._most_recent_size = None
253+        # filled in after __init__ if we're being created for the first time;
254+        # filled in by the servermap updater before publishing, otherwise.
255+        # set to this default value in case neither of those things happen,
256+        # or in case the servermap can't find any shares to tell us what
257+        # to publish as.
258+        # TODO: Set this back to None, and find out why the tests fail
259+        #       with it set to None.
260+        self._protocol_version = SDMF_VERSION
261 
262         # all users of this MutableFileNode go through the serializer. This
263         # takes advantage of the fact that Deferreds discard the callbacks
264hunk ./src/allmydata/mutable/filenode.py 472
265     def _did_upload(self, res, size):
266         self._most_recent_size = size
267         return res
268+
269+
270+    def set_version(self, version):
271+        # I can be set in two ways:
272+        #  1. When the node is created.
273+        #  2. (for an existing share) when the Servermap is updated
274+        #     before I am read.
275+        assert version in (MDMF_VERSION, SDMF_VERSION)
276+        self._protocol_version = version
277+
278+
279+    def get_version(self):
280+        return self._protocol_version
281hunk ./src/allmydata/util/hashutil.py 90
282 MUTABLE_READKEY_TAG = "allmydata_mutable_writekey_to_readkey_v1"
283 MUTABLE_DATAKEY_TAG = "allmydata_mutable_readkey_to_datakey_v1"
284 MUTABLE_STORAGEINDEX_TAG = "allmydata_mutable_readkey_to_storage_index_v1"
285+MUTABLE_SALT_TAG = "allmydata_mutable_segment_salt_v1"
286 
287 # dirnodes
288 DIRNODE_CHILD_WRITECAP_TAG = "allmydata_mutable_writekey_and_salt_to_dirnode_child_capkey_v1"
289hunk ./src/allmydata/util/hashutil.py 134
290 def plaintext_segment_hasher():
291     return tagged_hasher(PLAINTEXT_SEGMENT_TAG)
292 
293+def mutable_salt_hash(data):
294+    return tagged_hash(MUTABLE_SALT_TAG, data)
295+def mutable_salt_hasher():
296+    return tagged_hasher(MUTABLE_SALT_TAG)
297+
298 KEYLEN = 16
299 IVLEN = 16
300 
301}
302[nodemaker.py: create MDMF files when asked to
303Kevan Carstensen <kevan@isnotajoke.com>**20100624234833
304 Ignore-this: 26c16aaca9ddab7a7ce37a4530bc970
305] {
306hunk ./src/allmydata/nodemaker.py 3
307 import weakref
308 from zope.interface import implements
309-from allmydata.interfaces import INodeMaker
310+from allmydata.util.assertutil import precondition
311+from allmydata.interfaces import INodeMaker, MustBeDeepImmutableError, \
312+                                 SDMF_VERSION, MDMF_VERSION
313 from allmydata.immutable.filenode import ImmutableFileNode, LiteralFileNode
314 from allmydata.immutable.upload import Data
315 from allmydata.mutable.filenode import MutableFileNode
316hunk ./src/allmydata/nodemaker.py 88
317             return self._create_dirnode(filenode)
318         return None
319 
320-    def create_mutable_file(self, contents=None, keysize=None):
321+    def create_mutable_file(self, contents=None, keysize=None,
322+                            version=SDMF_VERSION):
323         n = MutableFileNode(self.storage_broker, self.secret_holder,
324                             self.default_encoding_parameters, self.history)
325hunk ./src/allmydata/nodemaker.py 92
326+        n.set_version(version)
327         d = self.key_generator.generate(keysize)
328         d.addCallback(n.create_with_keys, contents)
329         d.addCallback(lambda res: n)
330hunk ./src/allmydata/nodemaker.py 98
331         return d
332 
333-    def create_new_mutable_directory(self, initial_children={}):
334+    def create_new_mutable_directory(self, initial_children={},
335+                                     version=SDMF_VERSION):
336+        # initial_children must have metadata (i.e. {} instead of None)
337+        for (name, (node, metadata)) in initial_children.iteritems():
338+            precondition(isinstance(metadata, dict),
339+                         "create_new_mutable_directory requires metadata to be a dict, not None", metadata)
340+            node.raise_error()
341         d = self.create_mutable_file(lambda n:
342                                      pack_children(initial_children, n.get_writekey()))
343         d.addCallback(self._create_dirnode)
344merger 0.0 (
345hunk ./src/allmydata/nodemaker.py 106
346-                                     pack_children(n, initial_children))
347+                                     pack_children(initial_children, n.get_writekey()))
348hunk ./src/allmydata/nodemaker.py 106
349-                                     pack_children(n, initial_children))
350+                                     pack_children(n, initial_children),
351+                                     version)
352)
353}
354[storage/server.py: minor code cleanup
355Kevan Carstensen <kevan@isnotajoke.com>**20100624234905
356 Ignore-this: 2358c531c39e48d3c8e56b62b5768228
357] {
358hunk ./src/allmydata/storage/server.py 569
359                                          self)
360         return share
361 
362-    def remote_slot_readv(self, storage_index, shares, readv):
363+    def remote_slot_readv(self, storage_index, shares, readvs):
364         start = time.time()
365         self.count("readv")
366         si_s = si_b2a(storage_index)
367hunk ./src/allmydata/storage/server.py 590
368             if sharenum in shares or not shares:
369                 filename = os.path.join(bucketdir, sharenum_s)
370                 msf = MutableShareFile(filename, self)
371-                datavs[sharenum] = msf.readv(readv)
372+                datavs[sharenum] = msf.readv(readvs)
373         log.msg("returning shares %s" % (datavs.keys(),),
374                 facility="tahoe.storage", level=log.NOISY, parent=lp)
375         self.add_latency("readv", time.time() - start)
376}
377[test/test_mutable.py: alter some tests that were failing due to MDMF; minor code cleanup.
378Kevan Carstensen <kevan@isnotajoke.com>**20100624234924
379 Ignore-this: afb86ec1fbdbfe1a5ef6f46f350273c0
380] {
381hunk ./src/allmydata/test/test_mutable.py 151
382             chr(ord(original[byte_offset]) ^ 0x01) +
383             original[byte_offset+1:])
384 
385+def add_two(original, byte_offset):
386+    # It isn't enough to simply flip the bit for the version number,
387+    # because 1 is a valid version number. So we add two instead.
388+    return (original[:byte_offset] +
389+            chr(ord(original[byte_offset]) ^ 0x02) +
390+            original[byte_offset+1:])
391+
392 def corrupt(res, s, offset, shnums_to_corrupt=None, offset_offset=0):
393     # if shnums_to_corrupt is None, corrupt all shares. Otherwise it is a
394     # list of shnums to corrupt.
395hunk ./src/allmydata/test/test_mutable.py 187
396                 real_offset = offset1
397             real_offset = int(real_offset) + offset2 + offset_offset
398             assert isinstance(real_offset, int), offset
399-            shares[shnum] = flip_bit(data, real_offset)
400+            if offset1 == 0: # verbyte
401+                f = add_two
402+            else:
403+                f = flip_bit
404+            shares[shnum] = f(data, real_offset)
405     return res
406 
407 def make_storagebroker(s=None, num_peers=10):
408hunk ./src/allmydata/test/test_mutable.py 423
409         d.addCallback(_created)
410         return d
411 
412+
413     def test_modify_backoffer(self):
414         def _modifier(old_contents, servermap, first_time):
415             return old_contents + "line2"
416hunk ./src/allmydata/test/test_mutable.py 658
417         d.addCallback(_created)
418         return d
419 
420+
421     def _copy_shares(self, ignored, index):
422         shares = self._storage._peers
423         # we need a deep copy
424}
425[test/test_mutable.py: change the definition of corrupt() to work with MDMF as well as SDMF files, change users of corrupt to use the new definition
426Kevan Carstensen <kevan@isnotajoke.com>**20100626003520
427 Ignore-this: 836e59e2fde0535f6b4bea3468dc8244
428] {
429hunk ./src/allmydata/test/test_mutable.py 168
430                 and shnum not in shnums_to_corrupt):
431                 continue
432             data = shares[shnum]
433-            (version,
434-             seqnum,
435-             root_hash,
436-             IV,
437-             k, N, segsize, datalen,
438-             o) = unpack_header(data)
439-            if isinstance(offset, tuple):
440-                offset1, offset2 = offset
441-            else:
442-                offset1 = offset
443-                offset2 = 0
444-            if offset1 == "pubkey":
445-                real_offset = 107
446-            elif offset1 in o:
447-                real_offset = o[offset1]
448-            else:
449-                real_offset = offset1
450-            real_offset = int(real_offset) + offset2 + offset_offset
451-            assert isinstance(real_offset, int), offset
452-            if offset1 == 0: # verbyte
453-                f = add_two
454-            else:
455-                f = flip_bit
456-            shares[shnum] = f(data, real_offset)
457-    return res
458+            # We're feeding the reader all of the share data, so it
459+            # won't need to use the rref that we didn't provide, nor the
460+            # storage index that we didn't provide. We do this because
461+            # the reader will work for both MDMF and SDMF.
462+            reader = MDMFSlotReadProxy(None, None, shnum, data)
463+            # We need to get the offsets for the next part.
464+            d = reader.get_verinfo()
465+            def _do_corruption(verinfo, data, shnum):
466+                (seqnum,
467+                 root_hash,
468+                 IV,
469+                 segsize,
470+                 datalen,
471+                 k, n, prefix, o) = verinfo
472+                if isinstance(offset, tuple):
473+                    offset1, offset2 = offset
474+                else:
475+                    offset1 = offset
476+                    offset2 = 0
477+                if offset1 == "pubkey":
478+                    real_offset = 107
479+                elif offset1 in o:
480+                    real_offset = o[offset1]
481+                else:
482+                    real_offset = offset1
483+                real_offset = int(real_offset) + offset2 + offset_offset
484+                assert isinstance(real_offset, int), offset
485+                if offset1 == 0: # verbyte
486+                    f = add_two
487+                else:
488+                    f = flip_bit
489+                shares[shnum] = f(data, real_offset)
490+            d.addCallback(_do_corruption, data, shnum)
491+            ds.append(d)
492+    dl = defer.DeferredList(ds)
493+    dl.addCallback(lambda ignored: res)
494+    return dl
495 
496 def make_storagebroker(s=None, num_peers=10):
497     if not s:
498hunk ./src/allmydata/test/test_mutable.py 1177
499         return d
500 
501     def test_download_fails(self):
502-        corrupt(None, self._storage, "signature")
503-        d = self.shouldFail(UnrecoverableFileError, "test_download_anyway",
504+        d = corrupt(None, self._storage, "signature")
505+        d.addCallback(lambda ignored:
506+            self.shouldFail(UnrecoverableFileError, "test_download_anyway",
507                             "no recoverable versions",
508                             self._fn.download_best_version)
509         return d
510hunk ./src/allmydata/test/test_mutable.py 1232
511         return d
512 
513     def test_check_all_bad_sig(self):
514-        corrupt(None, self._storage, 1) # bad sig
515-        d = self._fn.check(Monitor())
516+        d = corrupt(None, self._storage, 1) # bad sig
517+        d.addCallback(lambda ignored:
518+            self._fn.check(Monitor()))
519         d.addCallback(self.check_bad, "test_check_all_bad_sig")
520         return d
521 
522hunk ./src/allmydata/test/test_mutable.py 1239
523     def test_check_all_bad_blocks(self):
524-        corrupt(None, self._storage, "share_data", [9]) # bad blocks
525+        d = corrupt(None, self._storage, "share_data", [9]) # bad blocks
526         # the Checker won't notice this.. it doesn't look at actual data
527hunk ./src/allmydata/test/test_mutable.py 1241
528-        d = self._fn.check(Monitor())
529+        d.addCallback(lambda ignored:
530+            self._fn.check(Monitor()))
531         d.addCallback(self.check_good, "test_check_all_bad_blocks")
532         return d
533 
534hunk ./src/allmydata/test/test_mutable.py 1252
535         return d
536 
537     def test_verify_all_bad_sig(self):
538-        corrupt(None, self._storage, 1) # bad sig
539-        d = self._fn.check(Monitor(), verify=True)
540+        d = corrupt(None, self._storage, 1) # bad sig
541+        d.addCallback(lambda ignored:
542+            self._fn.check(Monitor(), verify=True))
543         d.addCallback(self.check_bad, "test_verify_all_bad_sig")
544         return d
545 
546hunk ./src/allmydata/test/test_mutable.py 1259
547     def test_verify_one_bad_sig(self):
548-        corrupt(None, self._storage, 1, [9]) # bad sig
549-        d = self._fn.check(Monitor(), verify=True)
550+        d = corrupt(None, self._storage, 1, [9]) # bad sig
551+        d.addCallback(lambda ignored:
552+            self._fn.check(Monitor(), verify=True))
553         d.addCallback(self.check_bad, "test_verify_one_bad_sig")
554         return d
555 
556hunk ./src/allmydata/test/test_mutable.py 1266
557     def test_verify_one_bad_block(self):
558-        corrupt(None, self._storage, "share_data", [9]) # bad blocks
559+        d = corrupt(None, self._storage, "share_data", [9]) # bad blocks
560         # the Verifier *will* notice this, since it examines every byte
561hunk ./src/allmydata/test/test_mutable.py 1268
562-        d = self._fn.check(Monitor(), verify=True)
563+        d.addCallback(lambda ignored:
564+            self._fn.check(Monitor(), verify=True))
565         d.addCallback(self.check_bad, "test_verify_one_bad_block")
566         d.addCallback(self.check_expected_failure,
567                       CorruptShareError, "block hash tree failure",
568hunk ./src/allmydata/test/test_mutable.py 1277
569         return d
570 
571     def test_verify_one_bad_sharehash(self):
572-        corrupt(None, self._storage, "share_hash_chain", [9], 5)
573-        d = self._fn.check(Monitor(), verify=True)
574+        d = corrupt(None, self._storage, "share_hash_chain", [9], 5)
575+        d.addCallback(lambda ignored:
576+            self._fn.check(Monitor(), verify=True))
577         d.addCallback(self.check_bad, "test_verify_one_bad_sharehash")
578         d.addCallback(self.check_expected_failure,
579                       CorruptShareError, "corrupt hashes",
580hunk ./src/allmydata/test/test_mutable.py 1287
581         return d
582 
583     def test_verify_one_bad_encprivkey(self):
584-        corrupt(None, self._storage, "enc_privkey", [9]) # bad privkey
585-        d = self._fn.check(Monitor(), verify=True)
586+        d = corrupt(None, self._storage, "enc_privkey", [9]) # bad privkey
587+        d.addCallback(lambda ignored:
588+            self._fn.check(Monitor(), verify=True))
589         d.addCallback(self.check_bad, "test_verify_one_bad_encprivkey")
590         d.addCallback(self.check_expected_failure,
591                       CorruptShareError, "invalid privkey",
592hunk ./src/allmydata/test/test_mutable.py 1297
593         return d
594 
595     def test_verify_one_bad_encprivkey_uncheckable(self):
596-        corrupt(None, self._storage, "enc_privkey", [9]) # bad privkey
597+        d = corrupt(None, self._storage, "enc_privkey", [9]) # bad privkey
598         readonly_fn = self._fn.get_readonly()
599         # a read-only node has no way to validate the privkey
600hunk ./src/allmydata/test/test_mutable.py 1300
601-        d = readonly_fn.check(Monitor(), verify=True)
602+        d.addCallback(lambda ignored:
603+            readonly_fn.check(Monitor(), verify=True))
604         d.addCallback(self.check_good,
605                       "test_verify_one_bad_encprivkey_uncheckable")
606         return d
607}
608[Alter the ServermapUpdater to find MDMF files
609Kevan Carstensen <kevan@isnotajoke.com>**20100626234118
610 Ignore-this: 25f6278209c2983ba8f307cfe0fde0
611 
612 The servermapupdater should find MDMF files on a grid in the same way
613 that it finds SDMF files. This patch makes it do that.
614] {
615hunk ./src/allmydata/mutable/servermap.py 7
616 from itertools import count
617 from twisted.internet import defer
618 from twisted.python import failure
619-from foolscap.api import DeadReferenceError, RemoteException, eventually
620+from foolscap.api import DeadReferenceError, RemoteException, eventually, \
621+                         fireEventually
622 from allmydata.util import base32, hashutil, idlib, log
623 from allmydata.storage.server import si_b2a
624 from allmydata.interfaces import IServermapUpdaterStatus
625hunk ./src/allmydata/mutable/servermap.py 17
626 from allmydata.mutable.common import MODE_CHECK, MODE_ANYTHING, MODE_WRITE, MODE_READ, \
627      DictOfSets, CorruptShareError, NeedMoreDataError
628 from allmydata.mutable.layout import unpack_prefix_and_signature, unpack_header, unpack_share, \
629-     SIGNED_PREFIX_LENGTH
630+     SIGNED_PREFIX_LENGTH, MDMFSlotReadProxy
631 
632 class UpdateStatus:
633     implements(IServermapUpdaterStatus)
634hunk ./src/allmydata/mutable/servermap.py 254
635         """Return a set of versionids, one for each version that is currently
636         recoverable."""
637         versionmap = self.make_versionmap()
638-
639         recoverable_versions = set()
640         for (verinfo, shares) in versionmap.items():
641             (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
642hunk ./src/allmydata/mutable/servermap.py 366
643         self._servers_responded = set()
644 
645         # how much data should we read?
646+        # SDMF:
647         #  * if we only need the checkstring, then [0:75]
648         #  * if we need to validate the checkstring sig, then [543ish:799ish]
649         #  * if we need the verification key, then [107:436ish]
650hunk ./src/allmydata/mutable/servermap.py 374
651         #  * if we need the encrypted private key, we want [-1216ish:]
652         #   * but we can't read from negative offsets
653         #   * the offset table tells us the 'ish', also the positive offset
654-        # A future version of the SMDF slot format should consider using
655-        # fixed-size slots so we can retrieve less data. For now, we'll just
656-        # read 2000 bytes, which also happens to read enough actual data to
657-        # pre-fetch a 9-entry dirnode.
658+        # MDMF:
659+        #  * Checkstring? [0:72]
660+        #  * If we want to validate the checkstring, then [0:72], [143:?] --
661+        #    the offset table will tell us for sure.
662+        #  * If we need the verification key, we have to consult the offset
663+        #    table as well.
664+        # At this point, we don't know which we are. Our filenode can
665+        # tell us, but it might be lying -- in some cases, we're
666+        # responsible for telling it which kind of file it is.
667         self._read_size = 4000
668         if mode == MODE_CHECK:
669             # we use unpack_prefix_and_signature, so we need 1k
670hunk ./src/allmydata/mutable/servermap.py 432
671         self._queries_completed = 0
672 
673         sb = self._storage_broker
674+        # All of the peers, permuted by the storage index, as usual.
675         full_peerlist = sb.get_servers_for_index(self._storage_index)
676         self.full_peerlist = full_peerlist # for use later, immutable
677         self.extra_peers = full_peerlist[:] # peers are removed as we use them
678hunk ./src/allmydata/mutable/servermap.py 439
679         self._good_peers = set() # peers who had some shares
680         self._empty_peers = set() # peers who don't have any shares
681         self._bad_peers = set() # peers to whom our queries failed
682+        self._readers = {} # peerid -> dict(sharewriters), filled in
683+                           # after responses come in.
684 
685         k = self._node.get_required_shares()
686hunk ./src/allmydata/mutable/servermap.py 443
687+        # For what cases can these conditions work?
688         if k is None:
689             # make a guess
690             k = 3
691hunk ./src/allmydata/mutable/servermap.py 456
692         self.num_peers_to_query = k + self.EPSILON
693 
694         if self.mode == MODE_CHECK:
695+            # We want to query all of the peers.
696             initial_peers_to_query = dict(full_peerlist)
697             must_query = set(initial_peers_to_query.keys())
698             self.extra_peers = []
699hunk ./src/allmydata/mutable/servermap.py 464
700             # we're planning to replace all the shares, so we want a good
701             # chance of finding them all. We will keep searching until we've
702             # seen epsilon that don't have a share.
703+            # We don't query all of the peers because that could take a while.
704             self.num_peers_to_query = N + self.EPSILON
705             initial_peers_to_query, must_query = self._build_initial_querylist()
706             self.required_num_empty_peers = self.EPSILON
707hunk ./src/allmydata/mutable/servermap.py 474
708             # might also avoid the round trip required to read the encrypted
709             # private key.
710 
711-        else:
712+        else: # MODE_READ, MODE_ANYTHING
713+            # 2k peers is good enough.
714             initial_peers_to_query, must_query = self._build_initial_querylist()
715 
716         # this is a set of peers that we are required to get responses from:
717hunk ./src/allmydata/mutable/servermap.py 490
718         # before we can consider ourselves finished, and self.extra_peers
719         # contains the overflow (peers that we should tap if we don't get
720         # enough responses)
721+        # I guess that self._must_query is a subset of
722+        # initial_peers_to_query?
723+        assert set(must_query).issubset(set(initial_peers_to_query))
724 
725         self._send_initial_requests(initial_peers_to_query)
726         self._status.timings["initial_queries"] = time.time() - self._started
727hunk ./src/allmydata/mutable/servermap.py 549
728         # errors that aren't handled by _query_failed (and errors caused by
729         # _query_failed) get logged, but we still want to check for doneness.
730         d.addErrback(log.err)
731-        d.addBoth(self._check_for_done)
732         d.addErrback(self._fatal_error)
733hunk ./src/allmydata/mutable/servermap.py 550
734+        d.addCallback(self._check_for_done)
735         return d
736 
737     def _do_read(self, ss, peerid, storage_index, shnums, readv):
738hunk ./src/allmydata/mutable/servermap.py 569
739         d = ss.callRemote("slot_readv", storage_index, shnums, readv)
740         return d
741 
742+
743+    def _got_corrupt_share(self, e, shnum, peerid, data, lp):
744+        """
745+        I am called when a remote server returns a corrupt share in
746+        response to one of our queries. By corrupt, I mean a share
747+        without a valid signature. I then record the failure, notify the
748+        server of the corruption, and record the share as bad.
749+        """
750+        f = failure.Failure(e)
751+        self.log(format="bad share: %(f_value)s", f_value=str(f),
752+                 failure=f, parent=lp, level=log.WEIRD, umid="h5llHg")
753+        # Notify the server that its share is corrupt.
754+        self.notify_server_corruption(peerid, shnum, str(e))
755+        # By flagging this as a bad peer, we won't count any of
756+        # the other shares on that peer as valid, though if we
757+        # happen to find a valid version string amongst those
758+        # shares, we'll keep track of it so that we don't need
759+        # to validate the signature on those again.
760+        self._bad_peers.add(peerid)
761+        self._last_failure = f
762+        # XXX: Use the reader for this?
763+        checkstring = data[:SIGNED_PREFIX_LENGTH]
764+        self._servermap.mark_bad_share(peerid, shnum, checkstring)
765+        self._servermap.problems.append(f)
766+
767+
768+    def _cache_good_sharedata(self, verinfo, shnum, now, data):
769+        """
770+        If one of my queries returns successfully (which means that we
771+        were able to and successfully did validate the signature), I
772+        cache the data that we initially fetched from the storage
773+        server. This will help reduce the number of roundtrips that need
774+        to occur when the file is downloaded, or when the file is
775+        updated.
776+        """
777+        self._node._add_to_cache(verinfo, shnum, 0, data, now)
778+
779+
780     def _got_results(self, datavs, peerid, readsize, stuff, started):
781         lp = self.log(format="got result from [%(peerid)s], %(numshares)d shares",
782                       peerid=idlib.shortnodeid_b2a(peerid),
783hunk ./src/allmydata/mutable/servermap.py 630
784         else:
785             self._empty_peers.add(peerid)
786 
787-        last_verinfo = None
788-        last_shnum = None
789+        ss, storage_index = stuff
790+        ds = []
791+
792         for shnum,datav in datavs.items():
793             data = datav[0]
794hunk ./src/allmydata/mutable/servermap.py 635
795-            try:
796-                verinfo = self._got_results_one_share(shnum, data, peerid, lp)
797-                last_verinfo = verinfo
798-                last_shnum = shnum
799-                self._node._add_to_cache(verinfo, shnum, 0, data, now)
800-            except CorruptShareError, e:
801-                # log it and give the other shares a chance to be processed
802-                f = failure.Failure()
803-                self.log(format="bad share: %(f_value)s", f_value=str(f.value),
804-                         failure=f, parent=lp, level=log.WEIRD, umid="h5llHg")
805-                self.notify_server_corruption(peerid, shnum, str(e))
806-                self._bad_peers.add(peerid)
807-                self._last_failure = f
808-                checkstring = data[:SIGNED_PREFIX_LENGTH]
809-                self._servermap.mark_bad_share(peerid, shnum, checkstring)
810-                self._servermap.problems.append(f)
811-                pass
812-
813-        self._status.timings["cumulative_verify"] += (time.time() - now)
814+            reader = MDMFSlotReadProxy(ss,
815+                                       storage_index,
816+                                       shnum,
817+                                       data)
818+            self._readers.setdefault(peerid, dict())[shnum] = reader
819+            # our goal, with each response, is to validate the version
820+            # information and share data as best we can at this point --
821+            # we do this by validating the signature. To do this, we
822+            # need to do the following:
823+            #   - If we don't already have the public key, fetch the
824+            #     public key. We use this to validate the signature.
825+            if not self._node.get_pubkey():
826+                # fetch and set the public key.
827+                d = reader.get_verification_key()
828+                d.addCallback(lambda results, shnum=shnum, peerid=peerid:
829+                    self._try_to_set_pubkey(results, peerid, shnum, lp))
830+                # XXX: Make self._pubkey_query_failed?
831+                d.addErrback(lambda error, shnum=shnum, peerid=peerid:
832+                    self._got_corrupt_share(error, shnum, peerid, data, lp))
833+            else:
834+                # we already have the public key.
835+                d = defer.succeed(None)
836+            # Neither of these two branches return anything of
837+            # consequence, so the first entry in our deferredlist will
838+            # be None.
839 
840hunk ./src/allmydata/mutable/servermap.py 661
841-        if self._need_privkey and last_verinfo:
842-            # send them a request for the privkey. We send one request per
843-            # server.
844-            lp2 = self.log("sending privkey request",
845-                           parent=lp, level=log.NOISY)
846-            (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
847-             offsets_tuple) = last_verinfo
848-            o = dict(offsets_tuple)
849+            # - Next, we need the version information. We almost
850+            #   certainly got this by reading the first thousand or so
851+            #   bytes of the share on the storage server, so we
852+            #   shouldn't need to fetch anything at this step.
853+            d2 = reader.get_verinfo()
854+            d2.addErrback(lambda error, shnum=shnum, peerid=peerid:
855+                self._got_corrupt_share(error, shnum, peerid, data, lp))
856+            # - Next, we need the signature. For an SDMF share, it is
857+            #   likely that we fetched this when doing our initial fetch
858+            #   to get the version information. In MDMF, this lives at
859+            #   the end of the share, so unless the file is quite small,
860+            #   we'll need to do a remote fetch to get it.
861+            d3 = reader.get_signature()
862+            d3.addErrback(lambda error, shnum=shnum, peerid=peerid:
863+                self._got_corrupt_share(error, shnum, peerid, data, lp))
864+            #  Once we have all three of these responses, we can move on
865+            #  to validating the signature
866 
867hunk ./src/allmydata/mutable/servermap.py 679
868-            self._queries_outstanding.add(peerid)
869-            readv = [ (o['enc_privkey'], (o['EOF'] - o['enc_privkey'])) ]
870-            ss = self._servermap.connections[peerid]
871-            privkey_started = time.time()
872-            d = self._do_read(ss, peerid, self._storage_index,
873-                              [last_shnum], readv)
874-            d.addCallback(self._got_privkey_results, peerid, last_shnum,
875-                          privkey_started, lp2)
876-            d.addErrback(self._privkey_query_failed, peerid, last_shnum, lp2)
877-            d.addErrback(log.err)
878-            d.addCallback(self._check_for_done)
879-            d.addErrback(self._fatal_error)
880+            # Does the node already have a privkey? If not, we'll try to
881+            # fetch it here.
882+            if self._need_privkey:
883+                d4 = reader.get_encprivkey()
884+                d4.addCallback(lambda results, shnum=shnum, peerid=peerid:
885+                    self._try_to_validate_privkey(results, peerid, shnum, lp))
886+                d4.addErrback(lambda error, shnum=shnum, peerid=peerid:
887+                    self._privkey_query_failed(error, shnum, data, lp))
888+            else:
889+                d4 = defer.succeed(None)
890 
891hunk ./src/allmydata/mutable/servermap.py 690
892+            dl = defer.DeferredList([d, d2, d3, d4])
893+            dl.addCallback(lambda results, shnum=shnum, peerid=peerid:
894+                self._got_signature_one_share(results, shnum, peerid, lp))
895+            dl.addErrback(lambda error, shnum=shnum, data=data:
896+               self._got_corrupt_share(error, shnum, peerid, data, lp))
897+            dl.addCallback(lambda verinfo, shnum=shnum, peerid=peerid, data=data:
898+                self._cache_good_sharedata(verinfo, shnum, now, data))
899+            ds.append(dl)
900+        # dl is a deferred list that will fire when all of the shares
901+        # that we found on this peer are done processing. When dl fires,
902+        # we know that processing is done, so we can decrement the
903+        # semaphore-like thing that we incremented earlier.
904+        dl = defer.DeferredList(ds, fireOnOneErrback=True)
905+        # Are we done? Done means that there are no more queries to
906+        # send, that there are no outstanding queries, and that we
907+        # haven't received any queries that are still processing. If we
908+        # are done, self._check_for_done will cause the done deferred
909+        # that we returned to our caller to fire, which tells them that
910+        # they have a complete servermap, and that we won't be touching
911+        # the servermap anymore.
912+        dl.addCallback(self._check_for_done)
913+        dl.addErrback(self._fatal_error)
914         # all done!
915         self.log("_got_results done", parent=lp, level=log.NOISY)
916hunk ./src/allmydata/mutable/servermap.py 714
917+        return dl
918+
919+
920+    def _try_to_set_pubkey(self, pubkey_s, peerid, shnum, lp):
921+        if self._node.get_pubkey():
922+            return # don't go through this again if we don't have to
923+        fingerprint = hashutil.ssk_pubkey_fingerprint_hash(pubkey_s)
924+        assert len(fingerprint) == 32
925+        if fingerprint != self._node.get_fingerprint():
926+            raise CorruptShareError(peerid, shnum,
927+                                "pubkey doesn't match fingerprint")
928+        self._node._populate_pubkey(self._deserialize_pubkey(pubkey_s))
929+        assert self._node.get_pubkey()
930+
931 
932     def notify_server_corruption(self, peerid, shnum, reason):
933         ss = self._servermap.connections[peerid]
934hunk ./src/allmydata/mutable/servermap.py 734
935         ss.callRemoteOnly("advise_corrupt_share",
936                           "mutable", self._storage_index, shnum, reason)
937 
938-    def _got_results_one_share(self, shnum, data, peerid, lp):
939+
940+    def _got_signature_one_share(self, results, shnum, peerid, lp):
941+        # It is our job to give versioninfo to our caller. We need to
942+        # raise CorruptShareError if the share is corrupt for any
943+        # reason, something that our caller will handle.
944         self.log(format="_got_results: got shnum #%(shnum)d from peerid %(peerid)s",
945                  shnum=shnum,
946                  peerid=idlib.shortnodeid_b2a(peerid),
947hunk ./src/allmydata/mutable/servermap.py 744
948                  level=log.NOISY,
949                  parent=lp)
950-
951-        # this might raise NeedMoreDataError, if the pubkey and signature
952-        # live at some weird offset. That shouldn't happen, so I'm going to
953-        # treat it as a bad share.
954-        (seqnum, root_hash, IV, k, N, segsize, datalength,
955-         pubkey_s, signature, prefix) = unpack_prefix_and_signature(data)
956-
957-        if not self._node.get_pubkey():
958-            fingerprint = hashutil.ssk_pubkey_fingerprint_hash(pubkey_s)
959-            assert len(fingerprint) == 32
960-            if fingerprint != self._node.get_fingerprint():
961-                raise CorruptShareError(peerid, shnum,
962-                                        "pubkey doesn't match fingerprint")
963-            self._node._populate_pubkey(self._deserialize_pubkey(pubkey_s))
964-
965-        if self._need_privkey:
966-            self._try_to_extract_privkey(data, peerid, shnum, lp)
967-
968-        (ig_version, ig_seqnum, ig_root_hash, ig_IV, ig_k, ig_N,
969-         ig_segsize, ig_datalen, offsets) = unpack_header(data)
970+        _, verinfo, signature, __ = results
971+        (seqnum,
972+         root_hash,
973+         saltish,
974+         segsize,
975+         datalen,
976+         k,
977+         n,
978+         prefix,
979+         offsets) = verinfo[1]
980         offsets_tuple = tuple( [(key,value) for key,value in offsets.items()] )
981 
982hunk ./src/allmydata/mutable/servermap.py 756
983-        verinfo = (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
984+        # XXX: This should be done for us in the method, so
985+        # presumably you can go in there and fix it.
986+        verinfo = (seqnum,
987+                   root_hash,
988+                   saltish,
989+                   segsize,
990+                   datalen,
991+                   k,
992+                   n,
993+                   prefix,
994                    offsets_tuple)
995hunk ./src/allmydata/mutable/servermap.py 767
996+        # This tuple uniquely identifies a share on the grid; we use it
997+        # to keep track of the ones that we've already seen.
998 
999         if verinfo not in self._valid_versions:
1000hunk ./src/allmydata/mutable/servermap.py 771
1001-            # it's a new pair. Verify the signature.
1002-            valid = self._node.get_pubkey().verify(prefix, signature)
1003+            # This is a new version tuple, and we need to validate it
1004+            # against the public key before keeping track of it.
1005+            assert self._node.get_pubkey()
1006+            valid = self._node.get_pubkey().verify(prefix, signature[1])
1007             if not valid:
1008hunk ./src/allmydata/mutable/servermap.py 776
1009-                raise CorruptShareError(peerid, shnum, "signature is invalid")
1010+                raise CorruptShareError(peerid, shnum,
1011+                                        "signature is invalid")
1012 
1013hunk ./src/allmydata/mutable/servermap.py 779
1014-            # ok, it's a valid verinfo. Add it to the list of validated
1015-            # versions.
1016-            self.log(" found valid version %d-%s from %s-sh%d: %d-%d/%d/%d"
1017-                     % (seqnum, base32.b2a(root_hash)[:4],
1018-                        idlib.shortnodeid_b2a(peerid), shnum,
1019-                        k, N, segsize, datalength),
1020-                     parent=lp)
1021-            self._valid_versions.add(verinfo)
1022-        # We now know that this is a valid candidate verinfo.
1023+        # ok, it's a valid verinfo. Add it to the list of validated
1024+        # versions.
1025+        self.log(" found valid version %d-%s from %s-sh%d: %d-%d/%d/%d"
1026+                 % (seqnum, base32.b2a(root_hash)[:4],
1027+                    idlib.shortnodeid_b2a(peerid), shnum,
1028+                    k, n, segsize, datalen),
1029+                    parent=lp)
1030+        self._valid_versions.add(verinfo)
1031+        # We now know that this is a valid candidate verinfo. Whether or
1032+        # not this instance of it is valid is a matter for the next
1033+        # statement; at this point, we just know that if we see this
1034+        # version info again, that its signature checks out and that
1035+        # we're okay to skip the signature-checking step.
1036 
1037hunk ./src/allmydata/mutable/servermap.py 793
1038+        # (peerid, shnum) are bound in the method invocation.
1039         if (peerid, shnum) in self._servermap.bad_shares:
1040             # we've been told that the rest of the data in this share is
1041             # unusable, so don't add it to the servermap.
1042hunk ./src/allmydata/mutable/servermap.py 808
1043         self.versionmap.add(verinfo, (shnum, peerid, timestamp))
1044         return verinfo
1045 
1046+
1047     def _deserialize_pubkey(self, pubkey_s):
1048         verifier = rsa.create_verifying_key_from_string(pubkey_s)
1049         return verifier
1050hunk ./src/allmydata/mutable/servermap.py 813
1051 
1052-    def _try_to_extract_privkey(self, data, peerid, shnum, lp):
1053-        try:
1054-            r = unpack_share(data)
1055-        except NeedMoreDataError, e:
1056-            # this share won't help us. oh well.
1057-            offset = e.encprivkey_offset
1058-            length = e.encprivkey_length
1059-            self.log("shnum %d on peerid %s: share was too short (%dB) "
1060-                     "to get the encprivkey; [%d:%d] ought to hold it" %
1061-                     (shnum, idlib.shortnodeid_b2a(peerid), len(data),
1062-                      offset, offset+length),
1063-                     parent=lp)
1064-            # NOTE: if uncoordinated writes are taking place, someone might
1065-            # change the share (and most probably move the encprivkey) before
1066-            # we get a chance to do one of these reads and fetch it. This
1067-            # will cause us to see a NotEnoughSharesError(unable to fetch
1068-            # privkey) instead of an UncoordinatedWriteError . This is a
1069-            # nuisance, but it will go away when we move to DSA-based mutable
1070-            # files (since the privkey will be small enough to fit in the
1071-            # write cap).
1072-
1073-            return
1074-
1075-        (seqnum, root_hash, IV, k, N, segsize, datalen,
1076-         pubkey, signature, share_hash_chain, block_hash_tree,
1077-         share_data, enc_privkey) = r
1078-
1079-        return self._try_to_validate_privkey(enc_privkey, peerid, shnum, lp)
1080 
1081     def _try_to_validate_privkey(self, enc_privkey, peerid, shnum, lp):
1082hunk ./src/allmydata/mutable/servermap.py 815
1083-
1084+        """
1085+        Given a writekey from a remote server, I validate it against the
1086+        writekey stored in my node. If it is valid, then I set the
1087+        privkey and encprivkey properties of the node.
1088+        """
1089         alleged_privkey_s = self._node._decrypt_privkey(enc_privkey)
1090         alleged_writekey = hashutil.ssk_writekey_hash(alleged_privkey_s)
1091         if alleged_writekey != self._node.get_writekey():
1092hunk ./src/allmydata/mutable/servermap.py 892
1093         self._queries_completed += 1
1094         self._last_failure = f
1095 
1096-    def _got_privkey_results(self, datavs, peerid, shnum, started, lp):
1097-        now = time.time()
1098-        elapsed = now - started
1099-        self._status.add_per_server_time(peerid, "privkey", started, elapsed)
1100-        self._queries_outstanding.discard(peerid)
1101-        if not self._need_privkey:
1102-            return
1103-        if shnum not in datavs:
1104-            self.log("privkey wasn't there when we asked it",
1105-                     level=log.WEIRD, umid="VA9uDQ")
1106-            return
1107-        datav = datavs[shnum]
1108-        enc_privkey = datav[0]
1109-        self._try_to_validate_privkey(enc_privkey, peerid, shnum, lp)
1110 
1111     def _privkey_query_failed(self, f, peerid, shnum, lp):
1112         self._queries_outstanding.discard(peerid)
1113hunk ./src/allmydata/mutable/servermap.py 906
1114         self._servermap.problems.append(f)
1115         self._last_failure = f
1116 
1117+
1118     def _check_for_done(self, res):
1119         # exit paths:
1120         #  return self._send_more_queries(outstanding) : send some more queries
1121hunk ./src/allmydata/mutable/servermap.py 912
1122         #  return self._done() : all done
1123         #  return : keep waiting, no new queries
1124-
1125         lp = self.log(format=("_check_for_done, mode is '%(mode)s', "
1126                               "%(outstanding)d queries outstanding, "
1127                               "%(extra)d extra peers available, "
1128hunk ./src/allmydata/mutable/servermap.py 1117
1129         self._servermap.last_update_time = self._started
1130         # the servermap will not be touched after this
1131         self.log("servermap: %s" % self._servermap.summarize_versions())
1132+
1133         eventually(self._done_deferred.callback, self._servermap)
1134 
1135     def _fatal_error(self, f):
1136hunk ./src/allmydata/test/test_mutable.py 637
1137         d.addCallback(_created)
1138         return d
1139 
1140-    def publish_multiple(self):
1141+    def publish_mdmf(self):
1142+        # like publish_one, except that the result is guaranteed to be
1143+        # an MDMF file.
1144+        # self.CONTENTS should have more than one segment.
1145+        self.CONTENTS = "This is an MDMF file" * 100000
1146+        self._storage = FakeStorage()
1147+        self._nodemaker = make_nodemaker(self._storage)
1148+        self._storage_broker = self._nodemaker.storage_broker
1149+        d = self._nodemaker.create_mutable_file(self.CONTENTS, version=1)
1150+        def _created(node):
1151+            self._fn = node
1152+            self._fn2 = self._nodemaker.create_from_cap(node.get_uri())
1153+        d.addCallback(_created)
1154+        return d
1155+
1156+
1157+    def publish_sdmf(self):
1158+        # like publish_one, except that the result is guaranteed to be
1159+        # an SDMF file
1160+        self.CONTENTS = "This is an SDMF file" * 1000
1161+        self._storage = FakeStorage()
1162+        self._nodemaker = make_nodemaker(self._storage)
1163+        self._storage_broker = self._nodemaker.storage_broker
1164+        d = self._nodemaker.create_mutable_file(self.CONTENTS, version=0)
1165+        def _created(node):
1166+            self._fn = node
1167+            self._fn2 = self._nodemaker.create_from_cap(node.get_uri())
1168+        d.addCallback(_created)
1169+        return d
1170+
1171+
1172+    def publish_multiple(self, version=0):
1173         self.CONTENTS = ["Contents 0",
1174                          "Contents 1",
1175                          "Contents 2",
1176hunk ./src/allmydata/test/test_mutable.py 677
1177         self._copied_shares = {}
1178         self._storage = FakeStorage()
1179         self._nodemaker = make_nodemaker(self._storage)
1180-        d = self._nodemaker.create_mutable_file(self.CONTENTS[0]) # seqnum=1
1181+        d = self._nodemaker.create_mutable_file(self.CONTENTS[0], version=version) # seqnum=1
1182         def _created(node):
1183             self._fn = node
1184             # now create multiple versions of the same file, and accumulate
1185hunk ./src/allmydata/test/test_mutable.py 906
1186         return d
1187 
1188 
1189+    def test_servermapupdater_finds_mdmf_files(self):
1190+        # setUp already published an MDMF file for us. We just need to
1191+        # make sure that when we run the ServermapUpdater, the file is
1192+        # reported to have one recoverable version.
1193+        d = defer.succeed(None)
1194+        d.addCallback(lambda ignored:
1195+            self.publish_mdmf())
1196+        d.addCallback(lambda ignored:
1197+            self.make_servermap(mode=MODE_CHECK))
1198+        # Calling make_servermap also updates the servermap in the mode
1199+        # that we specify, so we just need to see what it says.
1200+        def _check_servermap(sm):
1201+            self.failUnlessEqual(len(sm.recoverable_versions()), 1)
1202+        d.addCallback(_check_servermap)
1203+        return d
1204+
1205+
1206+    def test_servermapupdater_finds_sdmf_files(self):
1207+        d = defer.succeed(None)
1208+        d.addCallback(lambda ignored:
1209+            self.publish_sdmf())
1210+        d.addCallback(lambda ignored:
1211+            self.make_servermap(mode=MODE_CHECK))
1212+        d.addCallback(lambda servermap:
1213+            self.failUnlessEqual(len(servermap.recoverable_versions()), 1))
1214+        return d
1215+
1216 
1217 class Roundtrip(unittest.TestCase, testutil.ShouldFailMixin, PublishMixin):
1218     def setUp(self):
1219hunk ./src/allmydata/test/test_mutable.py 1050
1220         return d
1221     test_no_servers_download.timeout = 15
1222 
1223+
1224     def _test_corrupt_all(self, offset, substring,
1225                           should_succeed=False, corrupt_early=True,
1226                           failure_checker=None):
1227}
1228[Make a segmented mutable uploader
1229Kevan Carstensen <kevan@isnotajoke.com>**20100626234204
1230 Ignore-this: d199af8ab0bc64d8ed2bc19c5437bfba
1231 
1232 The mutable file uploader should be able to publish files with one
1233 segment and files with multiple segments. This patch makes it do that.
1234 This is still incomplete, and rather ugly -- I need to flesh out error
1235 handling, I need to write tests, and I need to remove some of the uglier
1236 kludges in the process before I can call this done.
1237] {
1238hunk ./src/allmydata/mutable/publish.py 8
1239 from zope.interface import implements
1240 from twisted.internet import defer
1241 from twisted.python import failure
1242-from allmydata.interfaces import IPublishStatus
1243+from allmydata.interfaces import IPublishStatus, SDMF_VERSION, MDMF_VERSION
1244 from allmydata.util import base32, hashutil, mathutil, idlib, log
1245 from allmydata import hashtree, codec
1246 from allmydata.storage.server import si_b2a
1247hunk ./src/allmydata/mutable/publish.py 19
1248      UncoordinatedWriteError, NotEnoughServersError
1249 from allmydata.mutable.servermap import ServerMap
1250 from allmydata.mutable.layout import pack_prefix, pack_share, unpack_header, pack_checkstring, \
1251-     unpack_checkstring, SIGNED_PREFIX
1252+     unpack_checkstring, SIGNED_PREFIX, MDMFSlotWriteProxy
1253+
1254+KiB = 1024
1255+DEFAULT_MAX_SEGMENT_SIZE = 128 * KiB
1256 
1257 class PublishStatus:
1258     implements(IPublishStatus)
1259hunk ./src/allmydata/mutable/publish.py 112
1260         self._status.set_helper(False)
1261         self._status.set_progress(0.0)
1262         self._status.set_active(True)
1263+        # We use this to control how the file is written.
1264+        version = self._node.get_version()
1265+        assert version in (SDMF_VERSION, MDMF_VERSION)
1266+        self._version = version
1267 
1268     def get_status(self):
1269         return self._status
1270hunk ./src/allmydata/mutable/publish.py 134
1271         simultaneous write.
1272         """
1273 
1274-        # 1: generate shares (SDMF: files are small, so we can do it in RAM)
1275-        # 2: perform peer selection, get candidate servers
1276-        #  2a: send queries to n+epsilon servers, to determine current shares
1277-        #  2b: based upon responses, create target map
1278-        # 3: send slot_testv_and_readv_and_writev messages
1279-        # 4: as responses return, update share-dispatch table
1280-        # 4a: may need to run recovery algorithm
1281-        # 5: when enough responses are back, we're done
1282+        # 0. Setup encoding parameters, encoder, and other such things.
1283+        # 1. Encrypt, encode, and publish segments.
1284 
1285         self.log("starting publish, datalen is %s" % len(newdata))
1286         self._status.set_size(len(newdata))
1287hunk ./src/allmydata/mutable/publish.py 187
1288         self.bad_peers = set() # peerids who have errbacked/refused requests
1289 
1290         self.newdata = newdata
1291-        self.salt = os.urandom(16)
1292 
1293hunk ./src/allmydata/mutable/publish.py 188
1294+        # This will set self.segment_size, self.num_segments, and
1295+        # self.fec.
1296         self.setup_encoding_parameters()
1297 
1298         # if we experience any surprises (writes which were rejected because
1299hunk ./src/allmydata/mutable/publish.py 238
1300             self.bad_share_checkstrings[key] = old_checkstring
1301             self.connections[peerid] = self._servermap.connections[peerid]
1302 
1303-        # create the shares. We'll discard these as they are delivered. SDMF:
1304-        # we're allowed to hold everything in memory.
1305+        # Now, the process dovetails -- if this is an SDMF file, we need
1306+        # to write an SDMF file. Otherwise, we need to write an MDMF
1307+        # file.
1308+        if self._version == MDMF_VERSION:
1309+            return self._publish_mdmf()
1310+        else:
1311+            return self._publish_sdmf()
1312+        #return self.done_deferred
1313+
1314+    def _publish_mdmf(self):
1315+        # Next, we find homes for all of the shares that we don't have
1316+        # homes for yet.
1317+        # TODO: Make this part do peer selection.
1318+        self.update_goal()
1319+        self.writers = {}
1320+        # For each (peerid, shnum) in self.goal, we make an
1321+        # MDMFSlotWriteProxy for that peer. We'll use this to write
1322+        # shares to the peer.
1323+        for key in self.goal:
1324+            peerid, shnum = key
1325+            write_enabler = self._node.get_write_enabler(peerid)
1326+            renew_secret = self._node.get_renewal_secret(peerid)
1327+            cancel_secret = self._node.get_cancel_secret(peerid)
1328+            secrets = (write_enabler, renew_secret, cancel_secret)
1329+
1330+            self.writers[shnum] =  MDMFSlotWriteProxy(shnum,
1331+                                                      self.connections[peerid],
1332+                                                      self._storage_index,
1333+                                                      secrets,
1334+                                                      self._new_seqnum,
1335+                                                      self.required_shares,
1336+                                                      self.total_shares,
1337+                                                      self.segment_size,
1338+                                                      len(self.newdata))
1339+            if (peerid, shnum) in self._servermap.servermap:
1340+                old_versionid, old_timestamp = self._servermap.servermap[key]
1341+                (old_seqnum, old_root_hash, old_salt, old_segsize,
1342+                 old_datalength, old_k, old_N, old_prefix,
1343+                 old_offsets_tuple) = old_versionid
1344+                self.writers[shnum].set_checkstring(old_seqnum, old_root_hash)
1345+
1346+        # Now, we start pushing shares.
1347+        self._status.timings["setup"] = time.time() - self._started
1348+        def _start_pushing(res):
1349+            self._started_pushing = time.time()
1350+            return res
1351+
1352+        # First, we encrypt, encode, and publish the shares that we need
1353+        # to encrypt, encode, and publish.
1354+
1355+        # This will eventually hold the block hash chain for each share
1356+        # that we publish. We define it this way so that empty publishes
1357+        # will still have something to write to the remote slot.
1358+        self.blockhashes = dict([(i, []) for i in xrange(self.total_shares)])
1359+        self.sharehash_leaves = None # eventually [sharehashes]
1360+        self.sharehashes = {} # shnum -> [sharehash leaves necessary to
1361+                              # validate the share]
1362 
1363hunk ./src/allmydata/mutable/publish.py 296
1364+        d = defer.succeed(None)
1365+        self.log("Starting push")
1366+        for i in xrange(self.num_segments - 1):
1367+            d.addCallback(lambda ignored, i=i:
1368+                self.push_segment(i))
1369+            d.addCallback(self._turn_barrier)
1370+        # We have at least one segment, so we will have a tail segment
1371+        if self.num_segments > 0:
1372+            d.addCallback(lambda ignored:
1373+                self.push_tail_segment())
1374+
1375+        d.addCallback(lambda ignored:
1376+            self.push_encprivkey())
1377+        d.addCallback(lambda ignored:
1378+            self.push_blockhashes())
1379+        d.addCallback(lambda ignored:
1380+            self.push_sharehashes())
1381+        d.addCallback(lambda ignored:
1382+            self.push_toplevel_hashes_and_signature())
1383+        d.addCallback(lambda ignored:
1384+            self.finish_publishing())
1385+        return d
1386+
1387+
1388+    def _publish_sdmf(self):
1389         self._status.timings["setup"] = time.time() - self._started
1390hunk ./src/allmydata/mutable/publish.py 322
1391+        self.salt = os.urandom(16)
1392+
1393         d = self._encrypt_and_encode()
1394         d.addCallback(self._generate_shares)
1395         def _start_pushing(res):
1396hunk ./src/allmydata/mutable/publish.py 335
1397 
1398         return self.done_deferred
1399 
1400+
1401     def setup_encoding_parameters(self):
1402hunk ./src/allmydata/mutable/publish.py 337
1403-        segment_size = len(self.newdata)
1404+        if self._version == MDMF_VERSION:
1405+            segment_size = DEFAULT_MAX_SEGMENT_SIZE # 128 KiB by default
1406+        else:
1407+            segment_size = len(self.newdata) # SDMF is only one segment
1408         # this must be a multiple of self.required_shares
1409         segment_size = mathutil.next_multiple(segment_size,
1410                                               self.required_shares)
1411hunk ./src/allmydata/mutable/publish.py 350
1412                                                   segment_size)
1413         else:
1414             self.num_segments = 0
1415-        assert self.num_segments in [0, 1,] # SDMF restrictions
1416+        if self._version == SDMF_VERSION:
1417+            assert self.num_segments in (0, 1) # SDMF
1418+            return
1419+        # calculate the tail segment size.
1420+        self.tail_segment_size = len(self.newdata) % segment_size
1421+
1422+        if self.tail_segment_size == 0:
1423+            # The tail segment is the same size as the other segments.
1424+            self.tail_segment_size = segment_size
1425+
1426+        # We'll make an encoder ahead-of-time for the normal-sized
1427+        # segments (defined as any segment of segment_size size.
1428+        # (the part of the code that puts the tail segment will make its
1429+        #  own encoder for that part)
1430+        fec = codec.CRSEncoder()
1431+        fec.set_params(self.segment_size,
1432+                       self.required_shares, self.total_shares)
1433+        self.piece_size = fec.get_block_size()
1434+        self.fec = fec
1435+
1436+
1437+    def push_segment(self, segnum):
1438+        started = time.time()
1439+        segsize = self.segment_size
1440+        self.log("Pushing segment %d of %d" % (segnum + 1, self.num_segments))
1441+        data = self.newdata[segsize * segnum:segsize*(segnum + 1)]
1442+        assert len(data) == segsize
1443+
1444+        salt = os.urandom(16)
1445+
1446+        key = hashutil.ssk_readkey_data_hash(salt, self.readkey)
1447+        enc = AES(key)
1448+        crypttext = enc.process(data)
1449+        assert len(crypttext) == len(data)
1450+
1451+        now = time.time()
1452+        self._status.timings["encrypt"] = now - started
1453+        started = now
1454+
1455+        # now apply FEC
1456+
1457+        self._status.set_status("Encoding")
1458+        crypttext_pieces = [None] * self.required_shares
1459+        piece_size = self.piece_size
1460+        for i in range(len(crypttext_pieces)):
1461+            offset = i * piece_size
1462+            piece = crypttext[offset:offset+piece_size]
1463+            piece = piece + "\x00"*(piece_size - len(piece)) # padding
1464+            crypttext_pieces[i] = piece
1465+            assert len(piece) == piece_size
1466+        d = self.fec.encode(crypttext_pieces)
1467+        def _done_encoding(res):
1468+            elapsed = time.time() - started
1469+            self._status.timings["encode"] = elapsed
1470+            return res
1471+        d.addCallback(_done_encoding)
1472+
1473+        def _push_shares_and_salt(results):
1474+            shares, shareids = results
1475+            dl = []
1476+            for i in xrange(len(shares)):
1477+                sharedata = shares[i]
1478+                shareid = shareids[i]
1479+                block_hash = hashutil.block_hash(salt + sharedata)
1480+                self.blockhashes[shareid].append(block_hash)
1481+
1482+                # find the writer for this share
1483+                d = self.writers[shareid].put_block(sharedata, segnum, salt)
1484+                dl.append(d)
1485+            # TODO: Naturally, we need to check on the results of these.
1486+            return defer.DeferredList(dl)
1487+        d.addCallback(_push_shares_and_salt)
1488+        return d
1489+
1490+
1491+    def push_tail_segment(self):
1492+        # This is essentially the same as push_segment, except that we
1493+        # don't use the cached encoder that we use elsewhere.
1494+        self.log("Pushing tail segment")
1495+        started = time.time()
1496+        segsize = self.segment_size
1497+        data = self.newdata[segsize * (self.num_segments-1):]
1498+        assert len(data) == self.tail_segment_size
1499+        salt = os.urandom(16)
1500+
1501+        key = hashutil.ssk_readkey_data_hash(salt, self.readkey)
1502+        enc = AES(key)
1503+        crypttext = enc.process(data)
1504+        assert len(crypttext) == len(data)
1505+
1506+        now = time.time()
1507+        self._status.timings['encrypt'] = now - started
1508+        started = now
1509+
1510+        self._status.set_status("Encoding")
1511+        tail_fec = codec.CRSEncoder()
1512+        tail_fec.set_params(self.tail_segment_size,
1513+                            self.required_shares,
1514+                            self.total_shares)
1515+
1516+        crypttext_pieces = [None] * self.required_shares
1517+        piece_size = tail_fec.get_block_size()
1518+        for i in range(len(crypttext_pieces)):
1519+            offset = i * piece_size
1520+            piece = crypttext[offset:offset+piece_size]
1521+            piece = piece + "\x00"*(piece_size - len(piece)) # padding
1522+            crypttext_pieces[i] = piece
1523+            assert len(piece) == piece_size
1524+        d = tail_fec.encode(crypttext_pieces)
1525+        def _push_shares_and_salt(results):
1526+            shares, shareids = results
1527+            dl = []
1528+            for i in xrange(len(shares)):
1529+                sharedata = shares[i]
1530+                shareid = shareids[i]
1531+                block_hash = hashutil.block_hash(salt + sharedata)
1532+                self.blockhashes[shareid].append(block_hash)
1533+                # find the writer for this share
1534+                d = self.writers[shareid].put_block(sharedata,
1535+                                                    self.num_segments - 1,
1536+                                                    salt)
1537+                dl.append(d)
1538+            # TODO: Naturally, we need to check on the results of these.
1539+            return defer.DeferredList(dl)
1540+        d.addCallback(_push_shares_and_salt)
1541+        return d
1542+
1543+
1544+    def push_encprivkey(self):
1545+        started = time.time()
1546+        encprivkey = self._encprivkey
1547+        dl = []
1548+        def _spy_on_writer(results):
1549+            print results
1550+            return results
1551+        for shnum, writer in self.writers.iteritems():
1552+            d = writer.put_encprivkey(encprivkey)
1553+            dl.append(d)
1554+        d = defer.DeferredList(dl)
1555+        return d
1556+
1557+
1558+    def push_blockhashes(self):
1559+        started = time.time()
1560+        dl = []
1561+        def _spy_on_results(results):
1562+            print results
1563+            return results
1564+        self.sharehash_leaves = [None] * len(self.blockhashes)
1565+        for shnum, blockhashes in self.blockhashes.iteritems():
1566+            t = hashtree.HashTree(blockhashes)
1567+            self.blockhashes[shnum] = list(t)
1568+            # set the leaf for future use.
1569+            self.sharehash_leaves[shnum] = t[0]
1570+            d = self.writers[shnum].put_blockhashes(self.blockhashes[shnum])
1571+            dl.append(d)
1572+        d = defer.DeferredList(dl)
1573+        return d
1574+
1575+
1576+    def push_sharehashes(self):
1577+        share_hash_tree = hashtree.HashTree(self.sharehash_leaves)
1578+        share_hash_chain = {}
1579+        ds = []
1580+        def _spy_on_results(results):
1581+            print results
1582+            return results
1583+        for shnum in xrange(len(self.sharehash_leaves)):
1584+            needed_indices = share_hash_tree.needed_hashes(shnum)
1585+            self.sharehashes[shnum] = dict( [ (i, share_hash_tree[i])
1586+                                             for i in needed_indices] )
1587+            d = self.writers[shnum].put_sharehashes(self.sharehashes[shnum])
1588+            ds.append(d)
1589+        self.root_hash = share_hash_tree[0]
1590+        d = defer.DeferredList(ds)
1591+        return d
1592+
1593+
1594+    def push_toplevel_hashes_and_signature(self):
1595+        # We need to to three things here:
1596+        #   - Push the root hash and salt hash
1597+        #   - Get the checkstring of the resulting layout; sign that.
1598+        #   - Push the signature
1599+        ds = []
1600+        def _spy_on_results(results):
1601+            print results
1602+            return results
1603+        for shnum in xrange(self.total_shares):
1604+            d = self.writers[shnum].put_root_hash(self.root_hash)
1605+            ds.append(d)
1606+        d = defer.DeferredList(ds)
1607+        def _make_and_place_signature(ignored):
1608+            signable = self.writers[0].get_signable()
1609+            self.signature = self._privkey.sign(signable)
1610+
1611+            ds = []
1612+            for (shnum, writer) in self.writers.iteritems():
1613+                d = writer.put_signature(self.signature)
1614+                ds.append(d)
1615+            return defer.DeferredList(ds)
1616+        d.addCallback(_make_and_place_signature)
1617+        return d
1618+
1619+
1620+    def finish_publishing(self):
1621+        # We're almost done -- we just need to put the verification key
1622+        # and the offsets
1623+        ds = []
1624+        verification_key = self._pubkey.serialize()
1625+
1626+        def _spy_on_results(results):
1627+            print results
1628+            return results
1629+        for (shnum, writer) in self.writers.iteritems():
1630+            d = writer.put_verification_key(verification_key)
1631+            d.addCallback(lambda ignored, writer=writer:
1632+                writer.finish_publishing())
1633+            ds.append(d)
1634+        return defer.DeferredList(ds)
1635+
1636+
1637+    def _turn_barrier(self, res):
1638+        # putting this method in a Deferred chain imposes a guaranteed
1639+        # reactor turn between the pre- and post- portions of that chain.
1640+        # This can be useful to limit memory consumption: since Deferreds do
1641+        # not do tail recursion, code which uses defer.succeed(result) for
1642+        # consistency will cause objects to live for longer than you might
1643+        # normally expect.
1644+        return fireEventually(res)
1645+
1646 
1647     def _fatal_error(self, f):
1648         self.log("error during loop", failure=f, level=log.UNUSUAL)
1649hunk ./src/allmydata/mutable/publish.py 716
1650             self.log_goal(self.goal, "after update: ")
1651 
1652 
1653-
1654     def _encrypt_and_encode(self):
1655         # this returns a Deferred that fires with a list of (sharedata,
1656         # sharenum) tuples. TODO: cache the ciphertext, only produce the
1657hunk ./src/allmydata/mutable/publish.py 757
1658         d.addCallback(_done_encoding)
1659         return d
1660 
1661+
1662     def _generate_shares(self, shares_and_shareids):
1663         # this sets self.shares and self.root_hash
1664         self.log("_generate_shares")
1665hunk ./src/allmydata/mutable/publish.py 1145
1666             self._status.set_progress(1.0)
1667         eventually(self.done_deferred.callback, res)
1668 
1669-
1670hunk ./src/allmydata/test/test_mutable.py 248
1671         d.addCallback(_created)
1672         return d
1673 
1674+
1675+    def test_create_mdmf(self):
1676+        d = self.nodemaker.create_mutable_file(version=MDMF_VERSION)
1677+        def _created(n):
1678+            self.failUnless(isinstance(n, MutableFileNode))
1679+            self.failUnlessEqual(n.get_storage_index(), n._storage_index)
1680+            sb = self.nodemaker.storage_broker
1681+            peer0 = sorted(sb.get_all_serverids())[0]
1682+            shnums = self._storage._peers[peer0].keys()
1683+            self.failUnlessEqual(len(shnums), 1)
1684+        d.addCallback(_created)
1685+        return d
1686+
1687+
1688     def test_serialize(self):
1689         n = MutableFileNode(None, None, {"k": 3, "n": 10}, None)
1690         calls = []
1691hunk ./src/allmydata/test/test_mutable.py 334
1692         d.addCallback(_created)
1693         return d
1694 
1695+
1696+    def test_create_mdmf_with_initial_contents(self):
1697+        initial_contents = "foobarbaz" * 131072 # 900KiB
1698+        d = self.nodemaker.create_mutable_file(initial_contents,
1699+                                               version=MDMF_VERSION)
1700+        def _created(n):
1701+            d = n.download_best_version()
1702+            d.addCallback(lambda data:
1703+                self.failUnlessEqual(data, initial_contents))
1704+            d.addCallback(lambda ignored:
1705+                n.overwrite(initial_contents + "foobarbaz"))
1706+            d.addCallback(lambda ignored:
1707+                n.download_best_version())
1708+            d.addCallback(lambda data:
1709+                self.failUnlessEqual(data, initial_contents +
1710+                                           "foobarbaz"))
1711+            return d
1712+        d.addCallback(_created)
1713+        return d
1714+
1715+
1716     def test_create_with_initial_contents_function(self):
1717         data = "initial contents"
1718         def _make_contents(n):
1719hunk ./src/allmydata/test/test_mutable.py 370
1720         d.addCallback(lambda data2: self.failUnlessEqual(data2, data))
1721         return d
1722 
1723+
1724+    def test_create_mdmf_with_initial_contents_function(self):
1725+        data = "initial contents" * 100000
1726+        def _make_contents(n):
1727+            self.failUnless(isinstance(n, MutableFileNode))
1728+            key = n.get_writekey()
1729+            self.failUnless(isinstance(key, str), key)
1730+            self.failUnlessEqual(len(key), 16)
1731+            return data
1732+        d = self.nodemaker.create_mutable_file(_make_contents,
1733+                                               version=MDMF_VERSION)
1734+        d.addCallback(lambda n:
1735+            n.download_best_version())
1736+        d.addCallback(lambda data2:
1737+            self.failUnlessEqual(data2, data))
1738+        return d
1739+
1740+
1741     def test_create_with_too_large_contents(self):
1742         BIG = "a" * (self.OLD_MAX_SEGMENT_SIZE + 1)
1743         d = self.nodemaker.create_mutable_file(BIG)
1744}
1745[Write a segmented mutable downloader
1746Kevan Carstensen <kevan@isnotajoke.com>**20100626234314
1747 Ignore-this: d2bef531cde1b5c38f2eb28afdd4b17c
1748 
1749 The segmented mutable downloader can deal with MDMF files (files with
1750 one or more segments in MDMF format) and SDMF files (files with one
1751 segment in SDMF format). It is backwards compatible with the old
1752 file format.
1753 
1754 This patch also contains tests for the segmented mutable downloader.
1755] {
1756hunk ./src/allmydata/mutable/retrieve.py 8
1757 from twisted.internet import defer
1758 from twisted.python import failure
1759 from foolscap.api import DeadReferenceError, eventually, fireEventually
1760-from allmydata.interfaces import IRetrieveStatus, NotEnoughSharesError
1761-from allmydata.util import hashutil, idlib, log
1762+from allmydata.interfaces import IRetrieveStatus, NotEnoughSharesError, \
1763+                                 MDMF_VERSION, SDMF_VERSION
1764+from allmydata.util import hashutil, idlib, log, mathutil
1765 from allmydata import hashtree, codec
1766 from allmydata.storage.server import si_b2a
1767 from pycryptopp.cipher.aes import AES
1768hunk ./src/allmydata/mutable/retrieve.py 17
1769 from pycryptopp.publickey import rsa
1770 
1771 from allmydata.mutable.common import DictOfSets, CorruptShareError, UncoordinatedWriteError
1772-from allmydata.mutable.layout import SIGNED_PREFIX, unpack_share_data
1773+from allmydata.mutable.layout import SIGNED_PREFIX, unpack_share_data, \
1774+                                     MDMFSlotReadProxy
1775 
1776 class RetrieveStatus:
1777     implements(IRetrieveStatus)
1778hunk ./src/allmydata/mutable/retrieve.py 104
1779         self.verinfo = verinfo
1780         # during repair, we may be called upon to grab the private key, since
1781         # it wasn't picked up during a verify=False checker run, and we'll
1782-        # need it for repair to generate the a new version.
1783+        # need it for repair to generate a new version.
1784         self._need_privkey = fetch_privkey
1785         if self._node.get_privkey():
1786             self._need_privkey = False
1787hunk ./src/allmydata/mutable/retrieve.py 109
1788 
1789+        if self._need_privkey:
1790+            # TODO: Evaluate the need for this. We'll use it if we want
1791+            # to limit how many queries are on the wire for the privkey
1792+            # at once.
1793+            self._privkey_query_markers = [] # one Marker for each time we've
1794+                                             # tried to get the privkey.
1795+
1796         self._status = RetrieveStatus()
1797         self._status.set_storage_index(self._storage_index)
1798         self._status.set_helper(False)
1799hunk ./src/allmydata/mutable/retrieve.py 125
1800          offsets_tuple) = self.verinfo
1801         self._status.set_size(datalength)
1802         self._status.set_encoding(k, N)
1803+        self.readers = {}
1804 
1805     def get_status(self):
1806         return self._status
1807hunk ./src/allmydata/mutable/retrieve.py 149
1808         self.remaining_sharemap = DictOfSets()
1809         for (shnum, peerid, timestamp) in shares:
1810             self.remaining_sharemap.add(shnum, peerid)
1811+            # If the servermap update fetched anything, it fetched at least 1
1812+            # KiB, so we ask for that much.
1813+            # TODO: Change the cache methods to allow us to fetch all of the
1814+            # data that they have, then change this method to do that.
1815+            any_cache, timestamp = self._node._read_from_cache(self.verinfo,
1816+                                                               shnum,
1817+                                                               0,
1818+                                                               1000)
1819+            ss = self.servermap.connections[peerid]
1820+            reader = MDMFSlotReadProxy(ss,
1821+                                       self._storage_index,
1822+                                       shnum,
1823+                                       any_cache)
1824+            reader.peerid = peerid
1825+            self.readers[shnum] = reader
1826+
1827 
1828         self.shares = {} # maps shnum to validated blocks
1829hunk ./src/allmydata/mutable/retrieve.py 167
1830+        self._active_readers = [] # list of active readers for this dl.
1831+        self._validated_readers = set() # set of readers that we have
1832+                                        # validated the prefix of
1833+        self._block_hash_trees = {} # shnum => hashtree
1834+        # TODO: Make this into a file-backed consumer or something to
1835+        # conserve memory.
1836+        self._plaintext = ""
1837 
1838         # how many shares do we need?
1839hunk ./src/allmydata/mutable/retrieve.py 176
1840-        (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
1841+        (seqnum,
1842+         root_hash,
1843+         IV,
1844+         segsize,
1845+         datalength,
1846+         k,
1847+         N,
1848+         prefix,
1849          offsets_tuple) = self.verinfo
1850hunk ./src/allmydata/mutable/retrieve.py 185
1851-        assert len(self.remaining_sharemap) >= k
1852-        # we start with the lowest shnums we have available, since FEC is
1853-        # faster if we're using "primary shares"
1854-        self.active_shnums = set(sorted(self.remaining_sharemap.keys())[:k])
1855-        for shnum in self.active_shnums:
1856-            # we use an arbitrary peer who has the share. If shares are
1857-            # doubled up (more than one share per peer), we could make this
1858-            # run faster by spreading the load among multiple peers. But the
1859-            # algorithm to do that is more complicated than I want to write
1860-            # right now, and a well-provisioned grid shouldn't have multiple
1861-            # shares per peer.
1862-            peerid = list(self.remaining_sharemap[shnum])[0]
1863-            self.get_data(shnum, peerid)
1864 
1865hunk ./src/allmydata/mutable/retrieve.py 186
1866-        # control flow beyond this point: state machine. Receiving responses
1867-        # from queries is the input. We might send out more queries, or we
1868-        # might produce a result.
1869 
1870hunk ./src/allmydata/mutable/retrieve.py 187
1871+        # We need one share hash tree for the entire file; its leaves
1872+        # are the roots of the block hash trees for the shares that
1873+        # comprise it, and its root is in the verinfo.
1874+        self.share_hash_tree = hashtree.IncompleteHashTree(N)
1875+        self.share_hash_tree.set_hashes({0: root_hash})
1876+
1877+        # This will set up both the segment decoder and the tail segment
1878+        # decoder, as well as a variety of other instance variables that
1879+        # the download process will use.
1880+        self._setup_encoding_parameters()
1881+        assert len(self.remaining_sharemap) >= k
1882+
1883+        self.log("starting download")
1884+        self._add_active_peers()
1885+        # The download process beyond this is a state machine.
1886+        # _add_active_peers will select the peers that we want to use
1887+        # for the download, and then attempt to start downloading. After
1888+        # each segment, it will check for doneness, reacting to broken
1889+        # peers and corrupt shares as necessary. If it runs out of good
1890+        # peers before downloading all of the segments, _done_deferred
1891+        # will errback.  Otherwise, it will eventually callback with the
1892+        # contents of the mutable file.
1893         return self._done_deferred
1894 
1895hunk ./src/allmydata/mutable/retrieve.py 211
1896-    def get_data(self, shnum, peerid):
1897-        self.log(format="sending sh#%(shnum)d request to [%(peerid)s]",
1898-                 shnum=shnum,
1899-                 peerid=idlib.shortnodeid_b2a(peerid),
1900-                 level=log.NOISY)
1901-        ss = self.servermap.connections[peerid]
1902-        started = time.time()
1903-        (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
1904+
1905+    def _setup_encoding_parameters(self):
1906+        """
1907+        I set up the encoding parameters, including k, n, the number
1908+        of segments associated with this file, and the segment decoder.
1909+        """
1910+        (seqnum,
1911+         root_hash,
1912+         IV,
1913+         segsize,
1914+         datalength,
1915+         k,
1916+         n,
1917+         known_prefix,
1918          offsets_tuple) = self.verinfo
1919hunk ./src/allmydata/mutable/retrieve.py 226
1920-        offsets = dict(offsets_tuple)
1921+        self._required_shares = k
1922+        self._total_shares = n
1923+        self._segment_size = segsize
1924+        self._data_length = datalength
1925+
1926+        if not IV:
1927+            self._version = MDMF_VERSION
1928+        else:
1929+            self._version = SDMF_VERSION
1930+
1931+        if datalength and segsize:
1932+            self._num_segments = mathutil.div_ceil(datalength, segsize)
1933+            self._tail_data_size = datalength % segsize
1934+        else:
1935+            self._num_segments = 0
1936+            self._tail_data_size = 0
1937 
1938hunk ./src/allmydata/mutable/retrieve.py 243
1939-        # we read the checkstring, to make sure that the data we grab is from
1940-        # the right version.
1941-        readv = [ (0, struct.calcsize(SIGNED_PREFIX)) ]
1942+        self._segment_decoder = codec.CRSDecoder()
1943+        self._segment_decoder.set_params(segsize, k, n)
1944+        self._current_segment = 0
1945 
1946hunk ./src/allmydata/mutable/retrieve.py 247
1947-        # We also read the data, and the hashes necessary to validate them
1948-        # (share_hash_chain, block_hash_tree, share_data). We don't read the
1949-        # signature or the pubkey, since that was handled during the
1950-        # servermap phase, and we'll be comparing the share hash chain
1951-        # against the roothash that was validated back then.
1952+        if  not self._tail_data_size:
1953+            self._tail_data_size = segsize
1954 
1955hunk ./src/allmydata/mutable/retrieve.py 250
1956-        readv.append( (offsets['share_hash_chain'],
1957-                       offsets['enc_privkey'] - offsets['share_hash_chain'] ) )
1958+        self._tail_segment_size = mathutil.next_multiple(self._tail_data_size,
1959+                                                         self._required_shares)
1960+        if self._tail_segment_size == self._segment_size:
1961+            self._tail_decoder = self._segment_decoder
1962+        else:
1963+            self._tail_decoder = codec.CRSDecoder()
1964+            self._tail_decoder.set_params(self._tail_segment_size,
1965+                                          self._required_shares,
1966+                                          self._total_shares)
1967 
1968hunk ./src/allmydata/mutable/retrieve.py 260
1969-        # if we need the private key (for repair), we also fetch that
1970-        if self._need_privkey:
1971-            readv.append( (offsets['enc_privkey'],
1972-                           offsets['EOF'] - offsets['enc_privkey']) )
1973+        self.log("got encoding parameters: "
1974+                 "k: %d "
1975+                 "n: %d "
1976+                 "%d segments of %d bytes each (%d byte tail segment)" % \
1977+                 (k, n, self._num_segments, self._segment_size,
1978+                  self._tail_segment_size))
1979 
1980hunk ./src/allmydata/mutable/retrieve.py 267
1981-        m = Marker()
1982-        self._outstanding_queries[m] = (peerid, shnum, started)
1983+        for i in xrange(self._total_shares):
1984+            # So we don't have to do this later.
1985+            self._block_hash_trees[i] = hashtree.IncompleteHashTree(self._num_segments)
1986 
1987hunk ./src/allmydata/mutable/retrieve.py 271
1988-        # ask the cache first
1989-        got_from_cache = False
1990-        datavs = []
1991-        for (offset, length) in readv:
1992-            (data, timestamp) = self._node._read_from_cache(self.verinfo, shnum,
1993-                                                            offset, length)
1994-            if data is not None:
1995-                datavs.append(data)
1996-        if len(datavs) == len(readv):
1997-            self.log("got data from cache")
1998-            got_from_cache = True
1999-            d = fireEventually({shnum: datavs})
2000-            # datavs is a dict mapping shnum to a pair of strings
2001-        else:
2002-            d = self._do_read(ss, peerid, self._storage_index, [shnum], readv)
2003-        self.remaining_sharemap.discard(shnum, peerid)
2004+        # If we have more than one segment, we are an SDMF file, which
2005+        # means that we need to validate the salts as we receive them.
2006+        self._salt_hash_tree = hashtree.IncompleteHashTree(self._num_segments)
2007+        self._salt_hash_tree[0] = IV # from the prefix.
2008 
2009hunk ./src/allmydata/mutable/retrieve.py 276
2010-        d.addCallback(self._got_results, m, peerid, started, got_from_cache)
2011-        d.addErrback(self._query_failed, m, peerid)
2012-        # errors that aren't handled by _query_failed (and errors caused by
2013-        # _query_failed) get logged, but we still want to check for doneness.
2014-        def _oops(f):
2015-            self.log(format="problem in _query_failed for sh#%(shnum)d to %(peerid)s",
2016-                     shnum=shnum,
2017-                     peerid=idlib.shortnodeid_b2a(peerid),
2018-                     failure=f,
2019-                     level=log.WEIRD, umid="W0xnQA")
2020-        d.addErrback(_oops)
2021-        d.addBoth(self._check_for_done)
2022-        # any error during _check_for_done means the download fails. If the
2023-        # download is successful, _check_for_done will fire _done by itself.
2024-        d.addErrback(self._done)
2025-        d.addErrback(log.err)
2026-        return d # purely for testing convenience
2027 
2028hunk ./src/allmydata/mutable/retrieve.py 277
2029-    def _do_read(self, ss, peerid, storage_index, shnums, readv):
2030-        # isolate the callRemote to a separate method, so tests can subclass
2031-        # Publish and override it
2032-        d = ss.callRemote("slot_readv", storage_index, shnums, readv)
2033-        return d
2034+    def _add_active_peers(self):
2035+        """
2036+        I populate self._active_readers with enough active readers to
2037+        retrieve the contents of this mutable file. I am called before
2038+        downloading starts, and (eventually) after each validation
2039+        error, connection error, or other problem in the download.
2040+        """
2041+        # TODO: It would be cool to investigate other heuristics for
2042+        # reader selection. For instance, the cost (in time the user
2043+        # spends waiting for their file) of selecting a really slow peer
2044+        # that happens to have a primary share is probably more than
2045+        # selecting a really fast peer that doesn't have a primary
2046+        # share. Maybe the servermap could be extended to provide this
2047+        # information; it could keep track of latency information while
2048+        # it gathers more important data, and then this routine could
2049+        # use that to select active readers.
2050+        #
2051+        # (these and other questions would be easier to answer with a
2052+        #  robust, configurable tahoe-lafs simulator, which modeled node
2053+        #  failures, differences in node speed, and other characteristics
2054+        #  that we expect storage servers to have.  You could have
2055+        #  presets for really stable grids (like allmydata.com),
2056+        #  friendnets, make it easy to configure your own settings, and
2057+        #  then simulate the effect of big changes on these use cases
2058+        #  instead of just reasoning about what the effect might be. Out
2059+        #  of scope for MDMF, though.)
2060 
2061hunk ./src/allmydata/mutable/retrieve.py 304
2062-    def remove_peer(self, peerid):
2063-        for shnum in list(self.remaining_sharemap.keys()):
2064-            self.remaining_sharemap.discard(shnum, peerid)
2065+        # We need at least self._required_shares readers to download a
2066+        # segment.
2067+        needed = self._required_shares - len(self._active_readers)
2068+        # XXX: Why don't format= log messages work here?
2069+        self.log("adding %d peers to the active peers list" % needed)
2070 
2071hunk ./src/allmydata/mutable/retrieve.py 310
2072-    def _got_results(self, datavs, marker, peerid, started, got_from_cache):
2073-        now = time.time()
2074-        elapsed = now - started
2075-        if not got_from_cache:
2076-            self._status.add_fetch_timing(peerid, elapsed)
2077-        self.log(format="got results (%(shares)d shares) from [%(peerid)s]",
2078-                 shares=len(datavs),
2079-                 peerid=idlib.shortnodeid_b2a(peerid),
2080-                 level=log.NOISY)
2081-        self._outstanding_queries.pop(marker, None)
2082-        if not self._running:
2083-            return
2084+        # We favor lower numbered shares, since FEC is faster with
2085+        # primary shares than with other shares, and lower-numbered
2086+        # shares are more likely to be primary than higher numbered
2087+        # shares.
2088+        active_shnums = set(sorted(self.remaining_sharemap.keys()))
2089+        # We shouldn't consider adding shares that we already have; this
2090+        # will cause problems later.
2091+        active_shnums -= set([reader.shnum for reader in self._active_readers])
2092+        active_shnums = list(active_shnums)[:needed]
2093+        if len(active_shnums) < needed:
2094+            # We don't have enough readers to retrieve the file; fail.
2095+            return self._failed()
2096 
2097hunk ./src/allmydata/mutable/retrieve.py 323
2098-        # note that we only ask for a single share per query, so we only
2099-        # expect a single share back. On the other hand, we use the extra
2100-        # shares if we get them.. seems better than an assert().
2101+        for shnum in active_shnums:
2102+            self._active_readers.append(self.readers[shnum])
2103+            self.log("added reader for share %d" % shnum)
2104+        assert len(self._active_readers) == self._required_shares
2105+        # Conceptually, this is part of the _add_active_peers step. It
2106+        # validates the prefixes of newly added readers to make sure
2107+        # that they match what we are expecting for self.verinfo. If
2108+        # validation is successful, _validate_active_prefixes will call
2109+        # _download_current_segment for us. If validation is
2110+        # unsuccessful, then _validate_prefixes will remove the peer and
2111+        # call _add_active_peers again, where we will attempt to rectify
2112+        # the problem by choosing another peer.
2113+        return self._validate_active_prefixes()
2114 
2115hunk ./src/allmydata/mutable/retrieve.py 337
2116-        for shnum,datav in datavs.items():
2117-            (prefix, hash_and_data) = datav[:2]
2118-            try:
2119-                self._got_results_one_share(shnum, peerid,
2120-                                            prefix, hash_and_data)
2121-            except CorruptShareError, e:
2122-                # log it and give the other shares a chance to be processed
2123-                f = failure.Failure()
2124-                self.log(format="bad share: %(f_value)s",
2125-                         f_value=str(f.value), failure=f,
2126-                         level=log.WEIRD, umid="7fzWZw")
2127-                self.notify_server_corruption(peerid, shnum, str(e))
2128-                self.remove_peer(peerid)
2129-                self.servermap.mark_bad_share(peerid, shnum, prefix)
2130-                self._bad_shares.add( (peerid, shnum) )
2131-                self._status.problems[peerid] = f
2132-                self._last_failure = f
2133-                pass
2134-            if self._need_privkey and len(datav) > 2:
2135-                lp = None
2136-                self._try_to_validate_privkey(datav[2], peerid, shnum, lp)
2137-        # all done!
2138 
2139hunk ./src/allmydata/mutable/retrieve.py 338
2140-    def notify_server_corruption(self, peerid, shnum, reason):
2141-        ss = self.servermap.connections[peerid]
2142-        ss.callRemoteOnly("advise_corrupt_share",
2143-                          "mutable", self._storage_index, shnum, reason)
2144+    def _validate_active_prefixes(self):
2145+        """
2146+        I check to make sure that the prefixes on the peers that I am
2147+        currently reading from match the prefix that we want to see, as
2148+        said in self.verinfo.
2149 
2150hunk ./src/allmydata/mutable/retrieve.py 344
2151-    def _got_results_one_share(self, shnum, peerid,
2152-                               got_prefix, got_hash_and_data):
2153-        self.log("_got_results: got shnum #%d from peerid %s"
2154-                 % (shnum, idlib.shortnodeid_b2a(peerid)))
2155-        (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
2156+        If I find that all of the active peers have acceptable prefixes,
2157+        I pass control to _download_current_segment, which will use
2158+        those peers to do cool things. If I find that some of the active
2159+        peers have unacceptable prefixes, I will remove them from active
2160+        peers (and from further consideration) and call
2161+        _add_active_peers to attempt to rectify the situation. I keep
2162+        track of which peers I have already validated so that I don't
2163+        need to do so again.
2164+        """
2165+        assert self._active_readers, "No more active readers"
2166+
2167+        ds = []
2168+        new_readers = set(self._active_readers) - self._validated_readers
2169+        self.log('validating %d newly-added active readers' % len(new_readers))
2170+
2171+        for reader in new_readers:
2172+            # We force a remote read here -- otherwise, we are relying
2173+            # on cached data that we already verified as valid, and we
2174+            # won't detect an uncoordinated write that has occurred
2175+            # since the last servermap update.
2176+            d = reader.get_prefix(force_remote=True)
2177+            d.addCallback(self._try_to_validate_prefix, reader)
2178+            ds.append(d)
2179+        dl = defer.DeferredList(ds, consumeErrors=True)
2180+        def _check_results(results):
2181+            # Each result in results will be of the form (success, msg).
2182+            # We don't care about msg, but success will tell us whether
2183+            # or not the checkstring validated. If it didn't, we need to
2184+            # remove the offending (peer,share) from our active readers,
2185+            # and ensure that active readers is again populated.
2186+            bad_readers = []
2187+            for i, result in enumerate(results):
2188+                if not result[0]:
2189+                    reader = self._active_readers[i]
2190+                    f = result[1]
2191+                    assert isinstance(f, failure.Failure)
2192+
2193+                    self.log("The reader %s failed to "
2194+                             "properly validate: %s" % \
2195+                             (reader, str(f.value)))
2196+                    bad_readers.append((reader, f))
2197+                else:
2198+                    reader = self._active_readers[i]
2199+                    self.log("the reader %s checks out, so we'll use it" % \
2200+                             reader)
2201+                    self._validated_readers.add(reader)
2202+                    # Each time we validate a reader, we check to see if
2203+                    # we need the private key. If we do, we politely ask
2204+                    # for it and then continue computing. If we find
2205+                    # that we haven't gotten it at the end of
2206+                    # segment decoding, then we'll take more drastic
2207+                    # measures.
2208+                    if self._need_privkey:
2209+                        d = reader.get_encprivkey()
2210+                        d.addCallback(self._try_to_validate_privkey, reader)
2211+            if bad_readers:
2212+                # We do them all at once, or else we screw up list indexing.
2213+                for (reader, f) in bad_readers:
2214+                    self._mark_bad_share(reader, f)
2215+                return self._add_active_peers()
2216+            else:
2217+                return self._download_current_segment()
2218+            # The next step will assert that it has enough active
2219+            # readers to fetch shares; we just need to remove it.
2220+        dl.addCallback(_check_results)
2221+        return dl
2222+
2223+
2224+    def _try_to_validate_prefix(self, prefix, reader):
2225+        """
2226+        I check that the prefix returned by a candidate server for
2227+        retrieval matches the prefix that the servermap knows about
2228+        (and, hence, the prefix that was validated earlier). If it does,
2229+        I return True, which means that I approve of the use of the
2230+        candidate server for segment retrieval. If it doesn't, I return
2231+        False, which means that another server must be chosen.
2232+        """
2233+        (seqnum,
2234+         root_hash,
2235+         IV,
2236+         segsize,
2237+         datalength,
2238+         k,
2239+         N,
2240+         known_prefix,
2241          offsets_tuple) = self.verinfo
2242hunk ./src/allmydata/mutable/retrieve.py 430
2243-        assert len(got_prefix) == len(prefix), (len(got_prefix), len(prefix))
2244-        if got_prefix != prefix:
2245-            msg = "someone wrote to the data since we read the servermap: prefix changed"
2246-            raise UncoordinatedWriteError(msg)
2247-        (share_hash_chain, block_hash_tree,
2248-         share_data) = unpack_share_data(self.verinfo, got_hash_and_data)
2249+        if known_prefix != prefix:
2250+            self.log("prefix from share %d doesn't match" % reader.shnum)
2251+            raise UncoordinatedWriteError("Mismatched prefix -- this could "
2252+                                          "indicate an uncoordinated write")
2253+        # Otherwise, we're okay -- no issues.
2254 
2255hunk ./src/allmydata/mutable/retrieve.py 436
2256-        assert isinstance(share_data, str)
2257-        # build the block hash tree. SDMF has only one leaf.
2258-        leaves = [hashutil.block_hash(share_data)]
2259-        t = hashtree.HashTree(leaves)
2260-        if list(t) != block_hash_tree:
2261-            raise CorruptShareError(peerid, shnum, "block hash tree failure")
2262-        share_hash_leaf = t[0]
2263-        t2 = hashtree.IncompleteHashTree(N)
2264-        # root_hash was checked by the signature
2265-        t2.set_hashes({0: root_hash})
2266-        try:
2267-            t2.set_hashes(hashes=share_hash_chain,
2268-                          leaves={shnum: share_hash_leaf})
2269-        except (hashtree.BadHashError, hashtree.NotEnoughHashesError,
2270-                IndexError), e:
2271-            msg = "corrupt hashes: %s" % (e,)
2272-            raise CorruptShareError(peerid, shnum, msg)
2273-        self.log(" data valid! len=%d" % len(share_data))
2274-        # each query comes down to this: placing validated share data into
2275-        # self.shares
2276-        self.shares[shnum] = share_data
2277 
2278hunk ./src/allmydata/mutable/retrieve.py 437
2279-    def _try_to_validate_privkey(self, enc_privkey, peerid, shnum, lp):
2280+    def _remove_reader(self, reader):
2281+        """
2282+        At various points, we will wish to remove a peer from
2283+        consideration and/or use. These include, but are not necessarily
2284+        limited to:
2285 
2286hunk ./src/allmydata/mutable/retrieve.py 443
2287-        alleged_privkey_s = self._node._decrypt_privkey(enc_privkey)
2288-        alleged_writekey = hashutil.ssk_writekey_hash(alleged_privkey_s)
2289-        if alleged_writekey != self._node.get_writekey():
2290-            self.log("invalid privkey from %s shnum %d" %
2291-                     (idlib.nodeid_b2a(peerid)[:8], shnum),
2292-                     parent=lp, level=log.WEIRD, umid="YIw4tA")
2293-            return
2294+            - A connection error.
2295+            - A mismatched prefix (that is, a prefix that does not match
2296+              our conception of the version information string).
2297+            - A failing block hash, salt hash, or share hash, which can
2298+              indicate disk failure/bit flips, or network trouble.
2299 
2300hunk ./src/allmydata/mutable/retrieve.py 449
2301-        # it's good
2302-        self.log("got valid privkey from shnum %d on peerid %s" %
2303-                 (shnum, idlib.shortnodeid_b2a(peerid)),
2304-                 parent=lp)
2305-        privkey = rsa.create_signing_key_from_string(alleged_privkey_s)
2306-        self._node._populate_encprivkey(enc_privkey)
2307-        self._node._populate_privkey(privkey)
2308-        self._need_privkey = False
2309+        This method will do that. I will make sure that the
2310+        (shnum,reader) combination represented by my reader argument is
2311+        not used for anything else during this download. I will not
2312+        advise the reader of any corruption, something that my callers
2313+        may wish to do on their own.
2314+        """
2315+        # TODO: When you're done writing this, see if this is ever
2316+        # actually used for something that _mark_bad_share isn't. I have
2317+        # a feeling that they will be used for very similar things, and
2318+        # that having them both here is just going to be an epic amount
2319+        # of code duplication.
2320+        #
2321+        # (well, okay, not epic, but meaningful)
2322+        self.log("removing reader %s" % reader)
2323+        # Remove the reader from _active_readers
2324+        self._active_readers.remove(reader)
2325+        # TODO: self.readers.remove(reader)?
2326+        for shnum in list(self.remaining_sharemap.keys()):
2327+            self.remaining_sharemap.discard(shnum, reader.peerid)
2328 
2329hunk ./src/allmydata/mutable/retrieve.py 469
2330-    def _query_failed(self, f, marker, peerid):
2331-        self.log(format="query to [%(peerid)s] failed",
2332-                 peerid=idlib.shortnodeid_b2a(peerid),
2333-                 level=log.NOISY)
2334-        self._status.problems[peerid] = f
2335-        self._outstanding_queries.pop(marker, None)
2336-        if not self._running:
2337-            return
2338-        self._last_failure = f
2339-        self.remove_peer(peerid)
2340-        level = log.WEIRD
2341-        if f.check(DeadReferenceError):
2342-            level = log.UNUSUAL
2343-        self.log(format="error during query: %(f_value)s",
2344-                 f_value=str(f.value), failure=f, level=level, umid="gOJB5g")
2345 
2346hunk ./src/allmydata/mutable/retrieve.py 470
2347-    def _check_for_done(self, res):
2348-        # exit paths:
2349-        #  return : keep waiting, no new queries
2350-        #  return self._send_more_queries(outstanding) : send some more queries
2351-        #  fire self._done(plaintext) : download successful
2352-        #  raise exception : download fails
2353+    def _mark_bad_share(self, reader, f):
2354+        """
2355+        I mark the (peerid, shnum) encapsulated by my reader argument as
2356+        a bad share, which means that it will not be used anywhere else.
2357 
2358hunk ./src/allmydata/mutable/retrieve.py 475
2359-        self.log(format="_check_for_done: running=%(running)s, decoding=%(decoding)s",
2360-                 running=self._running, decoding=self._decoding,
2361-                 level=log.NOISY)
2362-        if not self._running:
2363-            return
2364-        if self._decoding:
2365-            return
2366-        (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
2367-         offsets_tuple) = self.verinfo
2368+        There are several reasons to want to mark something as a bad
2369+        share. These include:
2370 
2371hunk ./src/allmydata/mutable/retrieve.py 478
2372-        if len(self.shares) < k:
2373-            # we don't have enough shares yet
2374-            return self._maybe_send_more_queries(k)
2375-        if self._need_privkey:
2376-            # we got k shares, but none of them had a valid privkey. TODO:
2377-            # look further. Adding code to do this is a bit complicated, and
2378-            # I want to avoid that complication, and this should be pretty
2379-            # rare (k shares with bitflips in the enc_privkey but not in the
2380-            # data blocks). If we actually do get here, the subsequent repair
2381-            # will fail for lack of a privkey.
2382-            self.log("got k shares but still need_privkey, bummer",
2383-                     level=log.WEIRD, umid="MdRHPA")
2384+            - A connection error to the peer.
2385+            - A mismatched prefix (that is, a prefix that does not match
2386+              our local conception of the version information string).
2387+            - A failing block hash, salt hash, share hash, or other
2388+              integrity check.
2389 
2390hunk ./src/allmydata/mutable/retrieve.py 484
2391-        # we have enough to finish. All the shares have had their hashes
2392-        # checked, so if something fails at this point, we don't know how
2393-        # to fix it, so the download will fail.
2394+        This method will ensure that readers that we wish to mark bad
2395+        (for these reasons or other reasons) are not used for the rest
2396+        of the download. Additionally, it will attempt to tell the
2397+        remote peer (with no guarantee of success) that its share is
2398+        corrupt.
2399+        """
2400+        self.log("marking share %d on server %s as bad" % \
2401+                 (reader.shnum, reader))
2402+        self._remove_reader(reader)
2403+        self._bad_shares.add((reader.peerid, reader.shnum))
2404+        self._status.problems[reader.peerid] = f
2405+        self._last_failure = f
2406+        self.notify_server_corruption(reader.peerid, reader.shnum,
2407+                                      str(f.value))
2408 
2409hunk ./src/allmydata/mutable/retrieve.py 499
2410-        self._decoding = True # avoid reentrancy
2411-        self._status.set_status("decoding")
2412-        now = time.time()
2413-        elapsed = now - self._started
2414-        self._status.timings["fetch"] = elapsed
2415 
2416hunk ./src/allmydata/mutable/retrieve.py 500
2417-        d = defer.maybeDeferred(self._decode)
2418-        d.addCallback(self._decrypt, IV, self._node.get_readkey())
2419-        d.addBoth(self._done)
2420-        return d # purely for test convenience
2421+    def _download_current_segment(self):
2422+        """
2423+        I download, validate, decode, decrypt, and assemble the segment
2424+        that this Retrieve is currently responsible for downloading.
2425+        """
2426+        assert len(self._active_readers) >= self._required_shares
2427+        if self._current_segment < self._num_segments:
2428+            d = self._process_segment(self._current_segment)
2429+        else:
2430+            d = defer.succeed(None)
2431+        d.addCallback(self._check_for_done)
2432+        return d
2433 
2434hunk ./src/allmydata/mutable/retrieve.py 513
2435-    def _maybe_send_more_queries(self, k):
2436-        # we don't have enough shares yet. Should we send out more queries?
2437-        # There are some number of queries outstanding, each for a single
2438-        # share. If we can generate 'needed_shares' additional queries, we do
2439-        # so. If we can't, then we know this file is a goner, and we raise
2440-        # NotEnoughSharesError.
2441-        self.log(format=("_maybe_send_more_queries, have=%(have)d, k=%(k)d, "
2442-                         "outstanding=%(outstanding)d"),
2443-                 have=len(self.shares), k=k,
2444-                 outstanding=len(self._outstanding_queries),
2445-                 level=log.NOISY)
2446 
2447hunk ./src/allmydata/mutable/retrieve.py 514
2448-        remaining_shares = k - len(self.shares)
2449-        needed = remaining_shares - len(self._outstanding_queries)
2450-        if not needed:
2451-            # we have enough queries in flight already
2452+    def _process_segment(self, segnum):
2453+        """
2454+        I download, validate, decode, and decrypt one segment of the
2455+        file that this Retrieve is retrieving. This means coordinating
2456+        the process of getting k blocks of that file, validating them,
2457+        assembling them into one segment with the decoder, and then
2458+        decrypting them.
2459+        """
2460+        self.log("processing segment %d" % segnum)
2461 
2462hunk ./src/allmydata/mutable/retrieve.py 524
2463-            # TODO: but if they've been in flight for a long time, and we
2464-            # have reason to believe that new queries might respond faster
2465-            # (i.e. we've seen other queries come back faster, then consider
2466-            # sending out new queries. This could help with peers which have
2467-            # silently gone away since the servermap was updated, for which
2468-            # we're still waiting for the 15-minute TCP disconnect to happen.
2469-            self.log("enough queries are in flight, no more are needed",
2470-                     level=log.NOISY)
2471-            return
2472+        # TODO: The old code uses a marker. Should this code do that
2473+        # too? What did the Marker do?
2474+        assert len(self._active_readers) >= self._required_shares
2475+
2476+        # We need to ask each of our active readers for its block and
2477+        # salt. We will then validate those. If validation is
2478+        # successful, we will assemble the results into plaintext.
2479+        ds = []
2480+        for reader in self._active_readers:
2481+            d = reader.get_block_and_salt(segnum, queue=True)
2482+            d2 = self._get_needed_hashes(reader, segnum)
2483+            dl = defer.DeferredList([d, d2], consumeErrors=True)
2484+            dl.addCallback(self._validate_block, segnum, reader)
2485+            dl.addErrback(self._validation_or_decoding_failed, [reader])
2486+            ds.append(dl)
2487+            reader.flush()
2488+        dl = defer.DeferredList(ds)
2489+        dl.addCallback(self._maybe_decode_and_decrypt_segment, segnum)
2490+        return dl
2491 
2492hunk ./src/allmydata/mutable/retrieve.py 544
2493-        outstanding_shnums = set([shnum
2494-                                  for (peerid, shnum, started)
2495-                                  in self._outstanding_queries.values()])
2496-        # prefer low-numbered shares, they are more likely to be primary
2497-        available_shnums = sorted(self.remaining_sharemap.keys())
2498-        for shnum in available_shnums:
2499-            if shnum in outstanding_shnums:
2500-                # skip ones that are already in transit
2501-                continue
2502-            if shnum not in self.remaining_sharemap:
2503-                # no servers for that shnum. note that DictOfSets removes
2504-                # empty sets from the dict for us.
2505-                continue
2506-            peerid = list(self.remaining_sharemap[shnum])[0]
2507-            # get_data will remove that peerid from the sharemap, and add the
2508-            # query to self._outstanding_queries
2509-            self._status.set_status("Retrieving More Shares")
2510-            self.get_data(shnum, peerid)
2511-            needed -= 1
2512-            if not needed:
2513+
2514+    def _maybe_decode_and_decrypt_segment(self, blocks_and_salts, segnum):
2515+        """
2516+        I take the results of fetching and validating the blocks from a
2517+        callback chain in another method. If the results are such that
2518+        they tell me that validation and fetching succeeded without
2519+        incident, I will proceed with decoding and decryption.
2520+        Otherwise, I will do nothing.
2521+        """
2522+        self.log("trying to decode and decrypt segment %d" % segnum)
2523+        failures = False
2524+        for block_and_salt in blocks_and_salts:
2525+            if not block_and_salt[0] or block_and_salt[1] == None:
2526+                self.log("some validation operations failed; not proceeding")
2527+                failures = True
2528                 break
2529hunk ./src/allmydata/mutable/retrieve.py 560
2530+        if not failures:
2531+            self.log("everything looks ok, building segment %d" % segnum)
2532+            d = self._decode_blocks(blocks_and_salts, segnum)
2533+            d.addCallback(self._decrypt_segment)
2534+            d.addErrback(self._validation_or_decoding_failed,
2535+                         self._active_readers)
2536+            d.addCallback(self._set_segment)
2537+            return d
2538+        else:
2539+            return defer.succeed(None)
2540+
2541+
2542+    def _set_segment(self, segment):
2543+        """
2544+        Given a plaintext segment, I register that segment with the
2545+        target that is handling the file download.
2546+        """
2547+        self.log("got plaintext for segment %d" % self._current_segment)
2548+        self._plaintext += segment
2549+        self._current_segment += 1
2550 
2551hunk ./src/allmydata/mutable/retrieve.py 581
2552-        # at this point, we have as many outstanding queries as we can. If
2553-        # needed!=0 then we might not have enough to recover the file.
2554-        if needed:
2555-            format = ("ran out of peers: "
2556-                      "have %(have)d shares (k=%(k)d), "
2557-                      "%(outstanding)d queries in flight, "
2558-                      "need %(need)d more, "
2559-                      "found %(bad)d bad shares")
2560-            args = {"have": len(self.shares),
2561-                    "k": k,
2562-                    "outstanding": len(self._outstanding_queries),
2563-                    "need": needed,
2564-                    "bad": len(self._bad_shares),
2565-                    }
2566-            self.log(format=format,
2567-                     level=log.WEIRD, umid="ezTfjw", **args)
2568-            err = NotEnoughSharesError("%s, last failure: %s" %
2569-                                      (format % args, self._last_failure))
2570-            if self._bad_shares:
2571-                self.log("We found some bad shares this pass. You should "
2572-                         "update the servermap and try again to check "
2573-                         "more peers",
2574-                         level=log.WEIRD, umid="EFkOlA")
2575-                err.servermap = self.servermap
2576-            raise err
2577 
2578hunk ./src/allmydata/mutable/retrieve.py 582
2579+    def _validation_or_decoding_failed(self, f, readers):
2580+        """
2581+        I am called when a block or a salt fails to correctly validate, or when
2582+        the decryption or decoding operation fails for some reason.  I react to
2583+        this failure by notifying the remote server of corruption, and then
2584+        removing the remote peer from further activity.
2585+        """
2586+        assert isinstance(readers, list)
2587+        bad_shnums = [reader.shnum for reader in readers]
2588+
2589+        self.log("validation or decoding failed on share(s) %s, peer(s) %s "
2590+                 ", segment %d: %s" % \
2591+                 (bad_shnums, readers, self._current_segment, str(f)))
2592+        for reader in readers:
2593+            self._mark_bad_share(reader, f)
2594         return
2595 
2596hunk ./src/allmydata/mutable/retrieve.py 599
2597-    def _decode(self):
2598-        started = time.time()
2599-        (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
2600-         offsets_tuple) = self.verinfo
2601 
2602hunk ./src/allmydata/mutable/retrieve.py 600
2603-        # shares_dict is a dict mapping shnum to share data, but the codec
2604-        # wants two lists.
2605-        shareids = []; shares = []
2606-        for shareid, share in self.shares.items():
2607+    def _validate_block(self, results, segnum, reader):
2608+        """
2609+        I validate a block from one share on a remote server.
2610+        """
2611+        # Grab the part of the block hash tree that is necessary to
2612+        # validate this block, then generate the block hash root.
2613+        self.log("validating share %d for segment %d" % (reader.shnum,
2614+                                                             segnum))
2615+        # Did we fail to fetch either of the things that we were
2616+        # supposed to? Fail if so.
2617+        if not results[0][0] and results[1][0]:
2618+            # handled by the errback handler.
2619+
2620+            # These all get batched into one query, so the resulting
2621+            # failure should be the same for all of them, so we can just
2622+            # use the first one.
2623+            assert isinstance(results[0][1], failure.Failure)
2624+
2625+            f = results[0][1]
2626+            raise CorruptShareError(reader.peerid,
2627+                                    reader.shnum,
2628+                                    "Connection error: %s" % str(f))
2629+
2630+        block_and_salt, block_and_sharehashes = results
2631+        block, salt = block_and_salt[1]
2632+        blockhashes, sharehashes = block_and_sharehashes[1]
2633+
2634+        blockhashes = dict(enumerate(blockhashes[1]))
2635+        self.log("the reader gave me the following blockhashes: %s" % \
2636+                 blockhashes.keys())
2637+        self.log("the reader gave me the following sharehashes: %s" % \
2638+                 sharehashes[1].keys())
2639+        bht = self._block_hash_trees[reader.shnum]
2640+
2641+        if bht.needed_hashes(segnum, include_leaf=True):
2642+            try:
2643+                bht.set_hashes(blockhashes)
2644+            except (hashtree.BadHashError, hashtree.NotEnoughHashesError, \
2645+                    IndexError), e:
2646+                raise CorruptShareError(reader.peerid,
2647+                                        reader.shnum,
2648+                                        "block hash tree failure: %s" % e)
2649+
2650+        if self._version == MDMF_VERSION:
2651+            blockhash = hashutil.block_hash(salt + block)
2652+        else:
2653+            blockhash = hashutil.block_hash(block)
2654+        # If this works without an error, then validation is
2655+        # successful.
2656+        try:
2657+           bht.set_hashes(leaves={segnum: blockhash})
2658+        except (hashtree.BadHashError, hashtree.NotEnoughHashesError, \
2659+                IndexError), e:
2660+            raise CorruptShareError(reader.peerid,
2661+                                    reader.shnum,
2662+                                    "block hash tree failure: %s" % e)
2663+
2664+        # Reaching this point means that we know that this segment
2665+        # is correct. Now we need to check to see whether the share
2666+        # hash chain is also correct.
2667+        # SDMF wrote share hash chains that didn't contain the
2668+        # leaves, which would be produced from the block hash tree.
2669+        # So we need to validate the block hash tree first. If
2670+        # successful, then bht[0] will contain the root for the
2671+        # shnum, which will be a leaf in the share hash tree, which
2672+        # will allow us to validate the rest of the tree.
2673+        if self.share_hash_tree.needed_hashes(reader.shnum,
2674+                                               include_leaf=True):
2675+            try:
2676+                self.share_hash_tree.set_hashes(hashes=sharehashes[1],
2677+                                            leaves={reader.shnum: bht[0]})
2678+            except (hashtree.BadHashError, hashtree.NotEnoughHashesError, \
2679+                    IndexError), e:
2680+                raise CorruptShareError(reader.peerid,
2681+                                        reader.shnum,
2682+                                        "corrupt hashes: %s" % e)
2683+
2684+        # TODO: Validate the salt, too.
2685+        self.log('share %d is valid for segment %d' % (reader.shnum,
2686+                                                       segnum))
2687+        return {reader.shnum: (block, salt)}
2688+
2689+
2690+    def _get_needed_hashes(self, reader, segnum):
2691+        """
2692+        I get the hashes needed to validate segnum from the reader, then return
2693+        to my caller when this is done.
2694+        """
2695+        bht = self._block_hash_trees[reader.shnum]
2696+        needed = bht.needed_hashes(segnum, include_leaf=True)
2697+        # The root of the block hash tree is also a leaf in the share
2698+        # hash tree. So we don't need to fetch it from the remote
2699+        # server. In the case of files with one segment, this means that
2700+        # we won't fetch any block hash tree from the remote server,
2701+        # since the hash of each share of the file is the entire block
2702+        # hash tree, and is a leaf in the share hash tree. This is fine,
2703+        # since any share corruption will be detected in the share hash
2704+        # tree.
2705+        #needed.discard(0)
2706+        self.log("getting blockhashes for segment %d, share %d: %s" % \
2707+                 (segnum, reader.shnum, str(needed)))
2708+        d1 = reader.get_blockhashes(needed, queue=True, force_remote=True)
2709+        if self.share_hash_tree.needed_hashes(reader.shnum):
2710+            need = self.share_hash_tree.needed_hashes(reader.shnum)
2711+            self.log("also need sharehashes for share %d: %s" % (reader.shnum,
2712+                                                                 str(need)))
2713+            d2 = reader.get_sharehashes(need, queue=True, force_remote=True)
2714+        else:
2715+            d2 = defer.succeed({}) # the logic in the next method
2716+                                   # expects a dict
2717+        dl = defer.DeferredList([d1, d2], consumeErrors=True)
2718+        return dl
2719+
2720+
2721+    def _decode_blocks(self, blocks_and_salts, segnum):
2722+        """
2723+        I take a list of k blocks and salts, and decode that into a
2724+        single encrypted segment.
2725+        """
2726+        d = {}
2727+        # We want to merge our dictionaries to the form
2728+        # {shnum: blocks_and_salts}
2729+        #
2730+        # The dictionaries come from validate block that way, so we just
2731+        # need to merge them.
2732+        for block_and_salt in blocks_and_salts:
2733+            d.update(block_and_salt[1])
2734+
2735+        # All of these blocks should have the same salt; in SDMF, it is
2736+        # the file-wide IV, while in MDMF it is the per-segment salt. In
2737+        # either case, we just need to get one of them and use it.
2738+        #
2739+        # d.items()[0] is like (shnum, (block, salt))
2740+        # d.items()[0][1] is like (block, salt)
2741+        # d.items()[0][1][1] is the salt.
2742+        salt = d.items()[0][1][1]
2743+        # Next, extract just the blocks from the dict. We'll use the
2744+        # salt in the next step.
2745+        share_and_shareids = [(k, v[0]) for k, v in d.items()]
2746+        d2 = dict(share_and_shareids)
2747+        shareids = []
2748+        shares = []
2749+        for shareid, share in d2.items():
2750             shareids.append(shareid)
2751             shares.append(share)
2752 
2753hunk ./src/allmydata/mutable/retrieve.py 746
2754-        assert len(shareids) >= k, len(shareids)
2755+        assert len(shareids) >= self._required_shares, len(shareids)
2756         # zfec really doesn't want extra shares
2757hunk ./src/allmydata/mutable/retrieve.py 748
2758-        shareids = shareids[:k]
2759-        shares = shares[:k]
2760-
2761-        fec = codec.CRSDecoder()
2762-        fec.set_params(segsize, k, N)
2763-
2764-        self.log("params %s, we have %d shares" % ((segsize, k, N), len(shares)))
2765-        self.log("about to decode, shareids=%s" % (shareids,))
2766-        d = defer.maybeDeferred(fec.decode, shares, shareids)
2767-        def _done(buffers):
2768-            self._status.timings["decode"] = time.time() - started
2769-            self.log(" decode done, %d buffers" % len(buffers))
2770+        shareids = shareids[:self._required_shares]
2771+        shares = shares[:self._required_shares]
2772+        self.log("decoding segment %d" % segnum)
2773+        if segnum == self._num_segments - 1:
2774+            d = defer.maybeDeferred(self._tail_decoder.decode, shares, shareids)
2775+        else:
2776+            d = defer.maybeDeferred(self._segment_decoder.decode, shares, shareids)
2777+        def _process(buffers):
2778             segment = "".join(buffers)
2779hunk ./src/allmydata/mutable/retrieve.py 757
2780+            self.log(format="now decoding segment %(segnum)s of %(numsegs)s",
2781+                     segnum=segnum,
2782+                     numsegs=self._num_segments,
2783+                     level=log.NOISY)
2784             self.log(" joined length %d, datalength %d" %
2785hunk ./src/allmydata/mutable/retrieve.py 762
2786-                     (len(segment), datalength))
2787-            segment = segment[:datalength]
2788+                     (len(segment), self._data_length))
2789+            if segnum == self._num_segments - 1:
2790+                size_to_use = self._tail_data_size
2791+            else:
2792+                size_to_use = self._segment_size
2793+            segment = segment[:size_to_use]
2794             self.log(" segment len=%d" % len(segment))
2795hunk ./src/allmydata/mutable/retrieve.py 769
2796-            return segment
2797-        def _err(f):
2798-            self.log(" decode failed: %s" % f)
2799-            return f
2800-        d.addCallback(_done)
2801-        d.addErrback(_err)
2802+            return segment, salt
2803+        d.addCallback(_process)
2804         return d
2805 
2806hunk ./src/allmydata/mutable/retrieve.py 773
2807-    def _decrypt(self, crypttext, IV, readkey):
2808+
2809+    def _decrypt_segment(self, segment_and_salt):
2810+        """
2811+        I take a single segment and its salt, and decrypt it. I return
2812+        the plaintext of the segment that is in my argument.
2813+        """
2814+        segment, salt = segment_and_salt
2815         self._status.set_status("decrypting")
2816hunk ./src/allmydata/mutable/retrieve.py 781
2817+        self.log("decrypting segment %d" % self._current_segment)
2818         started = time.time()
2819hunk ./src/allmydata/mutable/retrieve.py 783
2820-        key = hashutil.ssk_readkey_data_hash(IV, readkey)
2821+        key = hashutil.ssk_readkey_data_hash(salt, self._node.get_readkey())
2822         decryptor = AES(key)
2823hunk ./src/allmydata/mutable/retrieve.py 785
2824-        plaintext = decryptor.process(crypttext)
2825+        plaintext = decryptor.process(segment)
2826         self._status.timings["decrypt"] = time.time() - started
2827         return plaintext
2828 
2829hunk ./src/allmydata/mutable/retrieve.py 789
2830-    def _done(self, res):
2831-        if not self._running:
2832+
2833+    def notify_server_corruption(self, peerid, shnum, reason):
2834+        ss = self.servermap.connections[peerid]
2835+        ss.callRemoteOnly("advise_corrupt_share",
2836+                          "mutable", self._storage_index, shnum, reason)
2837+
2838+
2839+    def _try_to_validate_privkey(self, enc_privkey, reader):
2840+
2841+        alleged_privkey_s = self._node._decrypt_privkey(enc_privkey)
2842+        alleged_writekey = hashutil.ssk_writekey_hash(alleged_privkey_s)
2843+        if alleged_writekey != self._node.get_writekey():
2844+            self.log("invalid privkey from %s shnum %d" %
2845+                     (reader, reader.shnum),
2846+                     level=log.WEIRD, umid="YIw4tA")
2847             return
2848hunk ./src/allmydata/mutable/retrieve.py 805
2849-        self._running = False
2850-        self._status.set_active(False)
2851-        self._status.timings["total"] = time.time() - self._started
2852-        # res is either the new contents, or a Failure
2853-        if isinstance(res, failure.Failure):
2854-            self.log("Retrieve done, with failure", failure=res,
2855-                     level=log.UNUSUAL)
2856-            self._status.set_status("Failed")
2857-        else:
2858-            self.log("Retrieve done, success!")
2859-            self._status.set_status("Finished")
2860-            self._status.set_progress(1.0)
2861-            # remember the encoding parameters, use them again next time
2862-            (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
2863-             offsets_tuple) = self.verinfo
2864-            self._node._populate_required_shares(k)
2865-            self._node._populate_total_shares(N)
2866-        eventually(self._done_deferred.callback, res)
2867 
2868hunk ./src/allmydata/mutable/retrieve.py 806
2869+        # it's good
2870+        self.log("got valid privkey from shnum %d on reader %s" %
2871+                 (reader.shnum, reader))
2872+        privkey = rsa.create_signing_key_from_string(alleged_privkey_s)
2873+        self._node._populate_encprivkey(enc_privkey)
2874+        self._node._populate_privkey(privkey)
2875+        self._need_privkey = False
2876+
2877+
2878+    def _check_for_done(self, res):
2879+        """
2880+        I check to see if this Retrieve object has successfully finished
2881+        its work.
2882+
2883+        I can exit in the following ways:
2884+            - If there are no more segments to download, then I exit by
2885+              causing self._done_deferred to fire with the plaintext
2886+              content requested by the caller.
2887+            - If there are still segments to be downloaded, and there
2888+              are enough active readers (readers which have not broken
2889+              and have not given us corrupt data) to continue
2890+              downloading, I send control back to
2891+              _download_current_segment.
2892+            - If there are still segments to be downloaded but there are
2893+              not enough active peers to download them, I ask
2894+              _add_active_peers to add more peers. If it is successful,
2895+              it will call _download_current_segment. If there are not
2896+              enough peers to retrieve the file, then that will cause
2897+              _done_deferred to errback.
2898+        """
2899+        self.log("checking for doneness")
2900+        if self._current_segment == self._num_segments:
2901+            # No more segments to download, we're done.
2902+            self.log("got plaintext, done")
2903+            return self._done()
2904+
2905+        if len(self._active_readers) >= self._required_shares:
2906+            # More segments to download, but we have enough good peers
2907+            # in self._active_readers that we can do that without issue,
2908+            # so go nab the next segment.
2909+            self.log("not done yet: on segment %d of %d" % \
2910+                     (self._current_segment + 1, self._num_segments))
2911+            return self._download_current_segment()
2912+
2913+        self.log("not done yet: on segment %d of %d, need to add peers" % \
2914+                 (self._current_segment + 1, self._num_segments))
2915+        return self._add_active_peers()
2916+
2917+
2918+    def _done(self):
2919+        """
2920+        I am called by _check_for_done when the download process has
2921+        finished successfully. After making some useful logging
2922+        statements, I return the decrypted contents to the owner of this
2923+        Retrieve object through self._done_deferred.
2924+        """
2925+        eventually(self._done_deferred.callback, self._plaintext)
2926+
2927+
2928+    def _failed(self):
2929+        """
2930+        I am called by _add_active_peers when there are not enough
2931+        active peers left to complete the download. After making some
2932+        useful logging statements, I return an exception to that effect
2933+        to the caller of this Retrieve object through
2934+        self._done_deferred.
2935+        """
2936+        format = ("ran out of peers: "
2937+                  "have %(have)d of %(total)d segments "
2938+                  "found %(bad)d bad shares "
2939+                  "encoding %(k)d-of-%(n)d")
2940+        args = {"have": self._current_segment,
2941+                "total": self._num_segments,
2942+                "k": self._required_shares,
2943+                "n": self._total_shares,
2944+                "bad": len(self._bad_shares)}
2945+        e = NotEnoughSharesError("%s, last failure: %s" % (format % args,
2946+                                                        str(self._last_failure)))
2947+        f = failure.Failure(e)
2948+        eventually(self._done_deferred.callback, f)
2949hunk ./src/allmydata/test/test_mutable.py 12
2950 from allmydata.util.hashutil import tagged_hash, ssk_writekey_hash, \
2951      ssk_pubkey_fingerprint_hash
2952 from allmydata.interfaces import IRepairResults, ICheckAndRepairResults, \
2953-     NotEnoughSharesError
2954+     NotEnoughSharesError, SDMF_VERSION, MDMF_VERSION
2955 from allmydata.monitor import Monitor
2956 from allmydata.test.common import ShouldFailMixin
2957 from allmydata.test.no_network import GridTestMixin
2958hunk ./src/allmydata/test/test_mutable.py 28
2959 from allmydata.mutable.retrieve import Retrieve
2960 from allmydata.mutable.publish import Publish
2961 from allmydata.mutable.servermap import ServerMap, ServermapUpdater
2962-from allmydata.mutable.layout import unpack_header, unpack_share
2963+from allmydata.mutable.layout import unpack_header, unpack_share, \
2964+                                     MDMFSlotReadProxy
2965 from allmydata.mutable.repairer import MustForceRepairError
2966 
2967 import allmydata.test.common_util as testutil
2968hunk ./src/allmydata/test/test_mutable.py 104
2969         d = fireEventually()
2970         d.addCallback(lambda res: _call())
2971         return d
2972+
2973     def callRemoteOnly(self, methname, *args, **kwargs):
2974         d = self.callRemote(methname, *args, **kwargs)
2975         d.addBoth(lambda ignore: None)
2976hunk ./src/allmydata/test/test_mutable.py 163
2977 def corrupt(res, s, offset, shnums_to_corrupt=None, offset_offset=0):
2978     # if shnums_to_corrupt is None, corrupt all shares. Otherwise it is a
2979     # list of shnums to corrupt.
2980+    ds = []
2981     for peerid in s._peers:
2982         shares = s._peers[peerid]
2983         for shnum in shares:
2984hunk ./src/allmydata/test/test_mutable.py 190
2985                 else:
2986                     offset1 = offset
2987                     offset2 = 0
2988-                if offset1 == "pubkey":
2989+                if offset1 == "pubkey" and IV:
2990                     real_offset = 107
2991hunk ./src/allmydata/test/test_mutable.py 192
2992+                elif offset1 == "share_data" and not IV:
2993+                    real_offset = 104
2994                 elif offset1 in o:
2995                     real_offset = o[offset1]
2996                 else:
2997hunk ./src/allmydata/test/test_mutable.py 327
2998         d.addCallback(_created)
2999         return d
3000 
3001+
3002+    def test_upload_and_download_mdmf(self):
3003+        d = self.nodemaker.create_mutable_file(version=MDMF_VERSION)
3004+        def _created(n):
3005+            d = defer.succeed(None)
3006+            d.addCallback(lambda ignored:
3007+                n.get_servermap(MODE_READ))
3008+            def _then(servermap):
3009+                dumped = servermap.dump(StringIO())
3010+                self.failUnlessIn("3-of-10", dumped.getvalue())
3011+            d.addCallback(_then)
3012+            # Now overwrite the contents with some new contents. We want
3013+            # to make them big enough to force the file to be uploaded
3014+            # in more than one segment.
3015+            big_contents = "contents1" * 100000 # about 900 KiB
3016+            d.addCallback(lambda ignored:
3017+                n.overwrite(big_contents))
3018+            d.addCallback(lambda ignored:
3019+                n.download_best_version())
3020+            d.addCallback(lambda data:
3021+                self.failUnlessEqual(data, big_contents))
3022+            # Overwrite the contents again with some new contents. As
3023+            # before, they need to be big enough to force multiple
3024+            # segments, so that we make the downloader deal with
3025+            # multiple segments.
3026+            bigger_contents = "contents2" * 1000000 # about 9MiB
3027+            d.addCallback(lambda ignored:
3028+                n.overwrite(bigger_contents))
3029+            d.addCallback(lambda ignored:
3030+                n.download_best_version())
3031+            d.addCallback(lambda data:
3032+                self.failUnlessEqual(data, bigger_contents))
3033+            return d
3034+        d.addCallback(_created)
3035+        return d
3036+
3037+
3038     def test_create_with_initial_contents(self):
3039         d = self.nodemaker.create_mutable_file("contents 1")
3040         def _created(n):
3041hunk ./src/allmydata/test/test_mutable.py 1147
3042 
3043 
3044     def _test_corrupt_all(self, offset, substring,
3045-                          should_succeed=False, corrupt_early=True,
3046-                          failure_checker=None):
3047+                          should_succeed=False,
3048+                          corrupt_early=True,
3049+                          failure_checker=None,
3050+                          fetch_privkey=False):
3051         d = defer.succeed(None)
3052         if corrupt_early:
3053             d.addCallback(corrupt, self._storage, offset)
3054hunk ./src/allmydata/test/test_mutable.py 1167
3055                     self.failUnlessIn(substring, "".join(allproblems))
3056                 return servermap
3057             if should_succeed:
3058-                d1 = self._fn.download_version(servermap, ver)
3059+                d1 = self._fn.download_version(servermap, ver,
3060+                                               fetch_privkey)
3061                 d1.addCallback(lambda new_contents:
3062                                self.failUnlessEqual(new_contents, self.CONTENTS))
3063             else:
3064hunk ./src/allmydata/test/test_mutable.py 1175
3065                 d1 = self.shouldFail(NotEnoughSharesError,
3066                                      "_corrupt_all(offset=%s)" % (offset,),
3067                                      substring,
3068-                                     self._fn.download_version, servermap, ver)
3069+                                     self._fn.download_version, servermap,
3070+                                                                ver,
3071+                                                                fetch_privkey)
3072             if failure_checker:
3073                 d1.addCallback(failure_checker)
3074             d1.addCallback(lambda res: servermap)
3075hunk ./src/allmydata/test/test_mutable.py 1186
3076         return d
3077 
3078     def test_corrupt_all_verbyte(self):
3079-        # when the version byte is not 0, we hit an UnknownVersionError error
3080-        # in unpack_share().
3081+        # when the version byte is not 0 or 1, we hit an UnknownVersionError
3082+        # error in unpack_share().
3083         d = self._test_corrupt_all(0, "UnknownVersionError")
3084         def _check_servermap(servermap):
3085             # and the dump should mention the problems
3086hunk ./src/allmydata/test/test_mutable.py 1193
3087             s = StringIO()
3088             dump = servermap.dump(s).getvalue()
3089-            self.failUnless("10 PROBLEMS" in dump, dump)
3090+            self.failUnless("30 PROBLEMS" in dump, dump)
3091         d.addCallback(_check_servermap)
3092         return d
3093 
3094hunk ./src/allmydata/test/test_mutable.py 1263
3095         return self._test_corrupt_all("enc_privkey", None, should_succeed=True)
3096 
3097 
3098+    def test_corrupt_all_encprivkey_late(self):
3099+        # this should work for the same reason as above, but we corrupt
3100+        # after the servermap update to exercise the error handling
3101+        # code.
3102+        # We need to remove the privkey from the node, or the retrieve
3103+        # process won't know to update it.
3104+        self._fn._privkey = None
3105+        return self._test_corrupt_all("enc_privkey",
3106+                                      None, # this shouldn't fail
3107+                                      should_succeed=True,
3108+                                      corrupt_early=False,
3109+                                      fetch_privkey=True)
3110+
3111+
3112     def test_corrupt_all_seqnum_late(self):
3113         # corrupting the seqnum between mapupdate and retrieve should result
3114         # in NotEnoughSharesError, since each share will look invalid
3115hunk ./src/allmydata/test/test_mutable.py 1283
3116         def _check(res):
3117             f = res[0]
3118             self.failUnless(f.check(NotEnoughSharesError))
3119-            self.failUnless("someone wrote to the data since we read the servermap" in str(f))
3120+            self.failUnless("uncoordinated write" in str(f))
3121         return self._test_corrupt_all(1, "ran out of peers",
3122                                       corrupt_early=False,
3123                                       failure_checker=_check)
3124hunk ./src/allmydata/test/test_mutable.py 1333
3125                       self.failUnlessEqual(new_contents, self.CONTENTS))
3126         return d
3127 
3128-    def test_corrupt_some(self):
3129-        # corrupt the data of first five shares (so the servermap thinks
3130-        # they're good but retrieve marks them as bad), so that the
3131-        # MODE_READ set of 6 will be insufficient, forcing node.download to
3132-        # retry with more servers.
3133-        corrupt(None, self._storage, "share_data", range(5))
3134-        d = self.make_servermap()
3135+
3136+    def _test_corrupt_some(self, offset, mdmf=False):
3137+        if mdmf:
3138+            d = self.publish_mdmf()
3139+        else:
3140+            d = defer.succeed(None)
3141+        d.addCallback(lambda ignored:
3142+            corrupt(None, self._storage, offset, range(5)))
3143+        d.addCallback(lambda ignored:
3144+            self.make_servermap())
3145         def _do_retrieve(servermap):
3146             ver = servermap.best_recoverable_version()
3147             self.failUnless(ver)
3148hunk ./src/allmydata/test/test_mutable.py 1349
3149             return self._fn.download_best_version()
3150         d.addCallback(_do_retrieve)
3151         d.addCallback(lambda new_contents:
3152-                      self.failUnlessEqual(new_contents, self.CONTENTS))
3153+            self.failUnlessEqual(new_contents, self.CONTENTS))
3154         return d
3155 
3156hunk ./src/allmydata/test/test_mutable.py 1352
3157+
3158+    def test_corrupt_some(self):
3159+        # corrupt the data of first five shares (so the servermap thinks
3160+        # they're good but retrieve marks them as bad), so that the
3161+        # MODE_READ set of 6 will be insufficient, forcing node.download to
3162+        # retry with more servers.
3163+        return self._test_corrupt_some("share_data")
3164+
3165+
3166     def test_download_fails(self):
3167         d = corrupt(None, self._storage, "signature")
3168         d.addCallback(lambda ignored:
3169hunk ./src/allmydata/test/test_mutable.py 1366
3170             self.shouldFail(UnrecoverableFileError, "test_download_anyway",
3171                             "no recoverable versions",
3172-                            self._fn.download_best_version)
3173+                            self._fn.download_best_version))
3174         return d
3175 
3176 
3177hunk ./src/allmydata/test/test_mutable.py 1370
3178+
3179+    def test_corrupt_mdmf_block_hash_tree(self):
3180+        d = self.publish_mdmf()
3181+        d.addCallback(lambda ignored:
3182+            self._test_corrupt_all(("block_hash_tree", 12 * 32),
3183+                                   "block hash tree failure",
3184+                                   corrupt_early=False,
3185+                                   should_succeed=False))
3186+        return d
3187+
3188+
3189+    def test_corrupt_mdmf_block_hash_tree_late(self):
3190+        d = self.publish_mdmf()
3191+        d.addCallback(lambda ignored:
3192+            self._test_corrupt_all(("block_hash_tree", 12 * 32),
3193+                                   "block hash tree failure",
3194+                                   corrupt_early=True,
3195+                                   should_succeed=False))
3196+        return d
3197+
3198+
3199+    def test_corrupt_mdmf_share_data(self):
3200+        d = self.publish_mdmf()
3201+        d.addCallback(lambda ignored:
3202+            # TODO: Find out what the block size is and corrupt a
3203+            # specific block, rather than just guessing.
3204+            self._test_corrupt_all(("share_data", 12 * 40),
3205+                                    "block hash tree failure",
3206+                                    corrupt_early=True,
3207+                                    should_succeed=False))
3208+        return d
3209+
3210+
3211+    def test_corrupt_some_mdmf(self):
3212+        return self._test_corrupt_some(("share_data", 12 * 40),
3213+                                       mdmf=True)
3214+
3215+
3216 class CheckerMixin:
3217     def check_good(self, r, where):
3218         self.failUnless(r.is_healthy(), where)
3219hunk ./src/allmydata/test/test_mutable.py 2116
3220             d.addCallback(lambda res:
3221                           self.shouldFail(NotEnoughSharesError,
3222                                           "test_retrieve_surprise",
3223-                                          "ran out of peers: have 0 shares (k=3)",
3224+                                          "ran out of peers: have 0 of 1",
3225                                           n.download_version,
3226                                           self.old_map,
3227                                           self.old_map.best_recoverable_version(),
3228hunk ./src/allmydata/test/test_mutable.py 2125
3229         d.addCallback(_created)
3230         return d
3231 
3232+
3233     def test_unexpected_shares(self):
3234         # upload the file, take a servermap, shut down one of the servers,
3235         # upload it again (causing shares to appear on a new server), then
3236hunk ./src/allmydata/test/test_mutable.py 2329
3237         self.basedir = "mutable/Problems/test_privkey_query_missing"
3238         self.set_up_grid(num_servers=20)
3239         nm = self.g.clients[0].nodemaker
3240-        LARGE = "These are Larger contents" * 2000 # about 50KB
3241+        LARGE = "These are Larger contents" * 2000 # about 50KiB
3242         nm._node_cache = DevNullDictionary() # disable the nodecache
3243 
3244         d = nm.create_mutable_file(LARGE)
3245hunk ./src/allmydata/test/test_mutable.py 2342
3246         d.addCallback(_created)
3247         d.addCallback(lambda res: self.n2.get_servermap(MODE_WRITE))
3248         return d
3249+
3250+
3251+    def test_block_and_hash_query_error(self):
3252+        # This tests for what happens when a query to a remote server
3253+        # fails in either the hash validation step or the block getting
3254+        # step (because of batching, this is the same actual query).
3255+        # We need to have the storage server persist up until the point
3256+        # that its prefix is validated, then suddenly die. This
3257+        # exercises some exception handling code in Retrieve.
3258+        self.basedir = "mutable/Problems/test_block_and_hash_query_error"
3259+        self.set_up_grid(num_servers=20)
3260+        nm = self.g.clients[0].nodemaker
3261+        CONTENTS = "contents" * 2000
3262+        d = nm.create_mutable_file(CONTENTS)
3263+        def _created(node):
3264+            self._node = node
3265+        d.addCallback(_created)
3266+        d.addCallback(lambda ignored:
3267+            self._node.get_servermap(MODE_READ))
3268+        def _then(servermap):
3269+            # we have our servermap. Now we set up the servers like the
3270+            # tests above -- the first one that gets a read call should
3271+            # start throwing errors, but only after returning its prefix
3272+            # for validation. Since we'll download without fetching the
3273+            # private key, the next query to the remote server will be
3274+            # for either a block and salt or for hashes, either of which
3275+            # will exercise the error handling code.
3276+            killer = FirstServerGetsKilled()
3277+            for (serverid, ss) in nm.storage_broker.get_all_servers():
3278+                ss.post_call_notifier = killer.notify
3279+            ver = servermap.best_recoverable_version()
3280+            assert ver
3281+            return self._node.download_version(servermap, ver)
3282+        d.addCallback(_then)
3283+        d.addCallback(lambda data:
3284+            self.failUnlessEqual(data, CONTENTS))
3285+        return d
3286}
3287[mutable/checker.py: check MDMF files
3288Kevan Carstensen <kevan@isnotajoke.com>**20100628225048
3289 Ignore-this: fb697b36285d60552df6ca5ac6a37629
3290 
3291 This patch adapts the mutable file checker and verifier to check and
3292 verify MDMF files. It does this by using the new segmented downloader,
3293 which is trained to perform verification operations on request. This
3294 removes some code duplication.
3295] {
3296hunk ./src/allmydata/mutable/checker.py 12
3297 from allmydata.mutable.common import MODE_CHECK, CorruptShareError
3298 from allmydata.mutable.servermap import ServerMap, ServermapUpdater
3299 from allmydata.mutable.layout import unpack_share, SIGNED_PREFIX_LENGTH
3300+from allmydata.mutable.retrieve import Retrieve # for verifying
3301 
3302 class MutableChecker:
3303 
3304hunk ./src/allmydata/mutable/checker.py 29
3305 
3306     def check(self, verify=False, add_lease=False):
3307         servermap = ServerMap()
3308+        # Updating the servermap in MODE_CHECK will stand a good chance
3309+        # of finding all of the shares, and getting a good idea of
3310+        # recoverability, etc, without verifying.
3311         u = ServermapUpdater(self._node, self._storage_broker, self._monitor,
3312                              servermap, MODE_CHECK, add_lease=add_lease)
3313         if self._history:
3314hunk ./src/allmydata/mutable/checker.py 55
3315         if num_recoverable:
3316             self.best_version = servermap.best_recoverable_version()
3317 
3318+        # The file is unhealthy and needs to be repaired if:
3319+        # - There are unrecoverable versions.
3320         if servermap.unrecoverable_versions():
3321             self.need_repair = True
3322hunk ./src/allmydata/mutable/checker.py 59
3323+        # - There isn't a recoverable version.
3324         if num_recoverable != 1:
3325             self.need_repair = True
3326hunk ./src/allmydata/mutable/checker.py 62
3327+        # - The best recoverable version is missing some shares.
3328         if self.best_version:
3329             available_shares = servermap.shares_available()
3330             (num_distinct_shares, k, N) = available_shares[self.best_version]
3331hunk ./src/allmydata/mutable/checker.py 73
3332 
3333     def _verify_all_shares(self, servermap):
3334         # read every byte of each share
3335+        #
3336+        # This logic is going to be very nearly the same as the
3337+        # downloader. I bet we could pass the downloader a flag that
3338+        # makes it do this, and piggyback onto that instead of
3339+        # duplicating a bunch of code.
3340+        #
3341+        # Like:
3342+        #  r = Retrieve(blah, blah, blah, verify=True)
3343+        #  d = r.download()
3344+        #  (wait, wait, wait, d.callback)
3345+        # 
3346+        #  Then, when it has finished, we can check the servermap (which
3347+        #  we provided to Retrieve) to figure out which shares are bad,
3348+        #  since the Retrieve process will have updated the servermap as
3349+        #  it went along.
3350+        #
3351+        #  By passing the verify=True flag to the constructor, we are
3352+        #  telling the downloader a few things.
3353+        #
3354+        #  1. It needs to download all N shares, not just K shares.
3355+        #  2. It doesn't need to decrypt or decode the shares, only
3356+        #     verify them.
3357         if not self.best_version:
3358             return
3359hunk ./src/allmydata/mutable/checker.py 97
3360-        versionmap = servermap.make_versionmap()
3361-        shares = versionmap[self.best_version]
3362-        (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
3363-         offsets_tuple) = self.best_version
3364-        offsets = dict(offsets_tuple)
3365-        readv = [ (0, offsets["EOF"]) ]
3366-        dl = []
3367-        for (shnum, peerid, timestamp) in shares:
3368-            ss = servermap.connections[peerid]
3369-            d = self._do_read(ss, peerid, self._storage_index, [shnum], readv)
3370-            d.addCallback(self._got_answer, peerid, servermap)
3371-            dl.append(d)
3372-        return defer.DeferredList(dl, fireOnOneErrback=True, consumeErrors=True)
3373 
3374hunk ./src/allmydata/mutable/checker.py 98
3375-    def _do_read(self, ss, peerid, storage_index, shnums, readv):
3376-        # isolate the callRemote to a separate method, so tests can subclass
3377-        # Publish and override it
3378-        d = ss.callRemote("slot_readv", storage_index, shnums, readv)
3379+        r = Retrieve(self._node, servermap, self.best_version, verify=True)
3380+        d = r.download()
3381+        d.addCallback(self._process_bad_shares)
3382         return d
3383 
3384hunk ./src/allmydata/mutable/checker.py 103
3385-    def _got_answer(self, datavs, peerid, servermap):
3386-        for shnum,datav in datavs.items():
3387-            data = datav[0]
3388-            try:
3389-                self._got_results_one_share(shnum, peerid, data)
3390-            except CorruptShareError:
3391-                f = failure.Failure()
3392-                self.need_repair = True
3393-                self.bad_shares.append( (peerid, shnum, f) )
3394-                prefix = data[:SIGNED_PREFIX_LENGTH]
3395-                servermap.mark_bad_share(peerid, shnum, prefix)
3396-                ss = servermap.connections[peerid]
3397-                self.notify_server_corruption(ss, shnum, str(f.value))
3398-
3399-    def check_prefix(self, peerid, shnum, data):
3400-        (seqnum, root_hash, IV, segsize, datalength, k, N, prefix,
3401-         offsets_tuple) = self.best_version
3402-        got_prefix = data[:SIGNED_PREFIX_LENGTH]
3403-        if got_prefix != prefix:
3404-            raise CorruptShareError(peerid, shnum,
3405-                                    "prefix mismatch: share changed while we were reading it")
3406-
3407-    def _got_results_one_share(self, shnum, peerid, data):
3408-        self.check_prefix(peerid, shnum, data)
3409-
3410-        # the [seqnum:signature] pieces are validated by _compare_prefix,
3411-        # which checks their signature against the pubkey known to be
3412-        # associated with this file.
3413 
3414hunk ./src/allmydata/mutable/checker.py 104
3415-        (seqnum, root_hash, IV, k, N, segsize, datalen, pubkey, signature,
3416-         share_hash_chain, block_hash_tree, share_data,
3417-         enc_privkey) = unpack_share(data)
3418-
3419-        # validate [share_hash_chain,block_hash_tree,share_data]
3420-
3421-        leaves = [hashutil.block_hash(share_data)]
3422-        t = hashtree.HashTree(leaves)
3423-        if list(t) != block_hash_tree:
3424-            raise CorruptShareError(peerid, shnum, "block hash tree failure")
3425-        share_hash_leaf = t[0]
3426-        t2 = hashtree.IncompleteHashTree(N)
3427-        # root_hash was checked by the signature
3428-        t2.set_hashes({0: root_hash})
3429-        try:
3430-            t2.set_hashes(hashes=share_hash_chain,
3431-                          leaves={shnum: share_hash_leaf})
3432-        except (hashtree.BadHashError, hashtree.NotEnoughHashesError,
3433-                IndexError), e:
3434-            msg = "corrupt hashes: %s" % (e,)
3435-            raise CorruptShareError(peerid, shnum, msg)
3436-
3437-        # validate enc_privkey: only possible if we have a write-cap
3438-        if not self._node.is_readonly():
3439-            alleged_privkey_s = self._node._decrypt_privkey(enc_privkey)
3440-            alleged_writekey = hashutil.ssk_writekey_hash(alleged_privkey_s)
3441-            if alleged_writekey != self._node.get_writekey():
3442-                raise CorruptShareError(peerid, shnum, "invalid privkey")
3443+    def _process_bad_shares(self, bad_shares):
3444+        if bad_shares:
3445+            self.need_repair = True
3446+        self.bad_shares = bad_shares
3447 
3448hunk ./src/allmydata/mutable/checker.py 109
3449-    def notify_server_corruption(self, ss, shnum, reason):
3450-        ss.callRemoteOnly("advise_corrupt_share",
3451-                          "mutable", self._storage_index, shnum, reason)
3452 
3453     def _count_shares(self, smap, version):
3454         available_shares = smap.shares_available()
3455hunk ./src/allmydata/test/test_mutable.py 193
3456                 if offset1 == "pubkey" and IV:
3457                     real_offset = 107
3458                 elif offset1 == "share_data" and not IV:
3459-                    real_offset = 104
3460+                    real_offset = 107
3461                 elif offset1 in o:
3462                     real_offset = o[offset1]
3463                 else:
3464hunk ./src/allmydata/test/test_mutable.py 395
3465             return d
3466         d.addCallback(_created)
3467         return d
3468+    test_create_mdmf_with_initial_contents.timeout = 20
3469 
3470 
3471     def test_create_with_initial_contents_function(self):
3472hunk ./src/allmydata/test/test_mutable.py 700
3473                                            k, N, segsize, datalen)
3474                 self.failUnless(p._pubkey.verify(sig_material, signature))
3475                 #self.failUnlessEqual(signature, p._privkey.sign(sig_material))
3476-                self.failUnless(isinstance(share_hash_chain, dict))
3477-                self.failUnlessEqual(len(share_hash_chain), 4) # ln2(10)++
3478+                self.failUnlessEqual(len(share_hash_chain), 4) # ln2(10)++
3479                 for shnum,share_hash in share_hash_chain.items():
3480                     self.failUnless(isinstance(shnum, int))
3481                     self.failUnless(isinstance(share_hash, str))
3482hunk ./src/allmydata/test/test_mutable.py 820
3483                     shares[peerid][shnum] = oldshares[index][peerid][shnum]
3484 
3485 
3486+
3487+
3488 class Servermap(unittest.TestCase, PublishMixin):
3489     def setUp(self):
3490         return self.publish_one()
3491hunk ./src/allmydata/test/test_mutable.py 951
3492         self._storage._peers = {} # delete all shares
3493         ms = self.make_servermap
3494         d = defer.succeed(None)
3495-
3496+#
3497         d.addCallback(lambda res: ms(mode=MODE_CHECK))
3498         d.addCallback(lambda sm: self.failUnlessNoneRecoverable(sm))
3499 
3500hunk ./src/allmydata/test/test_mutable.py 1440
3501         d.addCallback(self.check_good, "test_check_good")
3502         return d
3503 
3504+    def test_check_mdmf_good(self):
3505+        d = self.publish_mdmf()
3506+        d.addCallback(lambda ignored:
3507+            self._fn.check(Monitor()))
3508+        d.addCallback(self.check_good, "test_check_mdmf_good")
3509+        return d
3510+
3511     def test_check_no_shares(self):
3512         for shares in self._storage._peers.values():
3513             shares.clear()
3514hunk ./src/allmydata/test/test_mutable.py 1454
3515         d.addCallback(self.check_bad, "test_check_no_shares")
3516         return d
3517 
3518+    def test_check_mdmf_no_shares(self):
3519+        d = self.publish_mdmf()
3520+        def _then(ignored):
3521+            for share in self._storage._peers.values():
3522+                share.clear()
3523+        d.addCallback(_then)
3524+        d.addCallback(lambda ignored:
3525+            self._fn.check(Monitor()))
3526+        d.addCallback(self.check_bad, "test_check_mdmf_no_shares")
3527+        return d
3528+
3529     def test_check_not_enough_shares(self):
3530         for shares in self._storage._peers.values():
3531             for shnum in shares.keys():
3532hunk ./src/allmydata/test/test_mutable.py 1474
3533         d.addCallback(self.check_bad, "test_check_not_enough_shares")
3534         return d
3535 
3536+    def test_check_mdmf_not_enough_shares(self):
3537+        d = self.publish_mdmf()
3538+        def _then(ignored):
3539+            for shares in self._storage._peers.values():
3540+                for shnum in shares.keys():
3541+                    if shnum > 0:
3542+                        del shares[shnum]
3543+        d.addCallback(_then)
3544+        d.addCallback(lambda ignored:
3545+            self._fn.check(Monitor()))
3546+        d.addCallback(self.check_bad, "test_check_mdmf_not_enougH_shares")
3547+        return d
3548+
3549+
3550     def test_check_all_bad_sig(self):
3551         d = corrupt(None, self._storage, 1) # bad sig
3552         d.addCallback(lambda ignored:
3553hunk ./src/allmydata/test/test_mutable.py 1495
3554         d.addCallback(self.check_bad, "test_check_all_bad_sig")
3555         return d
3556 
3557+    def test_check_mdmf_all_bad_sig(self):
3558+        d = self.publish_mdmf()
3559+        d.addCallback(lambda ignored:
3560+            corrupt(None, self._storage, 1))
3561+        d.addCallback(lambda ignored:
3562+            self._fn.check(Monitor()))
3563+        d.addCallback(self.check_bad, "test_check_mdmf_all_bad_sig")
3564+        return d
3565+
3566     def test_check_all_bad_blocks(self):
3567         d = corrupt(None, self._storage, "share_data", [9]) # bad blocks
3568         # the Checker won't notice this.. it doesn't look at actual data
3569hunk ./src/allmydata/test/test_mutable.py 1512
3570         d.addCallback(self.check_good, "test_check_all_bad_blocks")
3571         return d
3572 
3573+
3574+    def test_check_mdmf_all_bad_blocks(self):
3575+        d = self.publish_mdmf()
3576+        d.addCallback(lambda ignored:
3577+            corrupt(None, self._storage, "share_data"))
3578+        d.addCallback(lambda ignored:
3579+            self._fn.check(Monitor()))
3580+        d.addCallback(self.check_good, "test_check_mdmf_all_bad_blocks")
3581+        return d
3582+
3583     def test_verify_good(self):
3584         d = self._fn.check(Monitor(), verify=True)
3585         d.addCallback(self.check_good, "test_verify_good")
3586hunk ./src/allmydata/test/test_mutable.py 1582
3587                       "test_verify_one_bad_encprivkey_uncheckable")
3588         return d
3589 
3590+
3591+    def test_verify_mdmf_good(self):
3592+        d = self.publish_mdmf()
3593+        d.addCallback(lambda ignored:
3594+            self._fn.check(Monitor(), verify=True))
3595+        d.addCallback(self.check_good, "test_verify_mdmf_good")
3596+        return d
3597+
3598+
3599+    def test_verify_mdmf_one_bad_block(self):
3600+        d = self.publish_mdmf()
3601+        d.addCallback(lambda ignored:
3602+            corrupt(None, self._storage, "share_data", [1]))
3603+        d.addCallback(lambda ignored:
3604+            self._fn.check(Monitor(), verify=True))
3605+        # We should find one bad block here
3606+        d.addCallback(self.check_bad, "test_verify_mdmf_one_bad_block")
3607+        d.addCallback(self.check_expected_failure,
3608+                      CorruptShareError, "block hash tree failure",
3609+                      "test_verify_mdmf_one_bad_block")
3610+        return d
3611+
3612+
3613+    def test_verify_mdmf_bad_encprivkey(self):
3614+        d = self.publish_mdmf()
3615+        d.addCallback(lambda ignored:
3616+            corrupt(None, self._storage, "enc_privkey", [1]))
3617+        d.addCallback(lambda ignored:
3618+            self._fn.check(Monitor(), verify=True))
3619+        d.addCallback(self.check_bad, "test_verify_mdmf_bad_encprivkey")
3620+        d.addCallback(self.check_expected_failure,
3621+                      CorruptShareError, "privkey",
3622+                      "test_verify_mdmf_bad_encprivkey")
3623+        return d
3624+
3625+
3626+    def test_verify_mdmf_bad_sig(self):
3627+        d = self.publish_mdmf()
3628+        d.addCallback(lambda ignored:
3629+            corrupt(None, self._storage, 1, [1]))
3630+        d.addCallback(lambda ignored:
3631+            self._fn.check(Monitor(), verify=True))
3632+        d.addCallback(self.check_bad, "test_verify_mdmf_bad_sig")
3633+        return d
3634+
3635+
3636+    def test_verify_mdmf_bad_encprivkey_uncheckable(self):
3637+        d = self.publish_mdmf()
3638+        d.addCallback(lambda ignored:
3639+            corrupt(None, self._storage, "enc_privkey", [1]))
3640+        d.addCallback(lambda ignored:
3641+            self._fn.get_readonly())
3642+        d.addCallback(lambda fn:
3643+            fn.check(Monitor(), verify=True))
3644+        d.addCallback(self.check_good,
3645+                      "test_verify_mdmf_bad_encprivkey_uncheckable")
3646+        return d
3647+
3648+
3649 class Repair(unittest.TestCase, PublishMixin, ShouldFailMixin):
3650 
3651     def get_shares(self, s):
3652hunk ./src/allmydata/test/test_mutable.py 1706
3653         current_shares = self.old_shares[-1]
3654         self.failUnlessEqual(old_shares, current_shares)
3655 
3656+
3657     def test_unrepairable_0shares(self):
3658         d = self.publish_one()
3659         def _delete_all_shares(ign):
3660hunk ./src/allmydata/test/test_mutable.py 1721
3661         d.addCallback(_check)
3662         return d
3663 
3664+    def test_mdmf_unrepairable_0shares(self):
3665+        d = self.publish_mdmf()
3666+        def _delete_all_shares(ign):
3667+            shares = self._storage._peers
3668+            for peerid in shares:
3669+                shares[peerid] = {}
3670+        d.addCallback(_delete_all_shares)
3671+        d.addCallback(lambda ign: self._fn.check(Monitor()))
3672+        d.addCallback(lambda check_results: self._fn.repair(check_results))
3673+        d.addCallback(lambda crr: self.failIf(crr.get_successful()))
3674+        return d
3675+
3676+
3677     def test_unrepairable_1share(self):
3678         d = self.publish_one()
3679         def _delete_all_shares(ign):
3680hunk ./src/allmydata/test/test_mutable.py 1750
3681         d.addCallback(_check)
3682         return d
3683 
3684+    def test_mdmf_unrepairable_1share(self):
3685+        d = self.publish_mdmf()
3686+        def _delete_all_shares(ign):
3687+            shares = self._storage._peers
3688+            for peerid in shares:
3689+                for shnum in list(shares[peerid]):
3690+                    if shnum > 0:
3691+                        del shares[peerid][shnum]
3692+        d.addCallback(_delete_all_shares)
3693+        d.addCallback(lambda ign: self._fn.check(Monitor()))
3694+        d.addCallback(lambda check_results: self._fn.repair(check_results))
3695+        def _check(crr):
3696+            self.failUnlessEqual(crr.get_successful(), False)
3697+        d.addCallback(_check)
3698+        return d
3699+
3700+    def test_repairable_5shares(self):
3701+        d = self.publish_mdmf()
3702+        def _delete_all_shares(ign):
3703+            shares = self._storage._peers
3704+            for peerid in shares:
3705+                for shnum in list(shares[peerid]):
3706+                    if shnum > 4:
3707+                        del shares[peerid][shnum]
3708+        d.addCallback(_delete_all_shares)
3709+        d.addCallback(lambda ign: self._fn.check(Monitor()))
3710+        d.addCallback(lambda check_results: self._fn.repair(check_results))
3711+        def _check(crr):
3712+            self.failUnlessEqual(crr.get_successful(), True)
3713+        d.addCallback(_check)
3714+        return d
3715+
3716+    def test_mdmf_repairable_5shares(self):
3717+        d = self.publish_mdmf()
3718+        def _delete_all_shares(ign):
3719+            shares = self._storage._peers
3720+            for peerid in shares:
3721+                for shnum in list(shares[peerid]):
3722+                    if shnum > 5:
3723+                        del shares[peerid][shnum]
3724+        d.addCallback(_delete_all_shares)
3725+        d.addCallback(lambda ign: self._fn.check(Monitor()))
3726+        d.addCallback(lambda check_results: self._fn.repair(check_results))
3727+        def _check(crr):
3728+            self.failUnlessEqual(crr.get_successful(), True)
3729+        d.addCallback(_check)
3730+        return d
3731+
3732+
3733     def test_merge(self):
3734         self.old_shares = []
3735         d = self.publish_multiple()
3736}
3737[mutable/retrieve.py: learn how to verify mutable files
3738Kevan Carstensen <kevan@isnotajoke.com>**20100628225201
3739 Ignore-this: 989af7800c47589620918461ec989483
3740] {
3741hunk ./src/allmydata/mutable/retrieve.py 86
3742     # Retrieve object will remain tied to a specific version of the file, and
3743     # will use a single ServerMap instance.
3744 
3745-    def __init__(self, filenode, servermap, verinfo, fetch_privkey=False):
3746+    def __init__(self, filenode, servermap, verinfo, fetch_privkey=False,
3747+                 verify=False):
3748         self._node = filenode
3749         assert self._node.get_pubkey()
3750         self._storage_index = filenode.get_storage_index()
3751hunk ./src/allmydata/mutable/retrieve.py 106
3752         # during repair, we may be called upon to grab the private key, since
3753         # it wasn't picked up during a verify=False checker run, and we'll
3754         # need it for repair to generate a new version.
3755-        self._need_privkey = fetch_privkey
3756-        if self._node.get_privkey():
3757+        self._need_privkey = fetch_privkey or verify
3758+        if self._node.get_privkey() and not verify:
3759             self._need_privkey = False
3760 
3761         if self._need_privkey:
3762hunk ./src/allmydata/mutable/retrieve.py 117
3763             self._privkey_query_markers = [] # one Marker for each time we've
3764                                              # tried to get the privkey.
3765 
3766+        # verify means that we are using the downloader logic to verify all
3767+        # of our shares. This tells the downloader a few things.
3768+        #
3769+        # 1. We need to download all of the shares.
3770+        # 2. We don't need to decode or decrypt the shares, since our
3771+        #    caller doesn't care about the plaintext, only the
3772+        #    information about which shares are or are not valid.
3773+        # 3. When we are validating readers, we need to validate the
3774+        #    signature on the prefix. Do we? We already do this in the
3775+        #    servermap update?
3776+        #
3777+        # (just work on 1 and 2 for now, I guess)
3778+        self._verify = False
3779+        if verify:
3780+            self._verify = True
3781+
3782         self._status = RetrieveStatus()
3783         self._status.set_storage_index(self._storage_index)
3784         self._status.set_helper(False)
3785hunk ./src/allmydata/mutable/retrieve.py 323
3786 
3787         # We need at least self._required_shares readers to download a
3788         # segment.
3789-        needed = self._required_shares - len(self._active_readers)
3790+        if self._verify:
3791+            needed = self._total_shares
3792+        else:
3793+            needed = self._required_shares - len(self._active_readers)
3794         # XXX: Why don't format= log messages work here?
3795         self.log("adding %d peers to the active peers list" % needed)
3796 
3797hunk ./src/allmydata/mutable/retrieve.py 339
3798         # will cause problems later.
3799         active_shnums -= set([reader.shnum for reader in self._active_readers])
3800         active_shnums = list(active_shnums)[:needed]
3801-        if len(active_shnums) < needed:
3802+        if len(active_shnums) < needed and not self._verify:
3803             # We don't have enough readers to retrieve the file; fail.
3804             return self._failed()
3805 
3806hunk ./src/allmydata/mutable/retrieve.py 346
3807         for shnum in active_shnums:
3808             self._active_readers.append(self.readers[shnum])
3809             self.log("added reader for share %d" % shnum)
3810-        assert len(self._active_readers) == self._required_shares
3811+        assert len(self._active_readers) >= self._required_shares
3812         # Conceptually, this is part of the _add_active_peers step. It
3813         # validates the prefixes of newly added readers to make sure
3814         # that they match what we are expecting for self.verinfo. If
3815hunk ./src/allmydata/mutable/retrieve.py 416
3816                     # that we haven't gotten it at the end of
3817                     # segment decoding, then we'll take more drastic
3818                     # measures.
3819-                    if self._need_privkey:
3820+                    if self._need_privkey and not self._node.is_readonly():
3821                         d = reader.get_encprivkey()
3822                         d.addCallback(self._try_to_validate_privkey, reader)
3823             if bad_readers:
3824hunk ./src/allmydata/mutable/retrieve.py 423
3825                 # We do them all at once, or else we screw up list indexing.
3826                 for (reader, f) in bad_readers:
3827                     self._mark_bad_share(reader, f)
3828-                return self._add_active_peers()
3829+                if self._verify:
3830+                    if len(self._active_readers) >= self._required_shares:
3831+                        return self._download_current_segment()
3832+                    else:
3833+                        return self._failed()
3834+                else:
3835+                    return self._add_active_peers()
3836             else:
3837                 return self._download_current_segment()
3838             # The next step will assert that it has enough active
3839hunk ./src/allmydata/mutable/retrieve.py 518
3840         """
3841         self.log("marking share %d on server %s as bad" % \
3842                  (reader.shnum, reader))
3843+        prefix = self.verinfo[-2]
3844+        self.servermap.mark_bad_share(reader.peerid,
3845+                                      reader.shnum,
3846+                                      prefix)
3847         self._remove_reader(reader)
3848hunk ./src/allmydata/mutable/retrieve.py 523
3849-        self._bad_shares.add((reader.peerid, reader.shnum))
3850+        self._bad_shares.add((reader.peerid, reader.shnum, f))
3851         self._status.problems[reader.peerid] = f
3852         self._last_failure = f
3853         self.notify_server_corruption(reader.peerid, reader.shnum,
3854hunk ./src/allmydata/mutable/retrieve.py 571
3855             ds.append(dl)
3856             reader.flush()
3857         dl = defer.DeferredList(ds)
3858-        dl.addCallback(self._maybe_decode_and_decrypt_segment, segnum)
3859+        if self._verify:
3860+            dl.addCallback(lambda ignored: "")
3861+            dl.addCallback(self._set_segment)
3862+        else:
3863+            dl.addCallback(self._maybe_decode_and_decrypt_segment, segnum)
3864         return dl
3865 
3866 
3867hunk ./src/allmydata/mutable/retrieve.py 701
3868         # shnum, which will be a leaf in the share hash tree, which
3869         # will allow us to validate the rest of the tree.
3870         if self.share_hash_tree.needed_hashes(reader.shnum,
3871-                                               include_leaf=True):
3872+                                              include_leaf=True) or \
3873+                                              self._verify:
3874             try:
3875                 self.share_hash_tree.set_hashes(hashes=sharehashes[1],
3876                                             leaves={reader.shnum: bht[0]})
3877hunk ./src/allmydata/mutable/retrieve.py 832
3878 
3879 
3880     def _try_to_validate_privkey(self, enc_privkey, reader):
3881-
3882         alleged_privkey_s = self._node._decrypt_privkey(enc_privkey)
3883         alleged_writekey = hashutil.ssk_writekey_hash(alleged_privkey_s)
3884         if alleged_writekey != self._node.get_writekey():
3885hunk ./src/allmydata/mutable/retrieve.py 838
3886             self.log("invalid privkey from %s shnum %d" %
3887                      (reader, reader.shnum),
3888                      level=log.WEIRD, umid="YIw4tA")
3889+            if self._verify:
3890+                self.servermap.mark_bad_share(reader.peerid, reader.shnum,
3891+                                              self.verinfo[-2])
3892+                e = CorruptShareError(reader.peerid,
3893+                                      reader.shnum,
3894+                                      "invalid privkey")
3895+                f = failure.Failure(e)
3896+                self._bad_shares.add((reader.peerid, reader.shnum, f))
3897             return
3898 
3899         # it's good
3900hunk ./src/allmydata/mutable/retrieve.py 904
3901         statements, I return the decrypted contents to the owner of this
3902         Retrieve object through self._done_deferred.
3903         """
3904-        eventually(self._done_deferred.callback, self._plaintext)
3905+        if self._verify:
3906+            ret = list(self._bad_shares)
3907+            self.log("done verifying, found %d bad shares" % len(ret))
3908+        else:
3909+            ret = self._plaintext
3910+        eventually(self._done_deferred.callback, ret)
3911 
3912 
3913     def _failed(self):
3914hunk ./src/allmydata/mutable/retrieve.py 920
3915         to the caller of this Retrieve object through
3916         self._done_deferred.
3917         """
3918-        format = ("ran out of peers: "
3919-                  "have %(have)d of %(total)d segments "
3920-                  "found %(bad)d bad shares "
3921-                  "encoding %(k)d-of-%(n)d")
3922-        args = {"have": self._current_segment,
3923-                "total": self._num_segments,
3924-                "k": self._required_shares,
3925-                "n": self._total_shares,
3926-                "bad": len(self._bad_shares)}
3927-        e = NotEnoughSharesError("%s, last failure: %s" % (format % args,
3928-                                                        str(self._last_failure)))
3929-        f = failure.Failure(e)
3930-        eventually(self._done_deferred.callback, f)
3931+        if self._verify:
3932+            ret = list(self._bad_shares)
3933+        else:
3934+            format = ("ran out of peers: "
3935+                      "have %(have)d of %(total)d segments "
3936+                      "found %(bad)d bad shares "
3937+                      "encoding %(k)d-of-%(n)d")
3938+            args = {"have": self._current_segment,
3939+                    "total": self._num_segments,
3940+                    "k": self._required_shares,
3941+                    "n": self._total_shares,
3942+                    "bad": len(self._bad_shares)}
3943+            e = NotEnoughSharesError("%s, last failure: %s" % \
3944+                                     (format % args, str(self._last_failure)))
3945+            f = failure.Failure(e)
3946+            ret = f
3947+        eventually(self._done_deferred.callback, ret)
3948}
3949[interfaces.py: add IMutableSlotWriter
3950Kevan Carstensen <kevan@isnotajoke.com>**20100630183305
3951 Ignore-this: ff9dca96ef1a009ae85485682f81ea5
3952] hunk ./src/allmydata/interfaces.py 418
3953         """
3954 
3955 
3956+class IMutableSlotWriter(Interface):
3957+    """
3958+    The interface for a writer around a mutable slot on a remote server.
3959+    """
3960+    def set_checkstring(checkstring, *args):
3961+        """
3962+        Set the checkstring that I will pass to the remote server when
3963+        writing.
3964+
3965+            @param checkstring A packed checkstring to use.
3966+
3967+        Note that implementations can differ in which semantics they
3968+        wish to support for set_checkstring -- they can, for example,
3969+        build the checkstring themselves from its constituents, or
3970+        some other thing.
3971+        """
3972+
3973+    def get_checkstring():
3974+        """
3975+        Get the checkstring that I think currently exists on the remote
3976+        server.
3977+        """
3978+
3979+    def put_block(data, segnum, salt):
3980+        """
3981+        Add a block and salt to the share.
3982+        """
3983+
3984+    def put_encprivey(encprivkey):
3985+        """
3986+        Add the encrypted private key to the share.
3987+        """
3988+
3989+    def put_blockhashes(blockhashes=list):
3990+        """
3991+        Add the block hash tree to the share.
3992+        """
3993+
3994+    def put_sharehashes(sharehashes=dict):
3995+        """
3996+        Add the share hash chain to the share.
3997+        """
3998+
3999+    def get_signable():
4000+        """
4001+        Return the part of the share that needs to be signed.
4002+        """
4003+
4004+    def put_signature(signature):
4005+        """
4006+        Add the signature to the share.
4007+        """
4008+
4009+    def put_verification_key(verification_key):
4010+        """
4011+        Add the verification key to the share.
4012+        """
4013+
4014+    def finish_publishing():
4015+        """
4016+        Do anything necessary to finish writing the share to a remote
4017+        server. I require that no further publishing needs to take place
4018+        after this method has been called.
4019+        """
4020+
4021+
4022 class IURI(Interface):
4023     def init_from_string(uri):
4024         """Accept a string (as created by my to_string() method) and populate
4025[test/test_mutable.py: temporarily disable two tests that are now irrelevant
4026Kevan Carstensen <kevan@isnotajoke.com>**20100701232806
4027 Ignore-this: 701e143567f3954812ca6960af1d6ac7
4028] {
4029hunk ./src/allmydata/test/test_mutable.py 651
4030             self.failUnlessEqual(len(share_ids), 10)
4031         d.addCallback(_done)
4032         return d
4033+    test_encrypt.todo = "Write an equivalent of this for the new uploader"
4034 
4035     def test_generate(self):
4036         nm = make_nodemaker()
4037hunk ./src/allmydata/test/test_mutable.py 713
4038                 self.failUnlessEqual(enc_privkey, self._fn.get_encprivkey())
4039         d.addCallback(_generated)
4040         return d
4041+    test_generate.todo = "Write an equivalent of this for the new uploader"
4042 
4043     # TODO: when we publish to 20 peers, we should get one share per peer on 10
4044     # when we publish to 3 peers, we should get either 3 or 4 shares per peer
4045}
4046[Add MDMF reader and writer, and SDMF writer
4047Kevan Carstensen <kevan@isnotajoke.com>**20100702225531
4048 Ignore-this: bf6276a91d27dcb4e779b0eb82ea1843
4049 
4050 The MDMF/SDMF reader MDMF writer, and SDMF writer are similar to the
4051 object proxies that exist for immutable files. They abstract away
4052 details of connection, state, and caching from their callers (in this
4053 case, the download, servermap updater, and uploader), and expose methods
4054 to get and set information on the remote server.
4055 
4056 MDMFSlotReadProxy reads a mutable file from the server, doing the right
4057 thing (in most cases) regardless of whether the file is MDMF or SDMF. It
4058 allows callers to tell it how to batch and flush reads.
4059 
4060 MDMFSlotWriteProxy writes an MDMF mutable file to a server.
4061 
4062 SDMFSlotWriteProxy writes an SDMF mutable file to a server.
4063 
4064 This patch also includes tests for MDMFSlotReadProxy,
4065 SDMFSlotWriteProxy, and MDMFSlotWriteProxy.
4066] {
4067hunk ./src/allmydata/mutable/layout.py 4
4068 
4069 import struct
4070 from allmydata.mutable.common import NeedMoreDataError, UnknownVersionError
4071+from allmydata.interfaces import HASH_SIZE, SALT_SIZE, SDMF_VERSION, \
4072+                                 MDMF_VERSION, IMutableSlotWriter
4073+from allmydata.util import mathutil, observer
4074+from twisted.python import failure
4075+from twisted.internet import defer
4076+from zope.interface import implements
4077+
4078+
4079+# These strings describe the format of the packed structs they help process
4080+# Here's what they mean:
4081+#
4082+#  PREFIX:
4083+#    >: Big-endian byte order; the most significant byte is first (leftmost).
4084+#    B: The version information; an 8 bit version identifier. Stored as
4085+#       an unsigned char. This is currently 00 00 00 00; our modifications
4086+#       will turn it into 00 00 00 01.
4087+#    Q: The sequence number; this is sort of like a revision history for
4088+#       mutable files; they start at 1 and increase as they are changed after
4089+#       being uploaded. Stored as an unsigned long long, which is 8 bytes in
4090+#       length.
4091+#  32s: The root hash of the share hash tree. We use sha-256d, so we use 32
4092+#       characters = 32 bytes to store the value.
4093+#  16s: The salt for the readkey. This is a 16-byte random value, stored as
4094+#       16 characters.
4095+#
4096+#  SIGNED_PREFIX additions, things that are covered by the signature:
4097+#    B: The "k" encoding parameter. We store this as an 8-bit character,
4098+#       which is convenient because our erasure coding scheme cannot
4099+#       encode if you ask for more than 255 pieces.
4100+#    B: The "N" encoding parameter. Stored as an 8-bit character for the
4101+#       same reasons as above.
4102+#    Q: The segment size of the uploaded file. This will essentially be the
4103+#       length of the file in SDMF. An unsigned long long, so we can store
4104+#       files of quite large size.
4105+#    Q: The data length of the uploaded file. Modulo padding, this will be
4106+#       the same of the data length field. Like the data length field, it is
4107+#       an unsigned long long and can be quite large.
4108+#
4109+#   HEADER additions:
4110+#     L: The offset of the signature of this. An unsigned long.
4111+#     L: The offset of the share hash chain. An unsigned long.
4112+#     L: The offset of the block hash tree. An unsigned long.
4113+#     L: The offset of the share data. An unsigned long.
4114+#     Q: The offset of the encrypted private key. An unsigned long long, to
4115+#        account for the possibility of a lot of share data.
4116+#     Q: The offset of the EOF. An unsigned long long, to account for the
4117+#        possibility of a lot of share data.
4118+#
4119+#  After all of these, we have the following:
4120+#    - The verification key: Occupies the space between the end of the header
4121+#      and the start of the signature (i.e.: data[HEADER_LENGTH:o['signature']].
4122+#    - The signature, which goes from the signature offset to the share hash
4123+#      chain offset.
4124+#    - The share hash chain, which goes from the share hash chain offset to
4125+#      the block hash tree offset.
4126+#    - The share data, which goes from the share data offset to the encrypted
4127+#      private key offset.
4128+#    - The encrypted private key offset, which goes until the end of the file.
4129+#
4130+#  The block hash tree in this encoding has only one share, so the offset of
4131+#  the share data will be 32 bits more than the offset of the block hash tree.
4132+#  Given this, we may need to check to see how many bytes a reasonably sized
4133+#  block hash tree will take up.
4134 
4135 PREFIX = ">BQ32s16s" # each version has a different prefix
4136 SIGNED_PREFIX = ">BQ32s16s BBQQ" # this is covered by the signature
4137hunk ./src/allmydata/mutable/layout.py 73
4138 SIGNED_PREFIX_LENGTH = struct.calcsize(SIGNED_PREFIX)
4139 HEADER = ">BQ32s16s BBQQ LLLLQQ" # includes offsets
4140 HEADER_LENGTH = struct.calcsize(HEADER)
4141+OFFSETS = ">LLLLQQ"
4142+OFFSETS_LENGTH = struct.calcsize(OFFSETS)
4143 
4144 def unpack_header(data):
4145     o = {}
4146hunk ./src/allmydata/mutable/layout.py 194
4147     return (share_hash_chain, block_hash_tree, share_data)
4148 
4149 
4150-def pack_checkstring(seqnum, root_hash, IV):
4151+def pack_checkstring(seqnum, root_hash, IV, version=0):
4152     return struct.pack(PREFIX,
4153hunk ./src/allmydata/mutable/layout.py 196
4154-                       0, # version,
4155+                       version,
4156                        seqnum,
4157                        root_hash,
4158                        IV)
4159hunk ./src/allmydata/mutable/layout.py 269
4160                            encprivkey])
4161     return final_share
4162 
4163+def pack_prefix(seqnum, root_hash, IV,
4164+                required_shares, total_shares,
4165+                segment_size, data_length):
4166+    prefix = struct.pack(SIGNED_PREFIX,
4167+                         0, # version,
4168+                         seqnum,
4169+                         root_hash,
4170+                         IV,
4171+                         required_shares,
4172+                         total_shares,
4173+                         segment_size,
4174+                         data_length,
4175+                         )
4176+    return prefix
4177+
4178+
4179+class SDMFSlotWriteProxy:
4180+    implements(IMutableSlotWriter)
4181+    """
4182+    I represent a remote write slot for an SDMF mutable file. I build a
4183+    share in memory, and then write it in one piece to the remote
4184+    server. This mimics how SDMF shares were built before MDMF (and the
4185+    new MDMF uploader), but provides that functionality in a way that
4186+    allows the MDMF uploader to be built without much special-casing for
4187+    file format, which makes the uploader code more readable.
4188+    """
4189+    def __init__(self,
4190+                 shnum,
4191+                 rref, # a remote reference to a storage server
4192+                 storage_index,
4193+                 secrets, # (write_enabler, renew_secret, cancel_secret)
4194+                 seqnum, # the sequence number of the mutable file
4195+                 required_shares,
4196+                 total_shares,
4197+                 segment_size,
4198+                 data_length): # the length of the original file
4199+        self.shnum = shnum
4200+        self._rref = rref
4201+        self._storage_index = storage_index
4202+        self._secrets = secrets
4203+        self._seqnum = seqnum
4204+        self._required_shares = required_shares
4205+        self._total_shares = total_shares
4206+        self._segment_size = segment_size
4207+        self._data_length = data_length
4208+
4209+        # This is an SDMF file, so it should have only one segment, so,
4210+        # modulo padding of the data length, the segment size and the
4211+        # data length should be the same.
4212+        expected_segment_size = mathutil.next_multiple(data_length,
4213+                                                       self._required_shares)
4214+        assert expected_segment_size == segment_size
4215+
4216+        self._block_size = self._segment_size / self._required_shares
4217+
4218+        # This is meant to mimic how SDMF files were built before MDMF
4219+        # entered the picture: we generate each share in its entirety,
4220+        # then push it off to the storage server in one write. When
4221+        # callers call set_*, they are just populating this dict.
4222+        # finish_publishing will stitch these pieces together into a
4223+        # coherent share, and then write the coherent share to the
4224+        # storage server.
4225+        self._share_pieces = {}
4226+
4227+        # This tells the write logic what checkstring to use when
4228+        # writing remote shares.
4229+        self._testvs = []
4230+
4231+        self._readvs = [(0, struct.calcsize(PREFIX))]
4232+
4233+
4234+    def set_checkstring(self, checkstring_or_seqnum,
4235+                              root_hash=None,
4236+                              salt=None):
4237+        """
4238+        Set the checkstring that I will pass to the remote server when
4239+        writing.
4240+
4241+            @param checkstring_or_seqnum: A packed checkstring to use,
4242+                   or a sequence number. I will treat this as a checkstr
4243+
4244+        Note that implementations can differ in which semantics they
4245+        wish to support for set_checkstring -- they can, for example,
4246+        build the checkstring themselves from its constituents, or
4247+        some other thing.
4248+        """
4249+        if root_hash and salt:
4250+            checkstring = struct.pack(PREFIX,
4251+                                      0,
4252+                                      checkstring_or_seqnum,
4253+                                      root_hash,
4254+                                      salt)
4255+        else:
4256+            checkstring = checkstring_or_seqnum
4257+        self._testvs = [(0, len(checkstring), "eq", checkstring)]
4258+
4259+
4260+    def get_checkstring(self):
4261+        """
4262+        Get the checkstring that I think currently exists on the remote
4263+        server.
4264+        """
4265+        if self._testvs:
4266+            return self._testvs[0][3]
4267+        return ""
4268+
4269+
4270+    def put_block(self, data, segnum, salt):
4271+        """
4272+        Add a block and salt to the share.
4273+        """
4274+        # SDMF files have only one segment
4275+        assert segnum == 0
4276+        assert len(data) == self._block_size
4277+        assert len(salt) == SALT_SIZE
4278+
4279+        self._share_pieces['sharedata'] = data
4280+        self._share_pieces['salt'] = salt
4281+
4282+        # TODO: Figure out something intelligent to return.
4283+        return defer.succeed(None)
4284+
4285+
4286+    def put_encprivkey(self, encprivkey):
4287+        """
4288+        Add the encrypted private key to the share.
4289+        """
4290+        self._share_pieces['encprivkey'] = encprivkey
4291+
4292+        return defer.succeed(None)
4293+
4294+
4295+    def put_blockhashes(self, blockhashes):
4296+        """
4297+        Add the block hash tree to the share.
4298+        """
4299+        assert isinstance(blockhashes, list)
4300+        for h in blockhashes:
4301+            assert len(h) == HASH_SIZE
4302+
4303+        # serialize the blockhashes, then set them.
4304+        blockhashes_s = "".join(blockhashes)
4305+        self._share_pieces['block_hash_tree'] = blockhashes_s
4306+
4307+        return defer.succeed(None)
4308+
4309+
4310+    def put_sharehashes(self, sharehashes):
4311+        """
4312+        Add the share hash chain to the share.
4313+        """
4314+        assert isinstance(sharehashes, dict)
4315+        for h in sharehashes.itervalues():
4316+            assert len(h) == HASH_SIZE
4317+
4318+        # serialize the sharehashes, then set them.
4319+        sharehashes_s = "".join([struct.pack(">H32s", i, sharehashes[i])
4320+                                 for i in sorted(sharehashes.keys())])
4321+        self._share_pieces['share_hash_chain'] = sharehashes_s
4322+
4323+        return defer.succeed(None)
4324+
4325+
4326+    def put_root_hash(self, root_hash):
4327+        """
4328+        Add the root hash to the share.
4329+        """
4330+        assert len(root_hash) == HASH_SIZE
4331+
4332+        self._share_pieces['root_hash'] = root_hash
4333+
4334+        return defer.succeed(None)
4335+
4336+
4337+    def put_salt(self, salt):
4338+        """
4339+        Add a salt to an empty SDMF file.
4340+        """
4341+        assert len(salt) == SALT_SIZE
4342+
4343+        self._share_pieces['salt'] = salt
4344+        self._share_pieces['sharedata'] = ""
4345+
4346+
4347+    def get_signable(self):
4348+        """
4349+        Return the part of the share that needs to be signed.
4350+
4351+        SDMF writers need to sign the packed representation of the
4352+        first eight fields of the remote share, that is:
4353+            - version number (0)
4354+            - sequence number
4355+            - root of the share hash tree
4356+            - salt
4357+            - k
4358+            - n
4359+            - segsize
4360+            - datalen
4361+
4362+        This method is responsible for returning that to callers.
4363+        """
4364+        return struct.pack(SIGNED_PREFIX,
4365+                           0,
4366+                           self._seqnum,
4367+                           self._share_pieces['root_hash'],
4368+                           self._share_pieces['salt'],
4369+                           self._required_shares,
4370+                           self._total_shares,
4371+                           self._segment_size,
4372+                           self._data_length)
4373+
4374+
4375+    def put_signature(self, signature):
4376+        """
4377+        Add the signature to the share.
4378+        """
4379+        self._share_pieces['signature'] = signature
4380+
4381+        return defer.succeed(None)
4382+
4383+
4384+    def put_verification_key(self, verification_key):
4385+        """
4386+        Add the verification key to the share.
4387+        """
4388+        self._share_pieces['verification_key'] = verification_key
4389+
4390+        return defer.succeed(None)
4391+
4392+
4393+    def get_verinfo(self):
4394+        """
4395+        I return my verinfo tuple. This is used by the ServermapUpdater
4396+        to keep track of versions of mutable files.
4397+
4398+        The verinfo tuple for MDMF files contains:
4399+            - seqnum
4400+            - root hash
4401+            - a blank (nothing)
4402+            - segsize
4403+            - datalen
4404+            - k
4405+            - n
4406+            - prefix (the thing that you sign)
4407+            - a tuple of offsets
4408+
4409+        We include the nonce in MDMF to simplify processing of version
4410+        information tuples.
4411+
4412+        The verinfo tuple for SDMF files is the same, but contains a
4413+        16-byte IV instead of a hash of salts.
4414+        """
4415+        return (self._seqnum,
4416+                self._share_pieces['root_hash'],
4417+                self._share_pieces['salt'],
4418+                self._segment_size,
4419+                self._data_length,
4420+                self._required_shares,
4421+                self._total_shares,
4422+                self.get_signable(),
4423+                self._get_offsets_tuple())
4424+
4425+    def _get_offsets_dict(self):
4426+        post_offset = HEADER_LENGTH
4427+        offsets = {}
4428+
4429+        verification_key_length = len(self._share_pieces['verification_key'])
4430+        o1 = offsets['signature'] = post_offset + verification_key_length
4431+
4432+        signature_length = len(self._share_pieces['signature'])
4433+        o2 = offsets['share_hash_chain'] = o1 + signature_length
4434+
4435+        share_hash_chain_length = len(self._share_pieces['share_hash_chain'])
4436+        o3 = offsets['block_hash_tree'] = o2 + share_hash_chain_length
4437+
4438+        block_hash_tree_length = len(self._share_pieces['block_hash_tree'])
4439+        o4 = offsets['share_data'] = o3 + block_hash_tree_length
4440+
4441+        share_data_length = len(self._share_pieces['sharedata'])
4442+        o5 = offsets['enc_privkey'] = o4 + share_data_length
4443+
4444+        encprivkey_length = len(self._share_pieces['encprivkey'])
4445+        offsets['EOF'] = o5 + encprivkey_length
4446+        return offsets
4447+
4448+
4449+    def _get_offsets_tuple(self):
4450+        offsets = self._get_offsets_dict()
4451+        return tuple([(key, value) for key, value in offsets.items()])
4452+
4453+
4454+    def _pack_offsets(self):
4455+        offsets = self._get_offsets_dict()
4456+        return struct.pack(">LLLLQQ",
4457+                           offsets['signature'],
4458+                           offsets['share_hash_chain'],
4459+                           offsets['block_hash_tree'],
4460+                           offsets['share_data'],
4461+                           offsets['enc_privkey'],
4462+                           offsets['EOF'])
4463+
4464+
4465+    def finish_publishing(self):
4466+        """
4467+        Do anything necessary to finish writing the share to a remote
4468+        server. I require that no further publishing needs to take place
4469+        after this method has been called.
4470+        """
4471+        for k in ["sharedata", "encprivkey", "signature", "verification_key",
4472+                  "share_hash_chain", "block_hash_tree"]:
4473+            assert k in self._share_pieces
4474+        # This is the only method that actually writes something to the
4475+        # remote server.
4476+        # First, we need to pack the share into data that we can write
4477+        # to the remote server in one write.
4478+        offsets = self._pack_offsets()
4479+        prefix = self.get_signable()
4480+        final_share = "".join([prefix,
4481+                               offsets,
4482+                               self._share_pieces['verification_key'],
4483+                               self._share_pieces['signature'],
4484+                               self._share_pieces['share_hash_chain'],
4485+                               self._share_pieces['block_hash_tree'],
4486+                               self._share_pieces['sharedata'],
4487+                               self._share_pieces['encprivkey']])
4488+
4489+        # Our only data vector is going to be writing the final share,
4490+        # in its entirely.
4491+        datavs = [(0, final_share)]
4492+
4493+        if not self._testvs:
4494+            # Our caller has not provided us with another checkstring
4495+            # yet, so we assume that we are writing a new share, and set
4496+            # a test vector that will allow a new share to be written.
4497+            self._testvs = []
4498+            self._testvs.append(tuple([0, 1, "eq", ""]))
4499+            new_share = True
4500+
4501+        tw_vectors = {}
4502+        tw_vectors[self.shnum] = (self._testvs, datavs, None)
4503+        return self._rref.callRemote("slot_testv_and_readv_and_writev",
4504+                                     self._storage_index,
4505+                                     self._secrets,
4506+                                     tw_vectors,
4507+                                     # TODO is it useful to read something?
4508+                                     self._readvs)
4509+
4510+
4511+MDMFHEADER = ">BQ32sBBQQ QQQQQQ"
4512+MDMFHEADERWITHOUTOFFSETS = ">BQ32sBBQQ"
4513+MDMFHEADERSIZE = struct.calcsize(MDMFHEADER)
4514+MDMFHEADERWITHOUTOFFSETSSIZE = struct.calcsize(MDMFHEADERWITHOUTOFFSETS)
4515+MDMFCHECKSTRING = ">BQ32s"
4516+MDMFSIGNABLEHEADER = ">BQ32sBBQQ"
4517+MDMFOFFSETS = ">QQQQQQ"
4518+MDMFOFFSETS_LENGTH = struct.calcsize(MDMFOFFSETS)
4519+
4520+class MDMFSlotWriteProxy:
4521+    implements(IMutableSlotWriter)
4522+
4523+    """
4524+    I represent a remote write slot for an MDMF mutable file.
4525+
4526+    I abstract away from my caller the details of block and salt
4527+    management, and the implementation of the on-disk format for MDMF
4528+    shares.
4529+    """
4530+    # Expected layout, MDMF:
4531+    # offset:     size:       name:
4532+    #-- signed part --
4533+    # 0           1           version number (01)
4534+    # 1           8           sequence number
4535+    # 9           32          share tree root hash
4536+    # 41          1           The "k" encoding parameter
4537+    # 42          1           The "N" encoding parameter
4538+    # 43          8           The segment size of the uploaded file
4539+    # 51          8           The data length of the original plaintext
4540+    #-- end signed part --
4541+    # 59          8           The offset of the encrypted private key
4542+    # 67          8           The offset of the block hash tree
4543+    # 75          8           The offset of the share hash chain
4544+    # 83          8           The offset of the signature
4545+    # 91          8           The offset of the verification key
4546+    # 99          8           The offset of the EOF
4547+    #
4548+    # followed by salts and share data, the encrypted private key, the
4549+    # block hash tree, the salt hash tree, the share hash chain, a
4550+    # signature over the first eight fields, and a verification key.
4551+    #
4552+    # The checkstring is the first three fields -- the version number,
4553+    # sequence number, root hash and root salt hash. This is consistent
4554+    # in meaning to what we have with SDMF files, except now instead of
4555+    # using the literal salt, we use a value derived from all of the
4556+    # salts -- the share hash root.
4557+    #
4558+    # The salt is stored before the block for each segment. The block
4559+    # hash tree is computed over the combination of block and salt for
4560+    # each segment. In this way, we get integrity checking for both
4561+    # block and salt with the current block hash tree arrangement.
4562+    #
4563+    # The ordering of the offsets is different to reflect the dependencies
4564+    # that we'll run into with an MDMF file. The expected write flow is
4565+    # something like this:
4566+    #
4567+    #   0: Initialize with the sequence number, encoding parameters and
4568+    #      data length. From this, we can deduce the number of segments,
4569+    #      and where they should go.. We can also figure out where the
4570+    #      encrypted private key should go, because we can figure out how
4571+    #      big the share data will be.
4572+    #
4573+    #   1: Encrypt, encode, and upload the file in chunks. Do something
4574+    #      like
4575+    #
4576+    #       put_block(data, segnum, salt)
4577+    #
4578+    #      to write a block and a salt to the disk. We can do both of
4579+    #      these operations now because we have enough of the offsets to
4580+    #      know where to put them.
4581+    #
4582+    #   2: Put the encrypted private key. Use:
4583+    #
4584+    #        put_encprivkey(encprivkey)
4585+    #
4586+    #      Now that we know the length of the private key, we can fill
4587+    #      in the offset for the block hash tree.
4588+    #
4589+    #   3: We're now in a position to upload the block hash tree for
4590+    #      a share. Put that using something like:
4591+    #       
4592+    #        put_blockhashes(block_hash_tree)
4593+    #
4594+    #      Note that block_hash_tree is a list of hashes -- we'll take
4595+    #      care of the details of serializing that appropriately. When
4596+    #      we get the block hash tree, we are also in a position to
4597+    #      calculate the offset for the share hash chain, and fill that
4598+    #      into the offsets table.
4599+    #
4600+    #   4: At the same time, we're in a position to upload the salt hash
4601+    #      tree. This is a Merkle tree over all of the salts. We use a
4602+    #      Merkle tree so that we can validate each block,salt pair as
4603+    #      we download them later. We do this using
4604+    #
4605+    #        put_salthashes(salt_hash_tree)
4606+    #
4607+    #      When you do this, I automatically put the root of the tree
4608+    #      (the hash at index 0 of the list) in its appropriate slot in
4609+    #      the signed prefix of the share.
4610+    #
4611+    #   5: We're now in a position to upload the share hash chain for
4612+    #      a share. Do that with something like:
4613+    #     
4614+    #        put_sharehashes(share_hash_chain)
4615+    #
4616+    #      share_hash_chain should be a dictionary mapping shnums to
4617+    #      32-byte hashes -- the wrapper handles serialization.
4618+    #      We'll know where to put the signature at this point, also.
4619+    #      The root of this tree will be put explicitly in the next
4620+    #      step.
4621+    #
4622+    #      TODO: Why? Why not just include it in the tree here?
4623+    #
4624+    #   6: Before putting the signature, we must first put the
4625+    #      root_hash. Do this with:
4626+    #
4627+    #        put_root_hash(root_hash).
4628+    #     
4629+    #      In terms of knowing where to put this value, it was always
4630+    #      possible to place it, but it makes sense semantically to
4631+    #      place it after the share hash tree, so that's why you do it
4632+    #      in this order.
4633+    #
4634+    #   6: With the root hash put, we can now sign the header. Use:
4635+    #
4636+    #        get_signable()
4637+    #
4638+    #      to get the part of the header that you want to sign, and use:
4639+    #       
4640+    #        put_signature(signature)
4641+    #
4642+    #      to write your signature to the remote server.
4643+    #
4644+    #   6: Add the verification key, and finish. Do:
4645+    #
4646+    #        put_verification_key(key)
4647+    #
4648+    #      and
4649+    #
4650+    #        finish_publish()
4651+    #
4652+    # Checkstring management:
4653+    #
4654+    # To write to a mutable slot, we have to provide test vectors to ensure
4655+    # that we are writing to the same data that we think we are. These
4656+    # vectors allow us to detect uncoordinated writes; that is, writes
4657+    # where both we and some other shareholder are writing to the
4658+    # mutable slot, and to report those back to the parts of the program
4659+    # doing the writing.
4660+    #
4661+    # With SDMF, this was easy -- all of the share data was written in
4662+    # one go, so it was easy to detect uncoordinated writes, and we only
4663+    # had to do it once. With MDMF, not all of the file is written at
4664+    # once.
4665+    #
4666+    # If a share is new, we write out as much of the header as we can
4667+    # before writing out anything else. This gives other writers a
4668+    # canary that they can use to detect uncoordinated writes, and, if
4669+    # they do the same thing, gives us the same canary. We them update
4670+    # the share. We won't be able to write out two fields of the header
4671+    # -- the share tree hash and the salt hash -- until we finish
4672+    # writing out the share. We only require the writer to provide the
4673+    # initial checkstring, and keep track of what it should be after
4674+    # updates ourselves.
4675+    #
4676+    # If we haven't written anything yet, then on the first write (which
4677+    # will probably be a block + salt of a share), we'll also write out
4678+    # the header. On subsequent passes, we'll expect to see the header.
4679+    # This changes in two places:
4680+    #
4681+    #   - When we write out the salt hash
4682+    #   - When we write out the root of the share hash tree
4683+    #
4684+    # since these values will change the header. It is possible that we
4685+    # can just make those be written in one operation to minimize
4686+    # disruption.
4687+    def __init__(self,
4688+                 shnum,
4689+                 rref, # a remote reference to a storage server
4690+                 storage_index,
4691+                 secrets, # (write_enabler, renew_secret, cancel_secret)
4692+                 seqnum, # the sequence number of the mutable file
4693+                 required_shares,
4694+                 total_shares,
4695+                 segment_size,
4696+                 data_length): # the length of the original file
4697+        self.shnum = shnum
4698+        self._rref = rref
4699+        self._storage_index = storage_index
4700+        self._seqnum = seqnum
4701+        self._required_shares = required_shares
4702+        assert self.shnum >= 0 and self.shnum < total_shares
4703+        self._total_shares = total_shares
4704+        # We build up the offset table as we write things. It is the
4705+        # last thing we write to the remote server.
4706+        self._offsets = {}
4707+        self._testvs = []
4708+        self._secrets = secrets
4709+        # The segment size needs to be a multiple of the k parameter --
4710+        # any padding should have been carried out by the publisher
4711+        # already.
4712+        assert segment_size % required_shares == 0
4713+        self._segment_size = segment_size
4714+        self._data_length = data_length
4715+
4716+        # These are set later -- we define them here so that we can
4717+        # check for their existence easily
4718+
4719+        # This is the root of the share hash tree -- the Merkle tree
4720+        # over the roots of the block hash trees computed for shares in
4721+        # this upload.
4722+        self._root_hash = None
4723+
4724+        # We haven't yet written anything to the remote bucket. By
4725+        # setting this, we tell the _write method as much. The write
4726+        # method will then know that it also needs to add a write vector
4727+        # for the checkstring (or what we have of it) to the first write
4728+        # request. We'll then record that value for future use.  If
4729+        # we're expecting something to be there already, we need to call
4730+        # set_checkstring before we write anything to tell the first
4731+        # write about that.
4732+        self._written = False
4733+
4734+        # When writing data to the storage servers, we get a read vector
4735+        # for free. We'll read the checkstring, which will help us
4736+        # figure out what's gone wrong if a write fails.
4737+        self._readv = [(0, struct.calcsize(MDMFCHECKSTRING))]
4738+
4739+        # We calculate the number of segments because it tells us
4740+        # where the salt part of the file ends/share segment begins,
4741+        # and also because it provides a useful amount of bounds checking.
4742+        self._num_segments = mathutil.div_ceil(self._data_length,
4743+                                               self._segment_size)
4744+        self._block_size = self._segment_size / self._required_shares
4745+        # We also calculate the share size, to help us with block
4746+        # constraints later.
4747+        tail_size = self._data_length % self._segment_size
4748+        if not tail_size:
4749+            self._tail_block_size = self._block_size
4750+        else:
4751+            self._tail_block_size = mathutil.next_multiple(tail_size,
4752+                                                           self._required_shares)
4753+            self._tail_block_size /= self._required_shares
4754+
4755+        # We already know where the sharedata starts; right after the end
4756+        # of the header (which is defined as the signable part + the offsets)
4757+        # We can also calculate where the encrypted private key begins
4758+        # from what we know know.
4759+        self._actual_block_size = self._block_size + SALT_SIZE
4760+        data_size = self._actual_block_size * (self._num_segments - 1)
4761+        data_size += self._tail_block_size
4762+        data_size += SALT_SIZE
4763+        self._offsets['enc_privkey'] = MDMFHEADERSIZE
4764+        self._offsets['enc_privkey'] += data_size
4765+        # We'll wait for the rest. Callers can now call my "put_block" and
4766+        # "set_checkstring" methods.
4767+
4768+
4769+    def set_checkstring(self,
4770+                        seqnum_or_checkstring,
4771+                        root_hash=None,
4772+                        salt=None):
4773+        """
4774+        Set checkstring checkstring for the given shnum.
4775+
4776+        This can be invoked in one of two ways.
4777+
4778+        With one argument, I assume that you are giving me a literal
4779+        checkstring -- e.g., the output of get_checkstring. I will then
4780+        set that checkstring as it is. This form is used by unit tests.
4781+
4782+        With two arguments, I assume that you are giving me a sequence
4783+        number and root hash to make a checkstring from. In that case, I
4784+        will build a checkstring and set it for you. This form is used
4785+        by the publisher.
4786+
4787+        By default, I assume that I am writing new shares to the grid.
4788+        If you don't explcitly set your own checkstring, I will use
4789+        one that requires that the remote share not exist. You will want
4790+        to use this method if you are updating a share in-place;
4791+        otherwise, writes will fail.
4792+        """
4793+        # You're allowed to overwrite checkstrings with this method;
4794+        # I assume that users know what they are doing when they call
4795+        # it.
4796+        if root_hash:
4797+            checkstring = struct.pack(MDMFCHECKSTRING,
4798+                                      1,
4799+                                      seqnum_or_checkstring,
4800+                                      root_hash)
4801+        else:
4802+            checkstring = seqnum_or_checkstring
4803+
4804+        if checkstring == "":
4805+            # We special-case this, since len("") = 0, but we need
4806+            # length of 1 for the case of an empty share to work on the
4807+            # storage server, which is what a checkstring that is the
4808+            # empty string means.
4809+            self._testvs = []
4810+        else:
4811+            self._testvs = []
4812+            self._testvs.append((0, len(checkstring), "eq", checkstring))
4813+
4814+
4815+    def __repr__(self):
4816+        return "MDMFSlotWriteProxy for share %d" % self.shnum
4817+
4818+
4819+    def get_checkstring(self):
4820+        """
4821+        Given a share number, I return a representation of what the
4822+        checkstring for that share on the server will look like.
4823+
4824+        I am mostly used for tests.
4825+        """
4826+        if self._root_hash:
4827+            roothash = self._root_hash
4828+        else:
4829+            roothash = "\x00" * 32
4830+        return struct.pack(MDMFCHECKSTRING,
4831+                           1,
4832+                           self._seqnum,
4833+                           roothash)
4834+
4835+
4836+    def put_block(self, data, segnum, salt):
4837+        """
4838+        Put the encrypted-and-encoded data segment in the slot, along
4839+        with the salt.
4840+        """
4841+        if segnum >= self._num_segments:
4842+            raise LayoutInvalid("I won't overwrite the private key")
4843+        if len(salt) != SALT_SIZE:
4844+            raise LayoutInvalid("I was given a salt of size %d, but "
4845+                                "I wanted a salt of size %d")
4846+        if segnum + 1 == self._num_segments:
4847+            if len(data) != self._tail_block_size:
4848+                raise LayoutInvalid("I was given the wrong size block to write")
4849+        elif len(data) != self._block_size:
4850+            raise LayoutInvalid("I was given the wrong size block to write")
4851+
4852+        # We want to write at len(MDMFHEADER) + segnum * block_size.
4853+
4854+        offset = MDMFHEADERSIZE + (self._actual_block_size * segnum)
4855+        data = salt + data
4856+
4857+        datavs = [tuple([offset, data])]
4858+        return self._write(datavs)
4859+
4860+
4861+    def put_encprivkey(self, encprivkey):
4862+        """
4863+        Put the encrypted private key in the remote slot.
4864+        """
4865+        assert self._offsets
4866+        assert self._offsets['enc_privkey']
4867+        # You shouldn't re-write the encprivkey after the block hash
4868+        # tree is written, since that could cause the private key to run
4869+        # into the block hash tree. Before it writes the block hash
4870+        # tree, the block hash tree writing method writes the offset of
4871+        # the salt hash tree. So that's a good indicator of whether or
4872+        # not the block hash tree has been written.
4873+        if "share_hash_chain" in self._offsets:
4874+            raise LayoutInvalid("You must write this before the block hash tree")
4875+
4876+        self._offsets['block_hash_tree'] = self._offsets['enc_privkey'] + len(encprivkey)
4877+        datavs = [(tuple([self._offsets['enc_privkey'], encprivkey]))]
4878+        def _on_failure():
4879+            del(self._offsets['block_hash_tree'])
4880+        return self._write(datavs, on_failure=_on_failure)
4881+
4882+
4883+    def put_blockhashes(self, blockhashes):
4884+        """
4885+        Put the block hash tree in the remote slot.
4886+
4887+        The encrypted private key must be put before the block hash
4888+        tree, since we need to know how large it is to know where the
4889+        block hash tree should go. The block hash tree must be put
4890+        before the salt hash tree, since its size determines the
4891+        offset of the share hash chain.
4892+        """
4893+        assert self._offsets
4894+        assert isinstance(blockhashes, list)
4895+        if "block_hash_tree" not in self._offsets:
4896+            raise LayoutInvalid("You must put the encrypted private key "
4897+                                "before you put the block hash tree")
4898+        # If written, the share hash chain causes the signature offset
4899+        # to be defined.
4900+        if "signature" in self._offsets:
4901+            raise LayoutInvalid("You must put the block hash tree before "
4902+                                "you put the share hash chain")
4903+        blockhashes_s = "".join(blockhashes)
4904+        self._offsets['share_hash_chain'] = self._offsets['block_hash_tree'] + len(blockhashes_s)
4905+        datavs = []
4906+        datavs.append(tuple([self._offsets['block_hash_tree'], blockhashes_s]))
4907+        def _on_failure():
4908+            del(self._offsets['share_hash_chain'])
4909+        return self._write(datavs, on_failure=_on_failure)
4910+
4911+
4912+    def put_sharehashes(self, sharehashes):
4913+        """
4914+        Put the share hash chain in the remote slot.
4915+
4916+        The salt hash tree must be put before the share hash chain,
4917+        since we need to know where the salt hash tree ends before we
4918+        can know where the share hash chain starts. The share hash chain
4919+        must be put before the signature, since the length of the packed
4920+        share hash chain determines the offset of the signature. Also,
4921+        semantically, you must know what the root of the salt hash tree
4922+        is before you can generate a valid signature.
4923+        """
4924+        assert isinstance(sharehashes, dict)
4925+        if "share_hash_chain" not in self._offsets:
4926+            raise LayoutInvalid("You need to put the salt hash tree before "
4927+                                "you can put the share hash chain")
4928+        # The signature comes after the share hash chain. If the
4929+        # signature has already been written, we must not write another
4930+        # share hash chain. The signature writes the verification key
4931+        # offset when it gets sent to the remote server, so we look for
4932+        # that.
4933+        if "verification_key" in self._offsets:
4934+            raise LayoutInvalid("You must write the share hash chain "
4935+                                "before you write the signature")
4936+        datavs = []
4937+        sharehashes_s = "".join([struct.pack(">H32s", i, sharehashes[i])
4938+                                  for i in sorted(sharehashes.keys())])
4939+        self._offsets['signature'] = self._offsets['share_hash_chain'] + len(sharehashes_s)
4940+        datavs.append(tuple([self._offsets['share_hash_chain'], sharehashes_s]))
4941+        def _on_failure():
4942+            del(self._offsets['signature'])
4943+        return self._write(datavs, on_failure=_on_failure)
4944+
4945+
4946+    def put_root_hash(self, roothash):
4947+        """
4948+        Put the root hash (the root of the share hash tree) in the
4949+        remote slot.
4950+        """
4951+        # It does not make sense to be able to put the root
4952+        # hash without first putting the share hashes, since you need
4953+        # the share hashes to generate the root hash.
4954+        #
4955+        # Signature is defined by the routine that places the share hash
4956+        # chain, so it's a good thing to look for in finding out whether
4957+        # or not the share hash chain exists on the remote server.
4958+        if "signature" not in self._offsets:
4959+            raise LayoutInvalid("You need to put the share hash chain "
4960+                                "before you can put the root share hash")
4961+        if len(roothash) != HASH_SIZE:
4962+            raise LayoutInvalid("hashes and salts must be exactly %d bytes"
4963+                                 % HASH_SIZE)
4964+        datavs = []
4965+        self._root_hash = roothash
4966+        # To write both of these values, we update the checkstring on
4967+        # the remote server, which includes them
4968+        checkstring = self.get_checkstring()
4969+        datavs.append(tuple([0, checkstring]))
4970+        # This write, if successful, changes the checkstring, so we need
4971+        # to update our internal checkstring to be consistent with the
4972+        # one on the server.
4973+        def _on_success():
4974+            self._testvs = [(0, len(checkstring), "eq", checkstring)]
4975+        def _on_failure():
4976+            self._root_hash = None
4977+        return self._write(datavs,
4978+                           on_success=_on_success,
4979+                           on_failure=_on_failure)
4980+
4981+
4982+    def get_signable(self):
4983+        """
4984+        Get the first seven fields of the mutable file; the parts that
4985+        are signed.
4986+        """
4987+        if not self._root_hash:
4988+            raise LayoutInvalid("You need to set the root hash "
4989+                                "before getting something to "
4990+                                "sign")
4991+        return struct.pack(MDMFSIGNABLEHEADER,
4992+                           1,
4993+                           self._seqnum,
4994+                           self._root_hash,
4995+                           self._required_shares,
4996+                           self._total_shares,
4997+                           self._segment_size,
4998+                           self._data_length)
4999+
5000+
5001+    def put_signature(self, signature):
5002+        """
5003+        Put the signature field to the remote slot.
5004+
5005+        I require that the root hash and share hash chain have been put
5006+        to the grid before I will write the signature to the grid.
5007+        """
5008+        if "signature" not in self._offsets:
5009+            raise LayoutInvalid("You must put the share hash chain "
5010+        # It does not make sense to put a signature without first
5011+        # putting the root hash and the salt hash (since otherwise
5012+        # the signature would be incomplete), so we don't allow that.
5013+                       "before putting the signature")
5014+        if not self._root_hash:
5015+            raise LayoutInvalid("You must complete the signed prefix "
5016+                                "before computing a signature")
5017+        # If we put the signature after we put the verification key, we
5018+        # could end up running into the verification key, and will
5019+        # probably screw up the offsets as well. So we don't allow that.
5020+        # The method that writes the verification key defines the EOF
5021+        # offset before writing the verification key, so look for that.
5022+        if "EOF" in self._offsets:
5023+            raise LayoutInvalid("You must write the signature before the verification key")
5024+
5025+        self._offsets['verification_key'] = self._offsets['signature'] + len(signature)
5026+        datavs = []
5027+        datavs.append(tuple([self._offsets['signature'], signature]))
5028+        def _on_failure():
5029+            del(self._offsets['verification_key'])
5030+        return self._write(datavs, on_failure=_on_failure)
5031+
5032+
5033+    def put_verification_key(self, verification_key):
5034+        """
5035+        Put the verification key into the remote slot.
5036+
5037+        I require that the signature have been written to the storage
5038+        server before I allow the verification key to be written to the
5039+        remote server.
5040+        """
5041+        if "verification_key" not in self._offsets:
5042+            raise LayoutInvalid("You must put the signature before you "
5043+                                "can put the verification key")
5044+        self._offsets['EOF'] = self._offsets['verification_key'] + len(verification_key)
5045+        datavs = []
5046+        datavs.append(tuple([self._offsets['verification_key'], verification_key]))
5047+        def _on_failure():
5048+            del(self._offsets['EOF'])
5049+        return self._write(datavs, on_failure=_on_failure)
5050+
5051+    def _get_offsets_tuple(self):
5052+        return tuple([(key, value) for key, value in self._offsets.items()])
5053+
5054+    def get_verinfo(self):
5055+        return (self._seqnum,
5056+                self._root_hash,
5057+                self._required_shares,
5058+                self._total_shares,
5059+                self._segment_size,
5060+                self._data_length,
5061+                self.get_signable(),
5062+                self._get_offsets_tuple())
5063+
5064+
5065+    def finish_publishing(self):
5066+        """
5067+        Write the offset table and encoding parameters to the remote
5068+        slot, since that's the only thing we have yet to publish at this
5069+        point.
5070+        """
5071+        if "EOF" not in self._offsets:
5072+            raise LayoutInvalid("You must put the verification key before "
5073+                                "you can publish the offsets")
5074+        offsets_offset = struct.calcsize(MDMFHEADERWITHOUTOFFSETS)
5075+        offsets = struct.pack(MDMFOFFSETS,
5076+                              self._offsets['enc_privkey'],
5077+                              self._offsets['block_hash_tree'],
5078+                              self._offsets['share_hash_chain'],
5079+                              self._offsets['signature'],
5080+                              self._offsets['verification_key'],
5081+                              self._offsets['EOF'])
5082+        datavs = []
5083+        datavs.append(tuple([offsets_offset, offsets]))
5084+        encoding_parameters_offset = struct.calcsize(MDMFCHECKSTRING)
5085+        params = struct.pack(">BBQQ",
5086+                             self._required_shares,
5087+                             self._total_shares,
5088+                             self._segment_size,
5089+                             self._data_length)
5090+        datavs.append(tuple([encoding_parameters_offset, params]))
5091+        return self._write(datavs)
5092+
5093+
5094+    def _write(self, datavs, on_failure=None, on_success=None):
5095+        """I write the data vectors in datavs to the remote slot."""
5096+        tw_vectors = {}
5097+        new_share = False
5098+        if not self._testvs:
5099+            self._testvs = []
5100+            self._testvs.append(tuple([0, 1, "eq", ""]))
5101+            new_share = True
5102+        if not self._written:
5103+            # Write a new checkstring to the share when we write it, so
5104+            # that we have something to check later.
5105+            new_checkstring = self.get_checkstring()
5106+            datavs.append((0, new_checkstring))
5107+            def _first_write():
5108+                self._written = True
5109+                self._testvs = [(0, len(new_checkstring), "eq", new_checkstring)]
5110+            on_success = _first_write
5111+        tw_vectors[self.shnum] = (self._testvs, datavs, None)
5112+        datalength = sum([len(x[1]) for x in datavs])
5113+        d = self._rref.callRemote("slot_testv_and_readv_and_writev",
5114+                                  self._storage_index,
5115+                                  self._secrets,
5116+                                  tw_vectors,
5117+                                  self._readv)
5118+        def _result(results):
5119+            if isinstance(results, failure.Failure) or not results[0]:
5120+                # Do nothing; the write was unsuccessful.
5121+                if on_failure: on_failure()
5122+            else:
5123+                if on_success: on_success()
5124+            return results
5125+        d.addCallback(_result)
5126+        return d
5127+
5128+
5129+class MDMFSlotReadProxy:
5130+    """
5131+    I read from a mutable slot filled with data written in the MDMF data
5132+    format (which is described above).
5133+
5134+    I can be initialized with some amount of data, which I will use (if
5135+    it is valid) to eliminate some of the need to fetch it from servers.
5136+    """
5137+    def __init__(self,
5138+                 rref,
5139+                 storage_index,
5140+                 shnum,
5141+                 data=""):
5142+        # Start the initialization process.
5143+        self._rref = rref
5144+        self._storage_index = storage_index
5145+        self.shnum = shnum
5146+
5147+        # Before doing anything, the reader is probably going to want to
5148+        # verify that the signature is correct. To do that, they'll need
5149+        # the verification key, and the signature. To get those, we'll
5150+        # need the offset table. So fetch the offset table on the
5151+        # assumption that that will be the first thing that a reader is
5152+        # going to do.
5153+
5154+        # The fact that these encoding parameters are None tells us
5155+        # that we haven't yet fetched them from the remote share, so we
5156+        # should. We could just not set them, but the checks will be
5157+        # easier to read if we don't have to use hasattr.
5158+        self._version_number = None
5159+        self._sequence_number = None
5160+        self._root_hash = None
5161+        # Filled in if we're dealing with an SDMF file. Unused
5162+        # otherwise.
5163+        self._salt = None
5164+        self._required_shares = None
5165+        self._total_shares = None
5166+        self._segment_size = None
5167+        self._data_length = None
5168+        self._offsets = None
5169+
5170+        # If the user has chosen to initialize us with some data, we'll
5171+        # try to satisfy subsequent data requests with that data before
5172+        # asking the storage server for it. If
5173+        self._data = data
5174+        # The way callers interact with cache in the filenode returns
5175+        # None if there isn't any cached data, but the way we index the
5176+        # cached data requires a string, so convert None to "".
5177+        if self._data == None:
5178+            self._data = ""
5179+
5180+        self._queue_observers = observer.ObserverList()
5181+        self._queue_errbacks = observer.ObserverList()
5182+        self._readvs = []
5183+
5184+
5185+    def _maybe_fetch_offsets_and_header(self, force_remote=False):
5186+        """
5187+        I fetch the offset table and the header from the remote slot if
5188+        I don't already have them. If I do have them, I do nothing and
5189+        return an empty Deferred.
5190+        """
5191+        if self._offsets:
5192+            return defer.succeed(None)
5193+        # At this point, we may be either SDMF or MDMF. Fetching 107
5194+        # bytes will be enough to get header and offsets for both SDMF and
5195+        # MDMF, though we'll be left with 4 more bytes than we
5196+        # need if this ends up being MDMF. This is probably less
5197+        # expensive than the cost of a second roundtrip.
5198+        readvs = [(0, 107)]
5199+        d = self._read(readvs, force_remote)
5200+        d.addCallback(self._process_encoding_parameters)
5201+        d.addCallback(self._process_offsets)
5202+        return d
5203+
5204+
5205+    def _process_encoding_parameters(self, encoding_parameters):
5206+        assert self.shnum in encoding_parameters
5207+        encoding_parameters = encoding_parameters[self.shnum][0]
5208+        # The first byte is the version number. It will tell us what
5209+        # to do next.
5210+        (verno,) = struct.unpack(">B", encoding_parameters[:1])
5211+        if verno == MDMF_VERSION:
5212+            read_size = MDMFHEADERWITHOUTOFFSETSSIZE
5213+            (verno,
5214+             seqnum,
5215+             root_hash,
5216+             k,
5217+             n,
5218+             segsize,
5219+             datalen) = struct.unpack(MDMFHEADERWITHOUTOFFSETS,
5220+                                      encoding_parameters[:read_size])
5221+            if segsize == 0 and datalen == 0:
5222+                # Empty file, no segments.
5223+                self._num_segments = 0
5224+            else:
5225+                self._num_segments = mathutil.div_ceil(datalen, segsize)
5226+
5227+        elif verno == SDMF_VERSION:
5228+            read_size = SIGNED_PREFIX_LENGTH
5229+            (verno,
5230+             seqnum,
5231+             root_hash,
5232+             salt,
5233+             k,
5234+             n,
5235+             segsize,
5236+             datalen) = struct.unpack(">BQ32s16s BBQQ",
5237+                                encoding_parameters[:SIGNED_PREFIX_LENGTH])
5238+            self._salt = salt
5239+            if segsize == 0 and datalen == 0:
5240+                # empty file
5241+                self._num_segments = 0
5242+            else:
5243+                # non-empty SDMF files have one segment.
5244+                self._num_segments = 1
5245+        else:
5246+            raise UnknownVersionError("You asked me to read mutable file "
5247+                                      "version %d, but I only understand "
5248+                                      "%d and %d" % (verno, SDMF_VERSION,
5249+                                                     MDMF_VERSION))
5250+
5251+        self._version_number = verno
5252+        self._sequence_number = seqnum
5253+        self._root_hash = root_hash
5254+        self._required_shares = k
5255+        self._total_shares = n
5256+        self._segment_size = segsize
5257+        self._data_length = datalen
5258+
5259+        self._block_size = self._segment_size / self._required_shares
5260+        # We can upload empty files, and need to account for this fact
5261+        # so as to avoid zero-division and zero-modulo errors.
5262+        if datalen > 0:
5263+            tail_size = self._data_length % self._segment_size
5264+        else:
5265+            tail_size = 0
5266+        if not tail_size:
5267+            self._tail_block_size = self._block_size
5268+        else:
5269+            self._tail_block_size = mathutil.next_multiple(tail_size,
5270+                                                    self._required_shares)
5271+            self._tail_block_size /= self._required_shares
5272+
5273+        return encoding_parameters
5274+
5275+
5276+    def _process_offsets(self, offsets):
5277+        if self._version_number == 0:
5278+            read_size = OFFSETS_LENGTH
5279+            read_offset = SIGNED_PREFIX_LENGTH
5280+            end = read_size + read_offset
5281+            (signature,
5282+             share_hash_chain,
5283+             block_hash_tree,
5284+             share_data,
5285+             enc_privkey,
5286+             EOF) = struct.unpack(">LLLLQQ",
5287+                                  offsets[read_offset:end])
5288+            self._offsets = {}
5289+            self._offsets['signature'] = signature
5290+            self._offsets['share_data'] = share_data
5291+            self._offsets['block_hash_tree'] = block_hash_tree
5292+            self._offsets['share_hash_chain'] = share_hash_chain
5293+            self._offsets['enc_privkey'] = enc_privkey
5294+            self._offsets['EOF'] = EOF
5295+
5296+        elif self._version_number == 1:
5297+            read_offset = MDMFHEADERWITHOUTOFFSETSSIZE
5298+            read_length = MDMFOFFSETS_LENGTH
5299+            end = read_offset + read_length
5300+            (encprivkey,
5301+             blockhashes,
5302+             sharehashes,
5303+             signature,
5304+             verification_key,
5305+             eof) = struct.unpack(MDMFOFFSETS,
5306+                                  offsets[read_offset:end])
5307+            self._offsets = {}
5308+            self._offsets['enc_privkey'] = encprivkey
5309+            self._offsets['block_hash_tree'] = blockhashes
5310+            self._offsets['share_hash_chain'] = sharehashes
5311+            self._offsets['signature'] = signature
5312+            self._offsets['verification_key'] = verification_key
5313+            self._offsets['EOF'] = eof
5314+
5315+
5316+    def get_block_and_salt(self, segnum, queue=False):
5317+        """
5318+        I return (block, salt), where block is the block data and
5319+        salt is the salt used to encrypt that segment.
5320+        """
5321+        d = self._maybe_fetch_offsets_and_header()
5322+        def _then(ignored):
5323+            if self._version_number == 1:
5324+                base_share_offset = MDMFHEADERSIZE
5325+            else:
5326+                base_share_offset = self._offsets['share_data']
5327+
5328+            if segnum + 1 > self._num_segments:
5329+                raise LayoutInvalid("Not a valid segment number")
5330+
5331+            if self._version_number == 0:
5332+                share_offset = base_share_offset + self._block_size * segnum
5333+            else:
5334+                share_offset = base_share_offset + (self._block_size + \
5335+                                                    SALT_SIZE) * segnum
5336+            if segnum + 1 == self._num_segments:
5337+                data = self._tail_block_size
5338+            else:
5339+                data = self._block_size
5340+
5341+            if self._version_number == 1:
5342+                data += SALT_SIZE
5343+
5344+            readvs = [(share_offset, data)]
5345+            return readvs
5346+        d.addCallback(_then)
5347+        d.addCallback(lambda readvs:
5348+            self._read(readvs, queue=queue))
5349+        def _process_results(results):
5350+            assert self.shnum in results
5351+            if self._version_number == 0:
5352+                # We only read the share data, but we know the salt from
5353+                # when we fetched the header
5354+                data = results[self.shnum]
5355+                if not data:
5356+                    data = ""
5357+                else:
5358+                    assert len(data) == 1
5359+                    data = data[0]
5360+                salt = self._salt
5361+            else:
5362+                data = results[self.shnum]
5363+                if not data:
5364+                    salt = data = ""
5365+                else:
5366+                    salt_and_data = results[self.shnum][0]
5367+                    salt = salt_and_data[:SALT_SIZE]
5368+                    data = salt_and_data[SALT_SIZE:]
5369+            return data, salt
5370+        d.addCallback(_process_results)
5371+        return d
5372+
5373+
5374+    def get_blockhashes(self, needed=None, queue=False, force_remote=False):
5375+        """
5376+        I return the block hash tree
5377+
5378+        I take an optional argument, needed, which is a set of indices
5379+        correspond to hashes that I should fetch. If this argument is
5380+        missing, I will fetch the entire block hash tree; otherwise, I
5381+        may attempt to fetch fewer hashes, based on what needed says
5382+        that I should do. Note that I may fetch as many hashes as I
5383+        want, so long as the set of hashes that I do fetch is a superset
5384+        of the ones that I am asked for, so callers should be prepared
5385+        to tolerate additional hashes.
5386+        """
5387+        # TODO: Return only the parts of the block hash tree necessary
5388+        # to validate the blocknum provided?
5389+        # This is a good idea, but it is hard to implement correctly. It
5390+        # is bad to fetch any one block hash more than once, so we
5391+        # probably just want to fetch the whole thing at once and then
5392+        # serve it.
5393+        if needed == set([]):
5394+            return defer.succeed([])
5395+        d = self._maybe_fetch_offsets_and_header()
5396+        def _then(ignored):
5397+            blockhashes_offset = self._offsets['block_hash_tree']
5398+            if self._version_number == 1:
5399+                blockhashes_length = self._offsets['share_hash_chain'] - blockhashes_offset
5400+            else:
5401+                blockhashes_length = self._offsets['share_data'] - blockhashes_offset
5402+            readvs = [(blockhashes_offset, blockhashes_length)]
5403+            return readvs
5404+        d.addCallback(_then)
5405+        d.addCallback(lambda readvs:
5406+            self._read(readvs, queue=queue, force_remote=force_remote))
5407+        def _build_block_hash_tree(results):
5408+            assert self.shnum in results
5409+
5410+            rawhashes = results[self.shnum][0]
5411+            results = [rawhashes[i:i+HASH_SIZE]
5412+                       for i in range(0, len(rawhashes), HASH_SIZE)]
5413+            return results
5414+        d.addCallback(_build_block_hash_tree)
5415+        return d
5416+
5417+
5418+    def get_sharehashes(self, needed=None, queue=False, force_remote=False):
5419+        """
5420+        I return the part of the share hash chain placed to validate
5421+        this share.
5422+
5423+        I take an optional argument, needed. Needed is a set of indices
5424+        that correspond to the hashes that I should fetch. If needed is
5425+        not present, I will fetch and return the entire share hash
5426+        chain. Otherwise, I may fetch and return any part of the share
5427+        hash chain that is a superset of the part that I am asked to
5428+        fetch. Callers should be prepared to deal with more hashes than
5429+        they've asked for.
5430+        """
5431+        if needed == set([]):
5432+            return defer.succeed([])
5433+        d = self._maybe_fetch_offsets_and_header()
5434+
5435+        def _make_readvs(ignored):
5436+            sharehashes_offset = self._offsets['share_hash_chain']
5437+            if self._version_number == 0:
5438+                sharehashes_length = self._offsets['block_hash_tree'] - sharehashes_offset
5439+            else:
5440+                sharehashes_length = self._offsets['signature'] - sharehashes_offset
5441+            readvs = [(sharehashes_offset, sharehashes_length)]
5442+            return readvs
5443+        d.addCallback(_make_readvs)
5444+        d.addCallback(lambda readvs:
5445+            self._read(readvs, queue=queue, force_remote=force_remote))
5446+        def _build_share_hash_chain(results):
5447+            assert self.shnum in results
5448+
5449+            sharehashes = results[self.shnum][0]
5450+            results = [sharehashes[i:i+(HASH_SIZE + 2)]
5451+                       for i in range(0, len(sharehashes), HASH_SIZE + 2)]
5452+            results = dict([struct.unpack(">H32s", data)
5453+                            for data in results])
5454+            return results
5455+        d.addCallback(_build_share_hash_chain)
5456+        return d
5457+
5458+
5459+    def get_encprivkey(self, queue=False):
5460+        """
5461+        I return the encrypted private key.
5462+        """
5463+        d = self._maybe_fetch_offsets_and_header()
5464+
5465+        def _make_readvs(ignored):
5466+            privkey_offset = self._offsets['enc_privkey']
5467+            if self._version_number == 0:
5468+                privkey_length = self._offsets['EOF'] - privkey_offset
5469+            else:
5470+                privkey_length = self._offsets['block_hash_tree'] - privkey_offset
5471+            readvs = [(privkey_offset, privkey_length)]
5472+            return readvs
5473+        d.addCallback(_make_readvs)
5474+        d.addCallback(lambda readvs:
5475+            self._read(readvs, queue=queue))
5476+        def _process_results(results):
5477+            assert self.shnum in results
5478+            privkey = results[self.shnum][0]
5479+            return privkey
5480+        d.addCallback(_process_results)
5481+        return d
5482+
5483+
5484+    def get_signature(self, queue=False):
5485+        """
5486+        I return the signature of my share.
5487+        """
5488+        d = self._maybe_fetch_offsets_and_header()
5489+
5490+        def _make_readvs(ignored):
5491+            signature_offset = self._offsets['signature']
5492+            if self._version_number == 1:
5493+                signature_length = self._offsets['verification_key'] - signature_offset
5494+            else:
5495+                signature_length = self._offsets['share_hash_chain'] - signature_offset
5496+            readvs = [(signature_offset, signature_length)]
5497+            return readvs
5498+        d.addCallback(_make_readvs)
5499+        d.addCallback(lambda readvs:
5500+            self._read(readvs, queue=queue))
5501+        def _process_results(results):
5502+            assert self.shnum in results
5503+            signature = results[self.shnum][0]
5504+            return signature
5505+        d.addCallback(_process_results)
5506+        return d
5507+
5508+
5509+    def get_verification_key(self, queue=False):
5510+        """
5511+        I return the verification key.
5512+        """
5513+        d = self._maybe_fetch_offsets_and_header()
5514+
5515+        def _make_readvs(ignored):
5516+            if self._version_number == 1:
5517+                vk_offset = self._offsets['verification_key']
5518+                vk_length = self._offsets['EOF'] - vk_offset
5519+            else:
5520+                vk_offset = struct.calcsize(">BQ32s16sBBQQLLLLQQ")
5521+                vk_length = self._offsets['signature'] - vk_offset
5522+            readvs = [(vk_offset, vk_length)]
5523+            return readvs
5524+        d.addCallback(_make_readvs)
5525+        d.addCallback(lambda readvs:
5526+            self._read(readvs, queue=queue))
5527+        def _process_results(results):
5528+            assert self.shnum in results
5529+            verification_key = results[self.shnum][0]
5530+            return verification_key
5531+        d.addCallback(_process_results)
5532+        return d
5533+
5534+
5535+    def get_encoding_parameters(self):
5536+        """
5537+        I return (k, n, segsize, datalen)
5538+        """
5539+        d = self._maybe_fetch_offsets_and_header()
5540+        d.addCallback(lambda ignored:
5541+            (self._required_shares,
5542+             self._total_shares,
5543+             self._segment_size,
5544+             self._data_length))
5545+        return d
5546+
5547+
5548+    def get_seqnum(self):
5549+        """
5550+        I return the sequence number for this share.
5551+        """
5552+        d = self._maybe_fetch_offsets_and_header()
5553+        d.addCallback(lambda ignored:
5554+            self._sequence_number)
5555+        return d
5556+
5557+
5558+    def get_root_hash(self):
5559+        """
5560+        I return the root of the block hash tree
5561+        """
5562+        d = self._maybe_fetch_offsets_and_header()
5563+        d.addCallback(lambda ignored: self._root_hash)
5564+        return d
5565+
5566+
5567+    def get_checkstring(self):
5568+        """
5569+        I return the packed representation of the following:
5570+
5571+            - version number
5572+            - sequence number
5573+            - root hash
5574+            - salt hash
5575+
5576+        which my users use as a checkstring to detect other writers.
5577+        """
5578+        d = self._maybe_fetch_offsets_and_header()
5579+        def _build_checkstring(ignored):
5580+            if self._salt:
5581+                checkstring = strut.pack(PREFIX,
5582+                                         self._version_number,
5583+                                         self._sequence_number,
5584+                                         self._root_hash,
5585+                                         self._salt)
5586+            else:
5587+                checkstring = struct.pack(MDMFCHECKSTRING,
5588+                                          self._version_number,
5589+                                          self._sequence_number,
5590+                                          self._root_hash)
5591+
5592+            return checkstring
5593+        d.addCallback(_build_checkstring)
5594+        return d
5595+
5596+
5597+    def get_prefix(self, force_remote):
5598+        d = self._maybe_fetch_offsets_and_header(force_remote)
5599+        d.addCallback(lambda ignored:
5600+            self._build_prefix())
5601+        return d
5602+
5603+
5604+    def _build_prefix(self):
5605+        # The prefix is another name for the part of the remote share
5606+        # that gets signed. It consists of everything up to and
5607+        # including the datalength, packed by struct.
5608+        if self._version_number == SDMF_VERSION:
5609+            return struct.pack(SIGNED_PREFIX,
5610+                           self._version_number,
5611+                           self._sequence_number,
5612+                           self._root_hash,
5613+                           self._salt,
5614+                           self._required_shares,
5615+                           self._total_shares,
5616+                           self._segment_size,
5617+                           self._data_length)
5618+
5619+        else:
5620+            return struct.pack(MDMFSIGNABLEHEADER,
5621+                           self._version_number,
5622+                           self._sequence_number,
5623+                           self._root_hash,
5624+                           self._required_shares,
5625+                           self._total_shares,
5626+                           self._segment_size,
5627+                           self._data_length)
5628+
5629+
5630+    def _get_offsets_tuple(self):
5631+        # The offsets tuple is another component of the version
5632+        # information tuple. It is basically our offsets dictionary,
5633+        # itemized and in a tuple.
5634+        return self._offsets.copy()
5635+
5636+
5637+    def get_verinfo(self):
5638+        """
5639+        I return my verinfo tuple. This is used by the ServermapUpdater
5640+        to keep track of versions of mutable files.
5641+
5642+        The verinfo tuple for MDMF files contains:
5643+            - seqnum
5644+            - root hash
5645+            - a blank (nothing)
5646+            - segsize
5647+            - datalen
5648+            - k
5649+            - n
5650+            - prefix (the thing that you sign)
5651+            - a tuple of offsets
5652+
5653+        We include the nonce in MDMF to simplify processing of version
5654+        information tuples.
5655+
5656+        The verinfo tuple for SDMF files is the same, but contains a
5657+        16-byte IV instead of a hash of salts.
5658+        """
5659+        d = self._maybe_fetch_offsets_and_header()
5660+        def _build_verinfo(ignored):
5661+            if self._version_number == SDMF_VERSION:
5662+                salt_to_use = self._salt
5663+            else:
5664+                salt_to_use = None
5665+            return (self._sequence_number,
5666+                    self._root_hash,
5667+                    salt_to_use,
5668+                    self._segment_size,
5669+                    self._data_length,
5670+                    self._required_shares,
5671+                    self._total_shares,
5672+                    self._build_prefix(),
5673+                    self._get_offsets_tuple())
5674+        d.addCallback(_build_verinfo)
5675+        return d
5676+
5677+
5678+    def flush(self):
5679+        """
5680+        I flush my queue of read vectors.
5681+        """
5682+        d = self._read(self._readvs)
5683+        def _then(results):
5684+            self._readvs = []
5685+            if isinstance(results, failure.Failure):
5686+                self._queue_errbacks.notify(results)
5687+            else:
5688+                self._queue_observers.notify(results)
5689+            self._queue_observers = observer.ObserverList()
5690+            self._queue_errbacks = observer.ObserverList()
5691+        d.addBoth(_then)
5692+
5693+
5694+    def _read(self, readvs, force_remote=False, queue=False):
5695+        unsatisfiable = filter(lambda x: x[0] + x[1] > len(self._data), readvs)
5696+        # TODO: It's entirely possible to tweak this so that it just
5697+        # fulfills the requests that it can, and not demand that all
5698+        # requests are satisfiable before running it.
5699+        if not unsatisfiable and not force_remote:
5700+            results = [self._data[offset:offset+length]
5701+                       for (offset, length) in readvs]
5702+            results = {self.shnum: results}
5703+            return defer.succeed(results)
5704+        else:
5705+            if queue:
5706+                start = len(self._readvs)
5707+                self._readvs += readvs
5708+                end = len(self._readvs)
5709+                def _get_results(results, start, end):
5710+                    if not self.shnum in results:
5711+                        return {self._shnum: [""]}
5712+                    return {self.shnum: results[self.shnum][start:end]}
5713+                d = defer.Deferred()
5714+                d.addCallback(_get_results, start, end)
5715+                self._queue_observers.subscribe(d.callback)
5716+                self._queue_errbacks.subscribe(d.errback)
5717+                return d
5718+            return self._rref.callRemote("slot_readv",
5719+                                         self._storage_index,
5720+                                         [self.shnum],
5721+                                         readvs)
5722+
5723+
5724+    def is_sdmf(self):
5725+        """I tell my caller whether or not my remote file is SDMF or MDMF
5726+        """
5727+        d = self._maybe_fetch_offsets_and_header()
5728+        d.addCallback(lambda ignored:
5729+            self._version_number == 0)
5730+        return d
5731+
5732+
5733+class LayoutInvalid(Exception):
5734+    """
5735+    This isn't a valid MDMF mutable file
5736+    """
5737hunk ./src/allmydata/test/test_storage.py 2
5738 
5739-import time, os.path, stat, re, simplejson, struct
5740+import time, os.path, stat, re, simplejson, struct, shutil
5741 
5742 from twisted.trial import unittest
5743 
5744hunk ./src/allmydata/test/test_storage.py 22
5745 from allmydata.storage.expirer import LeaseCheckingCrawler
5746 from allmydata.immutable.layout import WriteBucketProxy, WriteBucketProxy_v2, \
5747      ReadBucketProxy
5748-from allmydata.interfaces import BadWriteEnablerError
5749-from allmydata.test.common import LoggingServiceParent
5750+from allmydata.mutable.layout import MDMFSlotWriteProxy, MDMFSlotReadProxy, \
5751+                                     LayoutInvalid, MDMFSIGNABLEHEADER, \
5752+                                     SIGNED_PREFIX, MDMFHEADER, \
5753+                                     MDMFOFFSETS, SDMFSlotWriteProxy
5754+from allmydata.interfaces import BadWriteEnablerError, MDMF_VERSION, \
5755+                                 SDMF_VERSION
5756+from allmydata.test.common import LoggingServiceParent, ShouldFailMixin
5757 from allmydata.test.common_web import WebRenderingMixin
5758 from allmydata.web.storage import StorageStatus, remove_prefix
5759 
5760hunk ./src/allmydata/test/test_storage.py 106
5761 
5762 class RemoteBucket:
5763 
5764+    def __init__(self):
5765+        self.read_count = 0
5766+        self.write_count = 0
5767+
5768     def callRemote(self, methname, *args, **kwargs):
5769         def _call():
5770             meth = getattr(self.target, "remote_" + methname)
5771hunk ./src/allmydata/test/test_storage.py 114
5772             return meth(*args, **kwargs)
5773+
5774+        if methname == "slot_readv":
5775+            self.read_count += 1
5776+        if "writev" in methname:
5777+            self.write_count += 1
5778+
5779         return defer.maybeDeferred(_call)
5780 
5781hunk ./src/allmydata/test/test_storage.py 122
5782+
5783 class BucketProxy(unittest.TestCase):
5784     def make_bucket(self, name, size):
5785         basedir = os.path.join("storage", "BucketProxy", name)
5786hunk ./src/allmydata/test/test_storage.py 1313
5787         self.failUnless(os.path.exists(prefixdir), prefixdir)
5788         self.failIf(os.path.exists(bucketdir), bucketdir)
5789 
5790+
5791+class MDMFProxies(unittest.TestCase, ShouldFailMixin):
5792+    def setUp(self):
5793+        self.sparent = LoggingServiceParent()
5794+        self._lease_secret = itertools.count()
5795+        self.ss = self.create("MDMFProxies storage test server")
5796+        self.rref = RemoteBucket()
5797+        self.rref.target = self.ss
5798+        self.secrets = (self.write_enabler("we_secret"),
5799+                        self.renew_secret("renew_secret"),
5800+                        self.cancel_secret("cancel_secret"))
5801+        self.segment = "aaaaaa"
5802+        self.block = "aa"
5803+        self.salt = "a" * 16
5804+        self.block_hash = "a" * 32
5805+        self.block_hash_tree = [self.block_hash for i in xrange(6)]
5806+        self.share_hash = self.block_hash
5807+        self.share_hash_chain = dict([(i, self.share_hash) for i in xrange(6)])
5808+        self.signature = "foobarbaz"
5809+        self.verification_key = "vvvvvv"
5810+        self.encprivkey = "private"
5811+        self.root_hash = self.block_hash
5812+        self.salt_hash = self.root_hash
5813+        self.salt_hash_tree = [self.salt_hash for i in xrange(6)]
5814+        self.block_hash_tree_s = self.serialize_blockhashes(self.block_hash_tree)
5815+        self.share_hash_chain_s = self.serialize_sharehashes(self.share_hash_chain)
5816+        # blockhashes and salt hashes are serialized in the same way,
5817+        # only we lop off the first element and store that in the
5818+        # header.
5819+        self.salt_hash_tree_s = self.serialize_blockhashes(self.salt_hash_tree[1:])
5820+
5821+
5822+    def tearDown(self):
5823+        self.sparent.stopService()
5824+        shutil.rmtree(self.workdir("MDMFProxies storage test server"))
5825+
5826+
5827+    def write_enabler(self, we_tag):
5828+        return hashutil.tagged_hash("we_blah", we_tag)
5829+
5830+
5831+    def renew_secret(self, tag):
5832+        return hashutil.tagged_hash("renew_blah", str(tag))
5833+
5834+
5835+    def cancel_secret(self, tag):
5836+        return hashutil.tagged_hash("cancel_blah", str(tag))
5837+
5838+
5839+    def workdir(self, name):
5840+        basedir = os.path.join("storage", "MutableServer", name)
5841+        return basedir
5842+
5843+
5844+    def create(self, name):
5845+        workdir = self.workdir(name)
5846+        ss = StorageServer(workdir, "\x00" * 20)
5847+        ss.setServiceParent(self.sparent)
5848+        return ss
5849+
5850+
5851+    def build_test_mdmf_share(self, tail_segment=False, empty=False):
5852+        # Start with the checkstring
5853+        data = struct.pack(">BQ32s",
5854+                           1,
5855+                           0,
5856+                           self.root_hash)
5857+        self.checkstring = data
5858+        # Next, the encoding parameters
5859+        if tail_segment:
5860+            data += struct.pack(">BBQQ",
5861+                                3,
5862+                                10,
5863+                                6,
5864+                                33)
5865+        elif empty:
5866+            data += struct.pack(">BBQQ",
5867+                                3,
5868+                                10,
5869+                                0,
5870+                                0)
5871+        else:
5872+            data += struct.pack(">BBQQ",
5873+                                3,
5874+                                10,
5875+                                6,
5876+                                36)
5877+        # Now we'll build the offsets.
5878+        sharedata = ""
5879+        if not tail_segment and not empty:
5880+            for i in xrange(6):
5881+                sharedata += self.salt + self.block
5882+        elif tail_segment:
5883+            for i in xrange(5):
5884+                sharedata += self.salt + self.block
5885+            sharedata += self.salt + "a"
5886+
5887+        # The encrypted private key comes after the shares + salts
5888+        offset_size = struct.calcsize(MDMFOFFSETS)
5889+        encrypted_private_key_offset = len(data) + offset_size + len(sharedata)
5890+        # The blockhashes come after the private key
5891+        blockhashes_offset = encrypted_private_key_offset + len(self.encprivkey)
5892+        # The sharehashes come after the salt hashes
5893+        sharehashes_offset = blockhashes_offset + len(self.block_hash_tree_s)
5894+        # The signature comes after the share hash chain
5895+        signature_offset = sharehashes_offset + len(self.share_hash_chain_s)
5896+        # The verification key comes after the signature
5897+        verification_offset = signature_offset + len(self.signature)
5898+        # The EOF comes after the verification key
5899+        eof_offset = verification_offset + len(self.verification_key)
5900+        data += struct.pack(MDMFOFFSETS,
5901+                            encrypted_private_key_offset,
5902+                            blockhashes_offset,
5903+                            sharehashes_offset,
5904+                            signature_offset,
5905+                            verification_offset,
5906+                            eof_offset)
5907+        self.offsets = {}
5908+        self.offsets['enc_privkey'] = encrypted_private_key_offset
5909+        self.offsets['block_hash_tree'] = blockhashes_offset
5910+        self.offsets['share_hash_chain'] = sharehashes_offset
5911+        self.offsets['signature'] = signature_offset
5912+        self.offsets['verification_key'] = verification_offset
5913+        self.offsets['EOF'] = eof_offset
5914+        # Next, we'll add in the salts and share data,
5915+        data += sharedata
5916+        # the private key,
5917+        data += self.encprivkey
5918+        # the block hash tree,
5919+        data += self.block_hash_tree_s
5920+        # the share hash chain,
5921+        data += self.share_hash_chain_s
5922+        # the signature,
5923+        data += self.signature
5924+        # and the verification key
5925+        data += self.verification_key
5926+        return data
5927+
5928+
5929+    def write_test_share_to_server(self,
5930+                                   storage_index,
5931+                                   tail_segment=False,
5932+                                   empty=False):
5933+        """
5934+        I write some data for the read tests to read to self.ss
5935+
5936+        If tail_segment=True, then I will write a share that has a
5937+        smaller tail segment than other segments.
5938+        """
5939+        write = self.ss.remote_slot_testv_and_readv_and_writev
5940+        data = self.build_test_mdmf_share(tail_segment, empty)
5941+        # Finally, we write the whole thing to the storage server in one
5942+        # pass.
5943+        testvs = [(0, 1, "eq", "")]
5944+        tws = {}
5945+        tws[0] = (testvs, [(0, data)], None)
5946+        readv = [(0, 1)]
5947+        results = write(storage_index, self.secrets, tws, readv)
5948+        self.failUnless(results[0])
5949+
5950+
5951+    def build_test_sdmf_share(self, empty=False):
5952+        if empty:
5953+            sharedata = ""
5954+        else:
5955+            sharedata = self.segment * 6
5956+        self.sharedata = sharedata
5957+        blocksize = len(sharedata) / 3
5958+        block = sharedata[:blocksize]
5959+        self.blockdata = block
5960+        prefix = struct.pack(">BQ32s16s BBQQ",
5961+                             0, # version,
5962+                             0,
5963+                             self.root_hash,
5964+                             self.salt,
5965+                             3,
5966+                             10,
5967+                             len(sharedata),
5968+                             len(sharedata),
5969+                            )
5970+        post_offset = struct.calcsize(">BQ32s16sBBQQLLLLQQ")
5971+        signature_offset = post_offset + len(self.verification_key)
5972+        sharehashes_offset = signature_offset + len(self.signature)
5973+        blockhashes_offset = sharehashes_offset + len(self.share_hash_chain_s)
5974+        sharedata_offset = blockhashes_offset + len(self.block_hash_tree_s)
5975+        encprivkey_offset = sharedata_offset + len(block)
5976+        eof_offset = encprivkey_offset + len(self.encprivkey)
5977+        offsets = struct.pack(">LLLLQQ",
5978+                              signature_offset,
5979+                              sharehashes_offset,
5980+                              blockhashes_offset,
5981+                              sharedata_offset,
5982+                              encprivkey_offset,
5983+                              eof_offset)
5984+        final_share = "".join([prefix,
5985+                           offsets,
5986+                           self.verification_key,
5987+                           self.signature,
5988+                           self.share_hash_chain_s,
5989+                           self.block_hash_tree_s,
5990+                           block,
5991+                           self.encprivkey])
5992+        self.offsets = {}
5993+        self.offsets['signature'] = signature_offset
5994+        self.offsets['share_hash_chain'] = sharehashes_offset
5995+        self.offsets['block_hash_tree'] = blockhashes_offset
5996+        self.offsets['share_data'] = sharedata_offset
5997+        self.offsets['enc_privkey'] = encprivkey_offset
5998+        self.offsets['EOF'] = eof_offset
5999+        return final_share
6000+
6001+
6002+    def write_sdmf_share_to_server(self,
6003+                                   storage_index,
6004+                                   empty=False):
6005+        # Some tests need SDMF shares to verify that we can still
6006+        # read them. This method writes one, which resembles but is not
6007+        assert self.rref
6008+        write = self.ss.remote_slot_testv_and_readv_and_writev
6009+        share = self.build_test_sdmf_share(empty)
6010+        testvs = [(0, 1, "eq", "")]
6011+        tws = {}
6012+        tws[0] = (testvs, [(0, share)], None)
6013+        readv = []
6014+        results = write(storage_index, self.secrets, tws, readv)
6015+        self.failUnless(results[0])
6016+
6017+
6018+    def test_read(self):
6019+        self.write_test_share_to_server("si1")
6020+        mr = MDMFSlotReadProxy(self.rref, "si1", 0)
6021+        # Check that every method equals what we expect it to.
6022+        d = defer.succeed(None)
6023+        def _check_block_and_salt((block, salt)):
6024+            self.failUnlessEqual(block, self.block)
6025+            self.failUnlessEqual(salt, self.salt)
6026+
6027+        for i in xrange(6):
6028+            d.addCallback(lambda ignored, i=i:
6029+                mr.get_block_and_salt(i))
6030+            d.addCallback(_check_block_and_salt)
6031+
6032+        d.addCallback(lambda ignored:
6033+            mr.get_encprivkey())
6034+        d.addCallback(lambda encprivkey:
6035+            self.failUnlessEqual(self.encprivkey, encprivkey))
6036+
6037+        d.addCallback(lambda ignored:
6038+            mr.get_blockhashes())
6039+        d.addCallback(lambda blockhashes:
6040+            self.failUnlessEqual(self.block_hash_tree, blockhashes))
6041+
6042+        d.addCallback(lambda ignored:
6043+            mr.get_sharehashes())
6044+        d.addCallback(lambda sharehashes:
6045+            self.failUnlessEqual(self.share_hash_chain, sharehashes))
6046+
6047+        d.addCallback(lambda ignored:
6048+            mr.get_signature())
6049+        d.addCallback(lambda signature:
6050+            self.failUnlessEqual(signature, self.signature))
6051+
6052+        d.addCallback(lambda ignored:
6053+            mr.get_verification_key())
6054+        d.addCallback(lambda verification_key:
6055+            self.failUnlessEqual(verification_key, self.verification_key))
6056+
6057+        d.addCallback(lambda ignored:
6058+            mr.get_seqnum())
6059+        d.addCallback(lambda seqnum:
6060+            self.failUnlessEqual(seqnum, 0))
6061+
6062+        d.addCallback(lambda ignored:
6063+            mr.get_root_hash())
6064+        d.addCallback(lambda root_hash:
6065+            self.failUnlessEqual(self.root_hash, root_hash))
6066+
6067+        d.addCallback(lambda ignored:
6068+            mr.get_seqnum())
6069+        d.addCallback(lambda seqnum:
6070+            self.failUnlessEqual(0, seqnum))
6071+
6072+        d.addCallback(lambda ignored:
6073+            mr.get_encoding_parameters())
6074+        def _check_encoding_parameters((k, n, segsize, datalen)):
6075+            self.failUnlessEqual(k, 3)
6076+            self.failUnlessEqual(n, 10)
6077+            self.failUnlessEqual(segsize, 6)
6078+            self.failUnlessEqual(datalen, 36)
6079+        d.addCallback(_check_encoding_parameters)
6080+
6081+        d.addCallback(lambda ignored:
6082+            mr.get_checkstring())
6083+        d.addCallback(lambda checkstring:
6084+            self.failUnlessEqual(checkstring, checkstring))
6085+        return d
6086+
6087+
6088+    def test_read_with_different_tail_segment_size(self):
6089+        self.write_test_share_to_server("si1", tail_segment=True)
6090+        mr = MDMFSlotReadProxy(self.rref, "si1", 0)
6091+        d = mr.get_block_and_salt(5)
6092+        def _check_tail_segment(results):
6093+            block, salt = results
6094+            self.failUnlessEqual(len(block), 1)
6095+            self.failUnlessEqual(block, "a")
6096+        d.addCallback(_check_tail_segment)
6097+        return d
6098+
6099+
6100+    def test_get_block_with_invalid_segnum(self):
6101+        self.write_test_share_to_server("si1")
6102+        mr = MDMFSlotReadProxy(self.rref, "si1", 0)
6103+        d = defer.succeed(None)
6104+        d.addCallback(lambda ignored:
6105+            self.shouldFail(LayoutInvalid, "test invalid segnum",
6106+                            None,
6107+                            mr.get_block_and_salt, 7))
6108+        return d
6109+
6110+
6111+    def test_get_encoding_parameters_first(self):
6112+        self.write_test_share_to_server("si1")
6113+        mr = MDMFSlotReadProxy(self.rref, "si1", 0)
6114+        d = mr.get_encoding_parameters()
6115+        def _check_encoding_parameters((k, n, segment_size, datalen)):
6116+            self.failUnlessEqual(k, 3)
6117+            self.failUnlessEqual(n, 10)
6118+            self.failUnlessEqual(segment_size, 6)
6119+            self.failUnlessEqual(datalen, 36)
6120+        d.addCallback(_check_encoding_parameters)
6121+        return d
6122+
6123+
6124+    def test_get_seqnum_first(self):
6125+        self.write_test_share_to_server("si1")
6126+        mr = MDMFSlotReadProxy(self.rref, "si1", 0)
6127+        d = mr.get_seqnum()
6128+        d.addCallback(lambda seqnum:
6129+            self.failUnlessEqual(seqnum, 0))
6130+        return d
6131+
6132+
6133+    def test_get_root_hash_first(self):
6134+        self.write_test_share_to_server("si1")
6135+        mr = MDMFSlotReadProxy(self.rref, "si1", 0)
6136+        d = mr.get_root_hash()
6137+        d.addCallback(lambda root_hash:
6138+            self.failUnlessEqual(root_hash, self.root_hash))
6139+        return d
6140+
6141+
6142+    def test_get_checkstring_first(self):
6143+        self.write_test_share_to_server("si1")
6144+        mr = MDMFSlotReadProxy(self.rref, "si1", 0)
6145+        d = mr.get_checkstring()
6146+        d.addCallback(lambda checkstring:
6147+            self.failUnlessEqual(checkstring, self.checkstring))
6148+        return d
6149+
6150+
6151+    def test_write_read_vectors(self):
6152+        # When writing for us, the storage server will return to us a
6153+        # read vector, along with its result. If a write fails because
6154+        # the test vectors failed, this read vector can help us to
6155+        # diagnose the problem. This test ensures that the read vector
6156+        # is working appropriately.
6157+        mw = self._make_new_mw("si1", 0)
6158+        d = defer.succeed(None)
6159+
6160+        # Write one share. This should return a checkstring of nothing,
6161+        # since there is no data there.
6162+        d.addCallback(lambda ignored:
6163+            mw.put_block(self.block, 0, self.salt))
6164+        def _check_first_write(results):
6165+            result, readvs = results
6166+            self.failUnless(result)
6167+            self.failIf(readvs)
6168+        d.addCallback(_check_first_write)
6169+        # Now, there should be a different checkstring returned when
6170+        # we write other shares
6171+        d.addCallback(lambda ignored:
6172+            mw.put_block(self.block, 1, self.salt))
6173+        def _check_next_write(results):
6174+            result, readvs = results
6175+            self.failUnless(result)
6176+            self.expected_checkstring = mw.get_checkstring()
6177+            self.failUnlessIn(0, readvs)
6178+            self.failUnlessEqual(readvs[0][0], self.expected_checkstring)
6179+        d.addCallback(_check_next_write)
6180+        # Add the other four shares
6181+        for i in xrange(2, 6):
6182+            d.addCallback(lambda ignored, i=i:
6183+                mw.put_block(self.block, i, self.salt))
6184+            d.addCallback(_check_next_write)
6185+        # Add the encrypted private key
6186+        d.addCallback(lambda ignored:
6187+            mw.put_encprivkey(self.encprivkey))
6188+        d.addCallback(_check_next_write)
6189+        # Add the block hash tree and share hash tree
6190+        d.addCallback(lambda ignored:
6191+            mw.put_blockhashes(self.block_hash_tree))
6192+        d.addCallback(_check_next_write)
6193+        d.addCallback(lambda ignored:
6194+            mw.put_sharehashes(self.share_hash_chain))
6195+        d.addCallback(_check_next_write)
6196+        # Add the root hash and the salt hash. This should change the
6197+        # checkstring, but not in a way that we'll be able to see right
6198+        # now, since the read vectors are applied before the write
6199+        # vectors.
6200+        d.addCallback(lambda ignored:
6201+            mw.put_root_hash(self.root_hash))
6202+        def _check_old_testv_after_new_one_is_written(results):
6203+            result, readvs = results
6204+            self.failUnless(result)
6205+            self.failUnlessIn(0, readvs)
6206+            self.failUnlessEqual(self.expected_checkstring,
6207+                                 readvs[0][0])
6208+            new_checkstring = mw.get_checkstring()
6209+            self.failIfEqual(new_checkstring,
6210+                             readvs[0][0])
6211+        d.addCallback(_check_old_testv_after_new_one_is_written)
6212+        # Now add the signature. This should succeed, meaning that the
6213+        # data gets written and the read vector matches what the writer
6214+        # thinks should be there.
6215+        d.addCallback(lambda ignored:
6216+            mw.put_signature(self.signature))
6217+        d.addCallback(_check_next_write)
6218+        # The checkstring remains the same for the rest of the process.
6219+        return d
6220+
6221+
6222+    def test_blockhashes_after_share_hash_chain(self):
6223+        mw = self._make_new_mw("si1", 0)
6224+        d = defer.succeed(None)
6225+        # Put everything up to and including the share hash chain
6226+        for i in xrange(6):
6227+            d.addCallback(lambda ignored, i=i:
6228+                mw.put_block(self.block, i, self.salt))
6229+        d.addCallback(lambda ignored:
6230+            mw.put_encprivkey(self.encprivkey))
6231+        d.addCallback(lambda ignored:
6232+            mw.put_blockhashes(self.block_hash_tree))
6233+        d.addCallback(lambda ignored:
6234+            mw.put_sharehashes(self.share_hash_chain))
6235+
6236+        # Now try to put the block hash tree again.
6237+        d.addCallback(lambda ignored:
6238+            self.shouldFail(LayoutInvalid, "test repeat salthashes",
6239+                            None,
6240+                            mw.put_blockhashes, self.block_hash_tree))
6241+        return d
6242+
6243+
6244+    def test_encprivkey_after_blockhashes(self):
6245+        mw = self._make_new_mw("si1", 0)
6246+        d = defer.succeed(None)
6247+        # Put everything up to and including the block hash tree
6248+        for i in xrange(6):
6249+            d.addCallback(lambda ignored, i=i:
6250+                mw.put_block(self.block, i, self.salt))
6251+        d.addCallback(lambda ignored:
6252+            mw.put_encprivkey(self.encprivkey))
6253+        d.addCallback(lambda ignored:
6254+            mw.put_blockhashes(self.block_hash_tree))
6255+        d.addCallback(lambda ignored:
6256+            self.shouldFail(LayoutInvalid, "out of order private key",
6257+                            None,
6258+                            mw.put_encprivkey, self.encprivkey))
6259+        return d
6260+
6261+
6262+    def test_share_hash_chain_after_signature(self):
6263+        mw = self._make_new_mw("si1", 0)
6264+        d = defer.succeed(None)
6265+        # Put everything up to and including the signature
6266+        for i in xrange(6):
6267+            d.addCallback(lambda ignored, i=i:
6268+                mw.put_block(self.block, i, self.salt))
6269+        d.addCallback(lambda ignored:
6270+            mw.put_encprivkey(self.encprivkey))
6271+        d.addCallback(lambda ignored:
6272+            mw.put_blockhashes(self.block_hash_tree))
6273+        d.addCallback(lambda ignored:
6274+            mw.put_sharehashes(self.share_hash_chain))
6275+        d.addCallback(lambda ignored:
6276+            mw.put_root_hash(self.root_hash))
6277+        d.addCallback(lambda ignored:
6278+            mw.put_signature(self.signature))
6279+        # Now try to put the share hash chain again. This should fail
6280+        d.addCallback(lambda ignored:
6281+            self.shouldFail(LayoutInvalid, "out of order share hash chain",
6282+                            None,
6283+                            mw.put_sharehashes, self.share_hash_chain))
6284+        return d
6285+
6286+
6287+    def test_signature_after_verification_key(self):
6288+        mw = self._make_new_mw("si1", 0)
6289+        d = defer.succeed(None)
6290+        # Put everything up to and including the verification key.
6291+        for i in xrange(6):
6292+            d.addCallback(lambda ignored, i=i:
6293+                mw.put_block(self.block, i, self.salt))
6294+        d.addCallback(lambda ignored:
6295+            mw.put_encprivkey(self.encprivkey))
6296+        d.addCallback(lambda ignored:
6297+            mw.put_blockhashes(self.block_hash_tree))
6298+        d.addCallback(lambda ignored:
6299+            mw.put_sharehashes(self.share_hash_chain))
6300+        d.addCallback(lambda ignored:
6301+            mw.put_root_hash(self.root_hash))
6302+        d.addCallback(lambda ignored:
6303+            mw.put_signature(self.signature))
6304+        d.addCallback(lambda ignored:
6305+            mw.put_verification_key(self.verification_key))
6306+        # Now try to put the signature again. This should fail
6307+        d.addCallback(lambda ignored:
6308+            self.shouldFail(LayoutInvalid, "signature after verification",
6309+                            None,
6310+                            mw.put_signature, self.signature))
6311+        return d
6312+
6313+
6314+    def test_uncoordinated_write(self):
6315+        # Make two mutable writers, both pointing to the same storage
6316+        # server, both at the same storage index, and try writing to the
6317+        # same share.
6318+        mw1 = self._make_new_mw("si1", 0)
6319+        mw2 = self._make_new_mw("si1", 0)
6320+        d = defer.succeed(None)
6321+        def _check_success(results):
6322+            result, readvs = results
6323+            self.failUnless(result)
6324+
6325+        def _check_failure(results):
6326+            result, readvs = results
6327+            self.failIf(result)
6328+
6329+        d.addCallback(lambda ignored:
6330+            mw1.put_block(self.block, 0, self.salt))
6331+        d.addCallback(_check_success)
6332+        d.addCallback(lambda ignored:
6333+            mw2.put_block(self.block, 0, self.salt))
6334+        d.addCallback(_check_failure)
6335+        return d
6336+
6337+
6338+    def test_invalid_salt_size(self):
6339+        # Salts need to be 16 bytes in size. Writes that attempt to
6340+        # write more or less than this should be rejected.
6341+        mw = self._make_new_mw("si1", 0)
6342+        invalid_salt = "a" * 17 # 17 bytes
6343+        another_invalid_salt = "b" * 15 # 15 bytes
6344+        d = defer.succeed(None)
6345+        d.addCallback(lambda ignored:
6346+            self.shouldFail(LayoutInvalid, "salt too big",
6347+                            None,
6348+                            mw.put_block, self.block, 0, invalid_salt))
6349+        d.addCallback(lambda ignored:
6350+            self.shouldFail(LayoutInvalid, "salt too small",
6351+                            None,
6352+                            mw.put_block, self.block, 0,
6353+                            another_invalid_salt))
6354+        return d
6355+
6356+
6357+    def test_write_test_vectors(self):
6358+        # If we give the write proxy a bogus test vector at
6359+        # any point during the process, it should fail to write.
6360+        mw = self._make_new_mw("si1", 0)
6361+        mw.set_checkstring("this is a lie")
6362+        # The initial write should be expecting to find the improbable
6363+        # checkstring above in place; finding nothing, it should fail.
6364+        d = defer.succeed(None)
6365+        d.addCallback(lambda ignored:
6366+            mw.put_block(self.block, 0, self.salt))
6367+        def _check_failure(results):
6368+            result, readv = results
6369+            self.failIf(result)
6370+        d.addCallback(_check_failure)
6371+        # Now set the checkstring to the empty string, which
6372+        # indicates that no share is there.
6373+        d.addCallback(lambda ignored:
6374+            mw.set_checkstring(""))
6375+        d.addCallback(lambda ignored:
6376+            mw.put_block(self.block, 0, self.salt))
6377+        def _check_success(results):
6378+            result, readv = results
6379+            self.failUnless(result)
6380+        d.addCallback(_check_success)
6381+        # Now set the checkstring to something wrong
6382+        d.addCallback(lambda ignored:
6383+            mw.set_checkstring("something wrong"))
6384+        # This should fail to do anything
6385+        d.addCallback(lambda ignored:
6386+            mw.put_block(self.block, 1, self.salt))
6387+        d.addCallback(_check_failure)
6388+        # Now set it back to what it should be.
6389+        d.addCallback(lambda ignored:
6390+            mw.set_checkstring(mw.get_checkstring()))
6391+        for i in xrange(1, 6):
6392+            d.addCallback(lambda ignored, i=i:
6393+                mw.put_block(self.block, i, self.salt))
6394+            d.addCallback(_check_success)
6395+        d.addCallback(lambda ignored:
6396+            mw.put_encprivkey(self.encprivkey))
6397+        d.addCallback(_check_success)
6398+        d.addCallback(lambda ignored:
6399+            mw.put_blockhashes(self.block_hash_tree))
6400+        d.addCallback(_check_success)
6401+        d.addCallback(lambda ignored:
6402+            mw.put_sharehashes(self.share_hash_chain))
6403+        d.addCallback(_check_success)
6404+        def _keep_old_checkstring(ignored):
6405+            self.old_checkstring = mw.get_checkstring()
6406+            mw.set_checkstring("foobarbaz")
6407+        d.addCallback(_keep_old_checkstring)
6408+        d.addCallback(lambda ignored:
6409+            mw.put_root_hash(self.root_hash))
6410+        d.addCallback(_check_failure)
6411+        d.addCallback(lambda ignored:
6412+            self.failUnlessEqual(self.old_checkstring, mw.get_checkstring()))
6413+        def _restore_old_checkstring(ignored):
6414+            mw.set_checkstring(self.old_checkstring)
6415+        d.addCallback(_restore_old_checkstring)
6416+        d.addCallback(lambda ignored:
6417+            mw.put_root_hash(self.root_hash))
6418+        d.addCallback(_check_success)
6419+        # The checkstring should have been set appropriately for us on
6420+        # the last write; if we try to change it to something else,
6421+        # that change should cause the verification key step to fail.
6422+        d.addCallback(lambda ignored:
6423+            mw.set_checkstring("something else"))
6424+        d.addCallback(lambda ignored:
6425+            mw.put_signature(self.signature))
6426+        d.addCallback(_check_failure)
6427+        d.addCallback(lambda ignored:
6428+            mw.set_checkstring(mw.get_checkstring()))
6429+        d.addCallback(lambda ignored:
6430+            mw.put_signature(self.signature))
6431+        d.addCallback(_check_success)
6432+        d.addCallback(lambda ignored:
6433+            mw.put_verification_key(self.verification_key))
6434+        d.addCallback(_check_success)
6435+        return d
6436+
6437+
6438+    def test_offset_only_set_on_success(self):
6439+        # The write proxy should be smart enough to detect when a write
6440+        # has failed, and to temper its definition of progress based on
6441+        # that.
6442+        mw = self._make_new_mw("si1", 0)
6443+        d = defer.succeed(None)
6444+        for i in xrange(1, 6):
6445+            d.addCallback(lambda ignored, i=i:
6446+                mw.put_block(self.block, i, self.salt))
6447+        def _break_checkstring(ignored):
6448+            self._old_checkstring = mw.get_checkstring()
6449+            mw.set_checkstring("foobarbaz")
6450+
6451+        def _fix_checkstring(ignored):
6452+            mw.set_checkstring(self._old_checkstring)
6453+
6454+        d.addCallback(_break_checkstring)
6455+
6456+        # Setting the encrypted private key shouldn't work now, which is
6457+        # to be expected and is tested elsewhere. We also want to make
6458+        # sure that we can't add the block hash tree after a failed
6459+        # write of this sort.
6460+        d.addCallback(lambda ignored:
6461+            mw.put_encprivkey(self.encprivkey))
6462+        d.addCallback(lambda ignored:
6463+            self.shouldFail(LayoutInvalid, "test out-of-order blockhashes",
6464+                            None,
6465+                            mw.put_blockhashes, self.block_hash_tree))
6466+        d.addCallback(_fix_checkstring)
6467+        d.addCallback(lambda ignored:
6468+            mw.put_encprivkey(self.encprivkey))
6469+        d.addCallback(_break_checkstring)
6470+        d.addCallback(lambda ignored:
6471+            mw.put_blockhashes(self.block_hash_tree))
6472+        d.addCallback(lambda ignored:
6473+            self.shouldFail(LayoutInvalid, "test out-of-order sharehashes",
6474+                            None,
6475+                            mw.put_sharehashes, self.share_hash_chain))
6476+        d.addCallback(_fix_checkstring)
6477+        d.addCallback(lambda ignored:
6478+            mw.put_blockhashes(self.block_hash_tree))
6479+        d.addCallback(_break_checkstring)
6480+        d.addCallback(lambda ignored:
6481+            mw.put_sharehashes(self.share_hash_chain))
6482+        d.addCallback(lambda ignored:
6483+            self.shouldFail(LayoutInvalid, "out-of-order root hash",
6484+                            None,
6485+                            mw.put_root_hash, self.root_hash))
6486+        d.addCallback(_fix_checkstring)
6487+        d.addCallback(lambda ignored:
6488+            mw.put_sharehashes(self.share_hash_chain))
6489+        d.addCallback(_break_checkstring)
6490+        d.addCallback(lambda ignored:
6491+            mw.put_root_hash(self.root_hash))
6492+        d.addCallback(lambda ignored:
6493+            self.shouldFail(LayoutInvalid, "out-of-order signature",
6494+                            None,
6495+                            mw.put_signature, self.signature))
6496+        d.addCallback(_fix_checkstring)
6497+        d.addCallback(lambda ignored:
6498+            mw.put_root_hash(self.root_hash))
6499+        d.addCallback(_break_checkstring)
6500+        d.addCallback(lambda ignored:
6501+            mw.put_signature(self.signature))
6502+        d.addCallback(lambda ignored:
6503+            self.shouldFail(LayoutInvalid, "out-of-order verification key",
6504+                            None,
6505+                            mw.put_verification_key,
6506+                            self.verification_key))
6507+        d.addCallback(_fix_checkstring)
6508+        d.addCallback(lambda ignored:
6509+            mw.put_signature(self.signature))
6510+        d.addCallback(_break_checkstring)
6511+        d.addCallback(lambda ignored:
6512+            mw.put_verification_key(self.verification_key))
6513+        d.addCallback(lambda ignored:
6514+            self.shouldFail(LayoutInvalid, "out-of-order finish",
6515+                            None,
6516+                            mw.finish_publishing))
6517+        return d
6518+
6519+
6520+    def serialize_blockhashes(self, blockhashes):
6521+        return "".join(blockhashes)
6522+
6523+
6524+    def serialize_sharehashes(self, sharehashes):
6525+        ret = "".join([struct.pack(">H32s", i, sharehashes[i])
6526+                        for i in sorted(sharehashes.keys())])
6527+        return ret
6528+
6529+
6530+    def test_write(self):
6531+        # This translates to a file with 6 6-byte segments, and with 2-byte
6532+        # blocks.
6533+        mw = self._make_new_mw("si1", 0)
6534+        mw2 = self._make_new_mw("si1", 1)
6535+        # Test writing some blocks.
6536+        read = self.ss.remote_slot_readv
6537+        expected_sharedata_offset = struct.calcsize(MDMFHEADER)
6538+        written_block_size = 2 + len(self.salt)
6539+        written_block = self.block + self.salt
6540+        def _check_block_write(i, share):
6541+            self.failUnlessEqual(read("si1", [share], [(expected_sharedata_offset + (i * written_block_size), written_block_size)]),
6542+                                {share: [written_block]})
6543+        d = defer.succeed(None)
6544+        for i in xrange(6):
6545+            d.addCallback(lambda ignored, i=i:
6546+                mw.put_block(self.block, i, self.salt))
6547+            d.addCallback(lambda ignored, i=i:
6548+                _check_block_write(i, 0))
6549+        # Now try the same thing, but with share 1 instead of share 0.
6550+        for i in xrange(6):
6551+            d.addCallback(lambda ignored, i=i:
6552+                mw2.put_block(self.block, i, self.salt))
6553+            d.addCallback(lambda ignored, i=i:
6554+                _check_block_write(i, 1))
6555+
6556+        # Next, we make a fake encrypted private key, and put it onto the
6557+        # storage server.
6558+        d.addCallback(lambda ignored:
6559+            mw.put_encprivkey(self.encprivkey))
6560+        expected_private_key_offset = expected_sharedata_offset + \
6561+                                      len(written_block) * 6
6562+        self.failUnlessEqual(len(self.encprivkey), 7)
6563+        d.addCallback(lambda ignored:
6564+            self.failUnlessEqual(read("si1", [0], [(expected_private_key_offset, 7)]),
6565+                                 {0: [self.encprivkey]}))
6566+
6567+        # Next, we put a fake block hash tree.
6568+        d.addCallback(lambda ignored:
6569+            mw.put_blockhashes(self.block_hash_tree))
6570+        expected_block_hash_offset = expected_private_key_offset + len(self.encprivkey)
6571+        self.failUnlessEqual(len(self.block_hash_tree_s), 32 * 6)
6572+        d.addCallback(lambda ignored:
6573+            self.failUnlessEqual(read("si1", [0], [(expected_block_hash_offset, 32 * 6)]),
6574+                                 {0: [self.block_hash_tree_s]}))
6575+
6576+        # Next, put a fake share hash chain
6577+        d.addCallback(lambda ignored:
6578+            mw.put_sharehashes(self.share_hash_chain))
6579+        expected_share_hash_offset = expected_block_hash_offset + len(self.block_hash_tree_s)
6580+        d.addCallback(lambda ignored:
6581+            self.failUnlessEqual(read("si1", [0],[(expected_share_hash_offset, (32 + 2) * 6)]),
6582+                                 {0: [self.share_hash_chain_s]}))
6583+
6584+        # Next, we put what is supposed to be the root hash of
6585+        # our share hash tree but isn't       
6586+        d.addCallback(lambda ignored:
6587+            mw.put_root_hash(self.root_hash))
6588+        # The root hash gets inserted at byte 9 (its position is in the header,
6589+        # and is fixed).
6590+        def _check(ignored):
6591+            self.failUnlessEqual(read("si1", [0], [(9, 32)]),
6592+                                 {0: [self.root_hash]})
6593+        d.addCallback(_check)
6594+
6595+        # Next, we put a signature of the header block.
6596+        d.addCallback(lambda ignored:
6597+            mw.put_signature(self.signature))
6598+        expected_signature_offset = expected_share_hash_offset + len(self.share_hash_chain_s)
6599+        self.failUnlessEqual(len(self.signature), 9)
6600+        d.addCallback(lambda ignored:
6601+            self.failUnlessEqual(read("si1", [0], [(expected_signature_offset, 9)]),
6602+                                 {0: [self.signature]}))
6603+
6604+        # Next, we put the verification key
6605+        d.addCallback(lambda ignored:
6606+            mw.put_verification_key(self.verification_key))
6607+        expected_verification_key_offset = expected_signature_offset + len(self.signature)
6608+        self.failUnlessEqual(len(self.verification_key), 6)
6609+        d.addCallback(lambda ignored:
6610+            self.failUnlessEqual(read("si1", [0], [(expected_verification_key_offset, 6)]),
6611+                                 {0: [self.verification_key]}))
6612+
6613+        def _check_signable(ignored):
6614+            # Make sure that the signable is what we think it should be.
6615+            signable = mw.get_signable()
6616+            verno, seq, roothash, k, n, segsize, datalen = \
6617+                                            struct.unpack(">BQ32sBBQQ",
6618+                                                          signable)
6619+            self.failUnlessEqual(verno, 1)
6620+            self.failUnlessEqual(seq, 0)
6621+            self.failUnlessEqual(roothash, self.root_hash)
6622+            self.failUnlessEqual(k, 3)
6623+            self.failUnlessEqual(n, 10)
6624+            self.failUnlessEqual(segsize, 6)
6625+            self.failUnlessEqual(datalen, 36)
6626+        d.addCallback(_check_signable)
6627+        # Next, we cause the offset table to be published.
6628+        d.addCallback(lambda ignored:
6629+            mw.finish_publishing())
6630+        expected_eof_offset = expected_verification_key_offset + len(self.verification_key)
6631+
6632+        def _check_offsets(ignored):
6633+            # Check the version number to make sure that it is correct.
6634+            expected_version_number = struct.pack(">B", 1)
6635+            self.failUnlessEqual(read("si1", [0], [(0, 1)]),
6636+                                 {0: [expected_version_number]})
6637+            # Check the sequence number to make sure that it is correct
6638+            expected_sequence_number = struct.pack(">Q", 0)
6639+            self.failUnlessEqual(read("si1", [0], [(1, 8)]),
6640+                                 {0: [expected_sequence_number]})
6641+            # Check that the encoding parameters (k, N, segement size, data
6642+            # length) are what they should be. These are  3, 10, 6, 36
6643+            expected_k = struct.pack(">B", 3)
6644+            self.failUnlessEqual(read("si1", [0], [(41, 1)]),
6645+                                 {0: [expected_k]})
6646+            expected_n = struct.pack(">B", 10)
6647+            self.failUnlessEqual(read("si1", [0], [(42, 1)]),
6648+                                 {0: [expected_n]})
6649+            expected_segment_size = struct.pack(">Q", 6)
6650+            self.failUnlessEqual(read("si1", [0], [(43, 8)]),
6651+                                 {0: [expected_segment_size]})
6652+            expected_data_length = struct.pack(">Q", 36)
6653+            self.failUnlessEqual(read("si1", [0], [(51, 8)]),
6654+                                 {0: [expected_data_length]})
6655+            expected_offset = struct.pack(">Q", expected_private_key_offset)
6656+            self.failUnlessEqual(read("si1", [0], [(59, 8)]),
6657+                                 {0: [expected_offset]})
6658+            expected_offset = struct.pack(">Q", expected_block_hash_offset)
6659+            self.failUnlessEqual(read("si1", [0], [(67, 8)]),
6660+                                 {0: [expected_offset]})
6661+            expected_offset = struct.pack(">Q", expected_share_hash_offset)
6662+            self.failUnlessEqual(read("si1", [0], [(75, 8)]),
6663+                                 {0: [expected_offset]})
6664+            expected_offset = struct.pack(">Q", expected_signature_offset)
6665+            self.failUnlessEqual(read("si1", [0], [(83, 8)]),
6666+                                 {0: [expected_offset]})
6667+            expected_offset = struct.pack(">Q", expected_verification_key_offset)
6668+            self.failUnlessEqual(read("si1", [0], [(91, 8)]),
6669+                                 {0: [expected_offset]})
6670+            expected_offset = struct.pack(">Q", expected_eof_offset)
6671+            self.failUnlessEqual(read("si1", [0], [(99, 8)]),
6672+                                 {0: [expected_offset]})
6673+        d.addCallback(_check_offsets)
6674+        return d
6675+
6676+    def _make_new_mw(self, si, share, datalength=36):
6677+        # This is a file of size 36 bytes. Since it has a segment
6678+        # size of 6, we know that it has 6 byte segments, which will
6679+        # be split into blocks of 2 bytes because our FEC k
6680+        # parameter is 3.
6681+        mw = MDMFSlotWriteProxy(share, self.rref, si, self.secrets, 0, 3, 10,
6682+                                6, datalength)
6683+        return mw
6684+
6685+
6686+    def test_write_rejected_with_too_many_blocks(self):
6687+        mw = self._make_new_mw("si0", 0)
6688+
6689+        # Try writing too many blocks. We should not be able to write
6690+        # more than 6
6691+        # blocks into each share.
6692+        d = defer.succeed(None)
6693+        for i in xrange(6):
6694+            d.addCallback(lambda ignored, i=i:
6695+                mw.put_block(self.block, i, self.salt))
6696+        d.addCallback(lambda ignored:
6697+            self.shouldFail(LayoutInvalid, "too many blocks",
6698+                            None,
6699+                            mw.put_block, self.block, 7, self.salt))
6700+        return d
6701+
6702+
6703+    def test_write_rejected_with_invalid_salt(self):
6704+        # Try writing an invalid salt. Salts are 16 bytes -- any more or
6705+        # less should cause an error.
6706+        mw = self._make_new_mw("si1", 0)
6707+        bad_salt = "a" * 17 # 17 bytes
6708+        d = defer.succeed(None)
6709+        d.addCallback(lambda ignored:
6710+            self.shouldFail(LayoutInvalid, "test_invalid_salt",
6711+                            None, mw.put_block, self.block, 7, bad_salt))
6712+        return d
6713+
6714+
6715+    def test_write_rejected_with_invalid_root_hash(self):
6716+        # Try writing an invalid root hash. This should be SHA256d, and
6717+        # 32 bytes long as a result.
6718+        mw = self._make_new_mw("si2", 0)
6719+        # 17 bytes != 32 bytes
6720+        invalid_root_hash = "a" * 17
6721+        d = defer.succeed(None)
6722+        # Before this test can work, we need to put some blocks + salts,
6723+        # a block hash tree, and a share hash tree. Otherwise, we'll see
6724+        # failures that match what we are looking for, but are caused by
6725+        # the constraints imposed on operation ordering.
6726+        for i in xrange(6):
6727+            d.addCallback(lambda ignored, i=i:
6728+                mw.put_block(self.block, i, self.salt))
6729+        d.addCallback(lambda ignored:
6730+            mw.put_encprivkey(self.encprivkey))
6731+        d.addCallback(lambda ignored:
6732+            mw.put_blockhashes(self.block_hash_tree))
6733+        d.addCallback(lambda ignored:
6734+            mw.put_sharehashes(self.share_hash_chain))
6735+        d.addCallback(lambda ignored:
6736+            self.shouldFail(LayoutInvalid, "invalid root hash",
6737+                            None, mw.put_root_hash, invalid_root_hash))
6738+        return d
6739+
6740+
6741+    def test_write_rejected_with_invalid_blocksize(self):
6742+        # The blocksize implied by the writer that we get from
6743+        # _make_new_mw is 2bytes -- any more or any less than this
6744+        # should be cause for failure, unless it is the tail segment, in
6745+        # which case it may not be failure.
6746+        invalid_block = "a"
6747+        mw = self._make_new_mw("si3", 0, 33) # implies a tail segment with
6748+                                             # one byte blocks
6749+        # 1 bytes != 2 bytes
6750+        d = defer.succeed(None)
6751+        d.addCallback(lambda ignored, invalid_block=invalid_block:
6752+            self.shouldFail(LayoutInvalid, "test blocksize too small",
6753+                            None, mw.put_block, invalid_block, 0,
6754+                            self.salt))
6755+        invalid_block = invalid_block * 3
6756+        # 3 bytes != 2 bytes
6757+        d.addCallback(lambda ignored:
6758+            self.shouldFail(LayoutInvalid, "test blocksize too large",
6759+                            None,
6760+                            mw.put_block, invalid_block, 0, self.salt))
6761+        for i in xrange(5):
6762+            d.addCallback(lambda ignored, i=i:
6763+                mw.put_block(self.block, i, self.salt))
6764+        # Try to put an invalid tail segment
6765+        d.addCallback(lambda ignored:
6766+            self.shouldFail(LayoutInvalid, "test invalid tail segment",
6767+                            None,
6768+                            mw.put_block, self.block, 5, self.salt))
6769+        valid_block = "a"
6770+        d.addCallback(lambda ignored:
6771+            mw.put_block(valid_block, 5, self.salt))
6772+        return d
6773+
6774+
6775+    def test_write_enforces_order_constraints(self):
6776+        # We require that the MDMFSlotWriteProxy be interacted with in a
6777+        # specific way.
6778+        # That way is:
6779+        # 0: __init__
6780+        # 1: write blocks and salts
6781+        # 2: Write the encrypted private key
6782+        # 3: Write the block hashes
6783+        # 4: Write the share hashes
6784+        # 5: Write the root hash and salt hash
6785+        # 6: Write the signature and verification key
6786+        # 7: Write the file.
6787+        #
6788+        # Some of these can be performed out-of-order, and some can't.
6789+        # The dependencies that I want to test here are:
6790+        #  - Private key before block hashes
6791+        #  - share hashes and block hashes before root hash
6792+        #  - root hash before signature
6793+        #  - signature before verification key
6794+        mw0 = self._make_new_mw("si0", 0)
6795+        # Write some shares
6796+        d = defer.succeed(None)
6797+        for i in xrange(6):
6798+            d.addCallback(lambda ignored, i=i:
6799+                mw0.put_block(self.block, i, self.salt))
6800+        # Try to write the block hashes before writing the encrypted
6801+        # private key
6802+        d.addCallback(lambda ignored:
6803+            self.shouldFail(LayoutInvalid, "block hashes before key",
6804+                            None, mw0.put_blockhashes,
6805+                            self.block_hash_tree))
6806+
6807+        # Write the private key.
6808+        d.addCallback(lambda ignored:
6809+            mw0.put_encprivkey(self.encprivkey))
6810+
6811+
6812+        # Try to write the share hash chain without writing the block
6813+        # hash tree
6814+        d.addCallback(lambda ignored:
6815+            self.shouldFail(LayoutInvalid, "share hash chain before "
6816+                                           "salt hash tree",
6817+                            None,
6818+                            mw0.put_sharehashes, self.share_hash_chain))
6819+
6820+        # Try to write the root hash and without writing either the
6821+        # block hashes or the or the share hashes
6822+        d.addCallback(lambda ignored:
6823+            self.shouldFail(LayoutInvalid, "root hash before share hashes",
6824+                            None,
6825+                            mw0.put_root_hash, self.root_hash))
6826+
6827+        # Now write the block hashes and try again
6828+        d.addCallback(lambda ignored:
6829+            mw0.put_blockhashes(self.block_hash_tree))
6830+
6831+        d.addCallback(lambda ignored:
6832+            self.shouldFail(LayoutInvalid, "root hash before share hashes",
6833+                            None, mw0.put_root_hash, self.root_hash))
6834+
6835+        # We haven't yet put the root hash on the share, so we shouldn't
6836+        # be able to sign it.
6837+        d.addCallback(lambda ignored:
6838+            self.shouldFail(LayoutInvalid, "signature before root hash",
6839+                            None, mw0.put_signature, self.signature))
6840+
6841+        d.addCallback(lambda ignored:
6842+            self.failUnlessRaises(LayoutInvalid, mw0.get_signable))
6843+
6844+        # ..and, since that fails, we also shouldn't be able to put the
6845+        # verification key.
6846+        d.addCallback(lambda ignored:
6847+            self.shouldFail(LayoutInvalid, "key before signature",
6848+                            None, mw0.put_verification_key,
6849+                            self.verification_key))
6850+
6851+        # Now write the share hashes.
6852+        d.addCallback(lambda ignored:
6853+            mw0.put_sharehashes(self.share_hash_chain))
6854+        # We should be able to write the root hash now too
6855+        d.addCallback(lambda ignored:
6856+            mw0.put_root_hash(self.root_hash))
6857+
6858+        # We should still be unable to put the verification key
6859+        d.addCallback(lambda ignored:
6860+            self.shouldFail(LayoutInvalid, "key before signature",
6861+                            None, mw0.put_verification_key,
6862+                            self.verification_key))
6863+
6864+        d.addCallback(lambda ignored:
6865+            mw0.put_signature(self.signature))
6866+
6867+        # We shouldn't be able to write the offsets to the remote server
6868+        # until the offset table is finished; IOW, until we have written
6869+        # the verification key.
6870+        d.addCallback(lambda ignored:
6871+            self.shouldFail(LayoutInvalid, "offsets before verification key",
6872+                            None,
6873+                            mw0.finish_publishing))
6874+
6875+        d.addCallback(lambda ignored:
6876+            mw0.put_verification_key(self.verification_key))
6877+        return d
6878+
6879+
6880+    def test_end_to_end(self):
6881+        mw = self._make_new_mw("si1", 0)
6882+        # Write a share using the mutable writer, and make sure that the
6883+        # reader knows how to read everything back to us.
6884+        d = defer.succeed(None)
6885+        for i in xrange(6):
6886+            d.addCallback(lambda ignored, i=i:
6887+                mw.put_block(self.block, i, self.salt))
6888+        d.addCallback(lambda ignored:
6889+            mw.put_encprivkey(self.encprivkey))
6890+        d.addCallback(lambda ignored:
6891+            mw.put_blockhashes(self.block_hash_tree))
6892+        d.addCallback(lambda ignored:
6893+            mw.put_sharehashes(self.share_hash_chain))
6894+        d.addCallback(lambda ignored:
6895+            mw.put_root_hash(self.root_hash))
6896+        d.addCallback(lambda ignored:
6897+            mw.put_signature(self.signature))
6898+        d.addCallback(lambda ignored:
6899+            mw.put_verification_key(self.verification_key))
6900+        d.addCallback(lambda ignored:
6901+            mw.finish_publishing())
6902+
6903+        mr = MDMFSlotReadProxy(self.rref, "si1", 0)
6904+        def _check_block_and_salt((block, salt)):
6905+            self.failUnlessEqual(block, self.block)
6906+            self.failUnlessEqual(salt, self.salt)
6907+
6908+        for i in xrange(6):
6909+            d.addCallback(lambda ignored, i=i:
6910+                mr.get_block_and_salt(i))
6911+            d.addCallback(_check_block_and_salt)
6912+
6913+        d.addCallback(lambda ignored:
6914+            mr.get_encprivkey())
6915+        d.addCallback(lambda encprivkey:
6916+            self.failUnlessEqual(self.encprivkey, encprivkey))
6917+
6918+        d.addCallback(lambda ignored:
6919+            mr.get_blockhashes())
6920+        d.addCallback(lambda blockhashes:
6921+            self.failUnlessEqual(self.block_hash_tree, blockhashes))
6922+
6923+        d.addCallback(lambda ignored:
6924+            mr.get_sharehashes())
6925+        d.addCallback(lambda sharehashes:
6926+            self.failUnlessEqual(self.share_hash_chain, sharehashes))
6927+
6928+        d.addCallback(lambda ignored:
6929+            mr.get_signature())
6930+        d.addCallback(lambda signature:
6931+            self.failUnlessEqual(signature, self.signature))
6932+
6933+        d.addCallback(lambda ignored:
6934+            mr.get_verification_key())
6935+        d.addCallback(lambda verification_key:
6936+            self.failUnlessEqual(verification_key, self.verification_key))
6937+
6938+        d.addCallback(lambda ignored:
6939+            mr.get_seqnum())
6940+        d.addCallback(lambda seqnum:
6941+            self.failUnlessEqual(seqnum, 0))
6942+
6943+        d.addCallback(lambda ignored:
6944+            mr.get_root_hash())
6945+        d.addCallback(lambda root_hash:
6946+            self.failUnlessEqual(self.root_hash, root_hash))
6947+
6948+        d.addCallback(lambda ignored:
6949+            mr.get_encoding_parameters())
6950+        def _check_encoding_parameters((k, n, segsize, datalen)):
6951+            self.failUnlessEqual(k, 3)
6952+            self.failUnlessEqual(n, 10)
6953+            self.failUnlessEqual(segsize, 6)
6954+            self.failUnlessEqual(datalen, 36)
6955+        d.addCallback(_check_encoding_parameters)
6956+
6957+        d.addCallback(lambda ignored:
6958+            mr.get_checkstring())
6959+        d.addCallback(lambda checkstring:
6960+            self.failUnlessEqual(checkstring, mw.get_checkstring()))
6961+        return d
6962+
6963+
6964+    def test_is_sdmf(self):
6965+        # The MDMFSlotReadProxy should also know how to read SDMF files,
6966+        # since it will encounter them on the grid. Callers use the
6967+        # is_sdmf method to test this.
6968+        self.write_sdmf_share_to_server("si1")
6969+        mr = MDMFSlotReadProxy(self.rref, "si1", 0)
6970+        d = mr.is_sdmf()
6971+        d.addCallback(lambda issdmf:
6972+            self.failUnless(issdmf))
6973+        return d
6974+
6975+
6976+    def test_reads_sdmf(self):
6977+        # The slot read proxy should, naturally, know how to tell us
6978+        # about data in the SDMF format
6979+        self.write_sdmf_share_to_server("si1")
6980+        mr = MDMFSlotReadProxy(self.rref, "si1", 0)
6981+        d = defer.succeed(None)
6982+        d.addCallback(lambda ignored:
6983+            mr.is_sdmf())
6984+        d.addCallback(lambda issdmf:
6985+            self.failUnless(issdmf))
6986+
6987+        # What do we need to read?
6988+        #  - The sharedata
6989+        #  - The salt
6990+        d.addCallback(lambda ignored:
6991+            mr.get_block_and_salt(0))
6992+        def _check_block_and_salt(results):
6993+            block, salt = results
6994+            # Our original file is 36 bytes long. Then each share is 12
6995+            # bytes in size. The share is composed entirely of the
6996+            # letter a. self.block contains 2 as, so 6 * self.block is
6997+            # what we are looking for.
6998+            self.failUnlessEqual(block, self.block * 6)
6999+            self.failUnlessEqual(salt, self.salt)
7000+        d.addCallback(_check_block_and_salt)
7001+
7002+        #  - The blockhashes
7003+        d.addCallback(lambda ignored:
7004+            mr.get_blockhashes())
7005+        d.addCallback(lambda blockhashes:
7006+            self.failUnlessEqual(self.block_hash_tree,
7007+                                 blockhashes,
7008+                                 blockhashes))
7009+        #  - The sharehashes
7010+        d.addCallback(lambda ignored:
7011+            mr.get_sharehashes())
7012+        d.addCallback(lambda sharehashes:
7013+            self.failUnlessEqual(self.share_hash_chain,
7014+                                 sharehashes))
7015+        #  - The keys
7016+        d.addCallback(lambda ignored:
7017+            mr.get_encprivkey())
7018+        d.addCallback(lambda encprivkey:
7019+            self.failUnlessEqual(encprivkey, self.encprivkey, encprivkey))
7020+        d.addCallback(lambda ignored:
7021+            mr.get_verification_key())
7022+        d.addCallback(lambda verification_key:
7023+            self.failUnlessEqual(verification_key,
7024+                                 self.verification_key,
7025+                                 verification_key))
7026+        #  - The signature
7027+        d.addCallback(lambda ignored:
7028+            mr.get_signature())
7029+        d.addCallback(lambda signature:
7030+            self.failUnlessEqual(signature, self.signature, signature))
7031+
7032+        #  - The sequence number
7033+        d.addCallback(lambda ignored:
7034+            mr.get_seqnum())
7035+        d.addCallback(lambda seqnum:
7036+            self.failUnlessEqual(seqnum, 0, seqnum))
7037+
7038+        #  - The root hash
7039+        d.addCallback(lambda ignored:
7040+            mr.get_root_hash())
7041+        d.addCallback(lambda root_hash:
7042+            self.failUnlessEqual(root_hash, self.root_hash, root_hash))
7043+        return d
7044+
7045+
7046+    def test_only_reads_one_segment_sdmf(self):
7047+        # SDMF shares have only one segment, so it doesn't make sense to
7048+        # read more segments than that. The reader should know this and
7049+        # complain if we try to do that.
7050+        self.write_sdmf_share_to_server("si1")
7051+        mr = MDMFSlotReadProxy(self.rref, "si1", 0)
7052+        d = defer.succeed(None)
7053+        d.addCallback(lambda ignored:
7054+            mr.is_sdmf())
7055+        d.addCallback(lambda issdmf:
7056+            self.failUnless(issdmf))
7057+        d.addCallback(lambda ignored:
7058+            self.shouldFail(LayoutInvalid, "test bad segment",
7059+                            None,
7060+                            mr.get_block_and_salt, 1))
7061+        return d
7062+
7063+
7064+    def test_read_with_prefetched_mdmf_data(self):
7065+        # The MDMFSlotReadProxy will prefill certain fields if you pass
7066+        # it data that you have already fetched. This is useful for
7067+        # cases like the Servermap, which prefetches ~2kb of data while
7068+        # finding out which shares are on the remote peer so that it
7069+        # doesn't waste round trips.
7070+        mdmf_data = self.build_test_mdmf_share()
7071+        self.write_test_share_to_server("si1")
7072+        def _make_mr(ignored, length):
7073+            mr = MDMFSlotReadProxy(self.rref, "si1", 0, mdmf_data[:length])
7074+            return mr
7075+
7076+        d = defer.succeed(None)
7077+        # This should be enough to fill in both the encoding parameters
7078+        # and the table of offsets, which will complete the version
7079+        # information tuple.
7080+        d.addCallback(_make_mr, 107)
7081+        d.addCallback(lambda mr:
7082+            mr.get_verinfo())
7083+        def _check_verinfo(verinfo):
7084+            self.failUnless(verinfo)
7085+            self.failUnlessEqual(len(verinfo), 9)
7086+            (seqnum,
7087+             root_hash,
7088+             salt_hash,
7089+             segsize,
7090+             datalen,
7091+             k,
7092+             n,
7093+             prefix,
7094+             offsets) = verinfo
7095+            self.failUnlessEqual(seqnum, 0)
7096+            self.failUnlessEqual(root_hash, self.root_hash)
7097+            self.failUnlessEqual(segsize, 6)
7098+            self.failUnlessEqual(datalen, 36)
7099+            self.failUnlessEqual(k, 3)
7100+            self.failUnlessEqual(n, 10)
7101+            expected_prefix = struct.pack(MDMFSIGNABLEHEADER,
7102+                                          1,
7103+                                          seqnum,
7104+                                          root_hash,
7105+                                          k,
7106+                                          n,
7107+                                          segsize,
7108+                                          datalen)
7109+            self.failUnlessEqual(expected_prefix, prefix)
7110+            self.failUnlessEqual(self.rref.read_count, 0)
7111+        d.addCallback(_check_verinfo)
7112+        # This is not enough data to read a block and a share, so the
7113+        # wrapper should attempt to read this from the remote server.
7114+        d.addCallback(_make_mr, 107)
7115+        d.addCallback(lambda mr:
7116+            mr.get_block_and_salt(0))
7117+        def _check_block_and_salt((block, salt)):
7118+            self.failUnlessEqual(block, self.block)
7119+            self.failUnlessEqual(salt, self.salt)
7120+            self.failUnlessEqual(self.rref.read_count, 1)
7121+        # This should be enough data to read one block.
7122+        d.addCallback(_make_mr, 249)
7123+        d.addCallback(lambda mr:
7124+            mr.get_block_and_salt(0))
7125+        d.addCallback(_check_block_and_salt)
7126+        return d
7127+
7128+
7129+    def test_read_with_prefetched_sdmf_data(self):
7130+        sdmf_data = self.build_test_sdmf_share()
7131+        self.write_sdmf_share_to_server("si1")
7132+        def _make_mr(ignored, length):
7133+            mr = MDMFSlotReadProxy(self.rref, "si1", 0, sdmf_data[:length])
7134+            return mr
7135+
7136+        d = defer.succeed(None)
7137+        # This should be enough to get us the encoding parameters,
7138+        # offset table, and everything else we need to build a verinfo
7139+        # string.
7140+        d.addCallback(_make_mr, 107)
7141+        d.addCallback(lambda mr:
7142+            mr.get_verinfo())
7143+        def _check_verinfo(verinfo):
7144+            self.failUnless(verinfo)
7145+            self.failUnlessEqual(len(verinfo), 9)
7146+            (seqnum,
7147+             root_hash,
7148+             salt,
7149+             segsize,
7150+             datalen,
7151+             k,
7152+             n,
7153+             prefix,
7154+             offsets) = verinfo
7155+            self.failUnlessEqual(seqnum, 0)
7156+            self.failUnlessEqual(root_hash, self.root_hash)
7157+            self.failUnlessEqual(salt, self.salt)
7158+            self.failUnlessEqual(segsize, 36)
7159+            self.failUnlessEqual(datalen, 36)
7160+            self.failUnlessEqual(k, 3)
7161+            self.failUnlessEqual(n, 10)
7162+            expected_prefix = struct.pack(SIGNED_PREFIX,
7163+                                          0,
7164+                                          seqnum,
7165+                                          root_hash,
7166+                                          salt,
7167+                                          k,
7168+                                          n,
7169+                                          segsize,
7170+                                          datalen)
7171+            self.failUnlessEqual(expected_prefix, prefix)
7172+            self.failUnlessEqual(self.rref.read_count, 0)
7173+        d.addCallback(_check_verinfo)
7174+        # This shouldn't be enough to read any share data.
7175+        d.addCallback(_make_mr, 107)
7176+        d.addCallback(lambda mr:
7177+            mr.get_block_and_salt(0))
7178+        def _check_block_and_salt((block, salt)):
7179+            self.failUnlessEqual(block, self.block * 6)
7180+            self.failUnlessEqual(salt, self.salt)
7181+            # TODO: Fix the read routine so that it reads only the data
7182+            #       that it has cached if it can't read all of it.
7183+            self.failUnlessEqual(self.rref.read_count, 2)
7184+
7185+        # This should be enough to read share data.
7186+        d.addCallback(_make_mr, self.offsets['share_data'])
7187+        d.addCallback(lambda mr:
7188+            mr.get_block_and_salt(0))
7189+        d.addCallback(_check_block_and_salt)
7190+        return d
7191+
7192+
7193+    def test_read_with_empty_mdmf_file(self):
7194+        # Some tests upload a file with no contents to test things
7195+        # unrelated to the actual handling of the content of the file.
7196+        # The reader should behave intelligently in these cases.
7197+        self.write_test_share_to_server("si1", empty=True)
7198+        mr = MDMFSlotReadProxy(self.rref, "si1", 0)
7199+        # We should be able to get the encoding parameters, and they
7200+        # should be correct.
7201+        d = defer.succeed(None)
7202+        d.addCallback(lambda ignored:
7203+            mr.get_encoding_parameters())
7204+        def _check_encoding_parameters(params):
7205+            self.failUnlessEqual(len(params), 4)
7206+            k, n, segsize, datalen = params
7207+            self.failUnlessEqual(k, 3)
7208+            self.failUnlessEqual(n, 10)
7209+            self.failUnlessEqual(segsize, 0)
7210+            self.failUnlessEqual(datalen, 0)
7211+        d.addCallback(_check_encoding_parameters)
7212+
7213+        # We should not be able to fetch a block, since there are no
7214+        # blocks to fetch
7215+        d.addCallback(lambda ignored:
7216+            self.shouldFail(LayoutInvalid, "get block on empty file",
7217+                            None,
7218+                            mr.get_block_and_salt, 0))
7219+        return d
7220+
7221+
7222+    def test_read_with_empty_sdmf_file(self):
7223+        self.write_sdmf_share_to_server("si1", empty=True)
7224+        mr = MDMFSlotReadProxy(self.rref, "si1", 0)
7225+        # We should be able to get the encoding parameters, and they
7226+        # should be correct
7227+        d = defer.succeed(None)
7228+        d.addCallback(lambda ignored:
7229+            mr.get_encoding_parameters())
7230+        def _check_encoding_parameters(params):
7231+            self.failUnlessEqual(len(params), 4)
7232+            k, n, segsize, datalen = params
7233+            self.failUnlessEqual(k, 3)
7234+            self.failUnlessEqual(n, 10)
7235+            self.failUnlessEqual(segsize, 0)
7236+            self.failUnlessEqual(datalen, 0)
7237+        d.addCallback(_check_encoding_parameters)
7238+
7239+        # It does not make sense to get a block in this format, so we
7240+        # should not be able to.
7241+        d.addCallback(lambda ignored:
7242+            self.shouldFail(LayoutInvalid, "get block on an empty file",
7243+                            None,
7244+                            mr.get_block_and_salt, 0))
7245+        return d
7246+
7247+
7248+    def test_verinfo_with_sdmf_file(self):
7249+        self.write_sdmf_share_to_server("si1")
7250+        mr = MDMFSlotReadProxy(self.rref, "si1", 0)
7251+        # We should be able to get the version information.
7252+        d = defer.succeed(None)
7253+        d.addCallback(lambda ignored:
7254+            mr.get_verinfo())
7255+        def _check_verinfo(verinfo):
7256+            self.failUnless(verinfo)
7257+            self.failUnlessEqual(len(verinfo), 9)
7258+            (seqnum,
7259+             root_hash,
7260+             salt,
7261+             segsize,
7262+             datalen,
7263+             k,
7264+             n,
7265+             prefix,
7266+             offsets) = verinfo
7267+            self.failUnlessEqual(seqnum, 0)
7268+            self.failUnlessEqual(root_hash, self.root_hash)
7269+            self.failUnlessEqual(salt, self.salt)
7270+            self.failUnlessEqual(segsize, 36)
7271+            self.failUnlessEqual(datalen, 36)
7272+            self.failUnlessEqual(k, 3)
7273+            self.failUnlessEqual(n, 10)
7274+            expected_prefix = struct.pack(">BQ32s16s BBQQ",
7275+                                          0,
7276+                                          seqnum,
7277+                                          root_hash,
7278+                                          salt,
7279+                                          k,
7280+                                          n,
7281+                                          segsize,
7282+                                          datalen)
7283+            self.failUnlessEqual(prefix, expected_prefix)
7284+            self.failUnlessEqual(offsets, self.offsets)
7285+        d.addCallback(_check_verinfo)
7286+        return d
7287+
7288+
7289+    def test_verinfo_with_mdmf_file(self):
7290+        self.write_test_share_to_server("si1")
7291+        mr = MDMFSlotReadProxy(self.rref, "si1", 0)
7292+        d = defer.succeed(None)
7293+        d.addCallback(lambda ignored:
7294+            mr.get_verinfo())
7295+        def _check_verinfo(verinfo):
7296+            self.failUnless(verinfo)
7297+            self.failUnlessEqual(len(verinfo), 9)
7298+            (seqnum,
7299+             root_hash,
7300+             IV,
7301+             segsize,
7302+             datalen,
7303+             k,
7304+             n,
7305+             prefix,
7306+             offsets) = verinfo
7307+            self.failUnlessEqual(seqnum, 0)
7308+            self.failUnlessEqual(root_hash, self.root_hash)
7309+            self.failIf(IV)
7310+            self.failUnlessEqual(segsize, 6)
7311+            self.failUnlessEqual(datalen, 36)
7312+            self.failUnlessEqual(k, 3)
7313+            self.failUnlessEqual(n, 10)
7314+            expected_prefix = struct.pack(">BQ32s BBQQ",
7315+                                          1,
7316+                                          seqnum,
7317+                                          root_hash,
7318+                                          k,
7319+                                          n,
7320+                                          segsize,
7321+                                          datalen)
7322+            self.failUnlessEqual(prefix, expected_prefix)
7323+            self.failUnlessEqual(offsets, self.offsets)
7324+        d.addCallback(_check_verinfo)
7325+        return d
7326+
7327+
7328+    def test_reader_queue(self):
7329+        self.write_test_share_to_server('si1')
7330+        mr = MDMFSlotReadProxy(self.rref, "si1", 0)
7331+        d1 = mr.get_block_and_salt(0, queue=True)
7332+        d2 = mr.get_blockhashes(queue=True)
7333+        d3 = mr.get_sharehashes(queue=True)
7334+        d4 = mr.get_signature(queue=True)
7335+        d5 = mr.get_verification_key(queue=True)
7336+        dl = defer.DeferredList([d1, d2, d3, d4, d5])
7337+        mr.flush()
7338+        def _print(results):
7339+            self.failUnlessEqual(len(results), 5)
7340+            # We have one read for version information and offsets, and
7341+            # one for everything else.
7342+            self.failUnlessEqual(self.rref.read_count, 2)
7343+            block, salt = results[0][1] # results[0] is a boolean that says
7344+                                           # whether or not the operation
7345+                                           # worked.
7346+            self.failUnlessEqual(self.block, block)
7347+            self.failUnlessEqual(self.salt, salt)
7348+
7349+            blockhashes = results[1][1]
7350+            self.failUnlessEqual(self.block_hash_tree, blockhashes)
7351+
7352+            sharehashes = results[2][1]
7353+            self.failUnlessEqual(self.share_hash_chain, sharehashes)
7354+
7355+            signature = results[3][1]
7356+            self.failUnlessEqual(self.signature, signature)
7357+
7358+            verification_key = results[4][1]
7359+            self.failUnlessEqual(self.verification_key, verification_key)
7360+        dl.addCallback(_print)
7361+        return dl
7362+
7363+
7364+    def test_sdmf_writer(self):
7365+        # Go through the motions of writing an SDMF share to the storage
7366+        # server. Then read the storage server to see that the share got
7367+        # written in the way that we think it should have.
7368+
7369+        # We do this first so that the necessary instance variables get
7370+        # set the way we want them for the tests below.
7371+        data = self.build_test_sdmf_share()
7372+        sdmfr = SDMFSlotWriteProxy(0,
7373+                                   self.rref,
7374+                                   "si1",
7375+                                   self.secrets,
7376+                                   0, 3, 10, 36, 36)
7377+        # Put the block and salt.
7378+        sdmfr.put_block(self.blockdata, 0, self.salt)
7379+
7380+        # Put the encprivkey
7381+        sdmfr.put_encprivkey(self.encprivkey)
7382+
7383+        # Put the block and share hash chains
7384+        sdmfr.put_blockhashes(self.block_hash_tree)
7385+        sdmfr.put_sharehashes(self.share_hash_chain)
7386+        sdmfr.put_root_hash(self.root_hash)
7387+
7388+        # Put the signature
7389+        sdmfr.put_signature(self.signature)
7390+
7391+        # Put the verification key
7392+        sdmfr.put_verification_key(self.verification_key)
7393+
7394+        # Now check to make sure that nothing has been written yet.
7395+        self.failUnlessEqual(self.rref.write_count, 0)
7396+
7397+        # Now finish publishing
7398+        d = sdmfr.finish_publishing()
7399+        def _then(ignored):
7400+            self.failUnlessEqual(self.rref.write_count, 1)
7401+            read = self.ss.remote_slot_readv
7402+            self.failUnlessEqual(read("si1", [0], [(0, len(data))]),
7403+                                 {0: [data]})
7404+        d.addCallback(_then)
7405+        return d
7406+
7407+
7408+    def test_sdmf_writer_preexisting_share(self):
7409+        data = self.build_test_sdmf_share()
7410+        self.write_sdmf_share_to_server("si1")
7411+
7412+        # Now there is a share on the storage server. To successfully
7413+        # write, we need to set the checkstring correctly. When we
7414+        # don't, no write should occur.
7415+        sdmfw = SDMFSlotWriteProxy(0,
7416+                                   self.rref,
7417+                                   "si1",
7418+                                   self.secrets,
7419+                                   1, 3, 10, 36, 36)
7420+        sdmfw.put_block(self.blockdata, 0, self.salt)
7421+
7422+        # Put the encprivkey
7423+        sdmfw.put_encprivkey(self.encprivkey)
7424+
7425+        # Put the block and share hash chains
7426+        sdmfw.put_blockhashes(self.block_hash_tree)
7427+        sdmfw.put_sharehashes(self.share_hash_chain)
7428+
7429+        # Put the root hash
7430+        sdmfw.put_root_hash(self.root_hash)
7431+
7432+        # Put the signature
7433+        sdmfw.put_signature(self.signature)
7434+
7435+        # Put the verification key
7436+        sdmfw.put_verification_key(self.verification_key)
7437+
7438+        # We shouldn't have a checkstring yet
7439+        self.failUnlessEqual(sdmfw.get_checkstring(), "")
7440+
7441+        d = sdmfw.finish_publishing()
7442+        def _then(results):
7443+            self.failIf(results[0])
7444+            # this is the correct checkstring
7445+            self._expected_checkstring = results[1][0][0]
7446+            return self._expected_checkstring
7447+
7448+        d.addCallback(_then)
7449+        d.addCallback(sdmfw.set_checkstring)
7450+        d.addCallback(lambda ignored:
7451+            sdmfw.get_checkstring())
7452+        d.addCallback(lambda checkstring:
7453+            self.failUnlessEqual(checkstring, self._expected_checkstring))
7454+        d.addCallback(lambda ignored:
7455+            sdmfw.finish_publishing())
7456+        def _then_again(results):
7457+            self.failUnless(results[0])
7458+            read = self.ss.remote_slot_readv
7459+            self.failUnlessEqual(read("si1", [0], [(1, 8)]),
7460+                                 {0: [struct.pack(">Q", 1)]})
7461+            self.failUnlessEqual(read("si1", [0], [(9, len(data) - 9)]),
7462+                                 {0: [data[9:]]})
7463+        d.addCallback(_then_again)
7464+        return d
7465+
7466+
7467 class Stats(unittest.TestCase):
7468 
7469     def setUp(self):
7470}
7471[mutable/publish.py: cleanup + simplification
7472Kevan Carstensen <kevan@isnotajoke.com>**20100702225554
7473 Ignore-this: 36a58424ceceffb1ddc55cc5934399e2
7474] {
7475hunk ./src/allmydata/mutable/publish.py 19
7476      UncoordinatedWriteError, NotEnoughServersError
7477 from allmydata.mutable.servermap import ServerMap
7478 from allmydata.mutable.layout import pack_prefix, pack_share, unpack_header, pack_checkstring, \
7479-     unpack_checkstring, SIGNED_PREFIX, MDMFSlotWriteProxy
7480+     unpack_checkstring, SIGNED_PREFIX, MDMFSlotWriteProxy, \
7481+     SDMFSlotWriteProxy
7482 
7483 KiB = 1024
7484 DEFAULT_MAX_SEGMENT_SIZE = 128 * KiB
7485hunk ./src/allmydata/mutable/publish.py 24
7486+PUSHING_BLOCKS_STATE = 0
7487+PUSHING_EVERYTHING_ELSE_STATE = 1
7488+DONE_STATE = 2
7489 
7490 class PublishStatus:
7491     implements(IPublishStatus)
7492hunk ./src/allmydata/mutable/publish.py 229
7493 
7494         self.bad_share_checkstrings = {}
7495 
7496+        # This is set at the last step of the publishing process.
7497+        self.versioninfo = ""
7498+
7499         # we use the servermap to populate the initial goal: this way we will
7500         # try to update each existing share in place.
7501         for (peerid, shnum) in self._servermap.servermap:
7502hunk ./src/allmydata/mutable/publish.py 245
7503             self.bad_share_checkstrings[key] = old_checkstring
7504             self.connections[peerid] = self._servermap.connections[peerid]
7505 
7506-        # Now, the process dovetails -- if this is an SDMF file, we need
7507-        # to write an SDMF file. Otherwise, we need to write an MDMF
7508-        # file.
7509-        if self._version == MDMF_VERSION:
7510-            return self._publish_mdmf()
7511-        else:
7512-            return self._publish_sdmf()
7513-        #return self.done_deferred
7514-
7515-    def _publish_mdmf(self):
7516-        # Next, we find homes for all of the shares that we don't have
7517-        # homes for yet.
7518         # TODO: Make this part do peer selection.
7519         self.update_goal()
7520         self.writers = {}
7521hunk ./src/allmydata/mutable/publish.py 248
7522-        # For each (peerid, shnum) in self.goal, we make an
7523-        # MDMFSlotWriteProxy for that peer. We'll use this to write
7524+        if self._version == MDMF_VERSION:
7525+            writer_class = MDMFSlotWriteProxy
7526+        else:
7527+            writer_class = SDMFSlotWriteProxy
7528+
7529+        # For each (peerid, shnum) in self.goal, we make a
7530+        # write proxy for that peer. We'll use this to write
7531         # shares to the peer.
7532         for key in self.goal:
7533             peerid, shnum = key
7534hunk ./src/allmydata/mutable/publish.py 263
7535             cancel_secret = self._node.get_cancel_secret(peerid)
7536             secrets = (write_enabler, renew_secret, cancel_secret)
7537 
7538-            self.writers[shnum] =  MDMFSlotWriteProxy(shnum,
7539-                                                      self.connections[peerid],
7540-                                                      self._storage_index,
7541-                                                      secrets,
7542-                                                      self._new_seqnum,
7543-                                                      self.required_shares,
7544-                                                      self.total_shares,
7545-                                                      self.segment_size,
7546-                                                      len(self.newdata))
7547+            self.writers[shnum] =  writer_class(shnum,
7548+                                                self.connections[peerid],
7549+                                                self._storage_index,
7550+                                                secrets,
7551+                                                self._new_seqnum,
7552+                                                self.required_shares,
7553+                                                self.total_shares,
7554+                                                self.segment_size,
7555+                                                len(self.newdata))
7556+            self.writers[shnum].peerid = peerid
7557             if (peerid, shnum) in self._servermap.servermap:
7558                 old_versionid, old_timestamp = self._servermap.servermap[key]
7559                 (old_seqnum, old_root_hash, old_salt, old_segsize,
7560hunk ./src/allmydata/mutable/publish.py 278
7561                  old_datalength, old_k, old_N, old_prefix,
7562                  old_offsets_tuple) = old_versionid
7563-                self.writers[shnum].set_checkstring(old_seqnum, old_root_hash)
7564+                self.writers[shnum].set_checkstring(old_seqnum,
7565+                                                    old_root_hash,
7566+                                                    old_salt)
7567+            elif (peerid, shnum) in self.bad_share_checkstrings:
7568+                old_checkstring = self.bad_share_checkstrings[(peerid, shnum)]
7569+                self.writers[shnum].set_checkstring(old_checkstring)
7570+
7571+        # Our remote shares will not have a complete checkstring until
7572+        # after we are done writing share data and have started to write
7573+        # blocks. In the meantime, we need to know what to look for when
7574+        # writing, so that we can detect UncoordinatedWriteErrors.
7575+        self._checkstring = self.writers.values()[0].get_checkstring()
7576 
7577         # Now, we start pushing shares.
7578         self._status.timings["setup"] = time.time() - self._started
7579hunk ./src/allmydata/mutable/publish.py 293
7580-        def _start_pushing(res):
7581-            self._started_pushing = time.time()
7582-            return res
7583-
7584         # First, we encrypt, encode, and publish the shares that we need
7585         # to encrypt, encode, and publish.
7586 
7587hunk ./src/allmydata/mutable/publish.py 306
7588 
7589         d = defer.succeed(None)
7590         self.log("Starting push")
7591-        for i in xrange(self.num_segments - 1):
7592-            d.addCallback(lambda ignored, i=i:
7593-                self.push_segment(i))
7594-            d.addCallback(self._turn_barrier)
7595-        # We have at least one segment, so we will have a tail segment
7596-        if self.num_segments > 0:
7597-            d.addCallback(lambda ignored:
7598-                self.push_tail_segment())
7599-
7600-        d.addCallback(lambda ignored:
7601-            self.push_encprivkey())
7602-        d.addCallback(lambda ignored:
7603-            self.push_blockhashes())
7604-        d.addCallback(lambda ignored:
7605-            self.push_sharehashes())
7606-        d.addCallback(lambda ignored:
7607-            self.push_toplevel_hashes_and_signature())
7608-        d.addCallback(lambda ignored:
7609-            self.finish_publishing())
7610-        return d
7611-
7612-
7613-    def _publish_sdmf(self):
7614-        self._status.timings["setup"] = time.time() - self._started
7615-        self.salt = os.urandom(16)
7616 
7617hunk ./src/allmydata/mutable/publish.py 307
7618-        d = self._encrypt_and_encode()
7619-        d.addCallback(self._generate_shares)
7620-        def _start_pushing(res):
7621-            self._started_pushing = time.time()
7622-            return res
7623-        d.addCallback(_start_pushing)
7624-        d.addCallback(self.loop) # trigger delivery
7625-        d.addErrback(self._fatal_error)
7626+        self._state = PUSHING_BLOCKS_STATE
7627+        self._push()
7628 
7629         return self.done_deferred
7630 
7631hunk ./src/allmydata/mutable/publish.py 327
7632                                                   segment_size)
7633         else:
7634             self.num_segments = 0
7635+
7636+        self.log("building encoding parameters for file")
7637+        self.log("got segsize %d" % self.segment_size)
7638+        self.log("got %d segments" % self.num_segments)
7639+
7640         if self._version == SDMF_VERSION:
7641             assert self.num_segments in (0, 1) # SDMF
7642hunk ./src/allmydata/mutable/publish.py 334
7643-            return
7644         # calculate the tail segment size.
7645hunk ./src/allmydata/mutable/publish.py 335
7646-        self.tail_segment_size = len(self.newdata) % segment_size
7647 
7648hunk ./src/allmydata/mutable/publish.py 336
7649-        if self.tail_segment_size == 0:
7650+        if segment_size and self.newdata:
7651+            self.tail_segment_size = len(self.newdata) % segment_size
7652+        else:
7653+            self.tail_segment_size = 0
7654+
7655+        if self.tail_segment_size == 0 and segment_size:
7656             # The tail segment is the same size as the other segments.
7657             self.tail_segment_size = segment_size
7658 
7659hunk ./src/allmydata/mutable/publish.py 345
7660-        # We'll make an encoder ahead-of-time for the normal-sized
7661-        # segments (defined as any segment of segment_size size.
7662-        # (the part of the code that puts the tail segment will make its
7663-        #  own encoder for that part)
7664+        # Make FEC encoders
7665         fec = codec.CRSEncoder()
7666         fec.set_params(self.segment_size,
7667                        self.required_shares, self.total_shares)
7668hunk ./src/allmydata/mutable/publish.py 352
7669         self.piece_size = fec.get_block_size()
7670         self.fec = fec
7671 
7672+        if self.tail_segment_size == self.segment_size:
7673+            self.tail_fec = self.fec
7674+        else:
7675+            tail_fec = codec.CRSEncoder()
7676+            tail_fec.set_params(self.tail_segment_size,
7677+                                self.required_shares,
7678+                                self.total_shares)
7679+            self.tail_fec = tail_fec
7680+
7681+        self._current_segment = 0
7682+
7683+
7684+    def _push(self, ignored=None):
7685+        """
7686+        I manage state transitions. In particular, I see that we still
7687+        have a good enough number of writers to complete the upload
7688+        successfully.
7689+        """
7690+        # Can we still successfully publish this file?
7691+        # TODO: Keep track of outstanding queries before aborting the
7692+        #       process.
7693+        if len(self.writers) <= self.required_shares or self.surprised:
7694+            return self._failure()
7695+
7696+        # Figure out what we need to do next. Each of these needs to
7697+        # return a deferred so that we don't block execution when this
7698+        # is first called in the upload method.
7699+        if self._state == PUSHING_BLOCKS_STATE:
7700+            return self.push_segment(self._current_segment)
7701+
7702+        # XXX: Do we want more granularity in states? Is that useful at
7703+        #      all?
7704+        #      Yes -- quicker reaction to UCW.
7705+        elif self._state == PUSHING_EVERYTHING_ELSE_STATE:
7706+            return self.push_everything_else()
7707+
7708+        # If we make it to this point, we were successful in placing the
7709+        # file.
7710+        return self._done(None)
7711+
7712 
7713     def push_segment(self, segnum):
7714hunk ./src/allmydata/mutable/publish.py 394
7715+        if self.num_segments == 0 and self._version == SDMF_VERSION:
7716+            self._add_dummy_salts()
7717+
7718+        if segnum == self.num_segments:
7719+            # We don't have any more segments to push.
7720+            self._state = PUSHING_EVERYTHING_ELSE_STATE
7721+            return self._push()
7722+
7723+        d = self._encode_segment(segnum)
7724+        d.addCallback(self._push_segment, segnum)
7725+        def _increment_segnum(ign):
7726+            self._current_segment += 1
7727+        # XXX: I don't think we need to do addBoth here -- any errBacks
7728+        # should be handled within push_segment.
7729+        d.addBoth(_increment_segnum)
7730+        d.addBoth(self._push)
7731+
7732+
7733+    def _add_dummy_salts(self):
7734+        """
7735+        SDMF files need a salt even if they're empty, or the signature
7736+        won't make sense. This method adds a dummy salt to each of our
7737+        SDMF writers so that they can write the signature later.
7738+        """
7739+        salt = os.urandom(16)
7740+        assert self._version == SDMF_VERSION
7741+
7742+        for writer in self.writers.itervalues():
7743+            writer.put_salt(salt)
7744+
7745+
7746+    def _encode_segment(self, segnum):
7747+        """
7748+        I encrypt and encode the segment segnum.
7749+        """
7750         started = time.time()
7751hunk ./src/allmydata/mutable/publish.py 430
7752-        segsize = self.segment_size
7753+
7754+        if segnum + 1 == self.num_segments:
7755+            segsize = self.tail_segment_size
7756+        else:
7757+            segsize = self.segment_size
7758+
7759+
7760+        offset = self.segment_size * segnum
7761+        length = segsize + offset
7762         self.log("Pushing segment %d of %d" % (segnum + 1, self.num_segments))
7763hunk ./src/allmydata/mutable/publish.py 440
7764-        data = self.newdata[segsize * segnum:segsize*(segnum + 1)]
7765+        data = self.newdata[offset:length]
7766         assert len(data) == segsize
7767 
7768         salt = os.urandom(16)
7769hunk ./src/allmydata/mutable/publish.py 455
7770         started = now
7771 
7772         # now apply FEC
7773+        if segnum + 1 == self.num_segments:
7774+            fec = self.tail_fec
7775+        else:
7776+            fec = self.fec
7777 
7778         self._status.set_status("Encoding")
7779         crypttext_pieces = [None] * self.required_shares
7780hunk ./src/allmydata/mutable/publish.py 462
7781-        piece_size = self.piece_size
7782+        piece_size = fec.get_block_size()
7783         for i in range(len(crypttext_pieces)):
7784             offset = i * piece_size
7785             piece = crypttext[offset:offset+piece_size]
7786hunk ./src/allmydata/mutable/publish.py 469
7787             piece = piece + "\x00"*(piece_size - len(piece)) # padding
7788             crypttext_pieces[i] = piece
7789             assert len(piece) == piece_size
7790-        d = self.fec.encode(crypttext_pieces)
7791+        d = fec.encode(crypttext_pieces)
7792         def _done_encoding(res):
7793             elapsed = time.time() - started
7794             self._status.timings["encode"] = elapsed
7795hunk ./src/allmydata/mutable/publish.py 473
7796-            return res
7797+            return (res, salt)
7798         d.addCallback(_done_encoding)
7799hunk ./src/allmydata/mutable/publish.py 475
7800-
7801-        def _push_shares_and_salt(results):
7802-            shares, shareids = results
7803-            dl = []
7804-            for i in xrange(len(shares)):
7805-                sharedata = shares[i]
7806-                shareid = shareids[i]
7807-                block_hash = hashutil.block_hash(salt + sharedata)
7808-                self.blockhashes[shareid].append(block_hash)
7809-
7810-                # find the writer for this share
7811-                d = self.writers[shareid].put_block(sharedata, segnum, salt)
7812-                dl.append(d)
7813-            # TODO: Naturally, we need to check on the results of these.
7814-            return defer.DeferredList(dl)
7815-        d.addCallback(_push_shares_and_salt)
7816         return d
7817 
7818 
7819hunk ./src/allmydata/mutable/publish.py 478
7820-    def push_tail_segment(self):
7821-        # This is essentially the same as push_segment, except that we
7822-        # don't use the cached encoder that we use elsewhere.
7823-        self.log("Pushing tail segment")
7824+    def _push_segment(self, encoded_and_salt, segnum):
7825+        """
7826+        I push (data, salt) as segment number segnum.
7827+        """
7828+        results, salt = encoded_and_salt
7829+        shares, shareids = results
7830         started = time.time()
7831hunk ./src/allmydata/mutable/publish.py 485
7832-        segsize = self.segment_size
7833-        data = self.newdata[segsize * (self.num_segments-1):]
7834-        assert len(data) == self.tail_segment_size
7835-        salt = os.urandom(16)
7836-
7837-        key = hashutil.ssk_readkey_data_hash(salt, self.readkey)
7838-        enc = AES(key)
7839-        crypttext = enc.process(data)
7840-        assert len(crypttext) == len(data)
7841+        dl = []
7842+        for i in xrange(len(shares)):
7843+            sharedata = shares[i]
7844+            shareid = shareids[i]
7845+            if self._version == MDMF_VERSION:
7846+                hashed = salt + sharedata
7847+            else:
7848+                hashed = sharedata
7849+            block_hash = hashutil.block_hash(hashed)
7850+            self.blockhashes[shareid].append(block_hash)
7851 
7852hunk ./src/allmydata/mutable/publish.py 496
7853-        now = time.time()
7854-        self._status.timings['encrypt'] = now - started
7855-        started = now
7856+            # find the writer for this share
7857+            writer = self.writers[shareid]
7858+            d = writer.put_block(sharedata, segnum, salt)
7859+            d.addCallback(self._got_write_answer, writer, started)
7860+            d.addErrback(self._connection_problem, writer)
7861+            dl.append(d)
7862+            # TODO: Naturally, we need to check on the results of these.
7863+        return defer.DeferredList(dl)
7864 
7865hunk ./src/allmydata/mutable/publish.py 505
7866-        self._status.set_status("Encoding")
7867-        tail_fec = codec.CRSEncoder()
7868-        tail_fec.set_params(self.tail_segment_size,
7869-                            self.required_shares,
7870-                            self.total_shares)
7871 
7872hunk ./src/allmydata/mutable/publish.py 506
7873-        crypttext_pieces = [None] * self.required_shares
7874-        piece_size = tail_fec.get_block_size()
7875-        for i in range(len(crypttext_pieces)):
7876-            offset = i * piece_size
7877-            piece = crypttext[offset:offset+piece_size]
7878-            piece = piece + "\x00"*(piece_size - len(piece)) # padding
7879-            crypttext_pieces[i] = piece
7880-            assert len(piece) == piece_size
7881-        d = tail_fec.encode(crypttext_pieces)
7882-        def _push_shares_and_salt(results):
7883-            shares, shareids = results
7884-            dl = []
7885-            for i in xrange(len(shares)):
7886-                sharedata = shares[i]
7887-                shareid = shareids[i]
7888-                block_hash = hashutil.block_hash(salt + sharedata)
7889-                self.blockhashes[shareid].append(block_hash)
7890-                # find the writer for this share
7891-                d = self.writers[shareid].put_block(sharedata,
7892-                                                    self.num_segments - 1,
7893-                                                    salt)
7894-                dl.append(d)
7895-            # TODO: Naturally, we need to check on the results of these.
7896-            return defer.DeferredList(dl)
7897-        d.addCallback(_push_shares_and_salt)
7898+    def push_everything_else(self):
7899+        """
7900+        I put everything else associated with a share.
7901+        """
7902+        encprivkey = self._encprivkey
7903+        d = self.push_encprivkey()
7904+        d.addCallback(self.push_blockhashes)
7905+        d.addCallback(self.push_sharehashes)
7906+        d.addCallback(self.push_toplevel_hashes_and_signature)
7907+        d.addCallback(self.finish_publishing)
7908+        def _change_state(ignored):
7909+            self._state = DONE_STATE
7910+        d.addCallback(_change_state)
7911+        d.addCallback(self._push)
7912         return d
7913 
7914 
7915hunk ./src/allmydata/mutable/publish.py 527
7916         started = time.time()
7917         encprivkey = self._encprivkey
7918         dl = []
7919-        def _spy_on_writer(results):
7920-            print results
7921-            return results
7922-        for shnum, writer in self.writers.iteritems():
7923+        for writer in self.writers.itervalues():
7924             d = writer.put_encprivkey(encprivkey)
7925hunk ./src/allmydata/mutable/publish.py 529
7926+            d.addCallback(self._got_write_answer, writer, started)
7927+            d.addErrback(self._connection_problem, writer)
7928             dl.append(d)
7929         d = defer.DeferredList(dl)
7930         return d
7931hunk ./src/allmydata/mutable/publish.py 536
7932 
7933 
7934-    def push_blockhashes(self):
7935+    def push_blockhashes(self, ignored):
7936         started = time.time()
7937         dl = []
7938hunk ./src/allmydata/mutable/publish.py 539
7939-        def _spy_on_results(results):
7940-            print results
7941-            return results
7942         self.sharehash_leaves = [None] * len(self.blockhashes)
7943         for shnum, blockhashes in self.blockhashes.iteritems():
7944             t = hashtree.HashTree(blockhashes)
7945hunk ./src/allmydata/mutable/publish.py 545
7946             self.blockhashes[shnum] = list(t)
7947             # set the leaf for future use.
7948             self.sharehash_leaves[shnum] = t[0]
7949-            d = self.writers[shnum].put_blockhashes(self.blockhashes[shnum])
7950+            writer = self.writers[shnum]
7951+            d = writer.put_blockhashes(self.blockhashes[shnum])
7952+            d.addCallback(self._got_write_answer, writer, started)
7953+            d.addErrback(self._connection_problem, self.writers[shnum])
7954             dl.append(d)
7955         d = defer.DeferredList(dl)
7956         return d
7957hunk ./src/allmydata/mutable/publish.py 554
7958 
7959 
7960-    def push_sharehashes(self):
7961+    def push_sharehashes(self, ignored):
7962+        started = time.time()
7963         share_hash_tree = hashtree.HashTree(self.sharehash_leaves)
7964         share_hash_chain = {}
7965         ds = []
7966hunk ./src/allmydata/mutable/publish.py 559
7967-        def _spy_on_results(results):
7968-            print results
7969-            return results
7970         for shnum in xrange(len(self.sharehash_leaves)):
7971             needed_indices = share_hash_tree.needed_hashes(shnum)
7972             self.sharehashes[shnum] = dict( [ (i, share_hash_tree[i])
7973hunk ./src/allmydata/mutable/publish.py 563
7974                                              for i in needed_indices] )
7975-            d = self.writers[shnum].put_sharehashes(self.sharehashes[shnum])
7976+            writer = self.writers[shnum]
7977+            d = writer.put_sharehashes(self.sharehashes[shnum])
7978+            d.addCallback(self._got_write_answer, writer, started)
7979+            d.addErrback(self._connection_problem, writer)
7980             ds.append(d)
7981         self.root_hash = share_hash_tree[0]
7982         d = defer.DeferredList(ds)
7983hunk ./src/allmydata/mutable/publish.py 573
7984         return d
7985 
7986 
7987-    def push_toplevel_hashes_and_signature(self):
7988+    def push_toplevel_hashes_and_signature(self, ignored):
7989         # We need to to three things here:
7990         #   - Push the root hash and salt hash
7991         #   - Get the checkstring of the resulting layout; sign that.
7992hunk ./src/allmydata/mutable/publish.py 578
7993         #   - Push the signature
7994+        started = time.time()
7995         ds = []
7996hunk ./src/allmydata/mutable/publish.py 580
7997-        def _spy_on_results(results):
7998-            print results
7999-            return results
8000         for shnum in xrange(self.total_shares):
8001hunk ./src/allmydata/mutable/publish.py 581
8002-            d = self.writers[shnum].put_root_hash(self.root_hash)
8003+            writer = self.writers[shnum]
8004+            d = writer.put_root_hash(self.root_hash)
8005+            d.addCallback(self._got_write_answer, writer, started)
8006             ds.append(d)
8007         d = defer.DeferredList(ds)
8008hunk ./src/allmydata/mutable/publish.py 586
8009-        def _make_and_place_signature(ignored):
8010-            signable = self.writers[0].get_signable()
8011-            self.signature = self._privkey.sign(signable)
8012-
8013-            ds = []
8014-            for (shnum, writer) in self.writers.iteritems():
8015-                d = writer.put_signature(self.signature)
8016-                ds.append(d)
8017-            return defer.DeferredList(ds)
8018-        d.addCallback(_make_and_place_signature)
8019+        d.addCallback(self._update_checkstring)
8020+        d.addCallback(self._make_and_place_signature)
8021         return d
8022 
8023 
8024hunk ./src/allmydata/mutable/publish.py 591
8025-    def finish_publishing(self):
8026+    def _update_checkstring(self, ignored):
8027+        """
8028+        After putting the root hash, MDMF files will have the
8029+        checkstring written to the storage server. This means that we
8030+        can update our copy of the checkstring so we can detect
8031+        uncoordinated writes. SDMF files will have the same checkstring,
8032+        so we need not do anything.
8033+        """
8034+        self._checkstring = self.writers.values()[0].get_checkstring()
8035+
8036+
8037+    def _make_and_place_signature(self, ignored):
8038+        """
8039+        I create and place the signature.
8040+        """
8041+        started = time.time()
8042+        signable = self.writers[0].get_signable()
8043+        self.signature = self._privkey.sign(signable)
8044+
8045+        ds = []
8046+        for (shnum, writer) in self.writers.iteritems():
8047+            d = writer.put_signature(self.signature)
8048+            d.addCallback(self._got_write_answer, writer, started)
8049+            d.addErrback(self._connection_problem, writer)
8050+            ds.append(d)
8051+        return defer.DeferredList(ds)
8052+
8053+
8054+    def finish_publishing(self, ignored):
8055         # We're almost done -- we just need to put the verification key
8056         # and the offsets
8057hunk ./src/allmydata/mutable/publish.py 622
8058+        started = time.time()
8059         ds = []
8060         verification_key = self._pubkey.serialize()
8061 
8062hunk ./src/allmydata/mutable/publish.py 626
8063-        def _spy_on_results(results):
8064-            print results
8065-            return results
8066+
8067+        # TODO: Bad, since we remove from this same dict. We need to
8068+        # make a copy, or just use a non-iterated value.
8069         for (shnum, writer) in self.writers.iteritems():
8070             d = writer.put_verification_key(verification_key)
8071hunk ./src/allmydata/mutable/publish.py 631
8072+            d.addCallback(self._got_write_answer, writer, started)
8073+            d.addCallback(self._record_verinfo)
8074             d.addCallback(lambda ignored, writer=writer:
8075                 writer.finish_publishing())
8076hunk ./src/allmydata/mutable/publish.py 635
8077+            d.addCallback(self._got_write_answer, writer, started)
8078+            d.addErrback(self._connection_problem, writer)
8079             ds.append(d)
8080         return defer.DeferredList(ds)
8081 
8082hunk ./src/allmydata/mutable/publish.py 641
8083 
8084-    def _turn_barrier(self, res):
8085-        # putting this method in a Deferred chain imposes a guaranteed
8086-        # reactor turn between the pre- and post- portions of that chain.
8087-        # This can be useful to limit memory consumption: since Deferreds do
8088-        # not do tail recursion, code which uses defer.succeed(result) for
8089-        # consistency will cause objects to live for longer than you might
8090-        # normally expect.
8091-        return fireEventually(res)
8092+    def _record_verinfo(self, ignored):
8093+        self.versioninfo = self.writers.values()[0].get_verinfo()
8094 
8095 
8096hunk ./src/allmydata/mutable/publish.py 645
8097-    def _fatal_error(self, f):
8098-        self.log("error during loop", failure=f, level=log.UNUSUAL)
8099-        self._done(f)
8100+    def _connection_problem(self, f, writer):
8101+        """
8102+        We ran into a connection problem while working with writer, and
8103+        need to deal with that.
8104+        """
8105+        self.log("found problem: %s" % str(f))
8106+        self._last_failure = f
8107+        del(self.writers[writer.shnum])
8108 
8109hunk ./src/allmydata/mutable/publish.py 654
8110-    def _update_status(self):
8111-        self._status.set_status("Sending Shares: %d placed out of %d, "
8112-                                "%d messages outstanding" %
8113-                                (len(self.placed),
8114-                                 len(self.goal),
8115-                                 len(self.outstanding)))
8116-        self._status.set_progress(1.0 * len(self.placed) / len(self.goal))
8117 
8118     def loop(self, ignored=None):
8119         self.log("entering loop", level=log.NOISY)
8120hunk ./src/allmydata/mutable/publish.py 778
8121             self.log_goal(self.goal, "after update: ")
8122 
8123 
8124-    def _encrypt_and_encode(self):
8125-        # this returns a Deferred that fires with a list of (sharedata,
8126-        # sharenum) tuples. TODO: cache the ciphertext, only produce the
8127-        # shares that we care about.
8128-        self.log("_encrypt_and_encode")
8129-
8130-        self._status.set_status("Encrypting")
8131-        started = time.time()
8132+    def _got_write_answer(self, answer, writer, started):
8133+        if not answer:
8134+            # SDMF writers only pretend to write when readers set their
8135+            # blocks, salts, and so on -- they actually just write once,
8136+            # at the end of the upload process. In fake writes, they
8137+            # return defer.succeed(None). If we see that, we shouldn't
8138+            # bother checking it.
8139+            return
8140 
8141hunk ./src/allmydata/mutable/publish.py 787
8142-        key = hashutil.ssk_readkey_data_hash(self.salt, self.readkey)
8143-        enc = AES(key)
8144-        crypttext = enc.process(self.newdata)
8145-        assert len(crypttext) == len(self.newdata)
8146+        peerid = writer.peerid
8147+        lp = self.log("_got_write_answer from %s, share %d" %
8148+                      (idlib.shortnodeid_b2a(peerid), writer.shnum))
8149 
8150         now = time.time()
8151hunk ./src/allmydata/mutable/publish.py 792
8152-        self._status.timings["encrypt"] = now - started
8153-        started = now
8154-
8155-        # now apply FEC
8156-
8157-        self._status.set_status("Encoding")
8158-        fec = codec.CRSEncoder()
8159-        fec.set_params(self.segment_size,
8160-                       self.required_shares, self.total_shares)
8161-        piece_size = fec.get_block_size()
8162-        crypttext_pieces = [None] * self.required_shares
8163-        for i in range(len(crypttext_pieces)):
8164-            offset = i * piece_size
8165-            piece = crypttext[offset:offset+piece_size]
8166-            piece = piece + "\x00"*(piece_size - len(piece)) # padding
8167-            crypttext_pieces[i] = piece
8168-            assert len(piece) == piece_size
8169-
8170-        d = fec.encode(crypttext_pieces)
8171-        def _done_encoding(res):
8172-            elapsed = time.time() - started
8173-            self._status.timings["encode"] = elapsed
8174-            return res
8175-        d.addCallback(_done_encoding)
8176-        return d
8177-
8178-
8179-    def _generate_shares(self, shares_and_shareids):
8180-        # this sets self.shares and self.root_hash
8181-        self.log("_generate_shares")
8182-        self._status.set_status("Generating Shares")
8183-        started = time.time()
8184-
8185-        # we should know these by now
8186-        privkey = self._privkey
8187-        encprivkey = self._encprivkey
8188-        pubkey = self._pubkey
8189-
8190-        (shares, share_ids) = shares_and_shareids
8191-
8192-        assert len(shares) == len(share_ids)
8193-        assert len(shares) == self.total_shares
8194-        all_shares = {}
8195-        block_hash_trees = {}
8196-        share_hash_leaves = [None] * len(shares)
8197-        for i in range(len(shares)):
8198-            share_data = shares[i]
8199-            shnum = share_ids[i]
8200-            all_shares[shnum] = share_data
8201-
8202-            # build the block hash tree. SDMF has only one leaf.
8203-            leaves = [hashutil.block_hash(share_data)]
8204-            t = hashtree.HashTree(leaves)
8205-            block_hash_trees[shnum] = list(t)
8206-            share_hash_leaves[shnum] = t[0]
8207-        for leaf in share_hash_leaves:
8208-            assert leaf is not None
8209-        share_hash_tree = hashtree.HashTree(share_hash_leaves)
8210-        share_hash_chain = {}
8211-        for shnum in range(self.total_shares):
8212-            needed_hashes = share_hash_tree.needed_hashes(shnum)
8213-            share_hash_chain[shnum] = dict( [ (i, share_hash_tree[i])
8214-                                              for i in needed_hashes ] )
8215-        root_hash = share_hash_tree[0]
8216-        assert len(root_hash) == 32
8217-        self.log("my new root_hash is %s" % base32.b2a(root_hash))
8218-        self._new_version_info = (self._new_seqnum, root_hash, self.salt)
8219-
8220-        prefix = pack_prefix(self._new_seqnum, root_hash, self.salt,
8221-                             self.required_shares, self.total_shares,
8222-                             self.segment_size, len(self.newdata))
8223-
8224-        # now pack the beginning of the share. All shares are the same up
8225-        # to the signature, then they have divergent share hash chains,
8226-        # then completely different block hash trees + salt + share data,
8227-        # then they all share the same encprivkey at the end. The sizes
8228-        # of everything are the same for all shares.
8229-
8230-        sign_started = time.time()
8231-        signature = privkey.sign(prefix)
8232-        self._status.timings["sign"] = time.time() - sign_started
8233-
8234-        verification_key = pubkey.serialize()
8235-
8236-        final_shares = {}
8237-        for shnum in range(self.total_shares):
8238-            final_share = pack_share(prefix,
8239-                                     verification_key,
8240-                                     signature,
8241-                                     share_hash_chain[shnum],
8242-                                     block_hash_trees[shnum],
8243-                                     all_shares[shnum],
8244-                                     encprivkey)
8245-            final_shares[shnum] = final_share
8246-        elapsed = time.time() - started
8247-        self._status.timings["pack"] = elapsed
8248-        self.shares = final_shares
8249-        self.root_hash = root_hash
8250-
8251-        # we also need to build up the version identifier for what we're
8252-        # pushing. Extract the offsets from one of our shares.
8253-        assert final_shares
8254-        offsets = unpack_header(final_shares.values()[0])[-1]
8255-        offsets_tuple = tuple( [(key,value) for key,value in offsets.items()] )
8256-        verinfo = (self._new_seqnum, root_hash, self.salt,
8257-                   self.segment_size, len(self.newdata),
8258-                   self.required_shares, self.total_shares,
8259-                   prefix, offsets_tuple)
8260-        self.versioninfo = verinfo
8261-
8262-
8263-
8264-    def _send_shares(self, needed):
8265-        self.log("_send_shares")
8266-
8267-        # we're finally ready to send out our shares. If we encounter any
8268-        # surprises here, it's because somebody else is writing at the same
8269-        # time. (Note: in the future, when we remove the _query_peers() step
8270-        # and instead speculate about [or remember] which shares are where,
8271-        # surprises here are *not* indications of UncoordinatedWriteError,
8272-        # and we'll need to respond to them more gracefully.)
8273-
8274-        # needed is a set of (peerid, shnum) tuples. The first thing we do is
8275-        # organize it by peerid.
8276-
8277-        peermap = DictOfSets()
8278-        for (peerid, shnum) in needed:
8279-            peermap.add(peerid, shnum)
8280-
8281-        # the next thing is to build up a bunch of test vectors. The
8282-        # semantics of Publish are that we perform the operation if the world
8283-        # hasn't changed since the ServerMap was constructed (more or less).
8284-        # For every share we're trying to place, we create a test vector that
8285-        # tests to see if the server*share still corresponds to the
8286-        # map.
8287-
8288-        all_tw_vectors = {} # maps peerid to tw_vectors
8289-        sm = self._servermap.servermap
8290-
8291-        for key in needed:
8292-            (peerid, shnum) = key
8293-
8294-            if key in sm:
8295-                # an old version of that share already exists on the
8296-                # server, according to our servermap. We will create a
8297-                # request that attempts to replace it.
8298-                old_versionid, old_timestamp = sm[key]
8299-                (old_seqnum, old_root_hash, old_salt, old_segsize,
8300-                 old_datalength, old_k, old_N, old_prefix,
8301-                 old_offsets_tuple) = old_versionid
8302-                old_checkstring = pack_checkstring(old_seqnum,
8303-                                                   old_root_hash,
8304-                                                   old_salt)
8305-                testv = (0, len(old_checkstring), "eq", old_checkstring)
8306-
8307-            elif key in self.bad_share_checkstrings:
8308-                old_checkstring = self.bad_share_checkstrings[key]
8309-                testv = (0, len(old_checkstring), "eq", old_checkstring)
8310-
8311-            else:
8312-                # add a testv that requires the share not exist
8313-
8314-                # Unfortunately, foolscap-0.2.5 has a bug in the way inbound
8315-                # constraints are handled. If the same object is referenced
8316-                # multiple times inside the arguments, foolscap emits a
8317-                # 'reference' token instead of a distinct copy of the
8318-                # argument. The bug is that these 'reference' tokens are not
8319-                # accepted by the inbound constraint code. To work around
8320-                # this, we need to prevent python from interning the
8321-                # (constant) tuple, by creating a new copy of this vector
8322-                # each time.
8323-
8324-                # This bug is fixed in foolscap-0.2.6, and even though this
8325-                # version of Tahoe requires foolscap-0.3.1 or newer, we are
8326-                # supposed to be able to interoperate with older versions of
8327-                # Tahoe which are allowed to use older versions of foolscap,
8328-                # including foolscap-0.2.5 . In addition, I've seen other
8329-                # foolscap problems triggered by 'reference' tokens (see #541
8330-                # for details). So we must keep this workaround in place.
8331-
8332-                #testv = (0, 1, 'eq', "")
8333-                testv = tuple([0, 1, 'eq', ""])
8334-
8335-            testvs = [testv]
8336-            # the write vector is simply the share
8337-            writev = [(0, self.shares[shnum])]
8338-
8339-            if peerid not in all_tw_vectors:
8340-                all_tw_vectors[peerid] = {}
8341-                # maps shnum to (testvs, writevs, new_length)
8342-            assert shnum not in all_tw_vectors[peerid]
8343-
8344-            all_tw_vectors[peerid][shnum] = (testvs, writev, None)
8345-
8346-        # we read the checkstring back from each share, however we only use
8347-        # it to detect whether there was a new share that we didn't know
8348-        # about. The success or failure of the write will tell us whether
8349-        # there was a collision or not. If there is a collision, the first
8350-        # thing we'll do is update the servermap, which will find out what
8351-        # happened. We could conceivably reduce a roundtrip by using the
8352-        # readv checkstring to populate the servermap, but really we'd have
8353-        # to read enough data to validate the signatures too, so it wouldn't
8354-        # be an overall win.
8355-        read_vector = [(0, struct.calcsize(SIGNED_PREFIX))]
8356-
8357-        # ok, send the messages!
8358-        self.log("sending %d shares" % len(all_tw_vectors), level=log.NOISY)
8359-        started = time.time()
8360-        for (peerid, tw_vectors) in all_tw_vectors.items():
8361-
8362-            write_enabler = self._node.get_write_enabler(peerid)
8363-            renew_secret = self._node.get_renewal_secret(peerid)
8364-            cancel_secret = self._node.get_cancel_secret(peerid)
8365-            secrets = (write_enabler, renew_secret, cancel_secret)
8366-            shnums = tw_vectors.keys()
8367-
8368-            for shnum in shnums:
8369-                self.outstanding.add( (peerid, shnum) )
8370-
8371-            d = self._do_testreadwrite(peerid, secrets,
8372-                                       tw_vectors, read_vector)
8373-            d.addCallbacks(self._got_write_answer, self._got_write_error,
8374-                           callbackArgs=(peerid, shnums, started),
8375-                           errbackArgs=(peerid, shnums, started))
8376-            # tolerate immediate errback, like with DeadReferenceError
8377-            d.addBoth(fireEventually)
8378-            d.addCallback(self.loop)
8379-            d.addErrback(self._fatal_error)
8380-
8381-        self._update_status()
8382-        self.log("%d shares sent" % len(all_tw_vectors), level=log.NOISY)
8383+        elapsed = now - started
8384 
8385hunk ./src/allmydata/mutable/publish.py 794
8386-    def _do_testreadwrite(self, peerid, secrets,
8387-                          tw_vectors, read_vector):
8388-        storage_index = self._storage_index
8389-        ss = self.connections[peerid]
8390+        self._status.add_per_server_time(peerid, elapsed)
8391 
8392hunk ./src/allmydata/mutable/publish.py 796
8393-        #print "SS[%s] is %s" % (idlib.shortnodeid_b2a(peerid), ss), ss.tracker.interfaceName
8394-        d = ss.callRemote("slot_testv_and_readv_and_writev",
8395-                          storage_index,
8396-                          secrets,
8397-                          tw_vectors,
8398-                          read_vector)
8399-        return d
8400+        wrote, read_data = answer
8401 
8402hunk ./src/allmydata/mutable/publish.py 798
8403-    def _got_write_answer(self, answer, peerid, shnums, started):
8404-        lp = self.log("_got_write_answer from %s" %
8405-                      idlib.shortnodeid_b2a(peerid))
8406-        for shnum in shnums:
8407-            self.outstanding.discard( (peerid, shnum) )
8408+        surprise_shares = set(read_data.keys()) - set([writer.shnum])
8409 
8410hunk ./src/allmydata/mutable/publish.py 800
8411-        now = time.time()
8412-        elapsed = now - started
8413-        self._status.add_per_server_time(peerid, elapsed)
8414+        # We need to remove from surprise_shares any shares that we are
8415+        # knowingly also writing to that peer from other writers.
8416 
8417hunk ./src/allmydata/mutable/publish.py 803
8418-        wrote, read_data = answer
8419+        # TODO: Precompute this.
8420+        known_shnums = [x.shnum for x in self.writers.values()
8421+                        if x.peerid == peerid]
8422+        surprise_shares -= set(known_shnums)
8423+        self.log("found the following surprise shares: %s" %
8424+                 str(surprise_shares))
8425 
8426hunk ./src/allmydata/mutable/publish.py 810
8427-        surprise_shares = set(read_data.keys()) - set(shnums)
8428+        # Now surprise shares contains all of the shares that we did not
8429+        # expect to be there.
8430 
8431         surprised = False
8432         for shnum in surprise_shares:
8433hunk ./src/allmydata/mutable/publish.py 817
8434             # read_data is a dict mapping shnum to checkstring (SIGNED_PREFIX)
8435             checkstring = read_data[shnum][0]
8436-            their_version_info = unpack_checkstring(checkstring)
8437-            if their_version_info == self._new_version_info:
8438+            # What we want to do here is to see if their (seqnum,
8439+            # roothash, salt) is the same as our (seqnum, roothash,
8440+            # salt), or the equivalent for MDMF. The best way to do this
8441+            # is to store a packed representation of our checkstring
8442+            # somewhere, then not bother unpacking the other
8443+            # checkstring.
8444+            if checkstring == self._checkstring:
8445                 # they have the right share, somehow
8446 
8447                 if (peerid,shnum) in self.goal:
8448hunk ./src/allmydata/mutable/publish.py 902
8449             self.log("our testv failed, so the write did not happen",
8450                      parent=lp, level=log.WEIRD, umid="8sc26g")
8451             self.surprised = True
8452-            self.bad_peers.add(peerid) # don't ask them again
8453+            # TODO: This needs to
8454+            self.bad_peers.add(writer) # don't ask them again
8455             # use the checkstring to add information to the log message
8456             for (shnum,readv) in read_data.items():
8457                 checkstring = readv[0]
8458hunk ./src/allmydata/mutable/publish.py 928
8459             # self.loop() will take care of finding new homes
8460             return
8461 
8462-        for shnum in shnums:
8463-            self.placed.add( (peerid, shnum) )
8464-            # and update the servermap
8465-            self._servermap.add_new_share(peerid, shnum,
8466+        # and update the servermap
8467+        # self.versioninfo is set during the last phase of publishing.
8468+        # If we get there, we know that responses correspond to placed
8469+        # shares, and can safely execute these statements.
8470+        if self.versioninfo:
8471+            self.log("wrote successfully: adding new share to servermap")
8472+            self._servermap.add_new_share(peerid, writer.shnum,
8473                                           self.versioninfo, started)
8474hunk ./src/allmydata/mutable/publish.py 936
8475-
8476-        # self.loop() will take care of checking to see if we're done
8477-        return
8478+            self.placed.add( (peerid, writer.shnum) )
8479 
8480hunk ./src/allmydata/mutable/publish.py 938
8481-    def _got_write_error(self, f, peerid, shnums, started):
8482-        for shnum in shnums:
8483-            self.outstanding.discard( (peerid, shnum) )
8484-        self.bad_peers.add(peerid)
8485-        if self._first_write_error is None:
8486-            self._first_write_error = f
8487-        self.log(format="error while writing shares %(shnums)s to peerid %(peerid)s",
8488-                 shnums=list(shnums), peerid=idlib.shortnodeid_b2a(peerid),
8489-                 failure=f,
8490-                 level=log.UNUSUAL)
8491         # self.loop() will take care of checking to see if we're done
8492         return
8493 
8494hunk ./src/allmydata/mutable/publish.py 949
8495         now = time.time()
8496         self._status.timings["total"] = now - self._started
8497         self._status.set_active(False)
8498-        if isinstance(res, failure.Failure):
8499-            self.log("Publish done, with failure", failure=res,
8500-                     level=log.WEIRD, umid="nRsR9Q")
8501-            self._status.set_status("Failed")
8502-        elif self.surprised:
8503-            self.log("Publish done, UncoordinatedWriteError", level=log.UNUSUAL)
8504-            self._status.set_status("UncoordinatedWriteError")
8505-            # deliver a failure
8506-            res = failure.Failure(UncoordinatedWriteError())
8507-            # TODO: recovery
8508-        else:
8509-            self.log("Publish done, success")
8510-            self._status.set_status("Finished")
8511-            self._status.set_progress(1.0)
8512+        self.log("Publish done, success")
8513+        self._status.set_status("Finished")
8514+        self._status.set_progress(1.0)
8515         eventually(self.done_deferred.callback, res)
8516 
8517hunk ./src/allmydata/mutable/publish.py 954
8518+    def _failure(self):
8519+
8520+        if not self.surprised:
8521+            # We ran out of servers
8522+            self.log("Publish ran out of good servers, "
8523+                     "last failure was: %s" % str(self._last_failure))
8524+            e = NotEnoughServersError("Ran out of non-bad servers, "
8525+                                      "last failure was %s" %
8526+                                      str(self._last_failure))
8527+        else:
8528+            # We ran into shares that we didn't recognize, which means
8529+            # that we need to return an UncoordinatedWriteError.
8530+            self.log("Publish failed with UncoordinatedWriteError")
8531+            e = UncoordinatedWriteError()
8532+        f = failure.Failure(e)
8533+        eventually(self.done_deferred.callback, f)
8534}
8535[test/test_mutable.py: remove tests that are no longer relevant
8536Kevan Carstensen <kevan@isnotajoke.com>**20100702225710
8537 Ignore-this: 90a26b4cc4b2e190a635474ba7097e21
8538] hunk ./src/allmydata/test/test_mutable.py 627
8539         return d
8540 
8541 
8542-class MakeShares(unittest.TestCase):
8543-    def test_encrypt(self):
8544-        nm = make_nodemaker()
8545-        CONTENTS = "some initial contents"
8546-        d = nm.create_mutable_file(CONTENTS)
8547-        def _created(fn):
8548-            p = Publish(fn, nm.storage_broker, None)
8549-            p.salt = "SALT" * 4
8550-            p.readkey = "\x00" * 16
8551-            p.newdata = CONTENTS
8552-            p.required_shares = 3
8553-            p.total_shares = 10
8554-            p.setup_encoding_parameters()
8555-            return p._encrypt_and_encode()
8556-        d.addCallback(_created)
8557-        def _done(shares_and_shareids):
8558-            (shares, share_ids) = shares_and_shareids
8559-            self.failUnlessEqual(len(shares), 10)
8560-            for sh in shares:
8561-                self.failUnless(isinstance(sh, str))
8562-                self.failUnlessEqual(len(sh), 7)
8563-            self.failUnlessEqual(len(share_ids), 10)
8564-        d.addCallback(_done)
8565-        return d
8566-    test_encrypt.todo = "Write an equivalent of this for the new uploader"
8567-
8568-    def test_generate(self):
8569-        nm = make_nodemaker()
8570-        CONTENTS = "some initial contents"
8571-        d = nm.create_mutable_file(CONTENTS)
8572-        def _created(fn):
8573-            self._fn = fn
8574-            p = Publish(fn, nm.storage_broker, None)
8575-            self._p = p
8576-            p.newdata = CONTENTS
8577-            p.required_shares = 3
8578-            p.total_shares = 10
8579-            p.setup_encoding_parameters()
8580-            p._new_seqnum = 3
8581-            p.salt = "SALT" * 4
8582-            # make some fake shares
8583-            shares_and_ids = ( ["%07d" % i for i in range(10)], range(10) )
8584-            p._privkey = fn.get_privkey()
8585-            p._encprivkey = fn.get_encprivkey()
8586-            p._pubkey = fn.get_pubkey()
8587-            return p._generate_shares(shares_and_ids)
8588-        d.addCallback(_created)
8589-        def _generated(res):
8590-            p = self._p
8591-            final_shares = p.shares
8592-            root_hash = p.root_hash
8593-            self.failUnlessEqual(len(root_hash), 32)
8594-            self.failUnless(isinstance(final_shares, dict))
8595-            self.failUnlessEqual(len(final_shares), 10)
8596-            self.failUnlessEqual(sorted(final_shares.keys()), range(10))
8597-            for i,sh in final_shares.items():
8598-                self.failUnless(isinstance(sh, str))
8599-                # feed the share through the unpacker as a sanity-check
8600-                pieces = unpack_share(sh)
8601-                (u_seqnum, u_root_hash, IV, k, N, segsize, datalen,
8602-                 pubkey, signature, share_hash_chain, block_hash_tree,
8603-                 share_data, enc_privkey) = pieces
8604-                self.failUnlessEqual(u_seqnum, 3)
8605-                self.failUnlessEqual(u_root_hash, root_hash)
8606-                self.failUnlessEqual(k, 3)
8607-                self.failUnlessEqual(N, 10)
8608-                self.failUnlessEqual(segsize, 21)
8609-                self.failUnlessEqual(datalen, len(CONTENTS))
8610-                self.failUnlessEqual(pubkey, p._pubkey.serialize())
8611-                sig_material = struct.pack(">BQ32s16s BBQQ",
8612-                                           0, p._new_seqnum, root_hash, IV,
8613-                                           k, N, segsize, datalen)
8614-                self.failUnless(p._pubkey.verify(sig_material, signature))
8615-                #self.failUnlessEqual(signature, p._privkey.sign(sig_material))
8616-                self.failUnlessEqual(len(share_hash_chain), 4) # ln2(10)++
8617-                for shnum,share_hash in share_hash_chain.items():
8618-                    self.failUnless(isinstance(shnum, int))
8619-                    self.failUnless(isinstance(share_hash, str))
8620-                    self.failUnlessEqual(len(share_hash), 32)
8621-                self.failUnless(isinstance(block_hash_tree, list))
8622-                self.failUnlessEqual(len(block_hash_tree), 1) # very small tree
8623-                self.failUnlessEqual(IV, "SALT"*4)
8624-                self.failUnlessEqual(len(share_data), len("%07d" % 1))
8625-                self.failUnlessEqual(enc_privkey, self._fn.get_encprivkey())
8626-        d.addCallback(_generated)
8627-        return d
8628-    test_generate.todo = "Write an equivalent of this for the new uploader"
8629-
8630-    # TODO: when we publish to 20 peers, we should get one share per peer on 10
8631-    # when we publish to 3 peers, we should get either 3 or 4 shares per peer
8632-    # when we publish to zero peers, we should get a NotEnoughSharesError
8633-
8634 class PublishMixin:
8635     def publish_one(self):
8636         # publish a file and create shares, which can then be manipulated
8637[interfaces.py: create IMutableUploadable
8638Kevan Carstensen <kevan@isnotajoke.com>**20100706215217
8639 Ignore-this: bee202ec2bfbd8e41f2d4019cce176c7
8640] hunk ./src/allmydata/interfaces.py 1693
8641         """The upload is finished, and whatever filehandle was in use may be
8642         closed."""
8643 
8644+
8645+class IMutableUploadable(Interface):
8646+    """
8647+    I represent content that is due to be uploaded to a mutable filecap.
8648+    """
8649+    # This is somewhat simpler than the IUploadable interface above
8650+    # because mutable files do not need to be concerned with possibly
8651+    # generating a CHK, nor with per-file keys. It is a subset of the
8652+    # methods in IUploadable, though, so we could just as well implement
8653+    # the mutable uploadables as IUploadables that don't happen to use
8654+    # those methods (with the understanding that the unused methods will
8655+    # never be called on such objects)
8656+    def get_size():
8657+        """
8658+        Returns a Deferred that fires with the size of the content held
8659+        by the uploadable.
8660+        """
8661+
8662+    def read(length):
8663+        """
8664+        Returns a list of strings which, when concatenated, are the next
8665+        length bytes of the file, or fewer if there are fewer bytes
8666+        between the current location and the end of the file.
8667+        """
8668+
8669+    def close():
8670+        """
8671+        The process that used the Uploadable is finished using it, so
8672+        the uploadable may be closed.
8673+        """
8674+
8675 class IUploadResults(Interface):
8676     """I am returned by upload() methods. I contain a number of public
8677     attributes which can be read to determine the results of the upload. Some
8678[mutable/publish.py: add MutableDataHandle and MutableFileHandle
8679Kevan Carstensen <kevan@isnotajoke.com>**20100706215257
8680 Ignore-this: 295ea3bc2a962fd14fb7877fc76c011c
8681] {
8682hunk ./src/allmydata/mutable/publish.py 8
8683 from zope.interface import implements
8684 from twisted.internet import defer
8685 from twisted.python import failure
8686-from allmydata.interfaces import IPublishStatus, SDMF_VERSION, MDMF_VERSION
8687+from allmydata.interfaces import IPublishStatus, SDMF_VERSION, MDMF_VERSION, \
8688+                                 IMutableUploadable
8689 from allmydata.util import base32, hashutil, mathutil, idlib, log
8690 from allmydata import hashtree, codec
8691 from allmydata.storage.server import si_b2a
8692hunk ./src/allmydata/mutable/publish.py 971
8693             e = UncoordinatedWriteError()
8694         f = failure.Failure(e)
8695         eventually(self.done_deferred.callback, f)
8696+
8697+
8698+class MutableFileHandle:
8699+    """
8700+    I am a mutable uploadable built around a filehandle-like object,
8701+    usually either a StringIO instance or a handle to an actual file.
8702+    """
8703+    implements(IMutableUploadable)
8704+
8705+    def __init__(self, filehandle):
8706+        # The filehandle is defined as a generally file-like object that
8707+        # has these two methods. We don't care beyond that.
8708+        assert hasattr(filehandle, "read")
8709+        assert hasattr(filehandle, "close")
8710+
8711+        self._filehandle = filehandle
8712+
8713+
8714+    def get_size(self):
8715+        """
8716+        I return the amount of data in my filehandle.
8717+        """
8718+        if not hasattr(self, "_size"):
8719+            old_position = self._filehandle.tell()
8720+            # Seek to the end of the file by seeking 0 bytes from the
8721+            # file's end
8722+            self._filehandle.seek(0, os.SEEK_END)
8723+            self._size = self._filehandle.tell()
8724+            # Restore the previous position, in case this was called
8725+            # after a read.
8726+            self._filehandle.seek(old_position)
8727+            assert self._filehandle.tell() == old_position
8728+
8729+        assert hasattr(self, "_size")
8730+        return self._size
8731+
8732+
8733+    def read(self, length):
8734+        """
8735+        I return some data (up to length bytes) from my filehandle.
8736+
8737+        In most cases, I return length bytes. If I don't, it is because
8738+        length is longer than the distance between my current position
8739+        in the file that I represent and its end. In that case, I return
8740+        as many bytes as I can before going over the EOF.
8741+        """
8742+        return [self._filehandle.read(length)]
8743+
8744+
8745+    def close(self):
8746+        """
8747+        I close the underlying filehandle. Any further operations on the
8748+        filehandle fail at this point.
8749+        """
8750+        self._filehandle.close()
8751+
8752+
8753+class MutableDataHandle(MutableFileHandle):
8754+    """
8755+    I am a mutable uploadable built around a string, which I then cast
8756+    into a StringIO and treat as a filehandle.
8757+    """
8758+
8759+    def __init__(self, s):
8760+        # Take a string and return a file-like uploadable.
8761+        assert isinstance(s, str)
8762+
8763+        MutableFileHandle.__init__(self, StringIO(s))
8764}
8765[mutable/publish.py: reorganize in preparation of file-like uploadables
8766Kevan Carstensen <kevan@isnotajoke.com>**20100706215541
8767 Ignore-this: 5346c9f919ee5b73807c8f287c64e8ce
8768] {
8769hunk ./src/allmydata/mutable/publish.py 4
8770 
8771 
8772 import os, struct, time
8773+from StringIO import StringIO
8774 from itertools import count
8775 from zope.interface import implements
8776 from twisted.internet import defer
8777hunk ./src/allmydata/mutable/publish.py 118
8778         self._status.set_helper(False)
8779         self._status.set_progress(0.0)
8780         self._status.set_active(True)
8781-        # We use this to control how the file is written.
8782-        version = self._node.get_version()
8783-        assert version in (SDMF_VERSION, MDMF_VERSION)
8784-        self._version = version
8785+        self._version = self._node.get_version()
8786+        assert self._version in (SDMF_VERSION, MDMF_VERSION)
8787+
8788 
8789     def get_status(self):
8790         return self._status
8791hunk ./src/allmydata/mutable/publish.py 141
8792 
8793         # 0. Setup encoding parameters, encoder, and other such things.
8794         # 1. Encrypt, encode, and publish segments.
8795+        self.data = StringIO(newdata)
8796+        self.datalength = len(newdata)
8797 
8798hunk ./src/allmydata/mutable/publish.py 144
8799-        self.log("starting publish, datalen is %s" % len(newdata))
8800-        self._status.set_size(len(newdata))
8801+        self.log("starting publish, datalen is %s" % self.datalength)
8802+        self._status.set_size(self.datalength)
8803         self._status.set_status("Started")
8804         self._started = time.time()
8805 
8806hunk ./src/allmydata/mutable/publish.py 193
8807         self.full_peerlist = full_peerlist # for use later, immutable
8808         self.bad_peers = set() # peerids who have errbacked/refused requests
8809 
8810-        self.newdata = newdata
8811-
8812         # This will set self.segment_size, self.num_segments, and
8813         # self.fec.
8814         self.setup_encoding_parameters()
8815hunk ./src/allmydata/mutable/publish.py 272
8816                                                 self.required_shares,
8817                                                 self.total_shares,
8818                                                 self.segment_size,
8819-                                                len(self.newdata))
8820+                                                self.datalength)
8821             self.writers[shnum].peerid = peerid
8822             if (peerid, shnum) in self._servermap.servermap:
8823                 old_versionid, old_timestamp = self._servermap.servermap[key]
8824hunk ./src/allmydata/mutable/publish.py 318
8825         if self._version == MDMF_VERSION:
8826             segment_size = DEFAULT_MAX_SEGMENT_SIZE # 128 KiB by default
8827         else:
8828-            segment_size = len(self.newdata) # SDMF is only one segment
8829+            segment_size = self.datalength # SDMF is only one segment
8830         # this must be a multiple of self.required_shares
8831         segment_size = mathutil.next_multiple(segment_size,
8832                                               self.required_shares)
8833hunk ./src/allmydata/mutable/publish.py 324
8834         self.segment_size = segment_size
8835         if segment_size:
8836-            self.num_segments = mathutil.div_ceil(len(self.newdata),
8837+            self.num_segments = mathutil.div_ceil(self.datalength,
8838                                                   segment_size)
8839         else:
8840             self.num_segments = 0
8841hunk ./src/allmydata/mutable/publish.py 337
8842             assert self.num_segments in (0, 1) # SDMF
8843         # calculate the tail segment size.
8844 
8845-        if segment_size and self.newdata:
8846-            self.tail_segment_size = len(self.newdata) % segment_size
8847+        if segment_size and self.datalength:
8848+            self.tail_segment_size = self.datalength % segment_size
8849         else:
8850             self.tail_segment_size = 0
8851 
8852hunk ./src/allmydata/mutable/publish.py 438
8853             segsize = self.segment_size
8854 
8855 
8856-        offset = self.segment_size * segnum
8857-        length = segsize + offset
8858         self.log("Pushing segment %d of %d" % (segnum + 1, self.num_segments))
8859hunk ./src/allmydata/mutable/publish.py 439
8860-        data = self.newdata[offset:length]
8861+        data = self.data.read(segsize)
8862+
8863         assert len(data) == segsize
8864 
8865         salt = os.urandom(16)
8866hunk ./src/allmydata/mutable/publish.py 502
8867             d.addCallback(self._got_write_answer, writer, started)
8868             d.addErrback(self._connection_problem, writer)
8869             dl.append(d)
8870-            # TODO: Naturally, we need to check on the results of these.
8871         return defer.DeferredList(dl)
8872 
8873 
8874}
8875[test/test_mutable.py: write tests for MutableFileHandle and MutableDataHandle
8876Kevan Carstensen <kevan@isnotajoke.com>**20100706215649
8877 Ignore-this: df719a0c52b4bbe9be4fae206c7ab3e7
8878] {
8879hunk ./src/allmydata/test/test_mutable.py 2
8880 
8881-import struct
8882+import struct, os
8883 from cStringIO import StringIO
8884 from twisted.trial import unittest
8885 from twisted.internet import defer, reactor
8886hunk ./src/allmydata/test/test_mutable.py 26
8887      NeedMoreDataError, UnrecoverableFileError, UncoordinatedWriteError, \
8888      NotEnoughServersError, CorruptShareError
8889 from allmydata.mutable.retrieve import Retrieve
8890-from allmydata.mutable.publish import Publish
8891+from allmydata.mutable.publish import Publish, MutableFileHandle, \
8892+                                      MutableDataHandle
8893 from allmydata.mutable.servermap import ServerMap, ServermapUpdater
8894 from allmydata.mutable.layout import unpack_header, unpack_share, \
8895                                      MDMFSlotReadProxy
8896hunk ./src/allmydata/test/test_mutable.py 2465
8897         d.addCallback(lambda data:
8898             self.failUnlessEqual(data, CONTENTS))
8899         return d
8900+
8901+
8902+class FileHandle(unittest.TestCase):
8903+    def setUp(self):
8904+        self.test_data = "Test Data" * 50000
8905+        self.sio = StringIO(self.test_data)
8906+        self.uploadable = MutableFileHandle(self.sio)
8907+
8908+
8909+    def test_filehandle_read(self):
8910+        self.basedir = "mutable/FileHandle/test_filehandle_read"
8911+        chunk_size = 10
8912+        for i in xrange(0, len(self.test_data), chunk_size):
8913+            data = self.uploadable.read(chunk_size)
8914+            data = "".join(data)
8915+            start = i
8916+            end = i + chunk_size
8917+            self.failUnlessEqual(data, self.test_data[start:end])
8918+
8919+
8920+    def test_filehandle_get_size(self):
8921+        self.basedir = "mutable/FileHandle/test_filehandle_get_size"
8922+        actual_size = len(self.test_data)
8923+        size = self.uploadable.get_size()
8924+        self.failUnlessEqual(size, actual_size)
8925+
8926+
8927+    def test_filehandle_get_size_out_of_order(self):
8928+        # We should be able to call get_size whenever we want without
8929+        # disturbing the location of the seek pointer.
8930+        chunk_size = 100
8931+        data = self.uploadable.read(chunk_size)
8932+        self.failUnlessEqual("".join(data), self.test_data[:chunk_size])
8933+
8934+        # Now get the size.
8935+        size = self.uploadable.get_size()
8936+        self.failUnlessEqual(size, len(self.test_data))
8937+
8938+        # Now get more data. We should be right where we left off.
8939+        more_data = self.uploadable.read(chunk_size)
8940+        start = chunk_size
8941+        end = chunk_size * 2
8942+        self.failUnlessEqual("".join(more_data), self.test_data[start:end])
8943+
8944+
8945+    def test_filehandle_file(self):
8946+        # Make sure that the MutableFileHandle works on a file as well
8947+        # as a StringIO object, since in some cases it will be asked to
8948+        # deal with files.
8949+        self.basedir = self.mktemp()
8950+        # necessary? What am I doing wrong here?
8951+        os.mkdir(self.basedir)
8952+        f_path = os.path.join(self.basedir, "test_file")
8953+        f = open(f_path, "w")
8954+        f.write(self.test_data)
8955+        f.close()
8956+        f = open(f_path, "r")
8957+
8958+        uploadable = MutableFileHandle(f)
8959+
8960+        data = uploadable.read(len(self.test_data))
8961+        self.failUnlessEqual("".join(data), self.test_data)
8962+        size = uploadable.get_size()
8963+        self.failUnlessEqual(size, len(self.test_data))
8964+
8965+
8966+    def test_close(self):
8967+        # Make sure that the MutableFileHandle closes its handle when
8968+        # told to do so.
8969+        self.uploadable.close()
8970+        self.failUnless(self.sio.closed)
8971+
8972+
8973+class DataHandle(unittest.TestCase):
8974+    def setUp(self):
8975+        self.test_data = "Test Data" * 50000
8976+        self.uploadable = MutableDataHandle(self.test_data)
8977+
8978+
8979+    def test_datahandle_read(self):
8980+        chunk_size = 10
8981+        for i in xrange(0, len(self.test_data), chunk_size):
8982+            data = self.uploadable.read(chunk_size)
8983+            data = "".join(data)
8984+            start = i
8985+            end = i + chunk_size
8986+            self.failUnlessEqual(data, self.test_data[start:end])
8987+
8988+
8989+    def test_datahandle_get_size(self):
8990+        actual_size = len(self.test_data)
8991+        size = self.uploadable.get_size()
8992+        self.failUnlessEqual(size, actual_size)
8993+
8994+
8995+    def test_datahandle_get_size_out_of_order(self):
8996+        # We should be able to call get_size whenever we want without
8997+        # disturbing the location of the seek pointer.
8998+        chunk_size = 100
8999+        data = self.uploadable.read(chunk_size)
9000+        self.failUnlessEqual("".join(data), self.test_data[:chunk_size])
9001+
9002+        # Now get the size.
9003+        size = self.uploadable.get_size()
9004+        self.failUnlessEqual(size, len(self.test_data))
9005+
9006+        # Now get more data. We should be right where we left off.
9007+        more_data = self.uploadable.read(chunk_size)
9008+        start = chunk_size
9009+        end = chunk_size * 2
9010+        self.failUnlessEqual("".join(more_data), self.test_data[start:end])
9011}
9012[Alter tests to work with the new APIs
9013Kevan Carstensen <kevan@isnotajoke.com>**20100708000031
9014 Ignore-this: 1f377904ac61ce40e9a04716fbd2ad95
9015] {
9016hunk ./src/allmydata/test/common.py 12
9017 from allmydata import uri, dirnode, client
9018 from allmydata.introducer.server import IntroducerNode
9019 from allmydata.interfaces import IMutableFileNode, IImmutableFileNode, \
9020-     FileTooLargeError, NotEnoughSharesError, ICheckable
9021+     FileTooLargeError, NotEnoughSharesError, ICheckable, \
9022+     IMutableUploadable
9023 from allmydata.check_results import CheckResults, CheckAndRepairResults, \
9024      DeepCheckResults, DeepCheckAndRepairResults
9025 from allmydata.mutable.common import CorruptShareError
9026hunk ./src/allmydata/test/common.py 18
9027 from allmydata.mutable.layout import unpack_header
9028+from allmydata.mutable.publish import MutableDataHandle
9029 from allmydata.storage.server import storage_index_to_dir
9030 from allmydata.storage.mutable import MutableShareFile
9031 from allmydata.util import hashutil, log, fileutil, pollmixin
9032hunk ./src/allmydata/test/common.py 182
9033         self.init_from_cap(make_mutable_file_cap())
9034     def create(self, contents, key_generator=None, keysize=None):
9035         initial_contents = self._get_initial_contents(contents)
9036-        if len(initial_contents) > self.MUTABLE_SIZELIMIT:
9037+        if initial_contents.get_size() > self.MUTABLE_SIZELIMIT:
9038             raise FileTooLargeError("SDMF is limited to one segment, and "
9039hunk ./src/allmydata/test/common.py 184
9040-                                    "%d > %d" % (len(initial_contents),
9041+                                    "%d > %d" % (initial_contents.get_size(),
9042                                                  self.MUTABLE_SIZELIMIT))
9043hunk ./src/allmydata/test/common.py 186
9044-        self.all_contents[self.storage_index] = initial_contents
9045+        data = initial_contents.read(initial_contents.get_size())
9046+        data = "".join(data)
9047+        self.all_contents[self.storage_index] = data
9048         return defer.succeed(self)
9049     def _get_initial_contents(self, contents):
9050hunk ./src/allmydata/test/common.py 191
9051-        if isinstance(contents, str):
9052-            return contents
9053         if contents is None:
9054hunk ./src/allmydata/test/common.py 192
9055-            return ""
9056+            return MutableDataHandle("")
9057+
9058+        if IMutableUploadable.providedBy(contents):
9059+            return contents
9060+
9061         assert callable(contents), "%s should be callable, not %s" % \
9062                (contents, type(contents))
9063         return contents(self)
9064hunk ./src/allmydata/test/common.py 309
9065         return defer.succeed(self.all_contents[self.storage_index])
9066 
9067     def overwrite(self, new_contents):
9068-        if len(new_contents) > self.MUTABLE_SIZELIMIT:
9069+        if new_contents.get_size() > self.MUTABLE_SIZELIMIT:
9070             raise FileTooLargeError("SDMF is limited to one segment, and "
9071hunk ./src/allmydata/test/common.py 311
9072-                                    "%d > %d" % (len(new_contents),
9073+                                    "%d > %d" % (new_contents.get_size(),
9074                                                  self.MUTABLE_SIZELIMIT))
9075         assert not self.is_readonly()
9076hunk ./src/allmydata/test/common.py 314
9077-        self.all_contents[self.storage_index] = new_contents
9078+        new_data = new_contents.read(new_contents.get_size())
9079+        new_data = "".join(new_data)
9080+        self.all_contents[self.storage_index] = new_data
9081         return defer.succeed(None)
9082     def modify(self, modifier):
9083         # this does not implement FileTooLargeError, but the real one does
9084hunk ./src/allmydata/test/common.py 324
9085     def _modify(self, modifier):
9086         assert not self.is_readonly()
9087         old_contents = self.all_contents[self.storage_index]
9088-        self.all_contents[self.storage_index] = modifier(old_contents, None, True)
9089+        new_data = modifier(old_contents, None, True)
9090+        if new_data is not None:
9091+            new_data = new_data.read(new_data.get_size())
9092+            new_data = "".join(new_data)
9093+        self.all_contents[self.storage_index] = new_data
9094         return None
9095 
9096 def make_mutable_file_cap():
9097hunk ./src/allmydata/test/test_checker.py 11
9098 from allmydata.test.no_network import GridTestMixin
9099 from allmydata.immutable.upload import Data
9100 from allmydata.test.common_web import WebRenderingMixin
9101+from allmydata.mutable.publish import MutableDataHandle
9102 
9103 class FakeClient:
9104     def get_storage_broker(self):
9105hunk ./src/allmydata/test/test_checker.py 291
9106         def _stash_immutable(ur):
9107             self.imm = c0.create_node_from_uri(ur.uri)
9108         d.addCallback(_stash_immutable)
9109-        d.addCallback(lambda ign: c0.create_mutable_file("contents"))
9110+        d.addCallback(lambda ign:
9111+            c0.create_mutable_file(MutableDataHandle("contents")))
9112         def _stash_mutable(node):
9113             self.mut = node
9114         d.addCallback(_stash_mutable)
9115hunk ./src/allmydata/test/test_cli.py 12
9116 from allmydata.util import fileutil, hashutil, base32
9117 from allmydata import uri
9118 from allmydata.immutable import upload
9119+from allmydata.mutable.publish import MutableDataHandle
9120 from allmydata.dirnode import normalize
9121 
9122 # Test that the scripts can be imported -- although the actual tests of their
9123hunk ./src/allmydata/test/test_cli.py 1975
9124         self.set_up_grid()
9125         c0 = self.g.clients[0]
9126         DATA = "data" * 100
9127-        d = c0.create_mutable_file(DATA)
9128+        DATA_uploadable = MutableDataHandle(DATA)
9129+        d = c0.create_mutable_file(DATA_uploadable)
9130         def _stash_uri(n):
9131             self.uri = n.get_uri()
9132         d.addCallback(_stash_uri)
9133hunk ./src/allmydata/test/test_cli.py 2077
9134                                            upload.Data("literal",
9135                                                         convergence="")))
9136         d.addCallback(_stash_uri, "small")
9137-        d.addCallback(lambda ign: c0.create_mutable_file(DATA+"1"))
9138+        d.addCallback(lambda ign:
9139+            c0.create_mutable_file(MutableDataHandle(DATA+"1")))
9140         d.addCallback(lambda fn: self.rootnode.set_node(u"mutable", fn))
9141         d.addCallback(_stash_uri, "mutable")
9142 
9143hunk ./src/allmydata/test/test_cli.py 2096
9144         # root/small
9145         # root/mutable
9146 
9147+        # We haven't broken anything yet, so this should all be healthy.
9148         d.addCallback(lambda ign: self.do_cli("deep-check", "--verbose",
9149                                               self.rooturi))
9150         def _check2((rc, out, err)):
9151hunk ./src/allmydata/test/test_cli.py 2111
9152                             in lines, out)
9153         d.addCallback(_check2)
9154 
9155+        # Similarly, all of these results should be as we expect them to
9156+        # be for a healthy file layout.
9157         d.addCallback(lambda ign: self.do_cli("stats", self.rooturi))
9158         def _check_stats((rc, out, err)):
9159             self.failUnlessReallyEqual(err, "")
9160hunk ./src/allmydata/test/test_cli.py 2128
9161             self.failUnlessIn(" 317-1000 : 1    (1000 B, 1000 B)", lines)
9162         d.addCallback(_check_stats)
9163 
9164+        # Now we break things.
9165         def _clobber_shares(ignored):
9166             shares = self.find_uri_shares(self.uris[u"gööd"])
9167             self.failUnlessReallyEqual(len(shares), 10)
9168hunk ./src/allmydata/test/test_cli.py 2147
9169         d.addCallback(_clobber_shares)
9170 
9171         # root
9172-        # root/gööd  [9 shares]
9173+        # root/gööd  [1 missing share]
9174         # root/small
9175         # root/mutable [1 corrupt share]
9176 
9177hunk ./src/allmydata/test/test_cli.py 2153
9178         d.addCallback(lambda ign:
9179                       self.do_cli("deep-check", "--verbose", self.rooturi))
9180+        # This should reveal the missing share, but not the corrupt
9181+        # share, since we didn't tell the deep check operation to also
9182+        # verify.
9183         def _check3((rc, out, err)):
9184             self.failUnlessReallyEqual(err, "")
9185             self.failUnlessReallyEqual(rc, 0)
9186hunk ./src/allmydata/test/test_cli.py 2204
9187                                   "--verbose", "--verify", "--repair",
9188                                   self.rooturi))
9189         def _check6((rc, out, err)):
9190+            # We've just repaired the directory. There is no reason for
9191+            # that repair to be unsuccessful.
9192             self.failUnlessReallyEqual(err, "")
9193             self.failUnlessReallyEqual(rc, 0)
9194             lines = out.splitlines()
9195hunk ./src/allmydata/test/test_deepcheck.py 9
9196 from twisted.internet import threads # CLI tests use deferToThread
9197 from allmydata.immutable import upload
9198 from allmydata.mutable.common import UnrecoverableFileError
9199+from allmydata.mutable.publish import MutableDataHandle
9200 from allmydata.util import idlib
9201 from allmydata.util import base32
9202 from allmydata.scripts import runner
9203hunk ./src/allmydata/test/test_deepcheck.py 38
9204         self.basedir = "deepcheck/MutableChecker/good"
9205         self.set_up_grid()
9206         CONTENTS = "a little bit of data"
9207-        d = self.g.clients[0].create_mutable_file(CONTENTS)
9208+        CONTENTS_uploadable = MutableDataHandle(CONTENTS)
9209+        d = self.g.clients[0].create_mutable_file(CONTENTS_uploadable)
9210         def _created(node):
9211             self.node = node
9212             self.fileurl = "uri/" + urllib.quote(node.get_uri())
9213hunk ./src/allmydata/test/test_deepcheck.py 61
9214         self.basedir = "deepcheck/MutableChecker/corrupt"
9215         self.set_up_grid()
9216         CONTENTS = "a little bit of data"
9217-        d = self.g.clients[0].create_mutable_file(CONTENTS)
9218+        CONTENTS_uploadable = MutableDataHandle(CONTENTS)
9219+        d = self.g.clients[0].create_mutable_file(CONTENTS_uploadable)
9220         def _stash_and_corrupt(node):
9221             self.node = node
9222             self.fileurl = "uri/" + urllib.quote(node.get_uri())
9223hunk ./src/allmydata/test/test_deepcheck.py 99
9224         self.basedir = "deepcheck/MutableChecker/delete_share"
9225         self.set_up_grid()
9226         CONTENTS = "a little bit of data"
9227-        d = self.g.clients[0].create_mutable_file(CONTENTS)
9228+        CONTENTS_uploadable = MutableDataHandle(CONTENTS)
9229+        d = self.g.clients[0].create_mutable_file(CONTENTS_uploadable)
9230         def _stash_and_delete(node):
9231             self.node = node
9232             self.fileurl = "uri/" + urllib.quote(node.get_uri())
9233hunk ./src/allmydata/test/test_deepcheck.py 223
9234             self.root = n
9235             self.root_uri = n.get_uri()
9236         d.addCallback(_created_root)
9237-        d.addCallback(lambda ign: c0.create_mutable_file("mutable file contents"))
9238+        d.addCallback(lambda ign:
9239+            c0.create_mutable_file(MutableDataHandle("mutable file contents")))
9240         d.addCallback(lambda n: self.root.set_node(u"mutable", n))
9241         def _created_mutable(n):
9242             self.mutable = n
9243hunk ./src/allmydata/test/test_deepcheck.py 965
9244     def create_mangled(self, ignored, name):
9245         nodetype, mangletype = name.split("-", 1)
9246         if nodetype == "mutable":
9247-            d = self.g.clients[0].create_mutable_file("mutable file contents")
9248+            mutable_uploadable = MutableDataHandle("mutable file contents")
9249+            d = self.g.clients[0].create_mutable_file(mutable_uploadable)
9250             d.addCallback(lambda n: self.root.set_node(unicode(name), n))
9251         elif nodetype == "large":
9252             large = upload.Data("Lots of data\n" * 1000 + name + "\n", None)
9253hunk ./src/allmydata/test/test_dirnode.py 1305
9254     implements(IMutableFileNode)
9255     counter = 0
9256     def __init__(self, initial_contents=""):
9257-        self.data = self._get_initial_contents(initial_contents)
9258+        data = self._get_initial_contents(initial_contents)
9259+        self.data = data.read(data.get_size())
9260+        self.data = "".join(self.data)
9261+
9262         counter = FakeMutableFile.counter
9263         FakeMutableFile.counter += 1
9264         writekey = hashutil.ssk_writekey_hash(str(counter))
9265hunk ./src/allmydata/test/test_dirnode.py 1355
9266         pass
9267 
9268     def modify(self, modifier):
9269-        self.data = modifier(self.data, None, True)
9270+        data = modifier(self.data, None, True)
9271+        self.data = data.read(data.get_size())
9272+        self.data = "".join(self.data)
9273         return defer.succeed(None)
9274 
9275 class FakeNodeMaker(NodeMaker):
9276hunk ./src/allmydata/test/test_hung_server.py 10
9277 from allmydata.util.consumer import download_to_data
9278 from allmydata.immutable import upload
9279 from allmydata.mutable.common import UnrecoverableFileError
9280+from allmydata.mutable.publish import MutableDataHandle
9281 from allmydata.storage.common import storage_index_to_dir
9282 from allmydata.test.no_network import GridTestMixin
9283 from allmydata.test.common import ShouldFailMixin, _corrupt_share_data
9284hunk ./src/allmydata/test/test_hung_server.py 96
9285         self.servers = [(id, ss) for (id, ss) in nm.storage_broker.get_all_servers()]
9286 
9287         if mutable:
9288-            d = nm.create_mutable_file(mutable_plaintext)
9289+            uploadable = MutableDataHandle(mutable_plaintext)
9290+            d = nm.create_mutable_file(uploadable)
9291             def _uploaded_mutable(node):
9292                 self.uri = node.get_uri()
9293                 self.shares = self.find_uri_shares(self.uri)
9294hunk ./src/allmydata/test/test_mutable.py 297
9295             d.addCallback(lambda smap: smap.dump(StringIO()))
9296             d.addCallback(lambda sio:
9297                           self.failUnless("3-of-10" in sio.getvalue()))
9298-            d.addCallback(lambda res: n.overwrite("contents 1"))
9299+            d.addCallback(lambda res: n.overwrite(MutableDataHandle("contents 1")))
9300             d.addCallback(lambda res: self.failUnlessIdentical(res, None))
9301             d.addCallback(lambda res: n.download_best_version())
9302             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 1"))
9303hunk ./src/allmydata/test/test_mutable.py 304
9304             d.addCallback(lambda res: n.get_size_of_best_version())
9305             d.addCallback(lambda size:
9306                           self.failUnlessEqual(size, len("contents 1")))
9307-            d.addCallback(lambda res: n.overwrite("contents 2"))
9308+            d.addCallback(lambda res: n.overwrite(MutableDataHandle("contents 2")))
9309             d.addCallback(lambda res: n.download_best_version())
9310             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 2"))
9311             d.addCallback(lambda res: n.get_servermap(MODE_WRITE))
9312hunk ./src/allmydata/test/test_mutable.py 308
9313-            d.addCallback(lambda smap: n.upload("contents 3", smap))
9314+            d.addCallback(lambda smap: n.upload(MutableDataHandle("contents 3"), smap))
9315             d.addCallback(lambda res: n.download_best_version())
9316             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 3"))
9317             d.addCallback(lambda res: n.get_servermap(MODE_ANYTHING))
9318hunk ./src/allmydata/test/test_mutable.py 320
9319             # mapupdate-to-retrieve data caching (i.e. make the shares larger
9320             # than the default readsize, which is 2000 bytes). A 15kB file
9321             # will have 5kB shares.
9322-            d.addCallback(lambda res: n.overwrite("large size file" * 1000))
9323+            d.addCallback(lambda res: n.overwrite(MutableDataHandle("large size file" * 1000)))
9324             d.addCallback(lambda res: n.download_best_version())
9325             d.addCallback(lambda res:
9326                           self.failUnlessEqual(res, "large size file" * 1000))
9327hunk ./src/allmydata/test/test_mutable.py 343
9328             # to make them big enough to force the file to be uploaded
9329             # in more than one segment.
9330             big_contents = "contents1" * 100000 # about 900 KiB
9331+            big_contents_uploadable = MutableDataHandle(big_contents)
9332             d.addCallback(lambda ignored:
9333hunk ./src/allmydata/test/test_mutable.py 345
9334-                n.overwrite(big_contents))
9335+                n.overwrite(big_contents_uploadable))
9336             d.addCallback(lambda ignored:
9337                 n.download_best_version())
9338             d.addCallback(lambda data:
9339hunk ./src/allmydata/test/test_mutable.py 355
9340             # segments, so that we make the downloader deal with
9341             # multiple segments.
9342             bigger_contents = "contents2" * 1000000 # about 9MiB
9343+            bigger_contents_uploadable = MutableDataHandle(bigger_contents)
9344             d.addCallback(lambda ignored:
9345hunk ./src/allmydata/test/test_mutable.py 357
9346-                n.overwrite(bigger_contents))
9347+                n.overwrite(bigger_contents_uploadable))
9348             d.addCallback(lambda ignored:
9349                 n.download_best_version())
9350             d.addCallback(lambda data:
9351hunk ./src/allmydata/test/test_mutable.py 368
9352 
9353 
9354     def test_create_with_initial_contents(self):
9355-        d = self.nodemaker.create_mutable_file("contents 1")
9356+        upload1 = MutableDataHandle("contents 1")
9357+        d = self.nodemaker.create_mutable_file(upload1)
9358         def _created(n):
9359             d = n.download_best_version()
9360             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 1"))
9361hunk ./src/allmydata/test/test_mutable.py 373
9362-            d.addCallback(lambda res: n.overwrite("contents 2"))
9363+            upload2 = MutableDataHandle("contents 2")
9364+            d.addCallback(lambda res: n.overwrite(upload2))
9365             d.addCallback(lambda res: n.download_best_version())
9366             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 2"))
9367             return d
9368hunk ./src/allmydata/test/test_mutable.py 380
9369         d.addCallback(_created)
9370         return d
9371+    test_create_with_initial_contents.timeout = 15
9372 
9373 
9374     def test_create_mdmf_with_initial_contents(self):
9375hunk ./src/allmydata/test/test_mutable.py 385
9376         initial_contents = "foobarbaz" * 131072 # 900KiB
9377-        d = self.nodemaker.create_mutable_file(initial_contents,
9378+        initial_contents_uploadable = MutableDataHandle(initial_contents)
9379+        d = self.nodemaker.create_mutable_file(initial_contents_uploadable,
9380                                                version=MDMF_VERSION)
9381         def _created(n):
9382             d = n.download_best_version()
9383hunk ./src/allmydata/test/test_mutable.py 392
9384             d.addCallback(lambda data:
9385                 self.failUnlessEqual(data, initial_contents))
9386+            uploadable2 = MutableDataHandle(initial_contents + "foobarbaz")
9387             d.addCallback(lambda ignored:
9388hunk ./src/allmydata/test/test_mutable.py 394
9389-                n.overwrite(initial_contents + "foobarbaz"))
9390+                n.overwrite(uploadable2))
9391             d.addCallback(lambda ignored:
9392                 n.download_best_version())
9393             d.addCallback(lambda data:
9394hunk ./src/allmydata/test/test_mutable.py 413
9395             key = n.get_writekey()
9396             self.failUnless(isinstance(key, str), key)
9397             self.failUnlessEqual(len(key), 16) # AES key size
9398-            return data
9399+            return MutableDataHandle(data)
9400         d = self.nodemaker.create_mutable_file(_make_contents)
9401         def _created(n):
9402             return n.download_best_version()
9403hunk ./src/allmydata/test/test_mutable.py 429
9404             key = n.get_writekey()
9405             self.failUnless(isinstance(key, str), key)
9406             self.failUnlessEqual(len(key), 16)
9407-            return data
9408+            return MutableDataHandle(data)
9409         d = self.nodemaker.create_mutable_file(_make_contents,
9410                                                version=MDMF_VERSION)
9411         d.addCallback(lambda n:
9412hunk ./src/allmydata/test/test_mutable.py 441
9413 
9414     def test_create_with_too_large_contents(self):
9415         BIG = "a" * (self.OLD_MAX_SEGMENT_SIZE + 1)
9416-        d = self.nodemaker.create_mutable_file(BIG)
9417+        BIG_uploadable = MutableDataHandle(BIG)
9418+        d = self.nodemaker.create_mutable_file(BIG_uploadable)
9419         def _created(n):
9420hunk ./src/allmydata/test/test_mutable.py 444
9421-            d = n.overwrite(BIG)
9422+            other_BIG_uploadable = MutableDataHandle(BIG)
9423+            d = n.overwrite(other_BIG_uploadable)
9424             return d
9425         d.addCallback(_created)
9426         return d
9427hunk ./src/allmydata/test/test_mutable.py 459
9428 
9429     def test_modify(self):
9430         def _modifier(old_contents, servermap, first_time):
9431-            return old_contents + "line2"
9432+            new_contents = old_contents + "line2"
9433+            return MutableDataHandle(new_contents)
9434         def _non_modifier(old_contents, servermap, first_time):
9435hunk ./src/allmydata/test/test_mutable.py 462
9436-            return old_contents
9437+            return MutableDataHandle(old_contents)
9438         def _none_modifier(old_contents, servermap, first_time):
9439             return None
9440         def _error_modifier(old_contents, servermap, first_time):
9441hunk ./src/allmydata/test/test_mutable.py 468
9442             raise ValueError("oops")
9443         def _toobig_modifier(old_contents, servermap, first_time):
9444-            return "b" * (self.OLD_MAX_SEGMENT_SIZE+1)
9445+            new_content = "b" * (self.OLD_MAX_SEGMENT_SIZE + 1)
9446+            return MutableDataHandle(new_content)
9447         calls = []
9448         def _ucw_error_modifier(old_contents, servermap, first_time):
9449             # simulate an UncoordinatedWriteError once
9450hunk ./src/allmydata/test/test_mutable.py 476
9451             calls.append(1)
9452             if len(calls) <= 1:
9453                 raise UncoordinatedWriteError("simulated")
9454-            return old_contents + "line3"
9455+            new_contents = old_contents + "line3"
9456+            return MutableDataHandle(new_contents)
9457         def _ucw_error_non_modifier(old_contents, servermap, first_time):
9458             # simulate an UncoordinatedWriteError once, and don't actually
9459             # modify the contents on subsequent invocations
9460hunk ./src/allmydata/test/test_mutable.py 484
9461             calls.append(1)
9462             if len(calls) <= 1:
9463                 raise UncoordinatedWriteError("simulated")
9464-            return old_contents
9465+            return MutableDataHandle(old_contents)
9466 
9467hunk ./src/allmydata/test/test_mutable.py 486
9468-        d = self.nodemaker.create_mutable_file("line1")
9469+        initial_contents = "line1"
9470+        d = self.nodemaker.create_mutable_file(MutableDataHandle(initial_contents))
9471         def _created(n):
9472             d = n.modify(_modifier)
9473             d.addCallback(lambda res: n.download_best_version())
9474hunk ./src/allmydata/test/test_mutable.py 548
9475 
9476     def test_modify_backoffer(self):
9477         def _modifier(old_contents, servermap, first_time):
9478-            return old_contents + "line2"
9479+            return MutableDataHandle(old_contents + "line2")
9480         calls = []
9481         def _ucw_error_modifier(old_contents, servermap, first_time):
9482             # simulate an UncoordinatedWriteError once
9483hunk ./src/allmydata/test/test_mutable.py 555
9484             calls.append(1)
9485             if len(calls) <= 1:
9486                 raise UncoordinatedWriteError("simulated")
9487-            return old_contents + "line3"
9488+            return MutableDataHandle(old_contents + "line3")
9489         def _always_ucw_error_modifier(old_contents, servermap, first_time):
9490             raise UncoordinatedWriteError("simulated")
9491         def _backoff_stopper(node, f):
9492hunk ./src/allmydata/test/test_mutable.py 570
9493         giveuper._delay = 0.1
9494         giveuper.factor = 1
9495 
9496-        d = self.nodemaker.create_mutable_file("line1")
9497+        d = self.nodemaker.create_mutable_file(MutableDataHandle("line1"))
9498         def _created(n):
9499             d = n.modify(_modifier)
9500             d.addCallback(lambda res: n.download_best_version())
9501hunk ./src/allmydata/test/test_mutable.py 620
9502             d.addCallback(lambda smap: smap.dump(StringIO()))
9503             d.addCallback(lambda sio:
9504                           self.failUnless("3-of-10" in sio.getvalue()))
9505-            d.addCallback(lambda res: n.overwrite("contents 1"))
9506+            d.addCallback(lambda res: n.overwrite(MutableDataHandle("contents 1")))
9507             d.addCallback(lambda res: self.failUnlessIdentical(res, None))
9508             d.addCallback(lambda res: n.download_best_version())
9509             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 1"))
9510hunk ./src/allmydata/test/test_mutable.py 624
9511-            d.addCallback(lambda res: n.overwrite("contents 2"))
9512+            d.addCallback(lambda res: n.overwrite(MutableDataHandle("contents 2")))
9513             d.addCallback(lambda res: n.download_best_version())
9514             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 2"))
9515             d.addCallback(lambda res: n.get_servermap(MODE_WRITE))
9516hunk ./src/allmydata/test/test_mutable.py 628
9517-            d.addCallback(lambda smap: n.upload("contents 3", smap))
9518+            d.addCallback(lambda smap: n.upload(MutableDataHandle("contents 3"), smap))
9519             d.addCallback(lambda res: n.download_best_version())
9520             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 3"))
9521             d.addCallback(lambda res: n.get_servermap(MODE_ANYTHING))
9522hunk ./src/allmydata/test/test_mutable.py 646
9523         # publish a file and create shares, which can then be manipulated
9524         # later.
9525         self.CONTENTS = "New contents go here" * 1000
9526+        self.uploadable = MutableDataHandle(self.CONTENTS)
9527         self._storage = FakeStorage()
9528         self._nodemaker = make_nodemaker(self._storage)
9529         self._storage_broker = self._nodemaker.storage_broker
9530hunk ./src/allmydata/test/test_mutable.py 650
9531-        d = self._nodemaker.create_mutable_file(self.CONTENTS)
9532+        d = self._nodemaker.create_mutable_file(self.uploadable)
9533         def _created(node):
9534             self._fn = node
9535             self._fn2 = self._nodemaker.create_from_cap(node.get_uri())
9536hunk ./src/allmydata/test/test_mutable.py 662
9537         # an MDMF file.
9538         # self.CONTENTS should have more than one segment.
9539         self.CONTENTS = "This is an MDMF file" * 100000
9540+        self.uploadable = MutableDataHandle(self.CONTENTS)
9541         self._storage = FakeStorage()
9542         self._nodemaker = make_nodemaker(self._storage)
9543         self._storage_broker = self._nodemaker.storage_broker
9544hunk ./src/allmydata/test/test_mutable.py 666
9545-        d = self._nodemaker.create_mutable_file(self.CONTENTS, version=1)
9546+        d = self._nodemaker.create_mutable_file(self.uploadable, version=MDMF_VERSION)
9547         def _created(node):
9548             self._fn = node
9549             self._fn2 = self._nodemaker.create_from_cap(node.get_uri())
9550hunk ./src/allmydata/test/test_mutable.py 678
9551         # like publish_one, except that the result is guaranteed to be
9552         # an SDMF file
9553         self.CONTENTS = "This is an SDMF file" * 1000
9554+        self.uploadable = MutableDataHandle(self.CONTENTS)
9555         self._storage = FakeStorage()
9556         self._nodemaker = make_nodemaker(self._storage)
9557         self._storage_broker = self._nodemaker.storage_broker
9558hunk ./src/allmydata/test/test_mutable.py 682
9559-        d = self._nodemaker.create_mutable_file(self.CONTENTS, version=0)
9560+        d = self._nodemaker.create_mutable_file(self.uploadable, version=SDMF_VERSION)
9561         def _created(node):
9562             self._fn = node
9563             self._fn2 = self._nodemaker.create_from_cap(node.get_uri())
9564hunk ./src/allmydata/test/test_mutable.py 696
9565                          "Contents 2",
9566                          "Contents 3a",
9567                          "Contents 3b"]
9568+        self.uploadables = [MutableDataHandle(d) for d in self.CONTENTS]
9569         self._copied_shares = {}
9570         self._storage = FakeStorage()
9571         self._nodemaker = make_nodemaker(self._storage)
9572hunk ./src/allmydata/test/test_mutable.py 700
9573-        d = self._nodemaker.create_mutable_file(self.CONTENTS[0], version=version) # seqnum=1
9574+        d = self._nodemaker.create_mutable_file(self.uploadables[0], version=version) # seqnum=1
9575         def _created(node):
9576             self._fn = node
9577             # now create multiple versions of the same file, and accumulate
9578hunk ./src/allmydata/test/test_mutable.py 707
9579             # their shares, so we can mix and match them later.
9580             d = defer.succeed(None)
9581             d.addCallback(self._copy_shares, 0)
9582-            d.addCallback(lambda res: node.overwrite(self.CONTENTS[1])) #s2
9583+            d.addCallback(lambda res: node.overwrite(self.uploadables[1])) #s2
9584             d.addCallback(self._copy_shares, 1)
9585hunk ./src/allmydata/test/test_mutable.py 709
9586-            d.addCallback(lambda res: node.overwrite(self.CONTENTS[2])) #s3
9587+            d.addCallback(lambda res: node.overwrite(self.uploadables[2])) #s3
9588             d.addCallback(self._copy_shares, 2)
9589hunk ./src/allmydata/test/test_mutable.py 711
9590-            d.addCallback(lambda res: node.overwrite(self.CONTENTS[3])) #s4a
9591+            d.addCallback(lambda res: node.overwrite(self.uploadables[3])) #s4a
9592             d.addCallback(self._copy_shares, 3)
9593             # now we replace all the shares with version s3, and upload a new
9594             # version to get s4b.
9595hunk ./src/allmydata/test/test_mutable.py 717
9596             rollback = dict([(i,2) for i in range(10)])
9597             d.addCallback(lambda res: self._set_versions(rollback))
9598-            d.addCallback(lambda res: node.overwrite(self.CONTENTS[4])) #s4b
9599+            d.addCallback(lambda res: node.overwrite(self.uploadables[4])) #s4b
9600             d.addCallback(self._copy_shares, 4)
9601             # we leave the storage in state 4
9602             return d
9603hunk ./src/allmydata/test/test_mutable.py 826
9604         # create a new file, which is large enough to knock the privkey out
9605         # of the early part of the file
9606         LARGE = "These are Larger contents" * 200 # about 5KB
9607-        d.addCallback(lambda res: self._nodemaker.create_mutable_file(LARGE))
9608+        LARGE_uploadable = MutableDataHandle(LARGE)
9609+        d.addCallback(lambda res: self._nodemaker.create_mutable_file(LARGE_uploadable))
9610         def _created(large_fn):
9611             large_fn2 = self._nodemaker.create_from_cap(large_fn.get_uri())
9612             return self.make_servermap(MODE_WRITE, large_fn2)
9613hunk ./src/allmydata/test/test_mutable.py 1842
9614 class MultipleEncodings(unittest.TestCase):
9615     def setUp(self):
9616         self.CONTENTS = "New contents go here"
9617+        self.uploadable = MutableDataHandle(self.CONTENTS)
9618         self._storage = FakeStorage()
9619         self._nodemaker = make_nodemaker(self._storage, num_peers=20)
9620         self._storage_broker = self._nodemaker.storage_broker
9621hunk ./src/allmydata/test/test_mutable.py 1846
9622-        d = self._nodemaker.create_mutable_file(self.CONTENTS)
9623+        d = self._nodemaker.create_mutable_file(self.uploadable)
9624         def _created(node):
9625             self._fn = node
9626         d.addCallback(_created)
9627hunk ./src/allmydata/test/test_mutable.py 1872
9628         s = self._storage
9629         s._peers = {} # clear existing storage
9630         p2 = Publish(fn2, self._storage_broker, None)
9631-        d = p2.publish(data)
9632+        uploadable = MutableDataHandle(data)
9633+        d = p2.publish(uploadable)
9634         def _published(res):
9635             shares = s._peers
9636             s._peers = {}
9637hunk ./src/allmydata/test/test_mutable.py 2049
9638         self._set_versions(target)
9639 
9640         def _modify(oldversion, servermap, first_time):
9641-            return oldversion + " modified"
9642+            return MutableDataHandle(oldversion + " modified")
9643         d = self._fn.modify(_modify)
9644         d.addCallback(lambda res: self._fn.download_best_version())
9645         expected = self.CONTENTS[2] + " modified"
9646hunk ./src/allmydata/test/test_mutable.py 2175
9647         self.basedir = "mutable/Problems/test_publish_surprise"
9648         self.set_up_grid()
9649         nm = self.g.clients[0].nodemaker
9650-        d = nm.create_mutable_file("contents 1")
9651+        d = nm.create_mutable_file(MutableDataHandle("contents 1"))
9652         def _created(n):
9653             d = defer.succeed(None)
9654             d.addCallback(lambda res: n.get_servermap(MODE_WRITE))
9655hunk ./src/allmydata/test/test_mutable.py 2185
9656             d.addCallback(_got_smap1)
9657             # then modify the file, leaving the old map untouched
9658             d.addCallback(lambda res: log.msg("starting winning write"))
9659-            d.addCallback(lambda res: n.overwrite("contents 2"))
9660+            d.addCallback(lambda res: n.overwrite(MutableDataHandle("contents 2")))
9661             # now attempt to modify the file with the old servermap. This
9662             # will look just like an uncoordinated write, in which every
9663             # single share got updated between our mapupdate and our publish
9664hunk ./src/allmydata/test/test_mutable.py 2194
9665                           self.shouldFail(UncoordinatedWriteError,
9666                                           "test_publish_surprise", None,
9667                                           n.upload,
9668-                                          "contents 2a", self.old_map))
9669+                                          MutableDataHandle("contents 2a"), self.old_map))
9670             return d
9671         d.addCallback(_created)
9672         return d
9673hunk ./src/allmydata/test/test_mutable.py 2203
9674         self.basedir = "mutable/Problems/test_retrieve_surprise"
9675         self.set_up_grid()
9676         nm = self.g.clients[0].nodemaker
9677-        d = nm.create_mutable_file("contents 1")
9678+        d = nm.create_mutable_file(MutableDataHandle("contents 1"))
9679         def _created(n):
9680             d = defer.succeed(None)
9681             d.addCallback(lambda res: n.get_servermap(MODE_READ))
9682hunk ./src/allmydata/test/test_mutable.py 2213
9683             d.addCallback(_got_smap1)
9684             # then modify the file, leaving the old map untouched
9685             d.addCallback(lambda res: log.msg("starting winning write"))
9686-            d.addCallback(lambda res: n.overwrite("contents 2"))
9687+            d.addCallback(lambda res: n.overwrite(MutableDataHandle("contents 2")))
9688             # now attempt to retrieve the old version with the old servermap.
9689             # This will look like someone has changed the file since we
9690             # updated the servermap.
9691hunk ./src/allmydata/test/test_mutable.py 2241
9692         self.basedir = "mutable/Problems/test_unexpected_shares"
9693         self.set_up_grid()
9694         nm = self.g.clients[0].nodemaker
9695-        d = nm.create_mutable_file("contents 1")
9696+        d = nm.create_mutable_file(MutableDataHandle("contents 1"))
9697         def _created(n):
9698             d = defer.succeed(None)
9699             d.addCallback(lambda res: n.get_servermap(MODE_WRITE))
9700hunk ./src/allmydata/test/test_mutable.py 2253
9701                 self.g.remove_server(peer0)
9702                 # then modify the file, leaving the old map untouched
9703                 log.msg("starting winning write")
9704-                return n.overwrite("contents 2")
9705+                return n.overwrite(MutableDataHandle("contents 2"))
9706             d.addCallback(_got_smap1)
9707             # now attempt to modify the file with the old servermap. This
9708             # will look just like an uncoordinated write, in which every
9709hunk ./src/allmydata/test/test_mutable.py 2263
9710                           self.shouldFail(UncoordinatedWriteError,
9711                                           "test_surprise", None,
9712                                           n.upload,
9713-                                          "contents 2a", self.old_map))
9714+                                          MutableDataHandle("contents 2a"), self.old_map))
9715             return d
9716         d.addCallback(_created)
9717         return d
9718hunk ./src/allmydata/test/test_mutable.py 2267
9719+    test_unexpected_shares.timeout = 15
9720 
9721     def test_bad_server(self):
9722         # Break one server, then create the file: the initial publish should
9723hunk ./src/allmydata/test/test_mutable.py 2303
9724         d.addCallback(_break_peer0)
9725         # now "create" the file, using the pre-established key, and let the
9726         # initial publish finally happen
9727-        d.addCallback(lambda res: nm.create_mutable_file("contents 1"))
9728+        d.addCallback(lambda res: nm.create_mutable_file(MutableDataHandle("contents 1")))
9729         # that ought to work
9730         def _got_node(n):
9731             d = n.download_best_version()
9732hunk ./src/allmydata/test/test_mutable.py 2312
9733             def _break_peer1(res):
9734                 self.connection1.broken = True
9735             d.addCallback(_break_peer1)
9736-            d.addCallback(lambda res: n.overwrite("contents 2"))
9737+            d.addCallback(lambda res: n.overwrite(MutableDataHandle("contents 2")))
9738             # that ought to work too
9739             d.addCallback(lambda res: n.download_best_version())
9740             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 2"))
9741hunk ./src/allmydata/test/test_mutable.py 2344
9742         peerids = [serverid for (serverid,ss) in sb.get_all_servers()]
9743         self.g.break_server(peerids[0])
9744 
9745-        d = nm.create_mutable_file("contents 1")
9746+        d = nm.create_mutable_file(MutableDataHandle("contents 1"))
9747         def _created(n):
9748             d = n.download_best_version()
9749             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 1"))
9750hunk ./src/allmydata/test/test_mutable.py 2352
9751             def _break_second_server(res):
9752                 self.g.break_server(peerids[1])
9753             d.addCallback(_break_second_server)
9754-            d.addCallback(lambda res: n.overwrite("contents 2"))
9755+            d.addCallback(lambda res: n.overwrite(MutableDataHandle("contents 2")))
9756             # that ought to work too
9757             d.addCallback(lambda res: n.download_best_version())
9758             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 2"))
9759hunk ./src/allmydata/test/test_mutable.py 2371
9760         d = self.shouldFail(NotEnoughServersError,
9761                             "test_publish_all_servers_bad",
9762                             "Ran out of non-bad servers",
9763-                            nm.create_mutable_file, "contents")
9764+                            nm.create_mutable_file, MutableDataHandle("contents"))
9765         return d
9766 
9767     def test_publish_no_servers(self):
9768hunk ./src/allmydata/test/test_mutable.py 2383
9769         d = self.shouldFail(NotEnoughServersError,
9770                             "test_publish_no_servers",
9771                             "Ran out of non-bad servers",
9772-                            nm.create_mutable_file, "contents")
9773+                            nm.create_mutable_file, MutableDataHandle("contents"))
9774         return d
9775     test_publish_no_servers.timeout = 30
9776 
9777hunk ./src/allmydata/test/test_mutable.py 2401
9778         # we need some contents that are large enough to push the privkey out
9779         # of the early part of the file
9780         LARGE = "These are Larger contents" * 2000 # about 50KB
9781-        d = nm.create_mutable_file(LARGE)
9782+        LARGE_uploadable = MutableDataHandle(LARGE)
9783+        d = nm.create_mutable_file(LARGE_uploadable)
9784         def _created(n):
9785             self.uri = n.get_uri()
9786             self.n2 = nm.create_from_cap(self.uri)
9787hunk ./src/allmydata/test/test_mutable.py 2438
9788         self.set_up_grid(num_servers=20)
9789         nm = self.g.clients[0].nodemaker
9790         LARGE = "These are Larger contents" * 2000 # about 50KiB
9791+        LARGE_uploadable = MutableDataHandle(LARGE)
9792         nm._node_cache = DevNullDictionary() # disable the nodecache
9793 
9794hunk ./src/allmydata/test/test_mutable.py 2441
9795-        d = nm.create_mutable_file(LARGE)
9796+        d = nm.create_mutable_file(LARGE_uploadable)
9797         def _created(n):
9798             self.uri = n.get_uri()
9799             self.n2 = nm.create_from_cap(self.uri)
9800hunk ./src/allmydata/test/test_mutable.py 2464
9801         self.set_up_grid(num_servers=20)
9802         nm = self.g.clients[0].nodemaker
9803         CONTENTS = "contents" * 2000
9804-        d = nm.create_mutable_file(CONTENTS)
9805+        CONTENTS_uploadable = MutableDataHandle(CONTENTS)
9806+        d = nm.create_mutable_file(CONTENTS_uploadable)
9807         def _created(node):
9808             self._node = node
9809         d.addCallback(_created)
9810hunk ./src/allmydata/test/test_system.py 22
9811 from allmydata.monitor import Monitor
9812 from allmydata.mutable.common import NotWriteableError
9813 from allmydata.mutable import layout as mutable_layout
9814+from allmydata.mutable.publish import MutableDataHandle
9815 from foolscap.api import DeadReferenceError
9816 from twisted.python.failure import Failure
9817 from twisted.web.client import getPage
9818hunk ./src/allmydata/test/test_system.py 460
9819     def test_mutable(self):
9820         self.basedir = "system/SystemTest/test_mutable"
9821         DATA = "initial contents go here."  # 25 bytes % 3 != 0
9822+        DATA_uploadable = MutableDataHandle(DATA)
9823         NEWDATA = "new contents yay"
9824hunk ./src/allmydata/test/test_system.py 462
9825+        NEWDATA_uploadable = MutableDataHandle(NEWDATA)
9826         NEWERDATA = "this is getting old"
9827hunk ./src/allmydata/test/test_system.py 464
9828+        NEWERDATA_uploadable = MutableDataHandle(NEWERDATA)
9829 
9830         d = self.set_up_nodes(use_key_generator=True)
9831 
9832hunk ./src/allmydata/test/test_system.py 471
9833         def _create_mutable(res):
9834             c = self.clients[0]
9835             log.msg("starting create_mutable_file")
9836-            d1 = c.create_mutable_file(DATA)
9837+            d1 = c.create_mutable_file(DATA_uploadable)
9838             def _done(res):
9839                 log.msg("DONE: %s" % (res,))
9840                 self._mutable_node_1 = res
9841hunk ./src/allmydata/test/test_system.py 558
9842             self.failUnlessEqual(res, DATA)
9843             # replace the data
9844             log.msg("starting replace1")
9845-            d1 = newnode.overwrite(NEWDATA)
9846+            d1 = newnode.overwrite(NEWDATA_uploadable)
9847             d1.addCallback(lambda res: newnode.download_best_version())
9848             return d1
9849         d.addCallback(_check_download_3)
9850hunk ./src/allmydata/test/test_system.py 572
9851             newnode2 = self.clients[3].create_node_from_uri(uri)
9852             self._newnode3 = self.clients[3].create_node_from_uri(uri)
9853             log.msg("starting replace2")
9854-            d1 = newnode1.overwrite(NEWERDATA)
9855+            d1 = newnode1.overwrite(NEWERDATA_uploadable)
9856             d1.addCallback(lambda res: newnode2.download_best_version())
9857             return d1
9858         d.addCallback(_check_download_4)
9859hunk ./src/allmydata/test/test_system.py 642
9860         def _check_empty_file(res):
9861             # make sure we can create empty files, this usually screws up the
9862             # segsize math
9863-            d1 = self.clients[2].create_mutable_file("")
9864+            d1 = self.clients[2].create_mutable_file(MutableDataHandle(""))
9865             d1.addCallback(lambda newnode: newnode.download_best_version())
9866             d1.addCallback(lambda res: self.failUnlessEqual("", res))
9867             return d1
9868hunk ./src/allmydata/test/test_system.py 673
9869                                  self.key_generator_svc.key_generator.pool_size + size_delta)
9870 
9871         d.addCallback(check_kg_poolsize, 0)
9872-        d.addCallback(lambda junk: self.clients[3].create_mutable_file('hello, world'))
9873+        d.addCallback(lambda junk:
9874+            self.clients[3].create_mutable_file(MutableDataHandle('hello, world')))
9875         d.addCallback(check_kg_poolsize, -1)
9876         d.addCallback(lambda junk: self.clients[3].create_dirnode())
9877         d.addCallback(check_kg_poolsize, -2)
9878hunk ./src/allmydata/test/test_web.py 3183
9879         def _stash_mutable_uri(n, which):
9880             self.uris[which] = n.get_uri()
9881             assert isinstance(self.uris[which], str)
9882-        d.addCallback(lambda ign: c0.create_mutable_file(DATA+"3"))
9883+        d.addCallback(lambda ign:
9884+            c0.create_mutable_file(publish.MutableDataHandle(DATA+"3")))
9885         d.addCallback(_stash_mutable_uri, "corrupt")
9886         d.addCallback(lambda ign:
9887                       c0.upload(upload.Data("literal", convergence="")))
9888hunk ./src/allmydata/test/test_web.py 3330
9889         def _stash_mutable_uri(n, which):
9890             self.uris[which] = n.get_uri()
9891             assert isinstance(self.uris[which], str)
9892-        d.addCallback(lambda ign: c0.create_mutable_file(DATA+"3"))
9893+        d.addCallback(lambda ign:
9894+            c0.create_mutable_file(publish.MutableDataHandle(DATA+"3")))
9895         d.addCallback(_stash_mutable_uri, "corrupt")
9896 
9897         def _compute_fileurls(ignored):
9898hunk ./src/allmydata/test/test_web.py 3993
9899         def _stash_mutable_uri(n, which):
9900             self.uris[which] = n.get_uri()
9901             assert isinstance(self.uris[which], str)
9902-        d.addCallback(lambda ign: c0.create_mutable_file(DATA+"2"))
9903+        d.addCallback(lambda ign:
9904+            c0.create_mutable_file(publish.MutableDataHandle(DATA+"2")))
9905         d.addCallback(_stash_mutable_uri, "mutable")
9906 
9907         def _compute_fileurls(ignored):
9908hunk ./src/allmydata/test/test_web.py 4093
9909                                                         convergence="")))
9910         d.addCallback(_stash_uri, "small")
9911 
9912-        d.addCallback(lambda ign: c0.create_mutable_file("mutable"))
9913+        d.addCallback(lambda ign:
9914+            c0.create_mutable_file(publish.MutableDataHandle("mutable")))
9915         d.addCallback(lambda fn: self.rootnode.set_node(u"mutable", fn))
9916         d.addCallback(_stash_uri, "mutable")
9917 
9918}
9919[Alter mutable files to use file-like objects for publishing instead of strings.
9920Kevan Carstensen <kevan@isnotajoke.com>**20100708000732
9921 Ignore-this: 8dd07d95386b6d540bc21289f981ebd0
9922] {
9923hunk ./src/allmydata/dirnode.py 11
9924 from allmydata.mutable.common import NotWriteableError
9925 from allmydata.mutable.filenode import MutableFileNode
9926 from allmydata.unknown import UnknownNode, strip_prefix_for_ro
9927+from allmydata.mutable.publish import MutableDataHandle
9928 from allmydata.interfaces import IFilesystemNode, IDirectoryNode, IFileNode, \
9929      IImmutableFileNode, IMutableFileNode, \
9930      ExistingChildError, NoSuchChildError, ICheckable, IDeepCheckable, \
9931hunk ./src/allmydata/dirnode.py 104
9932 
9933         del children[self.name]
9934         new_contents = self.node._pack_contents(children)
9935-        return new_contents
9936+        uploadable = MutableDataHandle(new_contents)
9937+        return uploadable
9938 
9939 
9940 class MetadataSetter:
9941hunk ./src/allmydata/dirnode.py 130
9942 
9943         children[name] = (child, metadata)
9944         new_contents = self.node._pack_contents(children)
9945-        return new_contents
9946+        uploadable = MutableDataHandle(new_contents)
9947+        return uploadable
9948 
9949 
9950 class Adder:
9951hunk ./src/allmydata/dirnode.py 175
9952 
9953             children[name] = (child, metadata)
9954         new_contents = self.node._pack_contents(children)
9955-        return new_contents
9956+        uploadable = MutableDataHandle(new_contents)
9957+        return uploadable
9958 
9959 def _encrypt_rw_uri(writekey, rw_uri):
9960     precondition(isinstance(rw_uri, str), rw_uri)
9961hunk ./src/allmydata/mutable/filenode.py 7
9962 from zope.interface import implements
9963 from twisted.internet import defer, reactor
9964 from foolscap.api import eventually
9965-from allmydata.interfaces import IMutableFileNode, \
9966-     ICheckable, ICheckResults, NotEnoughSharesError, MDMF_VERSION, SDMF_VERSION
9967+from allmydata.interfaces import IMutableFileNode, ICheckable, ICheckResults, \
9968+                                 NotEnoughSharesError, \
9969+                                 MDMF_VERSION, SDMF_VERSION, IMutableUploadable
9970 from allmydata.util import hashutil, log
9971 from allmydata.util.assertutil import precondition
9972 from allmydata.uri import WriteableSSKFileURI, ReadonlySSKFileURI
9973hunk ./src/allmydata/mutable/filenode.py 16
9974 from allmydata.monitor import Monitor
9975 from pycryptopp.cipher.aes import AES
9976 
9977-from allmydata.mutable.publish import Publish
9978+from allmydata.mutable.publish import Publish, MutableFileHandle, \
9979+                                      MutableDataHandle
9980 from allmydata.mutable.common import MODE_READ, MODE_WRITE, UnrecoverableFileError, \
9981      ResponseCache, UncoordinatedWriteError
9982 from allmydata.mutable.servermap import ServerMap, ServermapUpdater
9983hunk ./src/allmydata/mutable/filenode.py 133
9984         return self._upload(initial_contents, None)
9985 
9986     def _get_initial_contents(self, contents):
9987-        if isinstance(contents, str):
9988-            return contents
9989         if contents is None:
9990hunk ./src/allmydata/mutable/filenode.py 134
9991-            return ""
9992+            return MutableDataHandle("")
9993+
9994+        if IMutableUploadable.providedBy(contents):
9995+            return contents
9996+
9997         assert callable(contents), "%s should be callable, not %s" % \
9998                (contents, type(contents))
9999         return contents(self)
10000hunk ./src/allmydata/mutable/filenode.py 353
10001     def overwrite(self, new_contents):
10002         return self._do_serialized(self._overwrite, new_contents)
10003     def _overwrite(self, new_contents):
10004+        assert IMutableUploadable.providedBy(new_contents)
10005+
10006         servermap = ServerMap()
10007         d = self._update_servermap(servermap, mode=MODE_WRITE)
10008         d.addCallback(lambda ignored: self._upload(new_contents, servermap))
10009hunk ./src/allmydata/mutable/filenode.py 431
10010                 # recovery when it observes UCWE, we need to do a second
10011                 # publish. See #551 for details. We'll basically loop until
10012                 # we managed an uncontested publish.
10013-                new_contents = old_contents
10014-            precondition(isinstance(new_contents, str),
10015-                         "Modifier function must return a string or None")
10016+                old_uploadable = MutableDataHandle(old_contents)
10017+                new_contents = old_uploadable
10018+            precondition((IMutableUploadable.providedBy(new_contents) or
10019+                          new_contents is None),
10020+                         "Modifier function must return an IMutableUploadable "
10021+                         "or None")
10022             return self._upload(new_contents, servermap)
10023         d.addCallback(_apply)
10024         return d
10025hunk ./src/allmydata/mutable/filenode.py 472
10026         return self._do_serialized(self._upload, new_contents, servermap)
10027     def _upload(self, new_contents, servermap):
10028         assert self._pubkey, "update_servermap must be called before publish"
10029+        assert IMutableUploadable.providedBy(new_contents)
10030+
10031         p = Publish(self, self._storage_broker, servermap)
10032         if self._history:
10033hunk ./src/allmydata/mutable/filenode.py 476
10034-            self._history.notify_publish(p.get_status(), len(new_contents))
10035+            self._history.notify_publish(p.get_status(), new_contents.get_size())
10036         d = p.publish(new_contents)
10037hunk ./src/allmydata/mutable/filenode.py 478
10038-        d.addCallback(self._did_upload, len(new_contents))
10039+        d.addCallback(self._did_upload, new_contents.get_size())
10040         return d
10041     def _did_upload(self, res, size):
10042         self._most_recent_size = size
10043hunk ./src/allmydata/mutable/publish.py 141
10044 
10045         # 0. Setup encoding parameters, encoder, and other such things.
10046         # 1. Encrypt, encode, and publish segments.
10047-        self.data = StringIO(newdata)
10048-        self.datalength = len(newdata)
10049+        assert IMutableUploadable.providedBy(newdata)
10050+
10051+        self.data = newdata
10052+        self.datalength = newdata.get_size()
10053 
10054         self.log("starting publish, datalen is %s" % self.datalength)
10055         self._status.set_size(self.datalength)
10056hunk ./src/allmydata/mutable/publish.py 442
10057 
10058         self.log("Pushing segment %d of %d" % (segnum + 1, self.num_segments))
10059         data = self.data.read(segsize)
10060+        # XXX: This is dumb. Why return a list?
10061+        data = "".join(data)
10062 
10063         assert len(data) == segsize
10064 
10065hunk ./src/allmydata/mutable/repairer.py 5
10066 from zope.interface import implements
10067 from twisted.internet import defer
10068 from allmydata.interfaces import IRepairResults, ICheckResults
10069+from allmydata.mutable.publish import MutableDataHandle
10070 
10071 class RepairResults:
10072     implements(IRepairResults)
10073hunk ./src/allmydata/mutable/repairer.py 108
10074             raise RepairRequiresWritecapError("Sorry, repair currently requires a writecap, to set the write-enabler properly.")
10075 
10076         d = self.node.download_version(smap, best_version, fetch_privkey=True)
10077+        d.addCallback(lambda data:
10078+            MutableDataHandle(data))
10079         d.addCallback(self.node.upload, smap)
10080         d.addCallback(self.get_results, smap)
10081         return d
10082hunk ./src/allmydata/nodemaker.py 9
10083 from allmydata.immutable.filenode import ImmutableFileNode, LiteralFileNode
10084 from allmydata.immutable.upload import Data
10085 from allmydata.mutable.filenode import MutableFileNode
10086+from allmydata.mutable.publish import MutableDataHandle
10087 from allmydata.dirnode import DirectoryNode, pack_children
10088 from allmydata.unknown import UnknownNode
10089 from allmydata import uri
10090merger 0.0 (
10091merger 0.0 (
10092hunk ./src/allmydata/nodemaker.py 107
10093-                                     pack_children(n, initial_children))
10094+                                     pack_children(n, initial_children),
10095+                                     version)
10096hunk ./src/allmydata/nodemaker.py 107
10097-                                     pack_children(n, initial_children))
10098+                                     pack_children(initial_children, n.get_writekey()))
10099)
10100hunk ./src/allmydata/nodemaker.py 107
10101-                                     pack_children(n, initial_children),
10102+                                     MutableDataHandle(
10103+                                        pack_children(n, initial_children)),
10104)
10105hunk ./src/allmydata/web/filenode.py 12
10106 from allmydata.interfaces import ExistingChildError
10107 from allmydata.monitor import Monitor
10108 from allmydata.immutable.upload import FileHandle
10109+from allmydata.mutable.publish import MutableFileHandle
10110 from allmydata.util import log, base32
10111 
10112 from allmydata.web.common import text_plain, WebError, RenderMixin, \
10113hunk ./src/allmydata/web/filenode.py 27
10114         # a new file is being uploaded in our place.
10115         mutable = boolean_of_arg(get_arg(req, "mutable", "false"))
10116         if mutable:
10117-            req.content.seek(0)
10118-            data = req.content.read()
10119+            data = MutableFileHandle(req.content)
10120             d = client.create_mutable_file(data)
10121             def _uploaded(newnode):
10122                 d2 = self.parentnode.set_node(self.name, newnode,
10123hunk ./src/allmydata/web/filenode.py 61
10124         d.addCallback(lambda res: childnode.get_uri())
10125         return d
10126 
10127-    def _read_data_from_formpost(self, req):
10128-        # SDMF: files are small, and we can only upload data, so we read
10129-        # the whole file into memory before uploading.
10130-        contents = req.fields["file"]
10131-        contents.file.seek(0)
10132-        data = contents.file.read()
10133-        return data
10134 
10135     def replace_me_with_a_formpost(self, req, client, replace):
10136         # create a new file, maybe mutable, maybe immutable
10137hunk ./src/allmydata/web/filenode.py 66
10138         mutable = boolean_of_arg(get_arg(req, "mutable", "false"))
10139 
10140+        # create an immutable file
10141+        contents = req.fields["file"]
10142         if mutable:
10143hunk ./src/allmydata/web/filenode.py 69
10144-            data = self._read_data_from_formpost(req)
10145-            d = client.create_mutable_file(data)
10146+            uploadable = MutableFileHandle(contents.file)
10147+            d = client.create_mutable_file(uploadable)
10148             def _uploaded(newnode):
10149                 d2 = self.parentnode.set_node(self.name, newnode,
10150                                               overwrite=replace)
10151hunk ./src/allmydata/web/filenode.py 78
10152                 return d2
10153             d.addCallback(_uploaded)
10154             return d
10155-        # create an immutable file
10156-        contents = req.fields["file"]
10157+
10158         uploadable = FileHandle(contents.file, convergence=client.convergence)
10159         d = self.parentnode.add_file(self.name, uploadable, overwrite=replace)
10160         d.addCallback(lambda newnode: newnode.get_uri())
10161hunk ./src/allmydata/web/filenode.py 84
10162         return d
10163 
10164+
10165 class PlaceHolderNodeHandler(RenderMixin, rend.Page, ReplaceMeMixin):
10166     def __init__(self, client, parentnode, name):
10167         rend.Page.__init__(self)
10168hunk ./src/allmydata/web/filenode.py 278
10169 
10170     def replace_my_contents(self, req):
10171         req.content.seek(0)
10172-        new_contents = req.content.read()
10173+        new_contents = MutableFileHandle(req.content)
10174         d = self.node.overwrite(new_contents)
10175         d.addCallback(lambda res: self.node.get_uri())
10176         return d
10177hunk ./src/allmydata/web/filenode.py 286
10178     def replace_my_contents_with_a_formpost(self, req):
10179         # we have a mutable file. Get the data from the formpost, and replace
10180         # the mutable file's contents with it.
10181-        new_contents = self._read_data_from_formpost(req)
10182+        new_contents = req.fields['file']
10183+        new_contents = MutableFileHandle(new_contents.file)
10184+
10185         d = self.node.overwrite(new_contents)
10186         d.addCallback(lambda res: self.node.get_uri())
10187         return d
10188hunk ./src/allmydata/web/unlinked.py 7
10189 from twisted.internet import defer
10190 from nevow import rend, url, tags as T
10191 from allmydata.immutable.upload import FileHandle
10192+from allmydata.mutable.publish import MutableFileHandle
10193 from allmydata.web.common import getxmlfile, get_arg, boolean_of_arg, \
10194      convert_children_json, WebError
10195 from allmydata.web import status
10196hunk ./src/allmydata/web/unlinked.py 23
10197 def PUTUnlinkedSSK(req, client):
10198     # SDMF: files are small, and we can only upload data
10199     req.content.seek(0)
10200-    data = req.content.read()
10201+    data = MutableFileHandle(req.content)
10202     d = client.create_mutable_file(data)
10203     d.addCallback(lambda n: n.get_uri())
10204     return d
10205hunk ./src/allmydata/web/unlinked.py 87
10206     # "POST /uri", to create an unlinked file.
10207     # SDMF: files are small, and we can only upload data
10208     contents = req.fields["file"]
10209-    contents.file.seek(0)
10210-    data = contents.file.read()
10211+    data = MutableFileHandle(contents.file)
10212     d = client.create_mutable_file(data)
10213     d.addCallback(lambda n: n.get_uri())
10214     return d
10215}
10216[test/test_sftp.py: alter a setup routine to work with new mutable file APIs.
10217Kevan Carstensen <kevan@isnotajoke.com>**20100708193522
10218 Ignore-this: 434bbe1347072076c0836d26fca8ac8a
10219] {
10220hunk ./src/allmydata/test/test_sftp.py 32
10221 
10222 from allmydata.util.consumer import download_to_data
10223 from allmydata.immutable import upload
10224+from allmydata.mutable import publish
10225 from allmydata.test.no_network import GridTestMixin
10226 from allmydata.test.common import ShouldFailMixin
10227 from allmydata.test.common_util import ReallyEqualMixin
10228hunk ./src/allmydata/test/test_sftp.py 84
10229         return d
10230 
10231     def _set_up_tree(self):
10232-        d = self.client.create_mutable_file("mutable file contents")
10233+        u = publish.MutableDataHandle("mutable file contents")
10234+        d = self.client.create_mutable_file(u)
10235         d.addCallback(lambda node: self.root.set_node(u"mutable", node))
10236         def _created_mutable(n):
10237             self.mutable = n
10238}
10239[mutable/publish.py: make MutableFileHandle seek to the beginning of its file handle before reading.
10240Kevan Carstensen <kevan@isnotajoke.com>**20100708193600
10241 Ignore-this: 453a737dc62a79c77b3d360fed9000ab
10242] hunk ./src/allmydata/mutable/publish.py 989
10243         assert hasattr(filehandle, "close")
10244 
10245         self._filehandle = filehandle
10246+        # We must start reading at the beginning of the file, or we risk
10247+        # encountering errors when the data read does not match the size
10248+        # reported to the uploader.
10249+        self._filehandle.seek(0)
10250 
10251 
10252     def get_size(self):
10253[Refactor download interfaces to be more uniform, per #993
10254Kevan Carstensen <kevan@isnotajoke.com>**20100709232912
10255 Ignore-this: 277c5699c4a2dd7c03ecfa0a28458f5b
10256] {
10257hunk ./src/allmydata/immutable/filenode.py 10
10258 from foolscap.api import eventually
10259 from allmydata.interfaces import IImmutableFileNode, ICheckable, \
10260      IDownloadTarget, IUploadResults
10261-from allmydata.util import dictutil, log, base32
10262+from allmydata.util import dictutil, log, base32, consumer
10263 from allmydata.uri import CHKFileURI, LiteralFileURI
10264 from allmydata.immutable.checker import Checker
10265 from allmydata.check_results import CheckResults, CheckAndRepairResults
10266hunk ./src/allmydata/immutable/filenode.py 318
10267                       self.download_cache.read(consumer, offset, size))
10268         return d
10269 
10270+    # IReadable, IFileNode
10271+
10272+    def get_best_readable_version(self):
10273+        """
10274+        Return an IReadable of the best version of this file. Since
10275+        immutable files can have only one version, we just return the
10276+        current filenode.
10277+        """
10278+        return self
10279+
10280+
10281+    def download_best_version(self):
10282+        """
10283+        Download the best version of this file, returning its contents
10284+        as a bytestring. Since there is only one version of an immutable
10285+        file, we download and return the contents of this file.
10286+        """
10287+        d = consumer.download_to_data(self)
10288+        return d
10289+
10290+    # for an immutable file, download_to_data (specified in IReadable)
10291+    # is the same as download_best_version (specified in IFileNode). For
10292+    # mutable files, the difference is more meaningful, since they can
10293+    # have multiple versions.
10294+    download_to_data = download_best_version
10295+
10296+
10297+    # get_size() (IReadable), get_current_size() (IFilesystemNode), and
10298+    # get_size_of_best_version(IFileNode) are all the same for immutable
10299+    # files.
10300+    get_size_of_best_version = get_current_size
10301+
10302+
10303 class LiteralProducer:
10304     implements(IPushProducer)
10305     def resumeProducing(self):
10306hunk ./src/allmydata/immutable/filenode.py 409
10307         d = basic.FileSender().beginFileTransfer(StringIO(data), consumer)
10308         d.addCallback(lambda lastSent: consumer)
10309         return d
10310+
10311+    # IReadable, IFileNode, IFilesystemNode
10312+    def get_best_readable_version(self):
10313+        return self
10314+
10315+
10316+    def download_best_version(self):
10317+        return defer.succeed(self.u.data)
10318+
10319+
10320+    download_to_data = download_best_version
10321+    get_size_of_best_version = get_current_size
10322hunk ./src/allmydata/interfaces.py 563
10323 class MustNotBeUnknownRWError(CapConstraintError):
10324     """Cannot add an unknown child cap specified in a rw_uri field."""
10325 
10326+
10327+class IReadable(Interface):
10328+    """I represent a readable object -- either an immutable file, or a
10329+    specific version of a mutable file.
10330+    """
10331+
10332+    def is_readonly():
10333+        """Return True if this reference provides mutable access to the given
10334+        file or directory (i.e. if you can modify it), or False if not. Note
10335+        that even if this reference is read-only, someone else may hold a
10336+        read-write reference to it.
10337+
10338+        For an IReadable returned by get_best_readable_version(), this will
10339+        always return True, but for instances of subinterfaces such as
10340+        IMutableFileVersion, it may return False."""
10341+
10342+    def is_mutable():
10343+        """Return True if this file or directory is mutable (by *somebody*,
10344+        not necessarily you), False if it is is immutable. Note that a file
10345+        might be mutable overall, but your reference to it might be
10346+        read-only. On the other hand, all references to an immutable file
10347+        will be read-only; there are no read-write references to an immutable
10348+        file."""
10349+
10350+    def get_storage_index():
10351+        """Return the storage index of the file."""
10352+
10353+    def get_size():
10354+        """Return the length (in bytes) of this readable object."""
10355+
10356+    def download_to_data():
10357+        """Download all of the file contents. I return a Deferred that fires
10358+        with the contents as a byte string."""
10359+
10360+    def read(consumer, offset=0, size=None):
10361+        """Download a portion (possibly all) of the file's contents, making
10362+        them available to the given IConsumer. Return a Deferred that fires
10363+        (with the consumer) when the consumer is unregistered (either because
10364+        the last byte has been given to it, or because the consumer threw an
10365+        exception during write(), possibly because it no longer wants to
10366+        receive data). The portion downloaded will start at 'offset' and
10367+        contain 'size' bytes (or the remainder of the file if size==None).
10368+
10369+        The consumer will be used in non-streaming mode: an IPullProducer
10370+        will be attached to it.
10371+
10372+        The consumer will not receive data right away: several network trips
10373+        must occur first. The order of events will be::
10374+
10375+         consumer.registerProducer(p, streaming)
10376+          (if streaming == False)::
10377+           consumer does p.resumeProducing()
10378+            consumer.write(data)
10379+           consumer does p.resumeProducing()
10380+            consumer.write(data).. (repeat until all data is written)
10381+         consumer.unregisterProducer()
10382+         deferred.callback(consumer)
10383+
10384+        If a download error occurs, or an exception is raised by
10385+        consumer.registerProducer() or consumer.write(), I will call
10386+        consumer.unregisterProducer() and then deliver the exception via
10387+        deferred.errback(). To cancel the download, the consumer should call
10388+        p.stopProducing(), which will result in an exception being delivered
10389+        via deferred.errback().
10390+
10391+        See src/allmydata/util/consumer.py for an example of a simple
10392+        download-to-memory consumer.
10393+        """
10394+
10395+
10396+class IMutableFileVersion(IReadable):
10397+    """I provide access to a particular version of a mutable file. The
10398+    access is read/write if I was obtained from a filenode derived from
10399+    a write cap, or read-only if the filenode was derived from a read cap.
10400+    """
10401+
10402+    def get_sequence_number():
10403+        """Return the sequence number of this version."""
10404+
10405+    def get_servermap():
10406+        """Return the IMutableFileServerMap instance that was used to create
10407+        this object.
10408+        """
10409+
10410+    def get_writekey():
10411+        """Return this filenode's writekey, or None if the node does not have
10412+        write-capability. This may be used to assist with data structures
10413+        that need to make certain data available only to writers, such as the
10414+        read-write child caps in dirnodes. The recommended process is to have
10415+        reader-visible data be submitted to the filenode in the clear (where
10416+        it will be encrypted by the filenode using the readkey), but encrypt
10417+        writer-visible data using this writekey.
10418+        """
10419+
10420+    # TODO: Can this be overwrite instead of replace?
10421+    def replace(new_contents):
10422+        """Replace the contents of the mutable file, provided that no other
10423+        node has published (or is attempting to publish, concurrently) a
10424+        newer version of the file than this one.
10425+
10426+        I will avoid modifying any share that is different than the version
10427+        given by get_sequence_number(). However, if another node is writing
10428+        to the file at the same time as me, I may manage to update some shares
10429+        while they update others. If I see any evidence of this, I will signal
10430+        UncoordinatedWriteError, and the file will be left in an inconsistent
10431+        state (possibly the version you provided, possibly the old version,
10432+        possibly somebody else's version, and possibly a mix of shares from
10433+        all of these).
10434+
10435+        The recommended response to UncoordinatedWriteError is to either
10436+        return it to the caller (since they failed to coordinate their
10437+        writes), or to attempt some sort of recovery. It may be sufficient to
10438+        wait a random interval (with exponential backoff) and repeat your
10439+        operation. If I do not signal UncoordinatedWriteError, then I was
10440+        able to write the new version without incident.
10441+
10442+        I return a Deferred that fires (with a PublishStatus object) when the
10443+        update has completed.
10444+        """
10445+
10446+    def modify(modifier_cb):
10447+        """Modify the contents of the file, by downloading this version,
10448+        applying the modifier function (or bound method), then uploading
10449+        the new version. This will succeed as long as no other node
10450+        publishes a version between the download and the upload.
10451+        I return a Deferred that fires (with a PublishStatus object) when
10452+        the update is complete.
10453+
10454+        The modifier callable will be given three arguments: a string (with
10455+        the old contents), a 'first_time' boolean, and a servermap. As with
10456+        download_to_data(), the old contents will be from this version,
10457+        but the modifier can use the servermap to make other decisions
10458+        (such as refusing to apply the delta if there are multiple parallel
10459+        versions, or if there is evidence of a newer unrecoverable version).
10460+        'first_time' will be True the first time the modifier is called,
10461+        and False on any subsequent calls.
10462+
10463+        The callable should return a string with the new contents. The
10464+        callable must be prepared to be called multiple times, and must
10465+        examine the input string to see if the change that it wants to make
10466+        is already present in the old version. If it does not need to make
10467+        any changes, it can either return None, or return its input string.
10468+
10469+        If the modifier raises an exception, it will be returned in the
10470+        errback.
10471+        """
10472+
10473+
10474 # The hierarchy looks like this:
10475 #  IFilesystemNode
10476 #   IFileNode
10477hunk ./src/allmydata/interfaces.py 801
10478     def raise_error():
10479         """Raise any error associated with this node."""
10480 
10481+    # XXX: These may not be appropriate outside the context of an IReadable.
10482     def get_size():
10483         """Return the length (in bytes) of the data this node represents. For
10484         directory nodes, I return the size of the backing store. I return
10485hunk ./src/allmydata/interfaces.py 818
10486 class IFileNode(IFilesystemNode):
10487     """I am a node which represents a file: a sequence of bytes. I am not a
10488     container, like IDirectoryNode."""
10489+    def get_best_readable_version():
10490+        """Return a Deferred that fires with an IReadable for the 'best'
10491+        available version of the file. The IReadable provides only read
10492+        access, even if this filenode was derived from a write cap.
10493 
10494hunk ./src/allmydata/interfaces.py 823
10495-class IImmutableFileNode(IFileNode):
10496-    def read(consumer, offset=0, size=None):
10497-        """Download a portion (possibly all) of the file's contents, making
10498-        them available to the given IConsumer. Return a Deferred that fires
10499-        (with the consumer) when the consumer is unregistered (either because
10500-        the last byte has been given to it, or because the consumer threw an
10501-        exception during write(), possibly because it no longer wants to
10502-        receive data). The portion downloaded will start at 'offset' and
10503-        contain 'size' bytes (or the remainder of the file if size==None).
10504-
10505-        The consumer will be used in non-streaming mode: an IPullProducer
10506-        will be attached to it.
10507+        For an immutable file, there is only one version. For a mutable
10508+        file, the 'best' version is the recoverable version with the
10509+        highest sequence number. If no uncoordinated writes have occurred,
10510+        and if enough shares are available, then this will be the most
10511+        recent version that has been uploaded. If no version is recoverable,
10512+        the Deferred will errback with an UnrecoverableFileError.
10513+        """
10514 
10515hunk ./src/allmydata/interfaces.py 831
10516-        The consumer will not receive data right away: several network trips
10517-        must occur first. The order of events will be::
10518+    def download_best_version():
10519+        """Download the contents of the version that would be returned
10520+        by get_best_readable_version(). This is equivalent to calling
10521+        download_to_data() on the IReadable given by that method.
10522 
10523hunk ./src/allmydata/interfaces.py 836
10524-         consumer.registerProducer(p, streaming)
10525-          (if streaming == False)::
10526-           consumer does p.resumeProducing()
10527-            consumer.write(data)
10528-           consumer does p.resumeProducing()
10529-            consumer.write(data).. (repeat until all data is written)
10530-         consumer.unregisterProducer()
10531-         deferred.callback(consumer)
10532+        I return a Deferred that fires with a byte string when the file
10533+        has been fully downloaded. To support streaming download, use
10534+        the 'read' method of IReadable. If no version is recoverable,
10535+        the Deferred will errback with an UnrecoverableFileError.
10536+        """
10537 
10538hunk ./src/allmydata/interfaces.py 842
10539-        If a download error occurs, or an exception is raised by
10540-        consumer.registerProducer() or consumer.write(), I will call
10541-        consumer.unregisterProducer() and then deliver the exception via
10542-        deferred.errback(). To cancel the download, the consumer should call
10543-        p.stopProducing(), which will result in an exception being delivered
10544-        via deferred.errback().
10545+    def get_size_of_best_version():
10546+        """Find the size of the version that would be returned by
10547+        get_best_readable_version().
10548 
10549hunk ./src/allmydata/interfaces.py 846
10550-        See src/allmydata/util/consumer.py for an example of a simple
10551-        download-to-memory consumer.
10552+        I return a Deferred that fires with an integer. If no version
10553+        is recoverable, the Deferred will errback with an
10554+        UnrecoverableFileError.
10555         """
10556 
10557hunk ./src/allmydata/interfaces.py 851
10558+
10559+class IImmutableFileNode(IFileNode, IReadable):
10560+    """I am a node representing an immutable file. Immutable files have
10561+    only one version"""
10562+
10563+
10564 class IMutableFileNode(IFileNode):
10565     """I provide access to a 'mutable file', which retains its identity
10566     regardless of what contents are put in it.
10567hunk ./src/allmydata/interfaces.py 916
10568     only be retrieved and updated all-at-once, as a single big string. Future
10569     versions of our mutable files will remove this restriction.
10570     """
10571-
10572-    def download_best_version():
10573-        """Download the 'best' available version of the file, meaning one of
10574-        the recoverable versions with the highest sequence number. If no
10575+    def get_best_mutable_version():
10576+        """Return a Deferred that fires with an IMutableFileVersion for
10577+        the 'best' available version of the file. The best version is
10578+        the recoverable version with the highest sequence number. If no
10579         uncoordinated writes have occurred, and if enough shares are
10580hunk ./src/allmydata/interfaces.py 921
10581-        available, then this will be the most recent version that has been
10582-        uploaded.
10583-
10584-        I update an internal servermap with MODE_READ, determine which
10585-        version of the file is indicated by
10586-        servermap.best_recoverable_version(), and return a Deferred that
10587-        fires with its contents. If no version is recoverable, the Deferred
10588-        will errback with UnrecoverableFileError.
10589-        """
10590-
10591-    def get_size_of_best_version():
10592-        """Find the size of the version that would be downloaded with
10593-        download_best_version(), without actually downloading the whole file.
10594+        available, then this will be the most recent version that has
10595+        been uploaded.
10596 
10597hunk ./src/allmydata/interfaces.py 924
10598-        I return a Deferred that fires with an integer.
10599+        If no version is recoverable, the Deferred will errback with an
10600+        UnrecoverableFileError.
10601         """
10602 
10603     def overwrite(new_contents):
10604hunk ./src/allmydata/interfaces.py 964
10605         errback.
10606         """
10607 
10608-
10609     def get_servermap(mode):
10610         """Return a Deferred that fires with an IMutableFileServerMap
10611         instance, updated using the given mode.
10612hunk ./src/allmydata/test/test_filenode.py 98
10613         def _check_segment(res):
10614             self.failUnlessEqual(res, DATA[1:1+5])
10615         d.addCallback(_check_segment)
10616+        d.addCallback(lambda ignored:
10617+            self.failUnlessEqual(fn1.get_best_readable_version(), fn1))
10618+        d.addCallback(lambda ignored:
10619+            fn1.get_size_of_best_version())
10620+        d.addCallback(lambda size:
10621+            self.failUnlessEqual(size, len(DATA)))
10622+        d.addCallback(lambda ignored:
10623+            fn1.download_to_data())
10624+        d.addCallback(lambda data:
10625+            self.failUnlessEqual(data, DATA))
10626+        d.addCallback(lambda ignored:
10627+            fn1.download_best_version())
10628+        d.addCallback(lambda data:
10629+            self.failUnlessEqual(data, DATA))
10630 
10631         return d
10632 
10633hunk ./src/allmydata/test/test_immutable.py 153
10634         return d
10635 
10636 
10637+    def test_download_to_data(self):
10638+        d = self.n.download_to_data()
10639+        d.addCallback(lambda data:
10640+            self.failUnlessEqual(data, common.TEST_DATA))
10641+        return d
10642+
10643+
10644+    def test_download_best_version(self):
10645+        d = self.n.download_best_version()
10646+        d.addCallback(lambda data:
10647+            self.failUnlessEqual(data, common.TEST_DATA))
10648+        return d
10649+
10650+
10651+    def test_get_best_readable_version(self):
10652+        n = self.n.get_best_readable_version()
10653+        self.failUnlessEqual(n, self.n)
10654+
10655+    def test_get_size_of_best_version(self):
10656+        d = self.n.get_size_of_best_version()
10657+        d.addCallback(lambda size:
10658+            self.failUnlessEqual(size, len(common.TEST_DATA)))
10659+        return d
10660+
10661+
10662 # XXX extend these tests to show bad behavior of various kinds from servers: raising exception from each remove_foo() method, for example
10663 
10664 # XXX test disconnect DeadReferenceError from get_buckets and get_block_whatsit
10665}
10666[frontends/sftpd.py: alter a mutable file overwrite to work with the new API
10667Kevan Carstensen <kevan@isnotajoke.com>**20100717014446
10668 Ignore-this: 2c57bbc8d9ab97a0e9af0634c00efc86
10669] {
10670hunk ./src/allmydata/frontends/sftpd.py 33
10671 from allmydata.interfaces import IFileNode, IDirectoryNode, ExistingChildError, \
10672      NoSuchChildError, ChildOfWrongTypeError
10673 from allmydata.mutable.common import NotWriteableError
10674+from allmydata.mutable.publish import MutableFileHandle
10675 from allmydata.immutable.upload import FileHandle
10676 from allmydata.dirnode import update_metadata
10677 from allmydata.util.fileutil import EncryptedTemporaryFile
10678merger 0.0 (
10679hunk ./src/allmydata/frontends/sftpd.py 664
10680-            # TODO: use download interface described in #993 when implemented.
10681hunk ./src/allmydata/frontends/sftpd.py 664
10682-            # TODO: use download interface described in #993 when implemented.
10683-            if filenode.is_mutable():
10684-                self.async.addCallback(lambda ign: filenode.download_best_version())
10685-                def _downloaded(data):
10686-                    self.consumer = OverwriteableFileConsumer(len(data), tempfile_maker)
10687-                    self.consumer.write(data)
10688-                    self.consumer.finish()
10689-                    return None
10690-                self.async.addCallback(_downloaded)
10691-            else:
10692-                download_size = filenode.get_size()
10693-                assert download_size is not None, "download_size is None"
10694+            self.async.addCallback(lambda ignored: filenode.get_best_readable_version())
10695+
10696+            def _read(version):
10697+                download_size = version.get_size()
10698+                assert download_size is not None
10699+
10700)
10701hunk ./src/allmydata/frontends/sftpd.py 677
10702                 download_size = filenode.get_size()
10703                 assert download_size is not None, "download_size is None"
10704                 self.consumer = OverwriteableFileConsumer(download_size, tempfile_maker)
10705-                def _read(ign):
10706-                    if noisy: self.log("_read immutable", level=NOISY)
10707-                    filenode.read(self.consumer, 0, None)
10708-                self.async.addCallback(_read)
10709+
10710+                if noisy: self.log("_read", level=NOISY)
10711+                version.read(self.consumer, 0, None)
10712+            self.async.addCallback(_read)
10713 
10714         eventually(self.async.callback, None)
10715 
10716hunk ./src/allmydata/frontends/sftpd.py 824
10717                     assert parent and childname, (parent, childname, self.metadata)
10718                     d2.addCallback(lambda ign: parent.set_metadata_for(childname, self.metadata))
10719 
10720-                d2.addCallback(lambda ign: self.consumer.get_current_size())
10721-                d2.addCallback(lambda size: self.consumer.read(0, size))
10722-                d2.addCallback(lambda new_contents: self.filenode.overwrite(new_contents))
10723+                d2.addCallback(lambda ign: self.filenode.overwrite(MutableFileHandle(self.consumer.get_file())))
10724             else:
10725                 def _add_file(ign):
10726                     self.log("_add_file childname=%r" % (childname,), level=OPERATIONAL)
10727}
10728[mutable/filenode.py: implement most of IVersion, per #993
10729Kevan Carstensen <kevan@isnotajoke.com>**20100717014516
10730 Ignore-this: d4551142b32ea97040ce0e98a394fde5
10731] {
10732hunk ./src/allmydata/mutable/filenode.py 8
10733 from twisted.internet import defer, reactor
10734 from foolscap.api import eventually
10735 from allmydata.interfaces import IMutableFileNode, ICheckable, ICheckResults, \
10736-                                 NotEnoughSharesError, \
10737-                                 MDMF_VERSION, SDMF_VERSION, IMutableUploadable
10738-from allmydata.util import hashutil, log
10739+     NotEnoughSharesError, MDMF_VERSION, SDMF_VERSION, IMutableUploadable, \
10740+     IMutableFileVersion
10741+from allmydata.util import hashutil, log, consumer
10742 from allmydata.util.assertutil import precondition
10743 from allmydata.uri import WriteableSSKFileURI, ReadonlySSKFileURI
10744 from allmydata.monitor import Monitor
10745hunk ./src/allmydata/mutable/filenode.py 17
10746 from pycryptopp.cipher.aes import AES
10747 
10748 from allmydata.mutable.publish import Publish, MutableFileHandle, \
10749-                                      MutableDataHandle
10750+                                      MutableData
10751 from allmydata.mutable.common import MODE_READ, MODE_WRITE, UnrecoverableFileError, \
10752      ResponseCache, UncoordinatedWriteError
10753 from allmydata.mutable.servermap import ServerMap, ServermapUpdater
10754hunk ./src/allmydata/mutable/filenode.py 134
10755 
10756     def _get_initial_contents(self, contents):
10757         if contents is None:
10758-            return MutableDataHandle("")
10759+            return MutableData("")
10760 
10761         if IMutableUploadable.providedBy(contents):
10762             return contents
10763hunk ./src/allmydata/mutable/filenode.py 208
10764 
10765     def get_size(self):
10766         return self._most_recent_size
10767+
10768     def get_current_size(self):
10769         d = self.get_size_of_best_version()
10770         d.addCallback(self._stash_size)
10771hunk ./src/allmydata/mutable/filenode.py 213
10772         return d
10773+
10774     def _stash_size(self, size):
10775         self._most_recent_size = size
10776         return size
10777hunk ./src/allmydata/mutable/filenode.py 272
10778             return cmp(self.__class__, them.__class__)
10779         return cmp(self._uri, them._uri)
10780 
10781-    def _do_serialized(self, cb, *args, **kwargs):
10782-        # note: to avoid deadlock, this callable is *not* allowed to invoke
10783-        # other serialized methods within this (or any other)
10784-        # MutableFileNode. The callable should be a bound method of this same
10785-        # MFN instance.
10786-        d = defer.Deferred()
10787-        self._serializer.addCallback(lambda ignore: cb(*args, **kwargs))
10788-        # we need to put off d.callback until this Deferred is finished being
10789-        # processed. Otherwise the caller's subsequent activities (like,
10790-        # doing other things with this node) can cause reentrancy problems in
10791-        # the Deferred code itself
10792-        self._serializer.addBoth(lambda res: eventually(d.callback, res))
10793-        # add a log.err just in case something really weird happens, because
10794-        # self._serializer stays around forever, therefore we won't see the
10795-        # usual Unhandled Error in Deferred that would give us a hint.
10796-        self._serializer.addErrback(log.err)
10797-        return d
10798 
10799     #################################
10800     # ICheckable
10801hunk ./src/allmydata/mutable/filenode.py 297
10802 
10803 
10804     #################################
10805-    # IMutableFileNode
10806+    # IFileNode
10807+
10808+    def get_best_readable_version(self):
10809+        """
10810+        I return a Deferred that fires with a MutableFileVersion
10811+        representing the best readable version of the file that I
10812+        represent
10813+        """
10814+        return self.get_readable_version()
10815+
10816+
10817+    def get_readable_version(self, servermap=None, version=None):
10818+        """
10819+        I return a Deferred that fires with an MutableFileVersion for my
10820+        version argument, if there is a recoverable file of that version
10821+        on the grid. If there is no recoverable version, I fire with an
10822+        UnrecoverableFileError.
10823+
10824+        If a servermap is provided, I look in there for the requested
10825+        version. If no servermap is provided, I create and update a new
10826+        one.
10827+
10828+        If no version is provided, then I return a MutableFileVersion
10829+        representing the best recoverable version of the file.
10830+        """
10831+        d = self._get_version_from_servermap(MODE_READ, servermap, version)
10832+        def _build_version((servermap, their_version)):
10833+            assert their_version in servermap.recoverable_versions()
10834+            assert their_version in servermap.make_versionmap()
10835+
10836+            mfv = MutableFileVersion(self,
10837+                                     servermap,
10838+                                     their_version,
10839+                                     self._storage_index,
10840+                                     self._storage_broker,
10841+                                     self._readkey,
10842+                                     history=self._history)
10843+            assert mfv.is_readonly()
10844+            # our caller can use this to download the contents of the
10845+            # mutable file.
10846+            return mfv
10847+        return d.addCallback(_build_version)
10848+
10849+
10850+    def _get_version_from_servermap(self,
10851+                                    mode,
10852+                                    servermap=None,
10853+                                    version=None):
10854+        """
10855+        I return a Deferred that fires with (servermap, version).
10856+
10857+        This function performs validation and a servermap update. If it
10858+        returns (servermap, version), the caller can assume that:
10859+            - servermap was last updated in mode.
10860+            - version is recoverable, and corresponds to the servermap.
10861+
10862+        If version and servermap are provided to me, I will validate
10863+        that version exists in the servermap, and that the servermap was
10864+        updated correctly.
10865+
10866+        If version is not provided, but servermap is, I will validate
10867+        the servermap and return the best recoverable version that I can
10868+        find in the servermap.
10869+
10870+        If the version is provided but the servermap isn't, I will
10871+        obtain a servermap that has been updated in the correct mode and
10872+        validate that version is found and recoverable.
10873+
10874+        If neither servermap nor version are provided, I will obtain a
10875+        servermap updated in the correct mode, and return the best
10876+        recoverable version that I can find in there.
10877+        """
10878+        # XXX: wording ^^^^
10879+        if servermap and servermap.last_update_mode == mode:
10880+            d = defer.succeed(servermap)
10881+        else:
10882+            d = self._get_servermap(mode)
10883+
10884+        def _get_version(servermap, version):
10885+            if version and version not in servermap.recoverable_versions():
10886+                version = None
10887+            else:
10888+                version = servermap.best_recoverable_version()
10889+            if not version:
10890+                raise UnrecoverableFileError("no recoverable versions")
10891+            return (servermap, version)
10892+        return d.addCallback(_get_version, version)
10893+
10894 
10895     def download_best_version(self):
10896hunk ./src/allmydata/mutable/filenode.py 387
10897+        """
10898+        I return a Deferred that fires with the contents of the best
10899+        version of this mutable file.
10900+        """
10901         return self._do_serialized(self._download_best_version)
10902hunk ./src/allmydata/mutable/filenode.py 392
10903+
10904+
10905     def _download_best_version(self):
10906hunk ./src/allmydata/mutable/filenode.py 395
10907-        servermap = ServerMap()
10908-        d = self._try_once_to_download_best_version(servermap, MODE_READ)
10909-        def _maybe_retry(f):
10910-            f.trap(NotEnoughSharesError)
10911-            # the download is worth retrying once. Make sure to use the
10912-            # old servermap, since it is what remembers the bad shares,
10913-            # but use MODE_WRITE to make it look for even more shares.
10914-            # TODO: consider allowing this to retry multiple times.. this
10915-            # approach will let us tolerate about 8 bad shares, I think.
10916-            return self._try_once_to_download_best_version(servermap,
10917-                                                           MODE_WRITE)
10918+        """
10919+        I am the serialized sibling of download_best_version.
10920+        """
10921+        d = self.get_best_readable_version()
10922+        d.addCallback(self._record_size)
10923+        d.addCallback(lambda version: version.download_to_data())
10924+
10925+        # It is possible that the download will fail because there
10926+        # aren't enough shares to be had. If so, we will try again after
10927+        # updating the servermap in MODE_WRITE, which may find more
10928+        # shares than updating in MODE_READ, as we just did. We can do
10929+        # this by getting the best mutable version and downloading from
10930+        # that -- the best mutable version will be a MutableFileVersion
10931+        # with a servermap that was last updated in MODE_WRITE, as we
10932+        # want. If this fails, then we give up.
10933+        def _maybe_retry(failure):
10934+            failure.trap(NotEnoughSharesError)
10935+
10936+            d = self.get_best_mutable_version()
10937+            d.addCallback(self._record_size)
10938+            d.addCallback(lambda version: version.download_to_data())
10939+            return d
10940+
10941         d.addErrback(_maybe_retry)
10942         return d
10943hunk ./src/allmydata/mutable/filenode.py 420
10944-    def _try_once_to_download_best_version(self, servermap, mode):
10945-        d = self._update_servermap(servermap, mode)
10946-        d.addCallback(self._once_updated_download_best_version, servermap)
10947-        return d
10948-    def _once_updated_download_best_version(self, ignored, servermap):
10949-        goal = servermap.best_recoverable_version()
10950-        if not goal:
10951-            raise UnrecoverableFileError("no recoverable versions")
10952-        return self._try_once_to_download_version(servermap, goal)
10953+
10954+
10955+    def _record_size(self, mfv):
10956+        """
10957+        I record the size of a mutable file version.
10958+        """
10959+        self._most_recent_size = mfv.get_size()
10960+        return mfv
10961+
10962 
10963     def get_size_of_best_version(self):
10964hunk ./src/allmydata/mutable/filenode.py 431
10965-        d = self.get_servermap(MODE_READ)
10966-        def _got_servermap(smap):
10967-            ver = smap.best_recoverable_version()
10968-            if not ver:
10969-                raise UnrecoverableFileError("no recoverable version")
10970-            return smap.size_of_version(ver)
10971-        d.addCallback(_got_servermap)
10972-        return d
10973+        """
10974+        I return the size of the best version of this mutable file.
10975+
10976+        This is equivalent to calling get_size() on the result of
10977+        get_best_readable_version().
10978+        """
10979+        d = self.get_best_readable_version()
10980+        return d.addCallback(lambda mfv: mfv.get_size())
10981+
10982+
10983+    #################################
10984+    # IMutableFileNode
10985+
10986+    def get_best_mutable_version(self, servermap=None):
10987+        """
10988+        I return a Deferred that fires with a MutableFileVersion
10989+        representing the best readable version of the file that I
10990+        represent. I am like get_best_readable_version, except that I
10991+        will try to make a writable version if I can.
10992+        """
10993+        return self.get_mutable_version(servermap=servermap)
10994+
10995+
10996+    def get_mutable_version(self, servermap=None, version=None):
10997+        """
10998+        I return a version of this mutable file. I return a Deferred
10999+        that fires with a MutableFileVersion
11000+
11001+        If version is provided, the Deferred will fire with a
11002+        MutableFileVersion initailized with that version. Otherwise, it
11003+        will fire with the best version that I can recover.
11004+
11005+        If servermap is provided, I will use that to find versions
11006+        instead of performing my own servermap update.
11007+        """
11008+        if self.is_readonly():
11009+            return self.get_readable_version(servermap=servermap,
11010+                                             version=version)
11011+
11012+        # get_mutable_version => write intent, so we require that the
11013+        # servermap is updated in MODE_WRITE
11014+        d = self._get_version_from_servermap(MODE_WRITE, servermap, version)
11015+        def _build_version((servermap, smap_version)):
11016+            # these should have been set by the servermap update.
11017+            assert self._secret_holder
11018+            assert self._writekey
11019+
11020+            mfv = MutableFileVersion(self,
11021+                                     servermap,
11022+                                     smap_version,
11023+                                     self._storage_index,
11024+                                     self._storage_broker,
11025+                                     self._readkey,
11026+                                     self._writekey,
11027+                                     self._secret_holder,
11028+                                     history=self._history)
11029+            assert not mfv.is_readonly()
11030+            return mfv
11031+
11032+        return d.addCallback(_build_version)
11033+
11034+
11035+    # XXX: I'm uncomfortable with the difference between upload and
11036+    #      overwrite, which, FWICT, is basically that you don't have to
11037+    #      do a servermap update before you overwrite. We split them up
11038+    #      that way anyway, so I guess there's no real difficulty in
11039+    #      offering both ways to callers, but it also makes the
11040+    #      public-facing API cluttery, and makes it hard to discern the
11041+    #      right way of doing things.
11042 
11043hunk ./src/allmydata/mutable/filenode.py 501
11044+    # In general, we leave it to callers to ensure that they aren't
11045+    # going to cause UncoordinatedWriteErrors when working with
11046+    # MutableFileVersions. We know that the next three operations
11047+    # (upload, overwrite, and modify) will all operate on the same
11048+    # version, so we say that only one of them can be going on at once,
11049+    # and serialize them to ensure that that actually happens, since as
11050+    # the caller in this situation it is our job to do that.
11051     def overwrite(self, new_contents):
11052hunk ./src/allmydata/mutable/filenode.py 509
11053+        """
11054+        I overwrite the contents of the best recoverable version of this
11055+        mutable file with new_contents. This is equivalent to calling
11056+        overwrite on the result of get_best_mutable_version with
11057+        new_contents as an argument. I return a Deferred that eventually
11058+        fires with the results of my replacement process.
11059+        """
11060         return self._do_serialized(self._overwrite, new_contents)
11061hunk ./src/allmydata/mutable/filenode.py 517
11062+
11063+
11064     def _overwrite(self, new_contents):
11065hunk ./src/allmydata/mutable/filenode.py 520
11066-        assert IMutableUploadable.providedBy(new_contents)
11067+        """
11068+        I am the serialized sibling of overwrite.
11069+        """
11070+        d = self.get_best_mutable_version()
11071+        return d.addCallback(lambda mfv: mfv.overwrite(new_contents))
11072+
11073+
11074+
11075+    def upload(self, new_contents, servermap):
11076+        """
11077+        I overwrite the contents of the best recoverable version of this
11078+        mutable file with new_contents, using servermap instead of
11079+        creating/updating our own servermap. I return a Deferred that
11080+        fires with the results of my upload.
11081+        """
11082+        return self._do_serialized(self._upload, new_contents, servermap)
11083+
11084+
11085+    def _upload(self, new_contents, servermap):
11086+        """
11087+        I am the serialized sibling of upload.
11088+        """
11089+        d = self.get_best_mutable_version(servermap)
11090+        return d.addCallback(lambda mfv: mfv.overwrite(new_contents))
11091+
11092+
11093+    def modify(self, modifier, backoffer=None):
11094+        """
11095+        I modify the contents of the best recoverable version of this
11096+        mutable file with the modifier. This is equivalent to calling
11097+        modify on the result of get_best_mutable_version. I return a
11098+        Deferred that eventually fires with an UploadResults instance
11099+        describing this process.
11100+        """
11101+        return self._do_serialized(self._modify, modifier, backoffer)
11102+
11103+
11104+    def _modify(self, modifier, backoffer):
11105+        """
11106+        I am the serialized sibling of modify.
11107+        """
11108+        d = self.get_best_mutable_version()
11109+        return d.addCallback(lambda mfv: mfv.modify(modifier, backoffer))
11110+
11111+
11112+    def download_version(self, servermap, version, fetch_privkey=False):
11113+        """
11114+        Download the specified version of this mutable file. I return a
11115+        Deferred that fires with the contents of the specified version
11116+        as a bytestring, or errbacks if the file is not recoverable.
11117+        """
11118+        d = self.get_readable_version(servermap, version)
11119+        return d.addCallback(lambda mfv: mfv.download_to_data(fetch_privkey))
11120+
11121+
11122+    def get_servermap(self, mode):
11123+        """
11124+        I return a servermap that has been updated in mode.
11125+
11126+        mode should be one of MODE_READ, MODE_WRITE, MODE_CHECK or
11127+        MODE_ANYTHING. See servermap.py for more on what these mean.
11128+        """
11129+        return self._do_serialized(self._get_servermap, mode)
11130+
11131 
11132hunk ./src/allmydata/mutable/filenode.py 585
11133+    def _get_servermap(self, mode):
11134+        """
11135+        I am a serialized twin to get_servermap.
11136+        """
11137         servermap = ServerMap()
11138hunk ./src/allmydata/mutable/filenode.py 590
11139-        d = self._update_servermap(servermap, mode=MODE_WRITE)
11140-        d.addCallback(lambda ignored: self._upload(new_contents, servermap))
11141+        return self._update_servermap(servermap, mode)
11142+
11143+
11144+    def _update_servermap(self, servermap, mode):
11145+        u = ServermapUpdater(self, self._storage_broker, Monitor(), servermap,
11146+                             mode)
11147+        if self._history:
11148+            self._history.notify_mapupdate(u.get_status())
11149+        return u.update()
11150+
11151+
11152+    def set_version(self, version):
11153+        # I can be set in two ways:
11154+        #  1. When the node is created.
11155+        #  2. (for an existing share) when the Servermap is updated
11156+        #     before I am read.
11157+        assert version in (MDMF_VERSION, SDMF_VERSION)
11158+        self._protocol_version = version
11159+
11160+
11161+    def get_version(self):
11162+        return self._protocol_version
11163+
11164+
11165+    def _do_serialized(self, cb, *args, **kwargs):
11166+        # note: to avoid deadlock, this callable is *not* allowed to invoke
11167+        # other serialized methods within this (or any other)
11168+        # MutableFileNode. The callable should be a bound method of this same
11169+        # MFN instance.
11170+        d = defer.Deferred()
11171+        self._serializer.addCallback(lambda ignore: cb(*args, **kwargs))
11172+        # we need to put off d.callback until this Deferred is finished being
11173+        # processed. Otherwise the caller's subsequent activities (like,
11174+        # doing other things with this node) can cause reentrancy problems in
11175+        # the Deferred code itself
11176+        self._serializer.addBoth(lambda res: eventually(d.callback, res))
11177+        # add a log.err just in case something really weird happens, because
11178+        # self._serializer stays around forever, therefore we won't see the
11179+        # usual Unhandled Error in Deferred that would give us a hint.
11180+        self._serializer.addErrback(log.err)
11181         return d
11182 
11183 
11184hunk ./src/allmydata/mutable/filenode.py 633
11185+    def _upload(self, new_contents, servermap):
11186+        """
11187+        A MutableFileNode still has to have some way of getting
11188+        published initially, which is what I am here for. After that,
11189+        all publishing, updating, modifying and so on happens through
11190+        MutableFileVersions.
11191+        """
11192+        assert self._pubkey, "update_servermap must be called before publish"
11193+
11194+        p = Publish(self, self._storage_broker, servermap)
11195+        if self._history:
11196+            self._history.notify_publish(p.get_status(),
11197+                                         new_contents.get_size())
11198+        d = p.publish(new_contents)
11199+        d.addCallback(self._did_upload, new_contents.get_size())
11200+        return d
11201+
11202+
11203+    def _did_upload(self, res, size):
11204+        self._most_recent_size = size
11205+        return res
11206+
11207+
11208+class MutableFileVersion:
11209+    """
11210+    I represent a specific version (most likely the best version) of a
11211+    mutable file.
11212+
11213+    Since I implement IReadable, instances which hold a
11214+    reference to an instance of me are guaranteed the ability (absent
11215+    connection difficulties or unrecoverable versions) to read the file
11216+    that I represent. Depending on whether I was initialized with a
11217+    write capability or not, I may also provide callers the ability to
11218+    overwrite or modify the contents of the mutable file that I
11219+    reference.
11220+    """
11221+    implements(IMutableFileVersion)
11222+
11223+    def __init__(self,
11224+                 node,
11225+                 servermap,
11226+                 version,
11227+                 storage_index,
11228+                 storage_broker,
11229+                 readcap,
11230+                 writekey=None,
11231+                 write_secrets=None,
11232+                 history=None):
11233+
11234+        self._node = node
11235+        self._servermap = servermap
11236+        self._version = version
11237+        self._storage_index = storage_index
11238+        self._write_secrets = write_secrets
11239+        self._history = history
11240+        self._storage_broker = storage_broker
11241+
11242+        #assert isinstance(readcap, IURI)
11243+        self._readcap = readcap
11244+
11245+        self._writekey = writekey
11246+        self._serializer = defer.succeed(None)
11247+        self._size = None
11248+
11249+
11250+    def get_sequence_number(self):
11251+        """
11252+        Get the sequence number of the mutable version that I represent.
11253+        """
11254+        return 0
11255+
11256+
11257+    # TODO: Terminology?
11258+    def get_writekey(self):
11259+        """
11260+        I return a writekey or None if I don't have a writekey.
11261+        """
11262+        return self._writekey
11263+
11264+
11265+    def overwrite(self, new_contents):
11266+        """
11267+        I overwrite the contents of this mutable file version with the
11268+        data in new_contents.
11269+        """
11270+        assert not self.is_readonly()
11271+
11272+        return self._do_serialized(self._overwrite, new_contents)
11273+
11274+
11275+    def _overwrite(self, new_contents):
11276+        assert IMutableUploadable.providedBy(new_contents)
11277+        assert self._servermap.last_update_mode == MODE_WRITE
11278+
11279+        return self._upload(new_contents)
11280+
11281+
11282     def modify(self, modifier, backoffer=None):
11283         """I use a modifier callback to apply a change to the mutable file.
11284         I implement the following pseudocode::
11285hunk ./src/allmydata/mutable/filenode.py 770
11286         backoffer should not invoke any methods on this MutableFileNode
11287         instance, and it needs to be highly conscious of deadlock issues.
11288         """
11289+        assert not self.is_readonly()
11290+
11291         return self._do_serialized(self._modify, modifier, backoffer)
11292hunk ./src/allmydata/mutable/filenode.py 773
11293+
11294+
11295     def _modify(self, modifier, backoffer):
11296hunk ./src/allmydata/mutable/filenode.py 776
11297-        servermap = ServerMap()
11298         if backoffer is None:
11299             backoffer = BackoffAgent().delay
11300hunk ./src/allmydata/mutable/filenode.py 778
11301-        return self._modify_and_retry(servermap, modifier, backoffer, True)
11302-    def _modify_and_retry(self, servermap, modifier, backoffer, first_time):
11303-        d = self._modify_once(servermap, modifier, first_time)
11304+        return self._modify_and_retry(modifier, backoffer, True)
11305+
11306+
11307+    def _modify_and_retry(self, modifier, backoffer, first_time):
11308+        """
11309+        I try to apply modifier to the contents of this version of the
11310+        mutable file. If I succeed, I return an UploadResults instance
11311+        describing my success. If I fail, I try again after waiting for
11312+        a little bit.
11313+        """
11314+        log.msg("doing modify")
11315+        d = self._modify_once(modifier, first_time)
11316         def _retry(f):
11317             f.trap(UncoordinatedWriteError)
11318             d2 = defer.maybeDeferred(backoffer, self, f)
11319hunk ./src/allmydata/mutable/filenode.py 794
11320             d2.addCallback(lambda ignored:
11321-                           self._modify_and_retry(servermap, modifier,
11322+                           self._modify_and_retry(modifier,
11323                                                   backoffer, False))
11324             return d2
11325         d.addErrback(_retry)
11326hunk ./src/allmydata/mutable/filenode.py 799
11327         return d
11328-    def _modify_once(self, servermap, modifier, first_time):
11329-        d = self._update_servermap(servermap, MODE_WRITE)
11330-        d.addCallback(self._once_updated_download_best_version, servermap)
11331+
11332+
11333+    def _modify_once(self, modifier, first_time):
11334+        """
11335+        I attempt to apply a modifier to the contents of the mutable
11336+        file.
11337+        """
11338+        assert self._servermap.last_update_mode == MODE_WRITE
11339+
11340+        # download_to_data is serialized, so we have to call this to
11341+        # avoid deadlock.
11342+        d = self._try_to_download_data()
11343         def _apply(old_contents):
11344hunk ./src/allmydata/mutable/filenode.py 812
11345-            new_contents = modifier(old_contents, servermap, first_time)
11346+            new_contents = modifier(old_contents, self._servermap, first_time)
11347             if new_contents is None or new_contents == old_contents:
11348hunk ./src/allmydata/mutable/filenode.py 814
11349+                log.msg("no changes")
11350                 # no changes need to be made
11351                 if first_time:
11352                     return
11353hunk ./src/allmydata/mutable/filenode.py 822
11354                 # recovery when it observes UCWE, we need to do a second
11355                 # publish. See #551 for details. We'll basically loop until
11356                 # we managed an uncontested publish.
11357-                old_uploadable = MutableDataHandle(old_contents)
11358+                old_uploadable = MutableData(old_contents)
11359                 new_contents = old_uploadable
11360             precondition((IMutableUploadable.providedBy(new_contents) or
11361                           new_contents is None),
11362hunk ./src/allmydata/mutable/filenode.py 828
11363                          "Modifier function must return an IMutableUploadable "
11364                          "or None")
11365-            return self._upload(new_contents, servermap)
11366+            return self._upload(new_contents)
11367         d.addCallback(_apply)
11368         return d
11369 
11370hunk ./src/allmydata/mutable/filenode.py 832
11371-    def get_servermap(self, mode):
11372-        return self._do_serialized(self._get_servermap, mode)
11373-    def _get_servermap(self, mode):
11374-        servermap = ServerMap()
11375-        return self._update_servermap(servermap, mode)
11376-    def _update_servermap(self, servermap, mode):
11377-        u = ServermapUpdater(self, self._storage_broker, Monitor(), servermap,
11378-                             mode)
11379-        if self._history:
11380-            self._history.notify_mapupdate(u.get_status())
11381-        return u.update()
11382 
11383hunk ./src/allmydata/mutable/filenode.py 833
11384-    def download_version(self, servermap, version, fetch_privkey=False):
11385-        return self._do_serialized(self._try_once_to_download_version,
11386-                                   servermap, version, fetch_privkey)
11387-    def _try_once_to_download_version(self, servermap, version,
11388-                                      fetch_privkey=False):
11389-        r = Retrieve(self, servermap, version, fetch_privkey)
11390+    def is_readonly(self):
11391+        """
11392+        I return True if this MutableFileVersion provides no write
11393+        access to the file that it encapsulates, and False if it
11394+        provides the ability to modify the file.
11395+        """
11396+        return self._writekey is None
11397+
11398+
11399+    def is_mutable(self):
11400+        """
11401+        I return True, since mutable files are always mutable by
11402+        somebody.
11403+        """
11404+        return True
11405+
11406+
11407+    def get_storage_index(self):
11408+        """
11409+        I return the storage index of the reference that I encapsulate.
11410+        """
11411+        return self._storage_index
11412+
11413+
11414+    def get_size(self):
11415+        """
11416+        I return the length, in bytes, of this readable object.
11417+        """
11418+        return self._servermap.size_of_version(self._version)
11419+
11420+
11421+    def download_to_data(self, fetch_privkey=False):
11422+        """
11423+        I return a Deferred that fires with the contents of this
11424+        readable object as a byte string.
11425+
11426+        """
11427+        c = consumer.MemoryConsumer()
11428+        d = self.read(c, fetch_privkey=fetch_privkey)
11429+        d.addCallback(lambda mc: "".join(mc.chunks))
11430+        return d
11431+
11432+
11433+    def _try_to_download_data(self):
11434+        """
11435+        I am an unserialized cousin of download_to_data; I am called
11436+        from the children of modify() to download the data associated
11437+        with this mutable version.
11438+        """
11439+        c = consumer.MemoryConsumer()
11440+        # modify will almost certainly write, so we need the privkey.
11441+        d = self._read(c, fetch_privkey=True)
11442+        d.addCallback(lambda mc: "".join(mc.chunks))
11443+        return d
11444+
11445+
11446+    def _update_servermap(self, mode=MODE_READ):
11447+        """
11448+        I update our Servermap according to my mode argument. I return a
11449+        Deferred that fires with None when this has finished. The
11450+        updated Servermap will be at self._servermap in that case.
11451+        """
11452+        d = self._node.get_servermap(mode)
11453+
11454+        def _got_servermap(servermap):
11455+            assert servermap.last_update_mode == mode
11456+
11457+            self._servermap = servermap
11458+        d.addCallback(_got_servermap)
11459+        return d
11460+
11461+
11462+    def read(self, consumer, offset=0, size=None, fetch_privkey=False):
11463+        """
11464+        I read a portion (possibly all) of the mutable file that I
11465+        reference into consumer.
11466+        """
11467+        return self._do_serialized(self._read, consumer, offset, size,
11468+                                   fetch_privkey)
11469+
11470+
11471+    def _read(self, consumer, offset=0, size=None, fetch_privkey=False):
11472+        """
11473+        I am the serialized companion of read.
11474+        """
11475+        r = Retrieve(self._node, self._servermap, self._version, fetch_privkey)
11476         if self._history:
11477             self._history.notify_retrieve(r.get_status())
11478hunk ./src/allmydata/mutable/filenode.py 921
11479-        d = r.download()
11480-        d.addCallback(self._downloaded_version)
11481+        d = r.download(consumer, offset, size)
11482+        return d
11483+
11484+
11485+    def _do_serialized(self, cb, *args, **kwargs):
11486+        # note: to avoid deadlock, this callable is *not* allowed to invoke
11487+        # other serialized methods within this (or any other)
11488+        # MutableFileNode. The callable should be a bound method of this same
11489+        # MFN instance.
11490+        d = defer.Deferred()
11491+        self._serializer.addCallback(lambda ignore: cb(*args, **kwargs))
11492+        # we need to put off d.callback until this Deferred is finished being
11493+        # processed. Otherwise the caller's subsequent activities (like,
11494+        # doing other things with this node) can cause reentrancy problems in
11495+        # the Deferred code itself
11496+        self._serializer.addBoth(lambda res: eventually(d.callback, res))
11497+        # add a log.err just in case something really weird happens, because
11498+        # self._serializer stays around forever, therefore we won't see the
11499+        # usual Unhandled Error in Deferred that would give us a hint.
11500+        self._serializer.addErrback(log.err)
11501         return d
11502hunk ./src/allmydata/mutable/filenode.py 942
11503-    def _downloaded_version(self, data):
11504-        self._most_recent_size = len(data)
11505-        return data
11506 
11507hunk ./src/allmydata/mutable/filenode.py 943
11508-    def upload(self, new_contents, servermap):
11509-        return self._do_serialized(self._upload, new_contents, servermap)
11510-    def _upload(self, new_contents, servermap):
11511-        assert self._pubkey, "update_servermap must be called before publish"
11512-        assert IMutableUploadable.providedBy(new_contents)
11513 
11514hunk ./src/allmydata/mutable/filenode.py 944
11515-        p = Publish(self, self._storage_broker, servermap)
11516+    def _upload(self, new_contents):
11517+        #assert self._pubkey, "update_servermap must be called before publish"
11518+        p = Publish(self._node, self._storage_broker, self._servermap)
11519         if self._history:
11520hunk ./src/allmydata/mutable/filenode.py 948
11521-            self._history.notify_publish(p.get_status(), new_contents.get_size())
11522+            self._history.notify_publish(p.get_status(),
11523+                                         new_contents.get_size())
11524         d = p.publish(new_contents)
11525         d.addCallback(self._did_upload, new_contents.get_size())
11526         return d
11527hunk ./src/allmydata/mutable/filenode.py 953
11528-    def _did_upload(self, res, size):
11529-        self._most_recent_size = size
11530-        return res
11531-
11532-
11533-    def set_version(self, version):
11534-        # I can be set in two ways:
11535-        #  1. When the node is created.
11536-        #  2. (for an existing share) when the Servermap is updated
11537-        #     before I am read.
11538-        assert version in (MDMF_VERSION, SDMF_VERSION)
11539-        self._protocol_version = version
11540 
11541 
11542hunk ./src/allmydata/mutable/filenode.py 955
11543-    def get_version(self):
11544-        return self._protocol_version
11545+    def _did_upload(self, res, size):
11546+        self._size = size
11547+        return res
11548}
11549[mutable/publish.py: enable segmented uploading for big files, change constants and wording, change MutableDataHandle to MutableData
11550Kevan Carstensen <kevan@isnotajoke.com>**20100717014549
11551 Ignore-this: f736c60c90ff09c98544af17146cf654
11552] {
11553hunk ./src/allmydata/mutable/publish.py 145
11554 
11555         self.data = newdata
11556         self.datalength = newdata.get_size()
11557+        if self.datalength >= DEFAULT_MAX_SEGMENT_SIZE:
11558+            self._version = MDMF_VERSION
11559+        else:
11560+            self._version = SDMF_VERSION
11561 
11562         self.log("starting publish, datalen is %s" % self.datalength)
11563         self._status.set_size(self.datalength)
11564hunk ./src/allmydata/mutable/publish.py 1007
11565             old_position = self._filehandle.tell()
11566             # Seek to the end of the file by seeking 0 bytes from the
11567             # file's end
11568-            self._filehandle.seek(0, os.SEEK_END)
11569+            self._filehandle.seek(0, 2) # 2 == os.SEEK_END in 2.5+
11570             self._size = self._filehandle.tell()
11571             # Restore the previous position, in case this was called
11572             # after a read.
11573hunk ./src/allmydata/mutable/publish.py 1022
11574         """
11575         I return some data (up to length bytes) from my filehandle.
11576 
11577-        In most cases, I return length bytes. If I don't, it is because
11578-        length is longer than the distance between my current position
11579-        in the file that I represent and its end. In that case, I return
11580-        as many bytes as I can before going over the EOF.
11581+        In most cases, I return length bytes, but sometimes I won't --
11582+        for example, if I am asked to read beyond the end of a file, or
11583+        an error occurs.
11584         """
11585         return [self._filehandle.read(length)]
11586 
11587hunk ./src/allmydata/mutable/publish.py 1037
11588         self._filehandle.close()
11589 
11590 
11591-class MutableDataHandle(MutableFileHandle):
11592+class MutableData(MutableFileHandle):
11593     """
11594     I am a mutable uploadable built around a string, which I then cast
11595     into a StringIO and treat as a filehandle.
11596}
11597[immutable/filenode.py: fix broken implementation of #993 interfaces
11598Kevan Carstensen <kevan@isnotajoke.com>**20100717015049
11599 Ignore-this: 19ac8cf5d31ac88d4a1998ac342db004
11600] {
11601hunk ./src/allmydata/immutable/filenode.py 326
11602         immutable files can have only one version, we just return the
11603         current filenode.
11604         """
11605-        return self
11606+        return defer.succeed(self)
11607 
11608 
11609     def download_best_version(self):
11610hunk ./src/allmydata/immutable/filenode.py 412
11611 
11612     # IReadable, IFileNode, IFilesystemNode
11613     def get_best_readable_version(self):
11614-        return self
11615+        return defer.succeed(self)
11616 
11617 
11618     def download_best_version(self):
11619}
11620[mutable/retrieve.py: alter Retrieve so that it can download parts of mutable files
11621Kevan Carstensen <kevan@isnotajoke.com>**20100717015123
11622 Ignore-this: f49a6d3c05afc784aff0a4c63964a3e5
11623] {
11624hunk ./src/allmydata/mutable/retrieve.py 7
11625 from zope.interface import implements
11626 from twisted.internet import defer
11627 from twisted.python import failure
11628+from twisted.internet.interfaces import IPushProducer, IConsumer
11629 from foolscap.api import DeadReferenceError, eventually, fireEventually
11630 from allmydata.interfaces import IRetrieveStatus, NotEnoughSharesError, \
11631                                  MDMF_VERSION, SDMF_VERSION
11632hunk ./src/allmydata/mutable/retrieve.py 86
11633     # times, and each will have a separate response chain. However the
11634     # Retrieve object will remain tied to a specific version of the file, and
11635     # will use a single ServerMap instance.
11636+    implements(IPushProducer)
11637 
11638     def __init__(self, filenode, servermap, verinfo, fetch_privkey=False,
11639                  verify=False):
11640hunk ./src/allmydata/mutable/retrieve.py 129
11641         # 3. When we are validating readers, we need to validate the
11642         #    signature on the prefix. Do we? We already do this in the
11643         #    servermap update?
11644-        #
11645-        # (just work on 1 and 2 for now, I guess)
11646         self._verify = False
11647         if verify:
11648             self._verify = True
11649hunk ./src/allmydata/mutable/retrieve.py 143
11650         self._status.set_size(datalength)
11651         self._status.set_encoding(k, N)
11652         self.readers = {}
11653+        self._paused = False
11654+        self._paused_deferred = None
11655+
11656 
11657     def get_status(self):
11658         return self._status
11659hunk ./src/allmydata/mutable/retrieve.py 157
11660             kwargs["facility"] = "tahoe.mutable.retrieve"
11661         return log.msg(*args, **kwargs)
11662 
11663-    def download(self):
11664+
11665+    ###################
11666+    # IPushProducer
11667+
11668+    def pauseProducing(self):
11669+        """
11670+        I am called by my download target if we have produced too much
11671+        data for it to handle. I make the downloader stop producing new
11672+        data until my resumeProducing method is called.
11673+        """
11674+        if self._paused:
11675+            return
11676+
11677+        # fired when the download is unpaused.
11678+        self._pause_deferred = defer.Deferred()
11679+        self._paused = True
11680+
11681+
11682+    def resumeProducing(self):
11683+        """
11684+        I am called by my download target once it is ready to begin
11685+        receiving data again.
11686+        """
11687+        if not self._paused:
11688+            return
11689+
11690+        self._paused = False
11691+        p = self._pause_deferred
11692+        self._pause_deferred = None
11693+        eventually(p.callback, None)
11694+
11695+
11696+    def _check_for_paused(self, res):
11697+        """
11698+        I am called just before a write to the consumer. I return a
11699+        Deferred that eventually fires with the data that is to be
11700+        written to the consumer. If the download has not been paused,
11701+        the Deferred fires immediately. Otherwise, the Deferred fires
11702+        when the downloader is unpaused.
11703+        """
11704+        if self._paused:
11705+            d = defer.Deferred()
11706+            self._pause_defered.addCallback(lambda ignored: d.callback(res))
11707+            return d
11708+        return defer.succeed(res)
11709+
11710+
11711+    def download(self, consumer=None, offset=0, size=None):
11712+        assert IConsumer.providedBy(consumer) or self._verify
11713+
11714+        if consumer:
11715+            self._consumer = consumer
11716+            # we provide IPushProducer, so streaming=True, per
11717+            # IConsumer.
11718+            self._consumer.registerProducer(self, streaming=True)
11719+
11720         self._done_deferred = defer.Deferred()
11721         self._started = time.time()
11722         self._status.set_status("Retrieving Shares")
11723hunk ./src/allmydata/mutable/retrieve.py 217
11724 
11725+        self._offset = offset
11726+        self._read_length = size
11727+
11728         # first, which servers can we use?
11729         versionmap = self.servermap.make_versionmap()
11730         shares = versionmap[self.verinfo]
11731hunk ./src/allmydata/mutable/retrieve.py 278
11732         assert len(self.remaining_sharemap) >= k
11733 
11734         self.log("starting download")
11735+        self._paused = False
11736         self._add_active_peers()
11737         # The download process beyond this is a state machine.
11738         # _add_active_peers will select the peers that we want to use
11739hunk ./src/allmydata/mutable/retrieve.py 324
11740 
11741         self._segment_decoder = codec.CRSDecoder()
11742         self._segment_decoder.set_params(segsize, k, n)
11743-        self._current_segment = 0
11744 
11745         if  not self._tail_data_size:
11746             self._tail_data_size = segsize
11747hunk ./src/allmydata/mutable/retrieve.py 349
11748             # So we don't have to do this later.
11749             self._block_hash_trees[i] = hashtree.IncompleteHashTree(self._num_segments)
11750 
11751-        # If we have more than one segment, we are an SDMF file, which
11752-        # means that we need to validate the salts as we receive them.
11753-        self._salt_hash_tree = hashtree.IncompleteHashTree(self._num_segments)
11754-        self._salt_hash_tree[0] = IV # from the prefix.
11755+        # Our last task is to tell the downloader where to start and
11756+        # where to stop. We use three parameters for that:
11757+        #   - self._start_segment: the segment that we need to start
11758+        #     downloading from.
11759+        #   - self._current_segment: the next segment that we need to
11760+        #     download.
11761+        #   - self._last_segment: The last segment that we were asked to
11762+        #     download.
11763+        #
11764+        #  We say that the download is complete when
11765+        #  self._current_segment > self._last_segment. We use
11766+        #  self._start_segment and self._last_segment to know when to
11767+        #  strip things off of segments, and how much to strip.
11768+        if self._offset:
11769+            self.log("got offset: %d" % self._offset)
11770+            # our start segment is the first segment containing the
11771+            # offset we were given.
11772+            start = mathutil.div_ceil(self._offset,
11773+                                      self._segment_size)
11774+            # this gets us the first segment after self._offset. Then
11775+            # our start segment is the one before it.
11776+            start -= 1
11777+
11778+            assert start < self._num_segments
11779+            self._start_segment = start
11780+            self.log("got start segment: %d" % self._start_segment)
11781+        else:
11782+            self._start_segment = 0
11783+
11784+
11785+        if self._read_length:
11786+            # our end segment is the last segment containing part of the
11787+            # segment that we were asked to read.
11788+            self.log("got read length %d" % self._read_length)
11789+            end_data = self._offset + self._read_length
11790+            end = mathutil.div_ceil(end_data,
11791+                                    self._segment_size)
11792+            end -= 1
11793+            assert end < self._num_segments
11794+            self._last_segment = end
11795+            self.log("got end segment: %d" % self._last_segment)
11796+        else:
11797+            self._last_segment = self._num_segments - 1
11798 
11799hunk ./src/allmydata/mutable/retrieve.py 393
11800+        self._current_segment = self._start_segment
11801 
11802     def _add_active_peers(self):
11803         """
11804hunk ./src/allmydata/mutable/retrieve.py 637
11805         that this Retrieve is currently responsible for downloading.
11806         """
11807         assert len(self._active_readers) >= self._required_shares
11808-        if self._current_segment < self._num_segments:
11809+        if self._current_segment <= self._last_segment:
11810             d = self._process_segment(self._current_segment)
11811         else:
11812             d = defer.succeed(None)
11813hunk ./src/allmydata/mutable/retrieve.py 701
11814             d.addCallback(self._decrypt_segment)
11815             d.addErrback(self._validation_or_decoding_failed,
11816                          self._active_readers)
11817+            # check to see whether we've been paused before writing
11818+            # anything.
11819+            d.addCallback(self._check_for_paused)
11820             d.addCallback(self._set_segment)
11821             return d
11822         else:
11823hunk ./src/allmydata/mutable/retrieve.py 716
11824         target that is handling the file download.
11825         """
11826         self.log("got plaintext for segment %d" % self._current_segment)
11827-        self._plaintext += segment
11828+        if self._current_segment == self._start_segment:
11829+            # We're on the first segment. It's possible that we want
11830+            # only some part of the end of this segment, and that we
11831+            # just downloaded the whole thing to get that part. If so,
11832+            # we need to account for that and give the reader just the
11833+            # data that they want.
11834+            n = self._offset % self._segment_size
11835+            self.log("stripping %d bytes off of the first segment" % n)
11836+            self.log("original segment length: %d" % len(segment))
11837+            segment = segment[n:]
11838+            self.log("new segment length: %d" % len(segment))
11839+
11840+        if self._current_segment == self._last_segment and self._read_length is not None:
11841+            # We're on the last segment. It's possible that we only want
11842+            # part of the beginning of this segment, and that we
11843+            # downloaded the whole thing anyway. Make sure to give the
11844+            # caller only the portion of the segment that they want to
11845+            # receive.
11846+            extra = self._read_length
11847+            if self._start_segment != self._last_segment:
11848+                extra -= self._segment_size - \
11849+                            (self._offset % self._segment_size)
11850+            extra %= self._segment_size
11851+            self.log("original segment length: %d" % len(segment))
11852+            segment = segment[:extra]
11853+            self.log("new segment length: %d" % len(segment))
11854+            self.log("only taking %d bytes of the last segment" % extra)
11855+
11856+        if not self._verify:
11857+            self._consumer.write(segment)
11858+        else:
11859+            # we don't care about the plaintext if we are doing a verify.
11860+            segment = None
11861         self._current_segment += 1
11862 
11863 
11864hunk ./src/allmydata/mutable/retrieve.py 848
11865                                         reader.shnum,
11866                                         "corrupt hashes: %s" % e)
11867 
11868-        # TODO: Validate the salt, too.
11869         self.log('share %d is valid for segment %d' % (reader.shnum,
11870                                                        segnum))
11871         return {reader.shnum: (block, salt)}
11872hunk ./src/allmydata/mutable/retrieve.py 1014
11873               _done_deferred to errback.
11874         """
11875         self.log("checking for doneness")
11876-        if self._current_segment == self._num_segments:
11877+        if self._current_segment > self._last_segment:
11878             # No more segments to download, we're done.
11879             self.log("got plaintext, done")
11880             return self._done()
11881hunk ./src/allmydata/mutable/retrieve.py 1043
11882             ret = list(self._bad_shares)
11883             self.log("done verifying, found %d bad shares" % len(ret))
11884         else:
11885-            ret = self._plaintext
11886+            # TODO: upload status here?
11887+            ret = self._consumer
11888+            self._consumer.unregisterProducer()
11889         eventually(self._done_deferred.callback, ret)
11890 
11891 
11892hunk ./src/allmydata/mutable/retrieve.py 1066
11893                       "encoding %(k)d-of-%(n)d")
11894             args = {"have": self._current_segment,
11895                     "total": self._num_segments,
11896+                    "need": self._last_segment,
11897                     "k": self._required_shares,
11898                     "n": self._total_shares,
11899                     "bad": len(self._bad_shares)}
11900}
11901[change MutableDataHandle to MutableData in code.
11902Kevan Carstensen <kevan@isnotajoke.com>**20100717015210
11903 Ignore-this: f85ae425eabc21b47ad60bd6bf1f7dec
11904] {
11905hunk ./src/allmydata/dirnode.py 11
11906 from allmydata.mutable.common import NotWriteableError
11907 from allmydata.mutable.filenode import MutableFileNode
11908 from allmydata.unknown import UnknownNode, strip_prefix_for_ro
11909-from allmydata.mutable.publish import MutableDataHandle
11910+from allmydata.mutable.publish import MutableData
11911 from allmydata.interfaces import IFilesystemNode, IDirectoryNode, IFileNode, \
11912      IImmutableFileNode, IMutableFileNode, \
11913      ExistingChildError, NoSuchChildError, ICheckable, IDeepCheckable, \
11914hunk ./src/allmydata/dirnode.py 104
11915 
11916         del children[self.name]
11917         new_contents = self.node._pack_contents(children)
11918-        uploadable = MutableDataHandle(new_contents)
11919+        uploadable = MutableData(new_contents)
11920         return uploadable
11921 
11922 
11923hunk ./src/allmydata/dirnode.py 130
11924 
11925         children[name] = (child, metadata)
11926         new_contents = self.node._pack_contents(children)
11927-        uploadable = MutableDataHandle(new_contents)
11928+        uploadable = MutableData(new_contents)
11929         return uploadable
11930 
11931 
11932hunk ./src/allmydata/dirnode.py 175
11933 
11934             children[name] = (child, metadata)
11935         new_contents = self.node._pack_contents(children)
11936-        uploadable = MutableDataHandle(new_contents)
11937+        uploadable = MutableData(new_contents)
11938         return uploadable
11939 
11940 def _encrypt_rw_uri(writekey, rw_uri):
11941hunk ./src/allmydata/mutable/repairer.py 5
11942 from zope.interface import implements
11943 from twisted.internet import defer
11944 from allmydata.interfaces import IRepairResults, ICheckResults
11945-from allmydata.mutable.publish import MutableDataHandle
11946+from allmydata.mutable.publish import MutableData
11947 
11948 class RepairResults:
11949     implements(IRepairResults)
11950hunk ./src/allmydata/mutable/repairer.py 109
11951 
11952         d = self.node.download_version(smap, best_version, fetch_privkey=True)
11953         d.addCallback(lambda data:
11954-            MutableDataHandle(data))
11955+            MutableData(data))
11956         d.addCallback(self.node.upload, smap)
11957         d.addCallback(self.get_results, smap)
11958         return d
11959hunk ./src/allmydata/nodemaker.py 9
11960 from allmydata.immutable.filenode import ImmutableFileNode, LiteralFileNode
11961 from allmydata.immutable.upload import Data
11962 from allmydata.mutable.filenode import MutableFileNode
11963-from allmydata.mutable.publish import MutableDataHandle
11964+from allmydata.mutable.publish import MutableData
11965 from allmydata.dirnode import DirectoryNode, pack_children
11966 from allmydata.unknown import UnknownNode
11967 from allmydata import uri
11968merger 0.0 (
11969merger 0.0 (
11970hunk ./src/allmydata/nodemaker.py 107
11971-                                     pack_children(n, initial_children),
11972+                                     MutableDataHandle(
11973+                                        pack_children(n, initial_children)),
11974merger 0.0 (
11975hunk ./src/allmydata/nodemaker.py 107
11976-                                     pack_children(n, initial_children))
11977+                                     pack_children(n, initial_children),
11978+                                     version)
11979hunk ./src/allmydata/nodemaker.py 107
11980-                                     pack_children(n, initial_children))
11981+                                     pack_children(initial_children, n.get_writekey()))
11982)
11983)
11984hunk ./src/allmydata/nodemaker.py 107
11985-                                     MutableDataHandle(
11986+                                     MutableData(
11987)
11988hunk ./src/allmydata/test/common.py 18
11989      DeepCheckResults, DeepCheckAndRepairResults
11990 from allmydata.mutable.common import CorruptShareError
11991 from allmydata.mutable.layout import unpack_header
11992-from allmydata.mutable.publish import MutableDataHandle
11993+from allmydata.mutable.publish import MutableData
11994 from allmydata.storage.server import storage_index_to_dir
11995 from allmydata.storage.mutable import MutableShareFile
11996 from allmydata.util import hashutil, log, fileutil, pollmixin
11997hunk ./src/allmydata/test/common.py 192
11998         return defer.succeed(self)
11999     def _get_initial_contents(self, contents):
12000         if contents is None:
12001-            return MutableDataHandle("")
12002+            return MutableData("")
12003 
12004         if IMutableUploadable.providedBy(contents):
12005             return contents
12006hunk ./src/allmydata/test/test_checker.py 11
12007 from allmydata.test.no_network import GridTestMixin
12008 from allmydata.immutable.upload import Data
12009 from allmydata.test.common_web import WebRenderingMixin
12010-from allmydata.mutable.publish import MutableDataHandle
12011+from allmydata.mutable.publish import MutableData
12012 
12013 class FakeClient:
12014     def get_storage_broker(self):
12015hunk ./src/allmydata/test/test_checker.py 292
12016             self.imm = c0.create_node_from_uri(ur.uri)
12017         d.addCallback(_stash_immutable)
12018         d.addCallback(lambda ign:
12019-            c0.create_mutable_file(MutableDataHandle("contents")))
12020+            c0.create_mutable_file(MutableData("contents")))
12021         def _stash_mutable(node):
12022             self.mut = node
12023         d.addCallback(_stash_mutable)
12024hunk ./src/allmydata/test/test_cli.py 12
12025 from allmydata.util import fileutil, hashutil, base32
12026 from allmydata import uri
12027 from allmydata.immutable import upload
12028-from allmydata.mutable.publish import MutableDataHandle
12029+from allmydata.mutable.publish import MutableData
12030 from allmydata.dirnode import normalize
12031 
12032 # Test that the scripts can be imported -- although the actual tests of their
12033hunk ./src/allmydata/test/test_cli.py 1975
12034         self.set_up_grid()
12035         c0 = self.g.clients[0]
12036         DATA = "data" * 100
12037-        DATA_uploadable = MutableDataHandle(DATA)
12038+        DATA_uploadable = MutableData(DATA)
12039         d = c0.create_mutable_file(DATA_uploadable)
12040         def _stash_uri(n):
12041             self.uri = n.get_uri()
12042hunk ./src/allmydata/test/test_cli.py 2078
12043                                                         convergence="")))
12044         d.addCallback(_stash_uri, "small")
12045         d.addCallback(lambda ign:
12046-            c0.create_mutable_file(MutableDataHandle(DATA+"1")))
12047+            c0.create_mutable_file(MutableData(DATA+"1")))
12048         d.addCallback(lambda fn: self.rootnode.set_node(u"mutable", fn))
12049         d.addCallback(_stash_uri, "mutable")
12050 
12051hunk ./src/allmydata/test/test_deepcheck.py 9
12052 from twisted.internet import threads # CLI tests use deferToThread
12053 from allmydata.immutable import upload
12054 from allmydata.mutable.common import UnrecoverableFileError
12055-from allmydata.mutable.publish import MutableDataHandle
12056+from allmydata.mutable.publish import MutableData
12057 from allmydata.util import idlib
12058 from allmydata.util import base32
12059 from allmydata.scripts import runner
12060hunk ./src/allmydata/test/test_deepcheck.py 38
12061         self.basedir = "deepcheck/MutableChecker/good"
12062         self.set_up_grid()
12063         CONTENTS = "a little bit of data"
12064-        CONTENTS_uploadable = MutableDataHandle(CONTENTS)
12065+        CONTENTS_uploadable = MutableData(CONTENTS)
12066         d = self.g.clients[0].create_mutable_file(CONTENTS_uploadable)
12067         def _created(node):
12068             self.node = node
12069hunk ./src/allmydata/test/test_deepcheck.py 61
12070         self.basedir = "deepcheck/MutableChecker/corrupt"
12071         self.set_up_grid()
12072         CONTENTS = "a little bit of data"
12073-        CONTENTS_uploadable = MutableDataHandle(CONTENTS)
12074+        CONTENTS_uploadable = MutableData(CONTENTS)
12075         d = self.g.clients[0].create_mutable_file(CONTENTS_uploadable)
12076         def _stash_and_corrupt(node):
12077             self.node = node
12078hunk ./src/allmydata/test/test_deepcheck.py 99
12079         self.basedir = "deepcheck/MutableChecker/delete_share"
12080         self.set_up_grid()
12081         CONTENTS = "a little bit of data"
12082-        CONTENTS_uploadable = MutableDataHandle(CONTENTS)
12083+        CONTENTS_uploadable = MutableData(CONTENTS)
12084         d = self.g.clients[0].create_mutable_file(CONTENTS_uploadable)
12085         def _stash_and_delete(node):
12086             self.node = node
12087hunk ./src/allmydata/test/test_deepcheck.py 224
12088             self.root_uri = n.get_uri()
12089         d.addCallback(_created_root)
12090         d.addCallback(lambda ign:
12091-            c0.create_mutable_file(MutableDataHandle("mutable file contents")))
12092+            c0.create_mutable_file(MutableData("mutable file contents")))
12093         d.addCallback(lambda n: self.root.set_node(u"mutable", n))
12094         def _created_mutable(n):
12095             self.mutable = n
12096hunk ./src/allmydata/test/test_deepcheck.py 965
12097     def create_mangled(self, ignored, name):
12098         nodetype, mangletype = name.split("-", 1)
12099         if nodetype == "mutable":
12100-            mutable_uploadable = MutableDataHandle("mutable file contents")
12101+            mutable_uploadable = MutableData("mutable file contents")
12102             d = self.g.clients[0].create_mutable_file(mutable_uploadable)
12103             d.addCallback(lambda n: self.root.set_node(unicode(name), n))
12104         elif nodetype == "large":
12105hunk ./src/allmydata/test/test_hung_server.py 10
12106 from allmydata.util.consumer import download_to_data
12107 from allmydata.immutable import upload
12108 from allmydata.mutable.common import UnrecoverableFileError
12109-from allmydata.mutable.publish import MutableDataHandle
12110+from allmydata.mutable.publish import MutableData
12111 from allmydata.storage.common import storage_index_to_dir
12112 from allmydata.test.no_network import GridTestMixin
12113 from allmydata.test.common import ShouldFailMixin, _corrupt_share_data
12114hunk ./src/allmydata/test/test_hung_server.py 96
12115         self.servers = [(id, ss) for (id, ss) in nm.storage_broker.get_all_servers()]
12116 
12117         if mutable:
12118-            uploadable = MutableDataHandle(mutable_plaintext)
12119+            uploadable = MutableData(mutable_plaintext)
12120             d = nm.create_mutable_file(uploadable)
12121             def _uploaded_mutable(node):
12122                 self.uri = node.get_uri()
12123hunk ./src/allmydata/test/test_mutable.py 27
12124      NotEnoughServersError, CorruptShareError
12125 from allmydata.mutable.retrieve import Retrieve
12126 from allmydata.mutable.publish import Publish, MutableFileHandle, \
12127-                                      MutableDataHandle
12128+                                      MutableData
12129 from allmydata.mutable.servermap import ServerMap, ServermapUpdater
12130 from allmydata.mutable.layout import unpack_header, unpack_share, \
12131                                      MDMFSlotReadProxy
12132hunk ./src/allmydata/test/test_mutable.py 297
12133             d.addCallback(lambda smap: smap.dump(StringIO()))
12134             d.addCallback(lambda sio:
12135                           self.failUnless("3-of-10" in sio.getvalue()))
12136-            d.addCallback(lambda res: n.overwrite(MutableDataHandle("contents 1")))
12137+            d.addCallback(lambda res: n.overwrite(MutableData("contents 1")))
12138             d.addCallback(lambda res: self.failUnlessIdentical(res, None))
12139             d.addCallback(lambda res: n.download_best_version())
12140             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 1"))
12141hunk ./src/allmydata/test/test_mutable.py 304
12142             d.addCallback(lambda res: n.get_size_of_best_version())
12143             d.addCallback(lambda size:
12144                           self.failUnlessEqual(size, len("contents 1")))
12145-            d.addCallback(lambda res: n.overwrite(MutableDataHandle("contents 2")))
12146+            d.addCallback(lambda res: n.overwrite(MutableData("contents 2")))
12147             d.addCallback(lambda res: n.download_best_version())
12148             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 2"))
12149             d.addCallback(lambda res: n.get_servermap(MODE_WRITE))
12150hunk ./src/allmydata/test/test_mutable.py 308
12151-            d.addCallback(lambda smap: n.upload(MutableDataHandle("contents 3"), smap))
12152+            d.addCallback(lambda smap: n.upload(MutableData("contents 3"), smap))
12153             d.addCallback(lambda res: n.download_best_version())
12154             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 3"))
12155             d.addCallback(lambda res: n.get_servermap(MODE_ANYTHING))
12156hunk ./src/allmydata/test/test_mutable.py 320
12157             # mapupdate-to-retrieve data caching (i.e. make the shares larger
12158             # than the default readsize, which is 2000 bytes). A 15kB file
12159             # will have 5kB shares.
12160-            d.addCallback(lambda res: n.overwrite(MutableDataHandle("large size file" * 1000)))
12161+            d.addCallback(lambda res: n.overwrite(MutableData("large size file" * 1000)))
12162             d.addCallback(lambda res: n.download_best_version())
12163             d.addCallback(lambda res:
12164                           self.failUnlessEqual(res, "large size file" * 1000))
12165hunk ./src/allmydata/test/test_mutable.py 343
12166             # to make them big enough to force the file to be uploaded
12167             # in more than one segment.
12168             big_contents = "contents1" * 100000 # about 900 KiB
12169-            big_contents_uploadable = MutableDataHandle(big_contents)
12170+            big_contents_uploadable = MutableData(big_contents)
12171             d.addCallback(lambda ignored:
12172                 n.overwrite(big_contents_uploadable))
12173             d.addCallback(lambda ignored:
12174hunk ./src/allmydata/test/test_mutable.py 355
12175             # segments, so that we make the downloader deal with
12176             # multiple segments.
12177             bigger_contents = "contents2" * 1000000 # about 9MiB
12178-            bigger_contents_uploadable = MutableDataHandle(bigger_contents)
12179+            bigger_contents_uploadable = MutableData(bigger_contents)
12180             d.addCallback(lambda ignored:
12181                 n.overwrite(bigger_contents_uploadable))
12182             d.addCallback(lambda ignored:
12183hunk ./src/allmydata/test/test_mutable.py 368
12184 
12185 
12186     def test_create_with_initial_contents(self):
12187-        upload1 = MutableDataHandle("contents 1")
12188+        upload1 = MutableData("contents 1")
12189         d = self.nodemaker.create_mutable_file(upload1)
12190         def _created(n):
12191             d = n.download_best_version()
12192hunk ./src/allmydata/test/test_mutable.py 373
12193             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 1"))
12194-            upload2 = MutableDataHandle("contents 2")
12195+            upload2 = MutableData("contents 2")
12196             d.addCallback(lambda res: n.overwrite(upload2))
12197             d.addCallback(lambda res: n.download_best_version())
12198             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 2"))
12199hunk ./src/allmydata/test/test_mutable.py 385
12200 
12201     def test_create_mdmf_with_initial_contents(self):
12202         initial_contents = "foobarbaz" * 131072 # 900KiB
12203-        initial_contents_uploadable = MutableDataHandle(initial_contents)
12204+        initial_contents_uploadable = MutableData(initial_contents)
12205         d = self.nodemaker.create_mutable_file(initial_contents_uploadable,
12206                                                version=MDMF_VERSION)
12207         def _created(n):
12208hunk ./src/allmydata/test/test_mutable.py 392
12209             d = n.download_best_version()
12210             d.addCallback(lambda data:
12211                 self.failUnlessEqual(data, initial_contents))
12212-            uploadable2 = MutableDataHandle(initial_contents + "foobarbaz")
12213+            uploadable2 = MutableData(initial_contents + "foobarbaz")
12214             d.addCallback(lambda ignored:
12215                 n.overwrite(uploadable2))
12216             d.addCallback(lambda ignored:
12217hunk ./src/allmydata/test/test_mutable.py 413
12218             key = n.get_writekey()
12219             self.failUnless(isinstance(key, str), key)
12220             self.failUnlessEqual(len(key), 16) # AES key size
12221-            return MutableDataHandle(data)
12222+            return MutableData(data)
12223         d = self.nodemaker.create_mutable_file(_make_contents)
12224         def _created(n):
12225             return n.download_best_version()
12226hunk ./src/allmydata/test/test_mutable.py 429
12227             key = n.get_writekey()
12228             self.failUnless(isinstance(key, str), key)
12229             self.failUnlessEqual(len(key), 16)
12230-            return MutableDataHandle(data)
12231+            return MutableData(data)
12232         d = self.nodemaker.create_mutable_file(_make_contents,
12233                                                version=MDMF_VERSION)
12234         d.addCallback(lambda n:
12235hunk ./src/allmydata/test/test_mutable.py 441
12236 
12237     def test_create_with_too_large_contents(self):
12238         BIG = "a" * (self.OLD_MAX_SEGMENT_SIZE + 1)
12239-        BIG_uploadable = MutableDataHandle(BIG)
12240+        BIG_uploadable = MutableData(BIG)
12241         d = self.nodemaker.create_mutable_file(BIG_uploadable)
12242         def _created(n):
12243hunk ./src/allmydata/test/test_mutable.py 444
12244-            other_BIG_uploadable = MutableDataHandle(BIG)
12245+            other_BIG_uploadable = MutableData(BIG)
12246             d = n.overwrite(other_BIG_uploadable)
12247             return d
12248         d.addCallback(_created)
12249hunk ./src/allmydata/test/test_mutable.py 460
12250     def test_modify(self):
12251         def _modifier(old_contents, servermap, first_time):
12252             new_contents = old_contents + "line2"
12253-            return MutableDataHandle(new_contents)
12254+            return MutableData(new_contents)
12255         def _non_modifier(old_contents, servermap, first_time):
12256hunk ./src/allmydata/test/test_mutable.py 462
12257-            return MutableDataHandle(old_contents)
12258+            return MutableData(old_contents)
12259         def _none_modifier(old_contents, servermap, first_time):
12260             return None
12261         def _error_modifier(old_contents, servermap, first_time):
12262hunk ./src/allmydata/test/test_mutable.py 469
12263             raise ValueError("oops")
12264         def _toobig_modifier(old_contents, servermap, first_time):
12265             new_content = "b" * (self.OLD_MAX_SEGMENT_SIZE + 1)
12266-            return MutableDataHandle(new_content)
12267+            return MutableData(new_content)
12268         calls = []
12269         def _ucw_error_modifier(old_contents, servermap, first_time):
12270             # simulate an UncoordinatedWriteError once
12271hunk ./src/allmydata/test/test_mutable.py 477
12272             if len(calls) <= 1:
12273                 raise UncoordinatedWriteError("simulated")
12274             new_contents = old_contents + "line3"
12275-            return MutableDataHandle(new_contents)
12276+            return MutableData(new_contents)
12277         def _ucw_error_non_modifier(old_contents, servermap, first_time):
12278             # simulate an UncoordinatedWriteError once, and don't actually
12279             # modify the contents on subsequent invocations
12280hunk ./src/allmydata/test/test_mutable.py 484
12281             calls.append(1)
12282             if len(calls) <= 1:
12283                 raise UncoordinatedWriteError("simulated")
12284-            return MutableDataHandle(old_contents)
12285+            return MutableData(old_contents)
12286 
12287         initial_contents = "line1"
12288hunk ./src/allmydata/test/test_mutable.py 487
12289-        d = self.nodemaker.create_mutable_file(MutableDataHandle(initial_contents))
12290+        d = self.nodemaker.create_mutable_file(MutableData(initial_contents))
12291         def _created(n):
12292             d = n.modify(_modifier)
12293             d.addCallback(lambda res: n.download_best_version())
12294hunk ./src/allmydata/test/test_mutable.py 548
12295 
12296     def test_modify_backoffer(self):
12297         def _modifier(old_contents, servermap, first_time):
12298-            return MutableDataHandle(old_contents + "line2")
12299+            return MutableData(old_contents + "line2")
12300         calls = []
12301         def _ucw_error_modifier(old_contents, servermap, first_time):
12302             # simulate an UncoordinatedWriteError once
12303hunk ./src/allmydata/test/test_mutable.py 555
12304             calls.append(1)
12305             if len(calls) <= 1:
12306                 raise UncoordinatedWriteError("simulated")
12307-            return MutableDataHandle(old_contents + "line3")
12308+            return MutableData(old_contents + "line3")
12309         def _always_ucw_error_modifier(old_contents, servermap, first_time):
12310             raise UncoordinatedWriteError("simulated")
12311         def _backoff_stopper(node, f):
12312hunk ./src/allmydata/test/test_mutable.py 570
12313         giveuper._delay = 0.1
12314         giveuper.factor = 1
12315 
12316-        d = self.nodemaker.create_mutable_file(MutableDataHandle("line1"))
12317+        d = self.nodemaker.create_mutable_file(MutableData("line1"))
12318         def _created(n):
12319             d = n.modify(_modifier)
12320             d.addCallback(lambda res: n.download_best_version())
12321hunk ./src/allmydata/test/test_mutable.py 620
12322             d.addCallback(lambda smap: smap.dump(StringIO()))
12323             d.addCallback(lambda sio:
12324                           self.failUnless("3-of-10" in sio.getvalue()))
12325-            d.addCallback(lambda res: n.overwrite(MutableDataHandle("contents 1")))
12326+            d.addCallback(lambda res: n.overwrite(MutableData("contents 1")))
12327             d.addCallback(lambda res: self.failUnlessIdentical(res, None))
12328             d.addCallback(lambda res: n.download_best_version())
12329             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 1"))
12330hunk ./src/allmydata/test/test_mutable.py 624
12331-            d.addCallback(lambda res: n.overwrite(MutableDataHandle("contents 2")))
12332+            d.addCallback(lambda res: n.overwrite(MutableData("contents 2")))
12333             d.addCallback(lambda res: n.download_best_version())
12334             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 2"))
12335             d.addCallback(lambda res: n.get_servermap(MODE_WRITE))
12336hunk ./src/allmydata/test/test_mutable.py 628
12337-            d.addCallback(lambda smap: n.upload(MutableDataHandle("contents 3"), smap))
12338+            d.addCallback(lambda smap: n.upload(MutableData("contents 3"), smap))
12339             d.addCallback(lambda res: n.download_best_version())
12340             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 3"))
12341             d.addCallback(lambda res: n.get_servermap(MODE_ANYTHING))
12342hunk ./src/allmydata/test/test_mutable.py 646
12343         # publish a file and create shares, which can then be manipulated
12344         # later.
12345         self.CONTENTS = "New contents go here" * 1000
12346-        self.uploadable = MutableDataHandle(self.CONTENTS)
12347+        self.uploadable = MutableData(self.CONTENTS)
12348         self._storage = FakeStorage()
12349         self._nodemaker = make_nodemaker(self._storage)
12350         self._storage_broker = self._nodemaker.storage_broker
12351hunk ./src/allmydata/test/test_mutable.py 662
12352         # an MDMF file.
12353         # self.CONTENTS should have more than one segment.
12354         self.CONTENTS = "This is an MDMF file" * 100000
12355-        self.uploadable = MutableDataHandle(self.CONTENTS)
12356+        self.uploadable = MutableData(self.CONTENTS)
12357         self._storage = FakeStorage()
12358         self._nodemaker = make_nodemaker(self._storage)
12359         self._storage_broker = self._nodemaker.storage_broker
12360hunk ./src/allmydata/test/test_mutable.py 678
12361         # like publish_one, except that the result is guaranteed to be
12362         # an SDMF file
12363         self.CONTENTS = "This is an SDMF file" * 1000
12364-        self.uploadable = MutableDataHandle(self.CONTENTS)
12365+        self.uploadable = MutableData(self.CONTENTS)
12366         self._storage = FakeStorage()
12367         self._nodemaker = make_nodemaker(self._storage)
12368         self._storage_broker = self._nodemaker.storage_broker
12369hunk ./src/allmydata/test/test_mutable.py 696
12370                          "Contents 2",
12371                          "Contents 3a",
12372                          "Contents 3b"]
12373-        self.uploadables = [MutableDataHandle(d) for d in self.CONTENTS]
12374+        self.uploadables = [MutableData(d) for d in self.CONTENTS]
12375         self._copied_shares = {}
12376         self._storage = FakeStorage()
12377         self._nodemaker = make_nodemaker(self._storage)
12378hunk ./src/allmydata/test/test_mutable.py 826
12379         # create a new file, which is large enough to knock the privkey out
12380         # of the early part of the file
12381         LARGE = "These are Larger contents" * 200 # about 5KB
12382-        LARGE_uploadable = MutableDataHandle(LARGE)
12383+        LARGE_uploadable = MutableData(LARGE)
12384         d.addCallback(lambda res: self._nodemaker.create_mutable_file(LARGE_uploadable))
12385         def _created(large_fn):
12386             large_fn2 = self._nodemaker.create_from_cap(large_fn.get_uri())
12387hunk ./src/allmydata/test/test_mutable.py 1842
12388 class MultipleEncodings(unittest.TestCase):
12389     def setUp(self):
12390         self.CONTENTS = "New contents go here"
12391-        self.uploadable = MutableDataHandle(self.CONTENTS)
12392+        self.uploadable = MutableData(self.CONTENTS)
12393         self._storage = FakeStorage()
12394         self._nodemaker = make_nodemaker(self._storage, num_peers=20)
12395         self._storage_broker = self._nodemaker.storage_broker
12396hunk ./src/allmydata/test/test_mutable.py 1872
12397         s = self._storage
12398         s._peers = {} # clear existing storage
12399         p2 = Publish(fn2, self._storage_broker, None)
12400-        uploadable = MutableDataHandle(data)
12401+        uploadable = MutableData(data)
12402         d = p2.publish(uploadable)
12403         def _published(res):
12404             shares = s._peers
12405hunk ./src/allmydata/test/test_mutable.py 2049
12406         self._set_versions(target)
12407 
12408         def _modify(oldversion, servermap, first_time):
12409-            return MutableDataHandle(oldversion + " modified")
12410+            return MutableData(oldversion + " modified")
12411         d = self._fn.modify(_modify)
12412         d.addCallback(lambda res: self._fn.download_best_version())
12413         expected = self.CONTENTS[2] + " modified"
12414hunk ./src/allmydata/test/test_mutable.py 2175
12415         self.basedir = "mutable/Problems/test_publish_surprise"
12416         self.set_up_grid()
12417         nm = self.g.clients[0].nodemaker
12418-        d = nm.create_mutable_file(MutableDataHandle("contents 1"))
12419+        d = nm.create_mutable_file(MutableData("contents 1"))
12420         def _created(n):
12421             d = defer.succeed(None)
12422             d.addCallback(lambda res: n.get_servermap(MODE_WRITE))
12423hunk ./src/allmydata/test/test_mutable.py 2185
12424             d.addCallback(_got_smap1)
12425             # then modify the file, leaving the old map untouched
12426             d.addCallback(lambda res: log.msg("starting winning write"))
12427-            d.addCallback(lambda res: n.overwrite(MutableDataHandle("contents 2")))
12428+            d.addCallback(lambda res: n.overwrite(MutableData("contents 2")))
12429             # now attempt to modify the file with the old servermap. This
12430             # will look just like an uncoordinated write, in which every
12431             # single share got updated between our mapupdate and our publish
12432hunk ./src/allmydata/test/test_mutable.py 2194
12433                           self.shouldFail(UncoordinatedWriteError,
12434                                           "test_publish_surprise", None,
12435                                           n.upload,
12436-                                          MutableDataHandle("contents 2a"), self.old_map))
12437+                                          MutableData("contents 2a"), self.old_map))
12438             return d
12439         d.addCallback(_created)
12440         return d
12441hunk ./src/allmydata/test/test_mutable.py 2203
12442         self.basedir = "mutable/Problems/test_retrieve_surprise"
12443         self.set_up_grid()
12444         nm = self.g.clients[0].nodemaker
12445-        d = nm.create_mutable_file(MutableDataHandle("contents 1"))
12446+        d = nm.create_mutable_file(MutableData("contents 1"))
12447         def _created(n):
12448             d = defer.succeed(None)
12449             d.addCallback(lambda res: n.get_servermap(MODE_READ))
12450hunk ./src/allmydata/test/test_mutable.py 2213
12451             d.addCallback(_got_smap1)
12452             # then modify the file, leaving the old map untouched
12453             d.addCallback(lambda res: log.msg("starting winning write"))
12454-            d.addCallback(lambda res: n.overwrite(MutableDataHandle("contents 2")))
12455+            d.addCallback(lambda res: n.overwrite(MutableData("contents 2")))
12456             # now attempt to retrieve the old version with the old servermap.
12457             # This will look like someone has changed the file since we
12458             # updated the servermap.
12459hunk ./src/allmydata/test/test_mutable.py 2241
12460         self.basedir = "mutable/Problems/test_unexpected_shares"
12461         self.set_up_grid()
12462         nm = self.g.clients[0].nodemaker
12463-        d = nm.create_mutable_file(MutableDataHandle("contents 1"))
12464+        d = nm.create_mutable_file(MutableData("contents 1"))
12465         def _created(n):
12466             d = defer.succeed(None)
12467             d.addCallback(lambda res: n.get_servermap(MODE_WRITE))
12468hunk ./src/allmydata/test/test_mutable.py 2253
12469                 self.g.remove_server(peer0)
12470                 # then modify the file, leaving the old map untouched
12471                 log.msg("starting winning write")
12472-                return n.overwrite(MutableDataHandle("contents 2"))
12473+                return n.overwrite(MutableData("contents 2"))
12474             d.addCallback(_got_smap1)
12475             # now attempt to modify the file with the old servermap. This
12476             # will look just like an uncoordinated write, in which every
12477hunk ./src/allmydata/test/test_mutable.py 2263
12478                           self.shouldFail(UncoordinatedWriteError,
12479                                           "test_surprise", None,
12480                                           n.upload,
12481-                                          MutableDataHandle("contents 2a"), self.old_map))
12482+                                          MutableData("contents 2a"), self.old_map))
12483             return d
12484         d.addCallback(_created)
12485         return d
12486hunk ./src/allmydata/test/test_mutable.py 2303
12487         d.addCallback(_break_peer0)
12488         # now "create" the file, using the pre-established key, and let the
12489         # initial publish finally happen
12490-        d.addCallback(lambda res: nm.create_mutable_file(MutableDataHandle("contents 1")))
12491+        d.addCallback(lambda res: nm.create_mutable_file(MutableData("contents 1")))
12492         # that ought to work
12493         def _got_node(n):
12494             d = n.download_best_version()
12495hunk ./src/allmydata/test/test_mutable.py 2312
12496             def _break_peer1(res):
12497                 self.connection1.broken = True
12498             d.addCallback(_break_peer1)
12499-            d.addCallback(lambda res: n.overwrite(MutableDataHandle("contents 2")))
12500+            d.addCallback(lambda res: n.overwrite(MutableData("contents 2")))
12501             # that ought to work too
12502             d.addCallback(lambda res: n.download_best_version())
12503             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 2"))
12504hunk ./src/allmydata/test/test_mutable.py 2344
12505         peerids = [serverid for (serverid,ss) in sb.get_all_servers()]
12506         self.g.break_server(peerids[0])
12507 
12508-        d = nm.create_mutable_file(MutableDataHandle("contents 1"))
12509+        d = nm.create_mutable_file(MutableData("contents 1"))
12510         def _created(n):
12511             d = n.download_best_version()
12512             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 1"))
12513hunk ./src/allmydata/test/test_mutable.py 2352
12514             def _break_second_server(res):
12515                 self.g.break_server(peerids[1])
12516             d.addCallback(_break_second_server)
12517-            d.addCallback(lambda res: n.overwrite(MutableDataHandle("contents 2")))
12518+            d.addCallback(lambda res: n.overwrite(MutableData("contents 2")))
12519             # that ought to work too
12520             d.addCallback(lambda res: n.download_best_version())
12521             d.addCallback(lambda res: self.failUnlessEqual(res, "contents 2"))
12522hunk ./src/allmydata/test/test_mutable.py 2371
12523         d = self.shouldFail(NotEnoughServersError,
12524                             "test_publish_all_servers_bad",
12525                             "Ran out of non-bad servers",
12526-                            nm.create_mutable_file, MutableDataHandle("contents"))
12527+                            nm.create_mutable_file, MutableData("contents"))
12528         return d
12529 
12530     def test_publish_no_servers(self):
12531hunk ./src/allmydata/test/test_mutable.py 2383
12532         d = self.shouldFail(NotEnoughServersError,
12533                             "test_publish_no_servers",
12534                             "Ran out of non-bad servers",
12535-                            nm.create_mutable_file, MutableDataHandle("contents"))
12536+                            nm.create_mutable_file, MutableData("contents"))
12537         return d
12538     test_publish_no_servers.timeout = 30
12539 
12540hunk ./src/allmydata/test/test_mutable.py 2401
12541         # we need some contents that are large enough to push the privkey out
12542         # of the early part of the file
12543         LARGE = "These are Larger contents" * 2000 # about 50KB
12544-        LARGE_uploadable = MutableDataHandle(LARGE)
12545+        LARGE_uploadable = MutableData(LARGE)
12546         d = nm.create_mutable_file(LARGE_uploadable)
12547         def _created(n):
12548             self.uri = n.get_uri()
12549hunk ./src/allmydata/test/test_mutable.py 2438
12550         self.set_up_grid(num_servers=20)
12551         nm = self.g.clients[0].nodemaker
12552         LARGE = "These are Larger contents" * 2000 # about 50KiB
12553-        LARGE_uploadable = MutableDataHandle(LARGE)
12554+        LARGE_uploadable = MutableData(LARGE)
12555         nm._node_cache = DevNullDictionary() # disable the nodecache
12556 
12557         d = nm.create_mutable_file(LARGE_uploadable)
12558hunk ./src/allmydata/test/test_mutable.py 2464
12559         self.set_up_grid(num_servers=20)
12560         nm = self.g.clients[0].nodemaker
12561         CONTENTS = "contents" * 2000
12562-        CONTENTS_uploadable = MutableDataHandle(CONTENTS)
12563+        CONTENTS_uploadable = MutableData(CONTENTS)
12564         d = nm.create_mutable_file(CONTENTS_uploadable)
12565         def _created(node):
12566             self._node = node
12567hunk ./src/allmydata/test/test_mutable.py 2565
12568 class DataHandle(unittest.TestCase):
12569     def setUp(self):
12570         self.test_data = "Test Data" * 50000
12571-        self.uploadable = MutableDataHandle(self.test_data)
12572+        self.uploadable = MutableData(self.test_data)
12573 
12574 
12575     def test_datahandle_read(self):
12576hunk ./src/allmydata/test/test_sftp.py 84
12577         return d
12578 
12579     def _set_up_tree(self):
12580-        u = publish.MutableDataHandle("mutable file contents")
12581+        u = publish.MutableData("mutable file contents")
12582         d = self.client.create_mutable_file(u)
12583         d.addCallback(lambda node: self.root.set_node(u"mutable", node))
12584         def _created_mutable(n):
12585hunk ./src/allmydata/test/test_system.py 22
12586 from allmydata.monitor import Monitor
12587 from allmydata.mutable.common import NotWriteableError
12588 from allmydata.mutable import layout as mutable_layout
12589-from allmydata.mutable.publish import MutableDataHandle
12590+from allmydata.mutable.publish import MutableData
12591 from foolscap.api import DeadReferenceError
12592 from twisted.python.failure import Failure
12593 from twisted.web.client import getPage
12594hunk ./src/allmydata/test/test_system.py 460
12595     def test_mutable(self):
12596         self.basedir = "system/SystemTest/test_mutable"
12597         DATA = "initial contents go here."  # 25 bytes % 3 != 0
12598-        DATA_uploadable = MutableDataHandle(DATA)
12599+        DATA_uploadable = MutableData(DATA)
12600         NEWDATA = "new contents yay"
12601hunk ./src/allmydata/test/test_system.py 462
12602-        NEWDATA_uploadable = MutableDataHandle(NEWDATA)
12603+        NEWDATA_uploadable = MutableData(NEWDATA)
12604         NEWERDATA = "this is getting old"
12605hunk ./src/allmydata/test/test_system.py 464
12606-        NEWERDATA_uploadable = MutableDataHandle(NEWERDATA)
12607+        NEWERDATA_uploadable = MutableData(NEWERDATA)
12608 
12609         d = self.set_up_nodes(use_key_generator=True)
12610 
12611hunk ./src/allmydata/test/test_system.py 642
12612         def _check_empty_file(res):
12613             # make sure we can create empty files, this usually screws up the
12614             # segsize math
12615-            d1 = self.clients[2].create_mutable_file(MutableDataHandle(""))
12616+            d1 = self.clients[2].create_mutable_file(MutableData(""))
12617             d1.addCallback(lambda newnode: newnode.download_best_version())
12618             d1.addCallback(lambda res: self.failUnlessEqual("", res))
12619             return d1
12620hunk ./src/allmydata/test/test_system.py 674
12621 
12622         d.addCallback(check_kg_poolsize, 0)
12623         d.addCallback(lambda junk:
12624-            self.clients[3].create_mutable_file(MutableDataHandle('hello, world')))
12625+            self.clients[3].create_mutable_file(MutableData('hello, world')))
12626         d.addCallback(check_kg_poolsize, -1)
12627         d.addCallback(lambda junk: self.clients[3].create_dirnode())
12628         d.addCallback(check_kg_poolsize, -2)
12629hunk ./src/allmydata/test/test_web.py 3184
12630             self.uris[which] = n.get_uri()
12631             assert isinstance(self.uris[which], str)
12632         d.addCallback(lambda ign:
12633-            c0.create_mutable_file(publish.MutableDataHandle(DATA+"3")))
12634+            c0.create_mutable_file(publish.MutableData(DATA+"3")))
12635         d.addCallback(_stash_mutable_uri, "corrupt")
12636         d.addCallback(lambda ign:
12637                       c0.upload(upload.Data("literal", convergence="")))
12638hunk ./src/allmydata/test/test_web.py 3331
12639             self.uris[which] = n.get_uri()
12640             assert isinstance(self.uris[which], str)
12641         d.addCallback(lambda ign:
12642-            c0.create_mutable_file(publish.MutableDataHandle(DATA+"3")))
12643+            c0.create_mutable_file(publish.MutableData(DATA+"3")))
12644         d.addCallback(_stash_mutable_uri, "corrupt")
12645 
12646         def _compute_fileurls(ignored):
12647hunk ./src/allmydata/test/test_web.py 3994
12648             self.uris[which] = n.get_uri()
12649             assert isinstance(self.uris[which], str)
12650         d.addCallback(lambda ign:
12651-            c0.create_mutable_file(publish.MutableDataHandle(DATA+"2")))
12652+            c0.create_mutable_file(publish.MutableData(DATA+"2")))
12653         d.addCallback(_stash_mutable_uri, "mutable")
12654 
12655         def _compute_fileurls(ignored):
12656hunk ./src/allmydata/test/test_web.py 4094
12657         d.addCallback(_stash_uri, "small")
12658 
12659         d.addCallback(lambda ign:
12660-            c0.create_mutable_file(publish.MutableDataHandle("mutable")))
12661+            c0.create_mutable_file(publish.MutableData("mutable")))
12662         d.addCallback(lambda fn: self.rootnode.set_node(u"mutable", fn))
12663         d.addCallback(_stash_uri, "mutable")
12664 
12665}
12666[tests: fix tests that were broken by #993
12667Kevan Carstensen <kevan@isnotajoke.com>**20100717015230
12668 Ignore-this: f0ace6538c6d824b7fe271c40a7ebf8d
12669] {
12670hunk ./src/allmydata/test/common.py 152
12671         consumer.write(data[start:end])
12672         return consumer
12673 
12674+
12675+    def get_best_readable_version(self):
12676+        return defer.succeed(self)
12677+
12678+
12679+    download_best_version = download_to_data
12680+
12681+
12682+    def download_to_data(self):
12683+        return download_to_data(self)
12684+
12685+
12686+    def get_size_of_best_version(self):
12687+        return defer.succeed(self.get_size)
12688+
12689+
12690 def make_chk_file_cap(size):
12691     return uri.CHKFileURI(key=os.urandom(16),
12692                           uri_extension_hash=os.urandom(32),
12693hunk ./src/allmydata/test/common.py 318
12694         return d
12695 
12696     def download_best_version(self):
12697+        return defer.succeed(self._download_best_version())
12698+
12699+
12700+    def _download_best_version(self, ignored=None):
12701         if isinstance(self.my_uri, uri.LiteralFileURI):
12702hunk ./src/allmydata/test/common.py 323
12703-            return defer.succeed(self.my_uri.data)
12704+            return self.my_uri.data
12705         if self.storage_index not in self.all_contents:
12706hunk ./src/allmydata/test/common.py 325
12707-            return defer.fail(NotEnoughSharesError(None, 0, 3))
12708-        return defer.succeed(self.all_contents[self.storage_index])
12709+            raise NotEnoughSharesError(None, 0, 3)
12710+        return self.all_contents[self.storage_index]
12711+
12712 
12713     def overwrite(self, new_contents):
12714         if new_contents.get_size() > self.MUTABLE_SIZELIMIT:
12715hunk ./src/allmydata/test/common.py 352
12716         self.all_contents[self.storage_index] = new_data
12717         return None
12718 
12719+    # As actually implemented, MutableFilenode and MutableFileVersion
12720+    # are distinct. However, nothing in the webapi uses (yet) that
12721+    # distinction -- it just uses the unified download interface
12722+    # provided by get_best_readable_version and read. When we start
12723+    # doing cooler things like LDMF, we will want to revise this code to
12724+    # be less simplistic.
12725+    def get_best_readable_version(self):
12726+        return defer.succeed(self)
12727+
12728+
12729+    def read(self, consumer, offset=0, size=None):
12730+        data = self._download_best_version()
12731+        if size:
12732+            data = data[offset:offset+size]
12733+        consumer.write(data)
12734+        return defer.succeed(consumer)
12735+
12736+
12737 def make_mutable_file_cap():
12738     return uri.WriteableSSKFileURI(writekey=os.urandom(16),
12739                                    fingerprint=os.urandom(32))
12740hunk ./src/allmydata/test/test_filenode.py 98
12741         def _check_segment(res):
12742             self.failUnlessEqual(res, DATA[1:1+5])
12743         d.addCallback(_check_segment)
12744-        d.addCallback(lambda ignored:
12745-            self.failUnlessEqual(fn1.get_best_readable_version(), fn1))
12746+        d.addCallback(lambda ignored: fn1.get_best_readable_version())
12747+        d.addCallback(lambda fn2: self.failUnlessEqual(fn1, fn2))
12748         d.addCallback(lambda ignored:
12749             fn1.get_size_of_best_version())
12750         d.addCallback(lambda size:
12751hunk ./src/allmydata/test/test_immutable.py 168
12752 
12753 
12754     def test_get_best_readable_version(self):
12755-        n = self.n.get_best_readable_version()
12756-        self.failUnlessEqual(n, self.n)
12757+        d = self.n.get_best_readable_version()
12758+        d.addCallback(lambda n2:
12759+            self.failUnlessEqual(n2, self.n))
12760+        return d
12761 
12762     def test_get_size_of_best_version(self):
12763         d = self.n.get_size_of_best_version()
12764hunk ./src/allmydata/test/test_mutable.py 8
12765 from twisted.internet import defer, reactor
12766 from allmydata import uri, client
12767 from allmydata.nodemaker import NodeMaker
12768-from allmydata.util import base32
12769+from allmydata.util import base32, consumer
12770 from allmydata.util.hashutil import tagged_hash, ssk_writekey_hash, \
12771      ssk_pubkey_fingerprint_hash
12772hunk ./src/allmydata/test/test_mutable.py 11
12773+from allmydata.util.deferredutil import gatherResults
12774 from allmydata.interfaces import IRepairResults, ICheckAndRepairResults, \
12775      NotEnoughSharesError, SDMF_VERSION, MDMF_VERSION
12776 from allmydata.monitor import Monitor
12777hunk ./src/allmydata/test/test_mutable.py 1000
12778         if version is None:
12779             version = servermap.best_recoverable_version()
12780         r = Retrieve(self._fn, servermap, version)
12781-        return r.download()
12782+        c = consumer.MemoryConsumer()
12783+        d = r.download(consumer=c)
12784+        d.addCallback(lambda mc: "".join(mc.chunks))
12785+        return d
12786+
12787 
12788     def test_basic(self):
12789         d = self.make_servermap()
12790hunk ./src/allmydata/test/test_mutable.py 1263
12791                             in str(servermap.problems[0]))
12792             ver = servermap.best_recoverable_version()
12793             r = Retrieve(self._fn, servermap, ver)
12794-            return r.download()
12795+            c = consumer.MemoryConsumer()
12796+            return r.download(c)
12797         d.addCallback(_do_retrieve)
12798hunk ./src/allmydata/test/test_mutable.py 1266
12799+        d.addCallback(lambda mc: "".join(mc.chunks))
12800         d.addCallback(lambda new_contents:
12801                       self.failUnlessEqual(new_contents, self.CONTENTS))
12802         return d
12803}
12804[test/test_immutable.py: add tests for #993-related modifications
12805Kevan Carstensen <kevan@isnotajoke.com>**20100717015402
12806 Ignore-this: d94ad98bd0d322ead85af2e0aa95be38
12807] hunk ./src/allmydata/test/test_mutable.py 2607
12808         start = chunk_size
12809         end = chunk_size * 2
12810         self.failUnlessEqual("".join(more_data), self.test_data[start:end])
12811+
12812+
12813+class Version(GridTestMixin, unittest.TestCase, testutil.ShouldFailMixin):
12814+    def setUp(self):
12815+        GridTestMixin.setUp(self)
12816+        self.basedir = self.mktemp()
12817+        self.set_up_grid()
12818+        self.c = self.g.clients[0]
12819+        self.nm = self.c.nodemaker
12820+        self.data = "test data" * 100000 # about 900 KiB; MDMF
12821+        self.small_data = "test data" * 10 # about 90 B; SDMF
12822+        return self.do_upload()
12823+
12824+
12825+    def do_upload(self):
12826+        d1 = self.nm.create_mutable_file(MutableData(self.data),
12827+                                         version=MDMF_VERSION)
12828+        d2 = self.nm.create_mutable_file(MutableData(self.small_data))
12829+        dl = gatherResults([d1, d2])
12830+        def _then((n1, n2)):
12831+            assert isinstance(n1, MutableFileNode)
12832+            assert isinstance(n2, MutableFileNode)
12833+
12834+            self.mdmf_node = n1
12835+            self.sdmf_node = n2
12836+        dl.addCallback(_then)
12837+        return dl
12838+
12839+
12840+    def test_get_readonly_mutable_version(self):
12841+        # Attempting to get a mutable version of a mutable file from a
12842+        # filenode initialized with a readcap should return a readonly
12843+        # version of that same node.
12844+        ro = self.mdmf_node.get_readonly()
12845+        d = ro.get_best_mutable_version()
12846+        d.addCallback(lambda version:
12847+            self.failUnless(version.is_readonly()))
12848+        d.addCallback(lambda ignored:
12849+            self.sdmf_node.get_readonly())
12850+        d.addCallback(lambda version:
12851+            self.failUnless(version.is_readonly()))
12852+        return d
12853+
12854+
12855+    def test_get_sequence_number(self):
12856+        d = self.mdmf_node.get_best_readable_version()
12857+        d.addCallback(lambda bv:
12858+            self.failUnlessEqual(bv.get_sequence_number(), 0))
12859+        d.addCallback(lambda ignored:
12860+            self.sdmf_node.get_best_readable_version())
12861+        d.addCallback(lambda bv:
12862+            self.failUnlessEqual(bv.get_sequence_number(), 0))
12863+        # Now update. The sequence number in both cases should be 1 in
12864+        # both cases.
12865+        def _do_update(ignored):
12866+            new_data = MutableData("foo bar baz" * 100000)
12867+            new_small_data = MutableData("foo bar baz" * 10)
12868+            d1 = self.mdmf_node.overwrite(new_data)
12869+            d2 = self.sdmf_node.overwrite(new_small_data)
12870+            dl = gatherResults([d1, d2])
12871+            return dl
12872+        d.addCallback(_do_update)
12873+        d.addCallback(lambda ignored:
12874+            self.mdmf_node.get_best_readable_version())
12875+        d.addCallback(lambda bv:
12876+            self.failUnlessEqual(bv.get_sequence_number(), 1))
12877+        d.addCallback(lambda ignored:
12878+            self.sdmf_node.get_best_readable_version())
12879+        d.addCallback(lambda bv:
12880+            self.failUnlessEqual(bv.get_sequence_number(), 1))
12881+        return d
12882+
12883+
12884+    def test_get_writekey(self):
12885+        d = self.mdmf_node.get_best_mutable_version()
12886+        d.addCallback(lambda bv:
12887+            self.failUnlessEqual(bv.get_writekey(),
12888+                                 self.mdmf_node.get_writekey()))
12889+        d.addCallback(lambda ignored:
12890+            self.sdmf_node.get_best_mutable_version())
12891+        d.addCallback(lambda bv:
12892+            self.failUnlessEqual(bv.get_writekey(),
12893+                                 self.sdmf_node.get_writekey()))
12894+        return d
12895+
12896+
12897+    def test_get_storage_index(self):
12898+        d = self.mdmf_node.get_best_mutable_version()
12899+        d.addCallback(lambda bv:
12900+            self.failUnlessEqual(bv.get_storage_index(),
12901+                                 self.mdmf_node.get_storage_index()))
12902+        d.addCallback(lambda ignored:
12903+            self.sdmf_node.get_best_mutable_version())
12904+        d.addCallback(lambda bv:
12905+            self.failUnlessEqual(bv.get_storage_index(),
12906+                                 self.sdmf_node.get_storage_index()))
12907+        return d
12908+
12909+
12910+    def test_get_readonly_version(self):
12911+        d = self.mdmf_node.get_best_readable_version()
12912+        d.addCallback(lambda bv:
12913+            self.failUnless(bv.is_readonly()))
12914+        d.addCallback(lambda ignored:
12915+            self.sdmf_node.get_best_readable_version())
12916+        d.addCallback(lambda bv:
12917+            self.failUnless(bv.is_readonly()))
12918+        return d
12919+
12920+
12921+    def test_get_mutable_version(self):
12922+        d = self.mdmf_node.get_best_mutable_version()
12923+        d.addCallback(lambda bv:
12924+            self.failIf(bv.is_readonly()))
12925+        d.addCallback(lambda ignored:
12926+            self.sdmf_node.get_best_mutable_version())
12927+        d.addCallback(lambda bv:
12928+            self.failIf(bv.is_readonly()))
12929+        return d
12930+
12931+
12932+    def test_toplevel_overwrite(self):
12933+        new_data = MutableData("foo bar baz" * 100000)
12934+        new_small_data = MutableData("foo bar baz" * 10)
12935+        d = self.mdmf_node.overwrite(new_data)
12936+        d.addCallback(lambda ignored:
12937+            self.mdmf_node.download_best_version())
12938+        d.addCallback(lambda data:
12939+            self.failUnlessEqual(data, "foo bar baz" * 100000))
12940+        d.addCallback(lambda ignored:
12941+            self.sdmf_node.overwrite(new_small_data))
12942+        d.addCallback(lambda ignored:
12943+            self.sdmf_node.download_best_version())
12944+        d.addCallback(lambda data:
12945+            self.failUnlessEqual(data, "foo bar baz" * 10))
12946+        return d
12947+
12948+
12949+    def test_toplevel_modify(self):
12950+        def modifier(old_contents, servermap, first_time):
12951+            return MutableData(old_contents + "modified")
12952+        d = self.mdmf_node.modify(modifier)
12953+        d.addCallback(lambda ignored:
12954+            self.mdmf_node.download_best_version())
12955+        d.addCallback(lambda data:
12956+            self.failUnlessIn("modified", data))
12957+        d.addCallback(lambda ignored:
12958+            self.sdmf_node.modify(modifier))
12959+        d.addCallback(lambda ignored:
12960+            self.sdmf_node.download_best_version())
12961+        d.addCallback(lambda data:
12962+            self.failUnlessIn("modified", data))
12963+        return d
12964+
12965+
12966+    def test_version_modify(self):
12967+        # TODO: When we can publish multiple versions, alter this test
12968+        # to modify a version other than the best usable version, then
12969+        # test to see that the best recoverable version is that.
12970+        def modifier(old_contents, servermap, first_time):
12971+            return MutableData(old_contents + "modified")
12972+        d = self.mdmf_node.modify(modifier)
12973+        d.addCallback(lambda ignored:
12974+            self.mdmf_node.download_best_version())
12975+        d.addCallback(lambda data:
12976+            self.failUnlessIn("modified", data))
12977+        d.addCallback(lambda ignored:
12978+            self.sdmf_node.modify(modifier))
12979+        d.addCallback(lambda ignored:
12980+            self.sdmf_node.download_best_version())
12981+        d.addCallback(lambda data:
12982+            self.failUnlessIn("modified", data))
12983+        return d
12984+
12985+
12986+    def test_download_version(self):
12987+        # This will only pass once we get the ability to publish
12988+        # multiple recoverable versions.
12989+        self.failUnless(False)
12990+
12991+
12992+    def test_partial_read(self):
12993+        # read only a few bytes at a time, and see that the results are
12994+        # what we expect.
12995+        d = self.mdmf_node.get_best_readable_version()
12996+        def _read_data(version):
12997+            c = consumer.MemoryConsumer()
12998+            d2 = defer.succeed(None)
12999+            for i in xrange(0, len(self.data), 10000):
13000+                d2.addCallback(lambda ignored, i=i: version.read(c, i, 10000))
13001+            d2.addCallback(lambda ignored:
13002+                self.failUnlessEqual(self.data, "".join(c.chunks)))
13003+            return d2
13004+        d.addCallback(_read_data)
13005+        return d
13006+
13007+
13008+    def test_read(self):
13009+        d = self.mdmf_node.get_best_readable_version()
13010+        def _read_data(version):
13011+            c = consumer.MemoryConsumer()
13012+            d2 = defer.succeed(None)
13013+            d2.addCallback(lambda ignored: version.read(c))
13014+            d2.addCallback(lambda ignored:
13015+                self.failUnlessEqual("".join(c.chunks), self.data))
13016+            return d2
13017+        d.addCallback(_read_data)
13018+        return d
13019+
13020+
13021+    def test_download_best_version(self):
13022+        d = self.mdmf_node.download_best_version()
13023+        d.addCallback(lambda data:
13024+            self.failUnlessEqual(data, self.data))
13025+        d.addCallback(lambda ignored:
13026+            self.sdmf_node.download_best_version())
13027+        d.addCallback(lambda data:
13028+            self.failUnlessEqual(data, self.small_data))
13029+        return d
13030[web/filenode.py: alter download code to use the new #993 interface.
13031Kevan Carstensen <kevan@isnotajoke.com>**20100717015426
13032 Ignore-this: 4e8b89f9b616755f1f644ecea4cda0bb
13033] {
13034hunk ./src/allmydata/web/filenode.py 167
13035             # properly. So we assume that at least the browser will agree
13036             # with itself, and echo back the same bytes that we were given.
13037             filename = get_arg(req, "filename", self.name) or "unknown"
13038-            if self.node.is_mutable():
13039-                # some day: d = self.node.get_best_version()
13040-                d = makeMutableDownloadable(self.node)
13041-            else:
13042-                d = defer.succeed(self.node)
13043+            d = self.node.get_best_readable_version()
13044             d.addCallback(lambda dn: FileDownloader(dn, filename))
13045             return d
13046         if t == "json":
13047hunk ./src/allmydata/web/filenode.py 191
13048         if t:
13049             raise WebError("GET file: bad t=%s" % t)
13050         filename = get_arg(req, "filename", self.name) or "unknown"
13051-        if self.node.is_mutable():
13052-            # some day: d = self.node.get_best_version()
13053-            d = makeMutableDownloadable(self.node)
13054-        else:
13055-            d = defer.succeed(self.node)
13056+        d = self.node.get_best_readable_version()
13057         d.addCallback(lambda dn: FileDownloader(dn, filename))
13058         return d
13059 
13060hunk ./src/allmydata/web/filenode.py 285
13061         d.addCallback(lambda res: self.node.get_uri())
13062         return d
13063 
13064-class MutableDownloadable:
13065-    #implements(IDownloadable)
13066-    def __init__(self, size, node):
13067-        self.size = size
13068-        self.node = node
13069-    def get_size(self):
13070-        return self.size
13071-    def is_mutable(self):
13072-        return True
13073-    def read(self, consumer, offset=0, size=None):
13074-        d = self.node.download_best_version()
13075-        d.addCallback(self._got_data, consumer, offset, size)
13076-        return d
13077-    def _got_data(self, contents, consumer, offset, size):
13078-        start = offset
13079-        if size is not None:
13080-            end = offset+size
13081-        else:
13082-            end = self.size
13083-        # SDMF: we can write the whole file in one big chunk
13084-        consumer.write(contents[start:end])
13085-        return consumer
13086-
13087-def makeMutableDownloadable(n):
13088-    d = defer.maybeDeferred(n.get_size_of_best_version)
13089-    d.addCallback(MutableDownloadable, n)
13090-    return d
13091 
13092 class FileDownloader(rend.Page):
13093     # since we override the rendering process (to let the tahoe Downloader
13094}
13095[test/common.py: remove FileTooLargeErrors that tested for an SDMF limitation that no longer exists
13096Kevan Carstensen <kevan@isnotajoke.com>**20100717015501
13097 Ignore-this: 8b17689d9391a4870a327c1d7c0b3225
13098] {
13099hunk ./src/allmydata/test/common.py 198
13100         self.init_from_cap(make_mutable_file_cap())
13101     def create(self, contents, key_generator=None, keysize=None):
13102         initial_contents = self._get_initial_contents(contents)
13103-        if initial_contents.get_size() > self.MUTABLE_SIZELIMIT:
13104-            raise FileTooLargeError("SDMF is limited to one segment, and "
13105-                                    "%d > %d" % (initial_contents.get_size(),
13106-                                                 self.MUTABLE_SIZELIMIT))
13107         data = initial_contents.read(initial_contents.get_size())
13108         data = "".join(data)
13109         self.all_contents[self.storage_index] = data
13110hunk ./src/allmydata/test/common.py 326
13111 
13112 
13113     def overwrite(self, new_contents):
13114-        if new_contents.get_size() > self.MUTABLE_SIZELIMIT:
13115-            raise FileTooLargeError("SDMF is limited to one segment, and "
13116-                                    "%d > %d" % (new_contents.get_size(),
13117-                                                 self.MUTABLE_SIZELIMIT))
13118         assert not self.is_readonly()
13119         new_data = new_contents.read(new_contents.get_size())
13120         new_data = "".join(new_data)
13121}
13122[nodemaker.py: resolve a conflict introduced in one of the 1.7.1 patches
13123Kevan Carstensen <kevan@isnotajoke.com>**20100720213109
13124 Ignore-this: 4e7d4e611f4cdf04824e9040167aa11
13125] hunk ./src/allmydata/nodemaker.py 107
13126                          "create_new_mutable_directory requires metadata to be a dict, not None", metadata)
13127             node.raise_error()
13128         d = self.create_mutable_file(lambda n:
13129-                                     pack_children(n, initial_children))
13130+                                     MutableData(pack_children(initial_children,
13131+                                                    n.get_writekey())),
13132+                                     version)
13133         d.addCallback(self._create_dirnode)
13134         return d
13135 
13136[frontends/sftpd.py: fix conflicts with trunk
13137Kevan Carstensen <kevan@isnotajoke.com>**20100727224651
13138 Ignore-this: 5636e7a27162bf3ca14d6c9dc07a015
13139] {
13140hunk ./src/allmydata/frontends/sftpd.py 664
13141         else:
13142             assert IFileNode.providedBy(filenode), filenode
13143 
13144-            # TODO: use download interface described in #993 when implemented.
13145-            if filenode.is_mutable():
13146-                self.async.addCallback(lambda ign: filenode.download_best_version())
13147-                def _downloaded(data):
13148-                    self.consumer = OverwriteableFileConsumer(len(data), tempfile_maker)
13149-                    self.consumer.write(data)
13150-                    self.consumer.finish()
13151-                    return None
13152-                self.async.addCallback(_downloaded)
13153-            else:
13154-                download_size = filenode.get_size()
13155-                assert download_size is not None, "download_size is None"
13156+            self.async.addCallback(lambda ignored: filenode.get_best_readable_version())
13157+
13158+            def _read(version):
13159+                if noisy: self.log("_read", level=NOISY)
13160+                download_size = version.get_size()
13161+                assert download_size is not None
13162+
13163                 self.consumer = OverwriteableFileConsumer(download_size, tempfile_maker)
13164 
13165hunk ./src/allmydata/frontends/sftpd.py 673
13166-                if noisy: self.log("_read", level=NOISY)
13167                 version.read(self.consumer, 0, None)
13168             self.async.addCallback(_read)
13169 
13170}
13171[interfaces.py: Create an IWritable interface
13172Kevan Carstensen <kevan@isnotajoke.com>**20100727224703
13173 Ignore-this: 3fd15da701c31c024963d7ee5c896124
13174] hunk ./src/allmydata/interfaces.py 633
13175         """
13176 
13177 
13178+class IWritable(Interface):
13179+    """
13180+    I define methods that callers can use to update SDMF and MDMF
13181+    mutable files on a Tahoe-LAFS grid.
13182+    """
13183+    # XXX: For the moment, we have only this. It is possible that we
13184+    #      want to move overwrite() and modify() in here too.
13185+    def update(data, offset):
13186+        """
13187+        I write the data from my data argument to the MDMF file,
13188+        starting at offset. I continue writing data until my data
13189+        argument is exhausted, appending data to the file as necessary.
13190+        """
13191+        # assert IMutableUploadable.providedBy(data)
13192+        # to append data: offset=node.get_size_of_best_version()
13193+        # do we want to support compacting MDMF?
13194+        # for an MDMF file, this can be done with O(data.get_size())
13195+        # memory. For an SDMF file, any modification takes
13196+        # O(node.get_size_of_best_version()).
13197+
13198+
13199 class IMutableFileVersion(IReadable):
13200     """I provide access to a particular version of a mutable file. The
13201     access is read/write if I was obtained from a filenode derived from
13202[mutable/layout.py: Alter MDMFSlotWriteProxy to perform all write operations in one actual write operation
13203Kevan Carstensen <kevan@isnotajoke.com>**20100727224725
13204 Ignore-this: 41d577e9d65eba9a38a4051c2a05d4be
13205] {
13206hunk ./src/allmydata/mutable/layout.py 814
13207         # last thing we write to the remote server.
13208         self._offsets = {}
13209         self._testvs = []
13210+        # This is a list of write vectors that will be sent to our
13211+        # remote server once we are directed to write things there.
13212+        self._writevs = []
13213         self._secrets = secrets
13214         # The segment size needs to be a multiple of the k parameter --
13215         # any padding should have been carried out by the publisher
13216hunk ./src/allmydata/mutable/layout.py 947
13217 
13218     def put_block(self, data, segnum, salt):
13219         """
13220-        Put the encrypted-and-encoded data segment in the slot, along
13221-        with the salt.
13222+        I queue a write vector for the data, salt, and segment number
13223+        provided to me. I return None, as I do not actually cause
13224+        anything to be written yet.
13225         """
13226         if segnum >= self._num_segments:
13227             raise LayoutInvalid("I won't overwrite the private key")
13228hunk ./src/allmydata/mutable/layout.py 967
13229         offset = MDMFHEADERSIZE + (self._actual_block_size * segnum)
13230         data = salt + data
13231 
13232-        datavs = [tuple([offset, data])]
13233-        return self._write(datavs)
13234+        self._writevs.append(tuple([offset, data]))
13235 
13236 
13237     def put_encprivkey(self, encprivkey):
13238hunk ./src/allmydata/mutable/layout.py 972
13239         """
13240-        Put the encrypted private key in the remote slot.
13241+        I queue a write vector for the encrypted private key provided to
13242+        me.
13243         """
13244         assert self._offsets
13245         assert self._offsets['enc_privkey']
13246hunk ./src/allmydata/mutable/layout.py 986
13247         if "share_hash_chain" in self._offsets:
13248             raise LayoutInvalid("You must write this before the block hash tree")
13249 
13250-        self._offsets['block_hash_tree'] = self._offsets['enc_privkey'] + len(encprivkey)
13251-        datavs = [(tuple([self._offsets['enc_privkey'], encprivkey]))]
13252-        def _on_failure():
13253-            del(self._offsets['block_hash_tree'])
13254-        return self._write(datavs, on_failure=_on_failure)
13255+        self._offsets['block_hash_tree'] = self._offsets['enc_privkey'] + \
13256+            len(encprivkey)
13257+        self._writevs.append(tuple([self._offsets['enc_privkey'], encprivkey]))
13258 
13259 
13260     def put_blockhashes(self, blockhashes):
13261hunk ./src/allmydata/mutable/layout.py 993
13262         """
13263-        Put the block hash tree in the remote slot.
13264+        I queue a write vector to put the block hash tree in blockhashes
13265+        onto the remote server.
13266 
13267hunk ./src/allmydata/mutable/layout.py 996
13268-        The encrypted private key must be put before the block hash
13269+        The encrypted private key must be queued before the block hash
13270         tree, since we need to know how large it is to know where the
13271         block hash tree should go. The block hash tree must be put
13272         before the salt hash tree, since its size determines the
13273hunk ./src/allmydata/mutable/layout.py 1014
13274                                 "you put the share hash chain")
13275         blockhashes_s = "".join(blockhashes)
13276         self._offsets['share_hash_chain'] = self._offsets['block_hash_tree'] + len(blockhashes_s)
13277-        datavs = []
13278-        datavs.append(tuple([self._offsets['block_hash_tree'], blockhashes_s]))
13279-        def _on_failure():
13280-            del(self._offsets['share_hash_chain'])
13281-        return self._write(datavs, on_failure=_on_failure)
13282+
13283+        self._writevs.append(tuple([self._offsets['block_hash_tree'],
13284+                                  blockhashes_s]))
13285 
13286 
13287     def put_sharehashes(self, sharehashes):
13288hunk ./src/allmydata/mutable/layout.py 1021
13289         """
13290-        Put the share hash chain in the remote slot.
13291+        I queue a write vector to put the share hash chain in my
13292+        argument onto the remote server.
13293 
13294hunk ./src/allmydata/mutable/layout.py 1024
13295-        The salt hash tree must be put before the share hash chain,
13296+        The salt hash tree must be queued before the share hash chain,
13297         since we need to know where the salt hash tree ends before we
13298         can know where the share hash chain starts. The share hash chain
13299         must be put before the signature, since the length of the packed
13300hunk ./src/allmydata/mutable/layout.py 1044
13301         if "verification_key" in self._offsets:
13302             raise LayoutInvalid("You must write the share hash chain "
13303                                 "before you write the signature")
13304-        datavs = []
13305         sharehashes_s = "".join([struct.pack(">H32s", i, sharehashes[i])
13306                                   for i in sorted(sharehashes.keys())])
13307         self._offsets['signature'] = self._offsets['share_hash_chain'] + len(sharehashes_s)
13308hunk ./src/allmydata/mutable/layout.py 1047
13309-        datavs.append(tuple([self._offsets['share_hash_chain'], sharehashes_s]))
13310-        def _on_failure():
13311-            del(self._offsets['signature'])
13312-        return self._write(datavs, on_failure=_on_failure)
13313+        self._writevs.append(tuple([self._offsets['share_hash_chain'],
13314+                            sharehashes_s]))
13315 
13316 
13317     def put_root_hash(self, roothash):
13318hunk ./src/allmydata/mutable/layout.py 1069
13319         if len(roothash) != HASH_SIZE:
13320             raise LayoutInvalid("hashes and salts must be exactly %d bytes"
13321                                  % HASH_SIZE)
13322-        datavs = []
13323         self._root_hash = roothash
13324         # To write both of these values, we update the checkstring on
13325         # the remote server, which includes them
13326hunk ./src/allmydata/mutable/layout.py 1073
13327         checkstring = self.get_checkstring()
13328-        datavs.append(tuple([0, checkstring]))
13329+        self._writevs.append(tuple([0, checkstring]))
13330         # This write, if successful, changes the checkstring, so we need
13331         # to update our internal checkstring to be consistent with the
13332         # one on the server.
13333hunk ./src/allmydata/mutable/layout.py 1077
13334-        def _on_success():
13335-            self._testvs = [(0, len(checkstring), "eq", checkstring)]
13336-        def _on_failure():
13337-            self._root_hash = None
13338-        return self._write(datavs,
13339-                           on_success=_on_success,
13340-                           on_failure=_on_failure)
13341 
13342 
13343     def get_signable(self):
13344hunk ./src/allmydata/mutable/layout.py 1100
13345 
13346     def put_signature(self, signature):
13347         """
13348-        Put the signature field to the remote slot.
13349+        I queue a write vector for the signature of the MDMF share.
13350 
13351         I require that the root hash and share hash chain have been put
13352         to the grid before I will write the signature to the grid.
13353hunk ./src/allmydata/mutable/layout.py 1123
13354             raise LayoutInvalid("You must write the signature before the verification key")
13355 
13356         self._offsets['verification_key'] = self._offsets['signature'] + len(signature)
13357-        datavs = []
13358-        datavs.append(tuple([self._offsets['signature'], signature]))
13359-        def _on_failure():
13360-            del(self._offsets['verification_key'])
13361-        return self._write(datavs, on_failure=_on_failure)
13362+        self._writevs.append(tuple([self._offsets['signature'], signature]))
13363 
13364 
13365     def put_verification_key(self, verification_key):
13366hunk ./src/allmydata/mutable/layout.py 1128
13367         """
13368-        Put the verification key into the remote slot.
13369+        I queue a write vector for the verification key.
13370 
13371         I require that the signature have been written to the storage
13372         server before I allow the verification key to be written to the
13373hunk ./src/allmydata/mutable/layout.py 1138
13374             raise LayoutInvalid("You must put the signature before you "
13375                                 "can put the verification key")
13376         self._offsets['EOF'] = self._offsets['verification_key'] + len(verification_key)
13377-        datavs = []
13378-        datavs.append(tuple([self._offsets['verification_key'], verification_key]))
13379-        def _on_failure():
13380-            del(self._offsets['EOF'])
13381-        return self._write(datavs, on_failure=_on_failure)
13382+        self._writevs.append(tuple([self._offsets['verification_key'],
13383+                            verification_key]))
13384+
13385 
13386     def _get_offsets_tuple(self):
13387         return tuple([(key, value) for key, value in self._offsets.items()])
13388hunk ./src/allmydata/mutable/layout.py 1145
13389 
13390+
13391     def get_verinfo(self):
13392         return (self._seqnum,
13393                 self._root_hash,
13394hunk ./src/allmydata/mutable/layout.py 1159
13395 
13396     def finish_publishing(self):
13397         """
13398-        Write the offset table and encoding parameters to the remote
13399-        slot, since that's the only thing we have yet to publish at this
13400-        point.
13401+        I add a write vector for the offsets table, and then cause all
13402+        of the write vectors that I've dealt with so far to be published
13403+        to the remote server, ending the write process.
13404         """
13405         if "EOF" not in self._offsets:
13406             raise LayoutInvalid("You must put the verification key before "
13407hunk ./src/allmydata/mutable/layout.py 1174
13408                               self._offsets['signature'],
13409                               self._offsets['verification_key'],
13410                               self._offsets['EOF'])
13411-        datavs = []
13412-        datavs.append(tuple([offsets_offset, offsets]))
13413+        self._writevs.append(tuple([offsets_offset, offsets]))
13414         encoding_parameters_offset = struct.calcsize(MDMFCHECKSTRING)
13415         params = struct.pack(">BBQQ",
13416                              self._required_shares,
13417hunk ./src/allmydata/mutable/layout.py 1181
13418                              self._total_shares,
13419                              self._segment_size,
13420                              self._data_length)
13421-        datavs.append(tuple([encoding_parameters_offset, params]))
13422-        return self._write(datavs)
13423+        self._writevs.append(tuple([encoding_parameters_offset, params]))
13424+        return self._write(self._writevs)
13425 
13426 
13427     def _write(self, datavs, on_failure=None, on_success=None):
13428}
13429[test/test_mutable.py: test that write operations occur all at once
13430Kevan Carstensen <kevan@isnotajoke.com>**20100727224817
13431 Ignore-this: 44cb37c6887ee9baa3e67645ece9555d
13432] {
13433hunk ./src/allmydata/test/test_mutable.py 100
13434         self.storage = storage
13435         self.queries = 0
13436     def callRemote(self, methname, *args, **kwargs):
13437+        self.queries += 1
13438         def _call():
13439             meth = getattr(self, methname)
13440             return meth(*args, **kwargs)
13441hunk ./src/allmydata/test/test_mutable.py 109
13442         return d
13443 
13444     def callRemoteOnly(self, methname, *args, **kwargs):
13445+        self.queries += 1
13446         d = self.callRemote(methname, *args, **kwargs)
13447         d.addBoth(lambda ignore: None)
13448         pass
13449hunk ./src/allmydata/test/test_mutable.py 370
13450         return d
13451 
13452 
13453+    def test_mdmf_write_count(self):
13454+        # Publishing an MDMF file should only cause one write for each
13455+        # share that is to be published. Otherwise, we introduce
13456+        # undesirable semantics that are a regression from SDMF
13457+        upload = MutableData("MDMF" * 100000) # about 400 KiB
13458+        d = self.nodemaker.create_mutable_file(upload,
13459+                                               version=MDMF_VERSION)
13460+        def _check_server_write_counts(ignored):
13461+            sb = self.nodemaker.storage_broker
13462+            peers = sb.test_servers.values()
13463+            for peer in peers:
13464+                self.failUnlessEqual(peer.queries, 1)
13465+        d.addCallback(_check_server_write_counts)
13466+        return d
13467+
13468+
13469     def test_create_with_initial_contents(self):
13470         upload1 = MutableData("contents 1")
13471         d = self.nodemaker.create_mutable_file(upload1)
13472}
13473[test/test_storage.py: modify proxy tests to work with the new writing semantics
13474Kevan Carstensen <kevan@isnotajoke.com>**20100727224853
13475 Ignore-this: 2b6bdde6dc9d8e4e7f096cdb725b40cf
13476] {
13477hunk ./src/allmydata/test/test_storage.py 1681
13478         # diagnose the problem. This test ensures that the read vector
13479         # is working appropriately.
13480         mw = self._make_new_mw("si1", 0)
13481-        d = defer.succeed(None)
13482 
13483hunk ./src/allmydata/test/test_storage.py 1682
13484-        # Write one share. This should return a checkstring of nothing,
13485-        # since there is no data there.
13486-        d.addCallback(lambda ignored:
13487-            mw.put_block(self.block, 0, self.salt))
13488-        def _check_first_write(results):
13489-            result, readvs = results
13490-            self.failUnless(result)
13491-            self.failIf(readvs)
13492-        d.addCallback(_check_first_write)
13493-        # Now, there should be a different checkstring returned when
13494-        # we write other shares
13495-        d.addCallback(lambda ignored:
13496-            mw.put_block(self.block, 1, self.salt))
13497-        def _check_next_write(results):
13498-            result, readvs = results
13499+        for i in xrange(6):
13500+            mw.put_block(self.block, i, self.salt)
13501+        mw.put_encprivkey(self.encprivkey)
13502+        mw.put_blockhashes(self.block_hash_tree)
13503+        mw.put_sharehashes(self.share_hash_chain)
13504+        mw.put_root_hash(self.root_hash)
13505+        mw.put_signature(self.signature)
13506+        mw.put_verification_key(self.verification_key)
13507+        d = mw.finish_publishing()
13508+        def _then(results):
13509+            self.failUnless(len(results), 2)
13510+            result, readv = results
13511             self.failUnless(result)
13512hunk ./src/allmydata/test/test_storage.py 1695
13513-            self.expected_checkstring = mw.get_checkstring()
13514-            self.failUnlessIn(0, readvs)
13515-            self.failUnlessEqual(readvs[0][0], self.expected_checkstring)
13516-        d.addCallback(_check_next_write)
13517-        # Add the other four shares
13518-        for i in xrange(2, 6):
13519-            d.addCallback(lambda ignored, i=i:
13520-                mw.put_block(self.block, i, self.salt))
13521-            d.addCallback(_check_next_write)
13522-        # Add the encrypted private key
13523-        d.addCallback(lambda ignored:
13524-            mw.put_encprivkey(self.encprivkey))
13525-        d.addCallback(_check_next_write)
13526-        # Add the block hash tree and share hash tree
13527-        d.addCallback(lambda ignored:
13528-            mw.put_blockhashes(self.block_hash_tree))
13529-        d.addCallback(_check_next_write)
13530-        d.addCallback(lambda ignored:
13531-            mw.put_sharehashes(self.share_hash_chain))
13532-        d.addCallback(_check_next_write)
13533-        # Add the root hash and the salt hash. This should change the
13534-        # checkstring, but not in a way that we'll be able to see right
13535-        # now, since the read vectors are applied before the write
13536-        # vectors.
13537+            self.failIf(readv)
13538+            self.old_checkstring = mw.get_checkstring()
13539+            mw.set_checkstring("")
13540+        d.addCallback(_then)
13541         d.addCallback(lambda ignored:
13542hunk ./src/allmydata/test/test_storage.py 1700
13543-            mw.put_root_hash(self.root_hash))
13544-        def _check_old_testv_after_new_one_is_written(results):
13545+            mw.finish_publishing())
13546+        def _then_again(results):
13547+            self.failUnlessEqual(len(results), 2)
13548             result, readvs = results
13549hunk ./src/allmydata/test/test_storage.py 1704
13550-            self.failUnless(result)
13551+            self.failIf(result)
13552             self.failUnlessIn(0, readvs)
13553hunk ./src/allmydata/test/test_storage.py 1706
13554-            self.failUnlessEqual(self.expected_checkstring,
13555-                                 readvs[0][0])
13556-            new_checkstring = mw.get_checkstring()
13557-            self.failIfEqual(new_checkstring,
13558-                             readvs[0][0])
13559-        d.addCallback(_check_old_testv_after_new_one_is_written)
13560-        # Now add the signature. This should succeed, meaning that the
13561-        # data gets written and the read vector matches what the writer
13562-        # thinks should be there.
13563-        d.addCallback(lambda ignored:
13564-            mw.put_signature(self.signature))
13565-        d.addCallback(_check_next_write)
13566+            readv = readvs[0][0]
13567+            self.failUnlessEqual(readv, self.old_checkstring)
13568+        d.addCallback(_then_again)
13569         # The checkstring remains the same for the rest of the process.
13570         return d
13571 
13572hunk ./src/allmydata/test/test_storage.py 1811
13573         # same share.
13574         mw1 = self._make_new_mw("si1", 0)
13575         mw2 = self._make_new_mw("si1", 0)
13576-        d = defer.succeed(None)
13577+
13578         def _check_success(results):
13579             result, readvs = results
13580             self.failUnless(result)
13581hunk ./src/allmydata/test/test_storage.py 1820
13582             result, readvs = results
13583             self.failIf(result)
13584 
13585-        d.addCallback(lambda ignored:
13586-            mw1.put_block(self.block, 0, self.salt))
13587+        def _write_share(mw):
13588+            for i in xrange(6):
13589+                mw.put_block(self.block, i, self.salt)
13590+            mw.put_encprivkey(self.encprivkey)
13591+            mw.put_blockhashes(self.block_hash_tree)
13592+            mw.put_sharehashes(self.share_hash_chain)
13593+            mw.put_root_hash(self.root_hash)
13594+            mw.put_signature(self.signature)
13595+            mw.put_verification_key(self.verification_key)
13596+            return mw.finish_publishing()
13597+        d = _write_share(mw1)
13598         d.addCallback(_check_success)
13599         d.addCallback(lambda ignored:
13600hunk ./src/allmydata/test/test_storage.py 1833
13601-            mw2.put_block(self.block, 0, self.salt))
13602+            _write_share(mw2))
13603         d.addCallback(_check_failure)
13604         return d
13605 
13606hunk ./src/allmydata/test/test_storage.py 1859
13607 
13608     def test_write_test_vectors(self):
13609         # If we give the write proxy a bogus test vector at
13610-        # any point during the process, it should fail to write.
13611+        # any point during the process, it should fail to write when we
13612+        # tell it to write.
13613+        def _check_failure(results):
13614+            self.failUnlessEqual(len(results), 2)
13615+            res, d = results
13616+            self.failIf(res)
13617+
13618+        def _check_success(results):
13619+            self.failUnlessEqual(len(results), 2)
13620+            res, d = results
13621+            self.failUnless(results)
13622+
13623         mw = self._make_new_mw("si1", 0)
13624         mw.set_checkstring("this is a lie")
13625hunk ./src/allmydata/test/test_storage.py 1873
13626-        # The initial write should be expecting to find the improbable
13627-        # checkstring above in place; finding nothing, it should fail.
13628-        d = defer.succeed(None)
13629-        d.addCallback(lambda ignored:
13630-            mw.put_block(self.block, 0, self.salt))
13631-        def _check_failure(results):
13632-            result, readv = results
13633-            self.failIf(result)
13634+        for i in xrange(6):
13635+            mw.put_block(self.block, i, self.salt)
13636+        mw.put_encprivkey(self.encprivkey)
13637+        mw.put_blockhashes(self.block_hash_tree)
13638+        mw.put_sharehashes(self.share_hash_chain)
13639+        mw.put_root_hash(self.root_hash)
13640+        mw.put_signature(self.signature)
13641+        mw.put_verification_key(self.verification_key)
13642+        d = mw.finish_publishing()
13643         d.addCallback(_check_failure)
13644hunk ./src/allmydata/test/test_storage.py 1883
13645-        # Now set the checkstring to the empty string, which
13646-        # indicates that no share is there.
13647         d.addCallback(lambda ignored:
13648             mw.set_checkstring(""))
13649         d.addCallback(lambda ignored:
13650hunk ./src/allmydata/test/test_storage.py 1886
13651-            mw.put_block(self.block, 0, self.salt))
13652-        def _check_success(results):
13653-            result, readv = results
13654-            self.failUnless(result)
13655-        d.addCallback(_check_success)
13656-        # Now set the checkstring to something wrong
13657-        d.addCallback(lambda ignored:
13658-            mw.set_checkstring("something wrong"))
13659-        # This should fail to do anything
13660-        d.addCallback(lambda ignored:
13661-            mw.put_block(self.block, 1, self.salt))
13662-        d.addCallback(_check_failure)
13663-        # Now set it back to what it should be.
13664-        d.addCallback(lambda ignored:
13665-            mw.set_checkstring(mw.get_checkstring()))
13666-        for i in xrange(1, 6):
13667-            d.addCallback(lambda ignored, i=i:
13668-                mw.put_block(self.block, i, self.salt))
13669-            d.addCallback(_check_success)
13670-        d.addCallback(lambda ignored:
13671-            mw.put_encprivkey(self.encprivkey))
13672-        d.addCallback(_check_success)
13673-        d.addCallback(lambda ignored:
13674-            mw.put_blockhashes(self.block_hash_tree))
13675-        d.addCallback(_check_success)
13676-        d.addCallback(lambda ignored:
13677-            mw.put_sharehashes(self.share_hash_chain))
13678-        d.addCallback(_check_success)
13679-        def _keep_old_checkstring(ignored):
13680-            self.old_checkstring = mw.get_checkstring()
13681-            mw.set_checkstring("foobarbaz")
13682-        d.addCallback(_keep_old_checkstring)
13683-        d.addCallback(lambda ignored:
13684-            mw.put_root_hash(self.root_hash))
13685-        d.addCallback(_check_failure)
13686-        d.addCallback(lambda ignored:
13687-            self.failUnlessEqual(self.old_checkstring, mw.get_checkstring()))
13688-        def _restore_old_checkstring(ignored):
13689-            mw.set_checkstring(self.old_checkstring)
13690-        d.addCallback(_restore_old_checkstring)
13691-        d.addCallback(lambda ignored:
13692-            mw.put_root_hash(self.root_hash))
13693-        d.addCallback(_check_success)
13694-        # The checkstring should have been set appropriately for us on
13695-        # the last write; if we try to change it to something else,
13696-        # that change should cause the verification key step to fail.
13697-        d.addCallback(lambda ignored:
13698-            mw.set_checkstring("something else"))
13699-        d.addCallback(lambda ignored:
13700-            mw.put_signature(self.signature))
13701-        d.addCallback(_check_failure)
13702-        d.addCallback(lambda ignored:
13703-            mw.set_checkstring(mw.get_checkstring()))
13704-        d.addCallback(lambda ignored:
13705-            mw.put_signature(self.signature))
13706-        d.addCallback(_check_success)
13707-        d.addCallback(lambda ignored:
13708-            mw.put_verification_key(self.verification_key))
13709+            mw.finish_publishing())
13710         d.addCallback(_check_success)
13711         return d
13712 
13713hunk ./src/allmydata/test/test_storage.py 1891
13714 
13715-    def test_offset_only_set_on_success(self):
13716-        # The write proxy should be smart enough to detect when a write
13717-        # has failed, and to temper its definition of progress based on
13718-        # that.
13719-        mw = self._make_new_mw("si1", 0)
13720-        d = defer.succeed(None)
13721-        for i in xrange(1, 6):
13722-            d.addCallback(lambda ignored, i=i:
13723-                mw.put_block(self.block, i, self.salt))
13724-        def _break_checkstring(ignored):
13725-            self._old_checkstring = mw.get_checkstring()
13726-            mw.set_checkstring("foobarbaz")
13727-
13728-        def _fix_checkstring(ignored):
13729-            mw.set_checkstring(self._old_checkstring)
13730-
13731-        d.addCallback(_break_checkstring)
13732-
13733-        # Setting the encrypted private key shouldn't work now, which is
13734-        # to be expected and is tested elsewhere. We also want to make
13735-        # sure that we can't add the block hash tree after a failed
13736-        # write of this sort.
13737-        d.addCallback(lambda ignored:
13738-            mw.put_encprivkey(self.encprivkey))
13739-        d.addCallback(lambda ignored:
13740-            self.shouldFail(LayoutInvalid, "test out-of-order blockhashes",
13741-                            None,
13742-                            mw.put_blockhashes, self.block_hash_tree))
13743-        d.addCallback(_fix_checkstring)
13744-        d.addCallback(lambda ignored:
13745-            mw.put_encprivkey(self.encprivkey))
13746-        d.addCallback(_break_checkstring)
13747-        d.addCallback(lambda ignored:
13748-            mw.put_blockhashes(self.block_hash_tree))
13749-        d.addCallback(lambda ignored:
13750-            self.shouldFail(LayoutInvalid, "test out-of-order sharehashes",
13751-                            None,
13752-                            mw.put_sharehashes, self.share_hash_chain))
13753-        d.addCallback(_fix_checkstring)
13754-        d.addCallback(lambda ignored:
13755-            mw.put_blockhashes(self.block_hash_tree))
13756-        d.addCallback(_break_checkstring)
13757-        d.addCallback(lambda ignored:
13758-            mw.put_sharehashes(self.share_hash_chain))
13759-        d.addCallback(lambda ignored:
13760-            self.shouldFail(LayoutInvalid, "out-of-order root hash",
13761-                            None,
13762-                            mw.put_root_hash, self.root_hash))
13763-        d.addCallback(_fix_checkstring)
13764-        d.addCallback(lambda ignored:
13765-            mw.put_sharehashes(self.share_hash_chain))
13766-        d.addCallback(_break_checkstring)
13767-        d.addCallback(lambda ignored:
13768-            mw.put_root_hash(self.root_hash))
13769-        d.addCallback(lambda ignored:
13770-            self.shouldFail(LayoutInvalid, "out-of-order signature",
13771-                            None,
13772-                            mw.put_signature, self.signature))
13773-        d.addCallback(_fix_checkstring)
13774-        d.addCallback(lambda ignored:
13775-            mw.put_root_hash(self.root_hash))
13776-        d.addCallback(_break_checkstring)
13777-        d.addCallback(lambda ignored:
13778-            mw.put_signature(self.signature))
13779-        d.addCallback(lambda ignored:
13780-            self.shouldFail(LayoutInvalid, "out-of-order verification key",
13781-                            None,
13782-                            mw.put_verification_key,
13783-                            self.verification_key))
13784-        d.addCallback(_fix_checkstring)
13785-        d.addCallback(lambda ignored:
13786-            mw.put_signature(self.signature))
13787-        d.addCallback(_break_checkstring)
13788-        d.addCallback(lambda ignored:
13789-            mw.put_verification_key(self.verification_key))
13790-        d.addCallback(lambda ignored:
13791-            self.shouldFail(LayoutInvalid, "out-of-order finish",
13792-                            None,
13793-                            mw.finish_publishing))
13794-        return d
13795-
13796-
13797     def serialize_blockhashes(self, blockhashes):
13798         return "".join(blockhashes)
13799 
13800hunk ./src/allmydata/test/test_storage.py 1905
13801         # This translates to a file with 6 6-byte segments, and with 2-byte
13802         # blocks.
13803         mw = self._make_new_mw("si1", 0)
13804-        mw2 = self._make_new_mw("si1", 1)
13805         # Test writing some blocks.
13806         read = self.ss.remote_slot_readv
13807         expected_sharedata_offset = struct.calcsize(MDMFHEADER)
13808hunk ./src/allmydata/test/test_storage.py 1910
13809         written_block_size = 2 + len(self.salt)
13810         written_block = self.block + self.salt
13811-        def _check_block_write(i, share):
13812-            self.failUnlessEqual(read("si1", [share], [(expected_sharedata_offset + (i * written_block_size), written_block_size)]),
13813-                                {share: [written_block]})
13814-        d = defer.succeed(None)
13815         for i in xrange(6):
13816hunk ./src/allmydata/test/test_storage.py 1911
13817-            d.addCallback(lambda ignored, i=i:
13818-                mw.put_block(self.block, i, self.salt))
13819-            d.addCallback(lambda ignored, i=i:
13820-                _check_block_write(i, 0))
13821-        # Now try the same thing, but with share 1 instead of share 0.
13822-        for i in xrange(6):
13823-            d.addCallback(lambda ignored, i=i:
13824-                mw2.put_block(self.block, i, self.salt))
13825-            d.addCallback(lambda ignored, i=i:
13826-                _check_block_write(i, 1))
13827+            mw.put_block(self.block, i, self.salt)
13828 
13829hunk ./src/allmydata/test/test_storage.py 1913
13830-        # Next, we make a fake encrypted private key, and put it onto the
13831-        # storage server.
13832-        d.addCallback(lambda ignored:
13833-            mw.put_encprivkey(self.encprivkey))
13834-        expected_private_key_offset = expected_sharedata_offset + \
13835+        mw.put_encprivkey(self.encprivkey)
13836+        mw.put_blockhashes(self.block_hash_tree)
13837+        mw.put_sharehashes(self.share_hash_chain)
13838+        mw.put_root_hash(self.root_hash)
13839+        mw.put_signature(self.signature)
13840+        mw.put_verification_key(self.verification_key)
13841+        d = mw.finish_publishing()
13842+        def _check_publish(results):
13843+            self.failUnlessEqual(len(results), 2)
13844+            result, ign = results
13845+            self.failUnless(result, "publish failed")
13846+            for i in xrange(6):
13847+                self.failUnlessEqual(read("si1", [0], [(expected_sharedata_offset + (i * written_block_size), written_block_size)]),
13848+                                {0: [written_block]})
13849+
13850+            expected_private_key_offset = expected_sharedata_offset + \
13851                                       len(written_block) * 6
13852hunk ./src/allmydata/test/test_storage.py 1930
13853-        self.failUnlessEqual(len(self.encprivkey), 7)
13854-        d.addCallback(lambda ignored:
13855+            self.failUnlessEqual(len(self.encprivkey), 7)
13856             self.failUnlessEqual(read("si1", [0], [(expected_private_key_offset, 7)]),
13857hunk ./src/allmydata/test/test_storage.py 1932
13858-                                 {0: [self.encprivkey]}))
13859+                                 {0: [self.encprivkey]})
13860 
13861hunk ./src/allmydata/test/test_storage.py 1934
13862-        # Next, we put a fake block hash tree.
13863-        d.addCallback(lambda ignored:
13864-            mw.put_blockhashes(self.block_hash_tree))
13865-        expected_block_hash_offset = expected_private_key_offset + len(self.encprivkey)
13866-        self.failUnlessEqual(len(self.block_hash_tree_s), 32 * 6)
13867-        d.addCallback(lambda ignored:
13868+            expected_block_hash_offset = expected_private_key_offset + len(self.encprivkey)
13869+            self.failUnlessEqual(len(self.block_hash_tree_s), 32 * 6)
13870             self.failUnlessEqual(read("si1", [0], [(expected_block_hash_offset, 32 * 6)]),
13871hunk ./src/allmydata/test/test_storage.py 1937
13872-                                 {0: [self.block_hash_tree_s]}))
13873+                                 {0: [self.block_hash_tree_s]})
13874 
13875hunk ./src/allmydata/test/test_storage.py 1939
13876-        # Next, put a fake share hash chain
13877-        d.addCallback(lambda ignored:
13878-            mw.put_sharehashes(self.share_hash_chain))
13879-        expected_share_hash_offset = expected_block_hash_offset + len(self.block_hash_tree_s)
13880-        d.addCallback(lambda ignored:
13881+            expected_share_hash_offset = expected_block_hash_offset + len(self.block_hash_tree_s)
13882             self.failUnlessEqual(read("si1", [0],[(expected_share_hash_offset, (32 + 2) * 6)]),
13883hunk ./src/allmydata/test/test_storage.py 1941
13884-                                 {0: [self.share_hash_chain_s]}))
13885+                                 {0: [self.share_hash_chain_s]})
13886 
13887hunk ./src/allmydata/test/test_storage.py 1943
13888-        # Next, we put what is supposed to be the root hash of
13889-        # our share hash tree but isn't       
13890-        d.addCallback(lambda ignored:
13891-            mw.put_root_hash(self.root_hash))
13892-        # The root hash gets inserted at byte 9 (its position is in the header,
13893-        # and is fixed).
13894-        def _check(ignored):
13895             self.failUnlessEqual(read("si1", [0], [(9, 32)]),
13896                                  {0: [self.root_hash]})
13897hunk ./src/allmydata/test/test_storage.py 1945
13898-        d.addCallback(_check)
13899-
13900-        # Next, we put a signature of the header block.
13901-        d.addCallback(lambda ignored:
13902-            mw.put_signature(self.signature))
13903-        expected_signature_offset = expected_share_hash_offset + len(self.share_hash_chain_s)
13904-        self.failUnlessEqual(len(self.signature), 9)
13905-        d.addCallback(lambda ignored:
13906+            expected_signature_offset = expected_share_hash_offset + len(self.share_hash_chain_s)
13907+            self.failUnlessEqual(len(self.signature), 9)
13908             self.failUnlessEqual(read("si1", [0], [(expected_signature_offset, 9)]),
13909hunk ./src/allmydata/test/test_storage.py 1948
13910-                                 {0: [self.signature]}))
13911+                                 {0: [self.signature]})
13912 
13913hunk ./src/allmydata/test/test_storage.py 1950
13914-        # Next, we put the verification key
13915-        d.addCallback(lambda ignored:
13916-            mw.put_verification_key(self.verification_key))
13917-        expected_verification_key_offset = expected_signature_offset + len(self.signature)
13918-        self.failUnlessEqual(len(self.verification_key), 6)
13919-        d.addCallback(lambda ignored:
13920+            expected_verification_key_offset = expected_signature_offset + len(self.signature)
13921+            self.failUnlessEqual(len(self.verification_key), 6)
13922             self.failUnlessEqual(read("si1", [0], [(expected_verification_key_offset, 6)]),
13923hunk ./src/allmydata/test/test_storage.py 1953
13924-                                 {0: [self.verification_key]}))
13925+                                 {0: [self.verification_key]})
13926 
13927hunk ./src/allmydata/test/test_storage.py 1955
13928-        def _check_signable(ignored):
13929-            # Make sure that the signable is what we think it should be.
13930             signable = mw.get_signable()
13931             verno, seq, roothash, k, n, segsize, datalen = \
13932                                             struct.unpack(">BQ32sBBQQ",
13933hunk ./src/allmydata/test/test_storage.py 1966
13934             self.failUnlessEqual(n, 10)
13935             self.failUnlessEqual(segsize, 6)
13936             self.failUnlessEqual(datalen, 36)
13937-        d.addCallback(_check_signable)
13938-        # Next, we cause the offset table to be published.
13939-        d.addCallback(lambda ignored:
13940-            mw.finish_publishing())
13941-        expected_eof_offset = expected_verification_key_offset + len(self.verification_key)
13942+            expected_eof_offset = expected_verification_key_offset + len(self.verification_key)
13943 
13944hunk ./src/allmydata/test/test_storage.py 1968
13945-        def _check_offsets(ignored):
13946             # Check the version number to make sure that it is correct.
13947             expected_version_number = struct.pack(">B", 1)
13948             self.failUnlessEqual(read("si1", [0], [(0, 1)]),
13949hunk ./src/allmydata/test/test_storage.py 2008
13950             expected_offset = struct.pack(">Q", expected_eof_offset)
13951             self.failUnlessEqual(read("si1", [0], [(99, 8)]),
13952                                  {0: [expected_offset]})
13953-        d.addCallback(_check_offsets)
13954+        d.addCallback(_check_publish)
13955         return d
13956 
13957     def _make_new_mw(self, si, share, datalength=36):
13958}
13959[mutable/publish.py: alter mutable publisher to work with new writing semantics
13960Kevan Carstensen <kevan@isnotajoke.com>**20100727225001
13961 Ignore-this: a6b4628e749e09bfcddf3309271b5831
13962] {
13963hunk ./src/allmydata/mutable/publish.py 389
13964         if self._state == PUSHING_BLOCKS_STATE:
13965             return self.push_segment(self._current_segment)
13966 
13967-        # XXX: Do we want more granularity in states? Is that useful at
13968-        #      all?
13969-        #      Yes -- quicker reaction to UCW.
13970         elif self._state == PUSHING_EVERYTHING_ELSE_STATE:
13971             return self.push_everything_else()
13972 
13973hunk ./src/allmydata/mutable/publish.py 490
13974         results, salt = encoded_and_salt
13975         shares, shareids = results
13976         started = time.time()
13977-        dl = []
13978         for i in xrange(len(shares)):
13979             sharedata = shares[i]
13980             shareid = shareids[i]
13981hunk ./src/allmydata/mutable/publish.py 502
13982 
13983             # find the writer for this share
13984             writer = self.writers[shareid]
13985-            d = writer.put_block(sharedata, segnum, salt)
13986-            d.addCallback(self._got_write_answer, writer, started)
13987-            d.addErrback(self._connection_problem, writer)
13988-            dl.append(d)
13989-        return defer.DeferredList(dl)
13990+            writer.put_block(sharedata, segnum, salt)
13991 
13992 
13993     def push_everything_else(self):
13994hunk ./src/allmydata/mutable/publish.py 510
13995         I put everything else associated with a share.
13996         """
13997         encprivkey = self._encprivkey
13998-        d = self.push_encprivkey()
13999-        d.addCallback(self.push_blockhashes)
14000-        d.addCallback(self.push_sharehashes)
14001-        d.addCallback(self.push_toplevel_hashes_and_signature)
14002-        d.addCallback(self.finish_publishing)
14003+        self.push_encprivkey()
14004+        self.push_blockhashes()
14005+        self.push_sharehashes()
14006+        self.push_toplevel_hashes_and_signature()
14007+        d = self.finish_publishing()
14008         def _change_state(ignored):
14009             self._state = DONE_STATE
14010         d.addCallback(_change_state)
14011hunk ./src/allmydata/mutable/publish.py 525
14012     def push_encprivkey(self):
14013         started = time.time()
14014         encprivkey = self._encprivkey
14015-        dl = []
14016         for writer in self.writers.itervalues():
14017hunk ./src/allmydata/mutable/publish.py 526
14018-            d = writer.put_encprivkey(encprivkey)
14019-            d.addCallback(self._got_write_answer, writer, started)
14020-            d.addErrback(self._connection_problem, writer)
14021-            dl.append(d)
14022-        d = defer.DeferredList(dl)
14023-        return d
14024+            writer.put_encprivkey(encprivkey)
14025 
14026 
14027hunk ./src/allmydata/mutable/publish.py 529
14028-    def push_blockhashes(self, ignored):
14029+    def push_blockhashes(self):
14030         started = time.time()
14031hunk ./src/allmydata/mutable/publish.py 531
14032-        dl = []
14033         self.sharehash_leaves = [None] * len(self.blockhashes)
14034         for shnum, blockhashes in self.blockhashes.iteritems():
14035             t = hashtree.HashTree(blockhashes)
14036hunk ./src/allmydata/mutable/publish.py 538
14037             # set the leaf for future use.
14038             self.sharehash_leaves[shnum] = t[0]
14039             writer = self.writers[shnum]
14040-            d = writer.put_blockhashes(self.blockhashes[shnum])
14041-            d.addCallback(self._got_write_answer, writer, started)
14042-            d.addErrback(self._connection_problem, self.writers[shnum])
14043-            dl.append(d)
14044-        d = defer.DeferredList(dl)
14045-        return d
14046+            writer.put_blockhashes(self.blockhashes[shnum])
14047 
14048 
14049hunk ./src/allmydata/mutable/publish.py 541
14050-    def push_sharehashes(self, ignored):
14051+    def push_sharehashes(self):
14052         started = time.time()
14053         share_hash_tree = hashtree.HashTree(self.sharehash_leaves)
14054         share_hash_chain = {}
14055hunk ./src/allmydata/mutable/publish.py 545
14056-        ds = []
14057         for shnum in xrange(len(self.sharehash_leaves)):
14058             needed_indices = share_hash_tree.needed_hashes(shnum)
14059             self.sharehashes[shnum] = dict( [ (i, share_hash_tree[i])
14060hunk ./src/allmydata/mutable/publish.py 550
14061                                              for i in needed_indices] )
14062             writer = self.writers[shnum]
14063-            d = writer.put_sharehashes(self.sharehashes[shnum])
14064-            d.addCallback(self._got_write_answer, writer, started)
14065-            d.addErrback(self._connection_problem, writer)
14066-            ds.append(d)
14067+            writer.put_sharehashes(self.sharehashes[shnum])
14068         self.root_hash = share_hash_tree[0]
14069hunk ./src/allmydata/mutable/publish.py 552
14070-        d = defer.DeferredList(ds)
14071-        return d
14072 
14073 
14074hunk ./src/allmydata/mutable/publish.py 554
14075-    def push_toplevel_hashes_and_signature(self, ignored):
14076+    def push_toplevel_hashes_and_signature(self):
14077         # We need to to three things here:
14078         #   - Push the root hash and salt hash
14079         #   - Get the checkstring of the resulting layout; sign that.
14080hunk ./src/allmydata/mutable/publish.py 560
14081         #   - Push the signature
14082         started = time.time()
14083-        ds = []
14084         for shnum in xrange(self.total_shares):
14085             writer = self.writers[shnum]
14086hunk ./src/allmydata/mutable/publish.py 562
14087-            d = writer.put_root_hash(self.root_hash)
14088-            d.addCallback(self._got_write_answer, writer, started)
14089-            ds.append(d)
14090-        d = defer.DeferredList(ds)
14091-        d.addCallback(self._update_checkstring)
14092-        d.addCallback(self._make_and_place_signature)
14093-        return d
14094+            writer.put_root_hash(self.root_hash)
14095+        self._update_checkstring()
14096+        self._make_and_place_signature()
14097 
14098 
14099hunk ./src/allmydata/mutable/publish.py 567
14100-    def _update_checkstring(self, ignored):
14101+    def _update_checkstring(self):
14102         """
14103         After putting the root hash, MDMF files will have the
14104         checkstring written to the storage server. This means that we
14105hunk ./src/allmydata/mutable/publish.py 578
14106         self._checkstring = self.writers.values()[0].get_checkstring()
14107 
14108 
14109-    def _make_and_place_signature(self, ignored):
14110+    def _make_and_place_signature(self):
14111         """
14112         I create and place the signature.
14113         """
14114hunk ./src/allmydata/mutable/publish.py 586
14115         signable = self.writers[0].get_signable()
14116         self.signature = self._privkey.sign(signable)
14117 
14118-        ds = []
14119         for (shnum, writer) in self.writers.iteritems():
14120hunk ./src/allmydata/mutable/publish.py 587
14121-            d = writer.put_signature(self.signature)
14122-            d.addCallback(self._got_write_answer, writer, started)
14123-            d.addErrback(self._connection_problem, writer)
14124-            ds.append(d)
14125-        return defer.DeferredList(ds)
14126+            writer.put_signature(self.signature)
14127 
14128 
14129hunk ./src/allmydata/mutable/publish.py 590
14130-    def finish_publishing(self, ignored):
14131+    def finish_publishing(self):
14132         # We're almost done -- we just need to put the verification key
14133         # and the offsets
14134         started = time.time()
14135hunk ./src/allmydata/mutable/publish.py 601
14136         # TODO: Bad, since we remove from this same dict. We need to
14137         # make a copy, or just use a non-iterated value.
14138         for (shnum, writer) in self.writers.iteritems():
14139-            d = writer.put_verification_key(verification_key)
14140-            d.addCallback(self._got_write_answer, writer, started)
14141-            d.addCallback(self._record_verinfo)
14142-            d.addCallback(lambda ignored, writer=writer:
14143-                writer.finish_publishing())
14144+            writer.put_verification_key(verification_key)
14145+            d = writer.finish_publishing()
14146             d.addCallback(self._got_write_answer, writer, started)
14147             d.addErrback(self._connection_problem, writer)
14148             ds.append(d)
14149hunk ./src/allmydata/mutable/publish.py 606
14150+        self._record_verinfo()
14151         return defer.DeferredList(ds)
14152 
14153 
14154hunk ./src/allmydata/mutable/publish.py 610
14155-    def _record_verinfo(self, ignored):
14156+    def _record_verinfo(self):
14157         self.versioninfo = self.writers.values()[0].get_verinfo()
14158 
14159 
14160}
14161[test/test_mutable.py: Add tests for new servermap behavior
14162Kevan Carstensen <kevan@isnotajoke.com>**20100728232434
14163 Ignore-this: aa6da7dbc9f86eb8840c8f0e779e644d
14164] {
14165hunk ./src/allmydata/test/test_mutable.py 773
14166     def setUp(self):
14167         return self.publish_one()
14168 
14169-    def make_servermap(self, mode=MODE_CHECK, fn=None, sb=None):
14170+    def make_servermap(self, mode=MODE_CHECK, fn=None, sb=None,
14171+                       update_range=None):
14172         if fn is None:
14173             fn = self._fn
14174         if sb is None:
14175hunk ./src/allmydata/test/test_mutable.py 780
14176             sb = self._storage_broker
14177         smu = ServermapUpdater(fn, sb, Monitor(),
14178-                               ServerMap(), mode)
14179+                               ServerMap(), mode, update_range=update_range)
14180         d = smu.update()
14181         return d
14182 
14183hunk ./src/allmydata/test/test_mutable.py 855
14184         d.addCallback(lambda sm: self.failUnlessOneRecoverable(sm, 10))
14185         return d
14186 
14187+
14188     def test_mark_bad(self):
14189         d = defer.succeed(None)
14190         ms = self.make_servermap
14191hunk ./src/allmydata/test/test_mutable.py 970
14192         return d
14193 
14194 
14195+    def test_fetch_update(self):
14196+        d = defer.succeed(None)
14197+        d.addCallback(lambda ignored:
14198+            self.publish_mdmf())
14199+        d.addCallback(lambda ignored:
14200+            self.make_servermap(mode=MODE_WRITE, update_range=(1, 2)))
14201+        def _check_servermap(sm):
14202+            # 10 shares
14203+            self.failUnlessEqual(len(sm.update_data), 10)
14204+            # one version
14205+            for data in sm.update_data.itervalues():
14206+                self.failUnlessEqual(len(data), 1)
14207+        d.addCallback(_check_servermap)
14208+        return d
14209+
14210+
14211     def test_servermapupdater_finds_sdmf_files(self):
14212         d = defer.succeed(None)
14213         d.addCallback(lambda ignored:
14214hunk ./src/allmydata/test/test_mutable.py 1756
14215 
14216     def test_mdmf_repairable_5shares(self):
14217         d = self.publish_mdmf()
14218-        def _delete_all_shares(ign):
14219+        def _delete_some_shares(ign):
14220             shares = self._storage._peers
14221             for peerid in shares:
14222                 for shnum in list(shares[peerid]):
14223hunk ./src/allmydata/test/test_mutable.py 1762
14224                     if shnum > 5:
14225                         del shares[peerid][shnum]
14226-        d.addCallback(_delete_all_shares)
14227+        d.addCallback(_delete_some_shares)
14228         d.addCallback(lambda ign: self._fn.check(Monitor()))
14229hunk ./src/allmydata/test/test_mutable.py 1764
14230+        def _check(cr):
14231+            self.failIf(cr.is_healthy())
14232+            self.failUnless(cr.is_recoverable())
14233+            return cr
14234+        d.addCallback(_check)
14235         d.addCallback(lambda check_results: self._fn.repair(check_results))
14236hunk ./src/allmydata/test/test_mutable.py 1770
14237-        def _check(crr):
14238+        def _check1(crr):
14239             self.failUnlessEqual(crr.get_successful(), True)
14240hunk ./src/allmydata/test/test_mutable.py 1772
14241-        d.addCallback(_check)
14242+        d.addCallback(_check1)
14243         return d
14244 
14245 
14246}
14247[mutable/filenode.py: add an update method.
14248Kevan Carstensen <kevan@isnotajoke.com>**20100730234029
14249 Ignore-this: 3ed4dcfb8a247812ed357216913334e7
14250] {
14251hunk ./src/allmydata/mutable/filenode.py 9
14252 from foolscap.api import eventually
14253 from allmydata.interfaces import IMutableFileNode, ICheckable, ICheckResults, \
14254      NotEnoughSharesError, MDMF_VERSION, SDMF_VERSION, IMutableUploadable, \
14255-     IMutableFileVersion
14256-from allmydata.util import hashutil, log, consumer
14257+     IMutableFileVersion, IWritable
14258+from allmydata.util import hashutil, log, consumer, deferredutil
14259 from allmydata.util.assertutil import precondition
14260 from allmydata.uri import WriteableSSKFileURI, ReadonlySSKFileURI
14261 from allmydata.monitor import Monitor
14262hunk ./src/allmydata/mutable/filenode.py 17
14263 from pycryptopp.cipher.aes import AES
14264 
14265 from allmydata.mutable.publish import Publish, MutableFileHandle, \
14266-                                      MutableData
14267+                                      MutableData,\
14268+                                      DEFAULT_MAX_SEGMENT_SIZE, \
14269+                                      TransformingUploadable
14270 from allmydata.mutable.common import MODE_READ, MODE_WRITE, UnrecoverableFileError, \
14271      ResponseCache, UncoordinatedWriteError
14272 from allmydata.mutable.servermap import ServerMap, ServermapUpdater
14273hunk ./src/allmydata/mutable/filenode.py 671
14274     overwrite or modify the contents of the mutable file that I
14275     reference.
14276     """
14277-    implements(IMutableFileVersion)
14278+    implements(IMutableFileVersion, IWritable)
14279 
14280     def __init__(self,
14281                  node,
14282hunk ./src/allmydata/mutable/filenode.py 960
14283     def _did_upload(self, res, size):
14284         self._size = size
14285         return res
14286+
14287+    def _update(self, data, offset):
14288+        """
14289+        Do an update of this mutable file version by inserting data at
14290+        offset within the file. If offset is the EOF, this is an append
14291+        operation. I return a Deferred that fires with the results of
14292+        the update operation when it has completed.
14293+
14294+        In cases where update does not append any data, or where it does
14295+        not append so many blocks that the block count crosses a
14296+        power-of-two boundary, this operation will use roughly
14297+        O(data.get_size()) memory/bandwidth/CPU to perform the update.
14298+        Otherwise, it must download, re-encode, and upload the entire
14299+        file again, which will use O(filesize) resources.
14300+        """
14301+        return self._do_serialized(self._update, data, offset)
14302+
14303+
14304+    def _update(self, data, offset):
14305+        """
14306+        I update the mutable file version represented by this particular
14307+        IMutableVersion by inserting the data in data at the offset
14308+        offset. I return a Deferred that fires when this has been
14309+        completed.
14310+        """
14311+        d = self._do_update_update(data, offset)
14312+        d.addCallback(self._decode_and_decrypt_segments)
14313+        d.addCallback(self._build_uploadable_and_finish)
14314+        return d
14315+
14316+
14317+    def _do_update_update(self, data, offset):
14318+        """
14319+        I start the Servermap update that gets us the data we need to
14320+        continue the update process. I return a Deferred that fires when
14321+        the servermap update is done.
14322+        """
14323+        assert IMutableUploadable.providedBy(data)
14324+        assert self.is_mutable()
14325+        # offset == self.get_size() is valid and means that we are
14326+        # appending data to the file.
14327+        assert offset <= self.get_size()
14328+
14329+        datasize = data.get_size()
14330+        # We'll need the segment that the data starts in, regardless of
14331+        # what we'll do later.
14332+        start_segment = mathutil.div_ceil(offset, DEFAULT_MAX_SEGMENT_SIZE)
14333+        start_segment -= 1
14334+
14335+        # We only need the end segment if the data we append does not go
14336+        # beyond the current end-of-file.
14337+        end_segment = start_segment
14338+        if offset + data.get_size() < self.get_size():
14339+            end_data = offset + data.get_size()
14340+            end_segment = mathutil.div_ceil(end_data, DEFAULT_MAX_SEGMENT_SIZE)
14341+            end_segment -= 1
14342+
14343+        # Now ask for the servermap to be updated in MODE_WRITE with
14344+        # this update range.
14345+        u = ServermapUpdater(self, self._storage_broker, Monitor(),
14346+                             self._servermap,
14347+                             mode=MODE_WRITE,
14348+                             update_range=(start_segment, end_segment))
14349+        return u.update()
14350+
14351+
14352+    def _decode_and_decrypt_segments(self, ignored, data, offset):
14353+        """
14354+        After the servermap update, I take the encrypted and encoded
14355+        data that the servermap fetched while doing its update and
14356+        transform it into decoded-and-decrypted plaintext that can be
14357+        used by the new uploadable. I return a Deferred that fires with
14358+        the segments.
14359+        """
14360+        r = Retrieve(self._node, self._servermap, self._version, fetch_privkey)
14361+        # decode: takes in our blocks and salts from the servermap,
14362+        # returns a Deferred that fires with the corresponding plaintext
14363+        # segments. Does not download -- simply takes advantage of
14364+        # existing infrastructure within the Retrieve class to avoid
14365+        # duplicating code.
14366+        sm = self._servermap
14367+        # XXX: If the methods in the servermap don't work as
14368+        # abstractions, you should rewrite them instead of going around
14369+        # them.
14370+        data = sm.update_data
14371+        data = [data[1] for i in update_data if i[0] == self._verinfo]
14372+        start_seg_data = [d[0] for d in data]
14373+        end_seg_data = [d[1] for d in data]
14374+        d1 = r.decode(start_seg_data)
14375+        d2 = r.decode(end_seg_data)
14376+        return deferredutil.gatherResults([d1, d2])
14377+
14378+
14379+    def _build_uploadable_and_finish(self, segments, data, offset):
14380+        """
14381+        After the process has the plaintext segments, I build the
14382+        TransformingUploadable that the publisher will eventually
14383+        re-upload to the grid. I then invoke the publisher with that
14384+        uploadable, and return a Deferred when the publish operation has
14385+        completed without issue.
14386+        """
14387+        u = TransformingUploadable(data, offset,
14388+                                   DEFAULT_MAX_SEGMENT_SIZE,
14389+                                   segments[0],
14390+                                   segments[1])
14391+        p = Publish(self._node, self._storage_broker, self._servermap)
14392+        return p.update(data, offset, blockhashes)
14393}
14394[mutable/publish.py: learn how to update as well as publish files.
14395Kevan Carstensen <kevan@isnotajoke.com>**20100730234056
14396 Ignore-this: 4c04857280da970f5c1c6c75466f1cd9
14397] {
14398hunk ./src/allmydata/mutable/publish.py 132
14399             kwargs["facility"] = "tahoe.mutable.publish"
14400         return log.msg(*args, **kwargs)
14401 
14402+
14403+    def update(self, data, offset, blockhashes):
14404+        """
14405+        I replace the contents of this file with the contents of data,
14406+        starting at offset. I return a Deferred that fires with None
14407+        when the replacement has been completed, or with an error if
14408+        something went wrong during the process.
14409+
14410+        Note that this process will not upload new shares. If the file
14411+        being updated is in need of repair, callers will have to repair
14412+        it on their own.
14413+        """
14414+        # How this works:
14415+        # 1: Make peer assignments. We'll assign each share that we know
14416+        # about on the grid to that peer that currently holds that
14417+        # share, and will not place any new shares.
14418+        # 2: Setup encoding parameters. Most of these will stay the same
14419+        # -- datalength will change, as will some of the offsets.
14420+        # 3. Upload the new segments.
14421+        # 4. Be done.
14422+        assert IMutableUploadable.providedBy(data)
14423+
14424+        self.data = data
14425+
14426+        # XXX: Use the MutableFileVersion instead.
14427+        self.datalength = self._node.get_size()
14428+        if offset + data.get_size() > self.datalength:
14429+            self.datalength = offset + data.get_size()
14430+
14431+        if self.datalength >= DEFAULT_MAX_SEGMENT_SIZE:
14432+            self._version = MDMF_VERSION
14433+        else:
14434+            self._version = SDMF_VERSION
14435+
14436+        self.log("starting update")
14437+        self.log("adding new data of length %d at offset %d" % \
14438+                    data.get_size(), offset)
14439+        self.log("new data length is %d" % self.datalength)
14440+        self._status.set_size(self.datalength)
14441+        self._status.set_status("Started")
14442+        self._started = time.time()
14443+
14444+        self.done_deferred = defer.Deferred()
14445+
14446+        self._writekey = self._node.get_writekey()
14447+        assert self._writekey, "need write capability to publish"
14448+
14449+        # first, which servers will we publish to? We require that the
14450+        # servermap was updated in MODE_WRITE, so we can depend upon the
14451+        # peerlist computed by that process instead of computing our own.
14452+        assert self._servermap
14453+        assert self._servermap.last_update_mode in (MODE_WRITE, MODE_CHECK)
14454+        # we will push a version that is one larger than anything present
14455+        # in the grid, according to the servermap.
14456+        self._new_seqnum = self._servermap.highest_seqnum() + 1
14457+        self._status.set_servermap(self._servermap)
14458+
14459+        self.log(format="new seqnum will be %(seqnum)d",
14460+                 seqnum=self._new_seqnum, level=log.NOISY)
14461+
14462+        # We're updating an existing file, so all of the following
14463+        # should be available.
14464+        self.readkey = self._node.get_readkey()
14465+        self.required_shares = self._node.get_required_shares()
14466+        assert self.required_shares is not None
14467+        self.total_shares = self._node.get_total_shares()
14468+        assert self.total_shares is not None
14469+        self._status.set_encoding(self.required_shares, self.total_shares)
14470+
14471+        self._pubkey = self._node.get_pubkey()
14472+        assert self._pubkey
14473+        self._privkey = self._node.get_privkey()
14474+        assert self._privkey
14475+        self._encprivkey = self._node.get_encprivkey()
14476+
14477+        sb = self._storage_broker
14478+        full_peerlist = sb.get_servers_for_index(self._storage_index)
14479+        self.full_peerlist = full_peerlist # for use later, immutable
14480+        self.bad_peers = set() # peerids who have errbacked/refused requests
14481+
14482+        # This will set self.segment_size, self.num_segments, and
14483+        # self.fec. TODO: Does it know how to do the offset? Probably
14484+        # not. So do that part next.
14485+        self.setup_encoding_parameters(offset=offset)
14486+
14487+        # if we experience any surprises (writes which were rejected because
14488+        # our test vector did not match, or shares which we didn't expect to
14489+        # see), we set this flag and report an UncoordinatedWriteError at the
14490+        # end of the publish process.
14491+        self.surprised = False
14492+
14493+        # as a failsafe, refuse to iterate through self.loop more than a
14494+        # thousand times.
14495+        self.looplimit = 1000
14496+
14497+        # we keep track of three tables. The first is our goal: which share
14498+        # we want to see on which servers. This is initially populated by the
14499+        # existing servermap.
14500+        self.goal = set() # pairs of (peerid, shnum) tuples
14501+
14502+        # the second table is our list of outstanding queries: those which
14503+        # are in flight and may or may not be delivered, accepted, or
14504+        # acknowledged. Items are added to this table when the request is
14505+        # sent, and removed when the response returns (or errbacks).
14506+        self.outstanding = set() # (peerid, shnum) tuples
14507+
14508+        # the third is a table of successes: share which have actually been
14509+        # placed. These are populated when responses come back with success.
14510+        # When self.placed == self.goal, we're done.
14511+        self.placed = set() # (peerid, shnum) tuples
14512+
14513+        # we also keep a mapping from peerid to RemoteReference. Each time we
14514+        # pull a connection out of the full peerlist, we add it to this for
14515+        # use later.
14516+        self.connections = {}
14517+
14518+        self.bad_share_checkstrings = {}
14519+
14520+        # This is set at the last step of the publishing process.
14521+        self.versioninfo = ""
14522+
14523+        # we use the servermap to populate the initial goal: this way we will
14524+        # try to update each existing share in place. Since we're
14525+        # updating, we ignore damaged and missing shares -- callers must
14526+        # do a repair to repair and recreate these.
14527+        for (peerid, shnum) in self._servermap.servermap:
14528+            self.goal.add( (peerid, shnum) )
14529+            self.connections[peerid] = self._servermap.connections[peerid]
14530+        self.writers = {}
14531+        if self._version == MDMF_VERSION:
14532+            writer_class = MDMFSlotWriteProxy
14533+        else:
14534+            writer_class = SDMFSlotWriteProxy
14535+
14536+        # For each (peerid, shnum) in self.goal, we make a
14537+        # write proxy for that peer. We'll use this to write
14538+        # shares to the peer.
14539+        for key in self.goal:
14540+            peerid, shnum = key
14541+            write_enabler = self._node.get_write_enabler(peerid)
14542+            renew_secret = self._node.get_renewal_secret(peerid)
14543+            cancel_secret = self._node.get_cancel_secret(peerid)
14544+            secrets = (write_enabler, renew_secret, cancel_secret)
14545+
14546+            self.writers[shnum] =  writer_class(shnum,
14547+                                                self.connections[peerid],
14548+                                                self._storage_index,
14549+                                                secrets,
14550+                                                self._new_seqnum,
14551+                                                self.required_shares,
14552+                                                self.total_shares,
14553+                                                self.segment_size,
14554+                                                self.datalength)
14555+            self.writers[shnum].peerid = peerid
14556+            assert (peerid, shnum) in self._servermap.servermap
14557+            old_versionid, old_timestamp = self._servermap.servermap[key]
14558+            (old_seqnum, old_root_hash, old_salt, old_segsize,
14559+             old_datalength, old_k, old_N, old_prefix,
14560+             old_offsets_tuple) = old_versionid
14561+            self.writers[shnum].set_checkstring(old_seqnum,
14562+                                                old_root_hash,
14563+                                                old_salt)
14564+
14565+        # Our remote shares will not have a complete checkstring until
14566+        # after we are done writing share data and have started to write
14567+        # blocks. In the meantime, we need to know what to look for when
14568+        # writing, so that we can detect UncoordinatedWriteErrors.
14569+        self._checkstring = self.writers.values()[0].get_checkstring()
14570+
14571+        # Now, we start pushing shares.
14572+        self._status.timings["setup"] = time.time() - self._started
14573+        # First, we encrypt, encode, and publish the shares that we need
14574+        # to encrypt, encode, and publish.
14575+
14576+        # Our update process fetched these for us. We need to update
14577+        # them in place as publishing happens.
14578+        self.blockhashes = {} # (shnum, [blochashes])
14579+        for (i, bht) in blockhashes.iteritems():
14580+            self.blockhashes[i] = bht
14581+
14582+        # These are filled in later, after we've modified the block hash
14583+        # tree suitably.
14584+        self.sharehash_leaves = None # eventually [sharehashes]
14585+        self.sharehashes = {} # shnum -> [sharehash leaves necessary to
14586+                              # validate the share]
14587+
14588+        d = defer.succeed(None)
14589+        self.log("Starting push")
14590+
14591+        self._state = PUSHING_BLOCKS_STATE
14592+        self._push()
14593+
14594+        return self.done_deferred
14595+
14596+
14597     def publish(self, newdata):
14598         """Publish the filenode's current contents.  Returns a Deferred that
14599         fires (with None) when the publish has done as much work as it's ever
14600hunk ./src/allmydata/mutable/publish.py 502
14601         # that we publish. We define it this way so that empty publishes
14602         # will still have something to write to the remote slot.
14603         self.blockhashes = dict([(i, []) for i in xrange(self.total_shares)])
14604+        for i in xrange(self.total_shares):
14605+            blocks = self.blockhashes[i]
14606+            for j in xrange(self.num_segments):
14607+                blocks.append(None)
14608         self.sharehash_leaves = None # eventually [sharehashes]
14609         self.sharehashes = {} # shnum -> [sharehash leaves necessary to
14610                               # validate the share]
14611hunk ./src/allmydata/mutable/publish.py 519
14612         return self.done_deferred
14613 
14614 
14615-    def setup_encoding_parameters(self):
14616+    def setup_encoding_parameters(self, offset=0):
14617         if self._version == MDMF_VERSION:
14618             segment_size = DEFAULT_MAX_SEGMENT_SIZE # 128 KiB by default
14619         else:
14620hunk ./src/allmydata/mutable/publish.py 528
14621         segment_size = mathutil.next_multiple(segment_size,
14622                                               self.required_shares)
14623         self.segment_size = segment_size
14624+
14625+        # Calculate the starting segment for the upload.
14626         if segment_size:
14627             self.num_segments = mathutil.div_ceil(self.datalength,
14628                                                   segment_size)
14629hunk ./src/allmydata/mutable/publish.py 533
14630+            self.starting_segment = mathutil.div_ceil(offset,
14631+                                                      segment_size)
14632+            if offset == 0:
14633+                self.starting_segment = 0
14634+
14635         else:
14636             self.num_segments = 0
14637hunk ./src/allmydata/mutable/publish.py 540
14638+            self.starting_segment = 0
14639+
14640 
14641         self.log("building encoding parameters for file")
14642         self.log("got segsize %d" % self.segment_size)
14643hunk ./src/allmydata/mutable/publish.py 576
14644                                 self.total_shares)
14645             self.tail_fec = tail_fec
14646 
14647-        self._current_segment = 0
14648+        self._current_segment = self.starting_segment
14649+        self.end_segment = self.num_segments - 1
14650+        # Now figure out where the last segment should be.
14651+        if self.data.get_size() != self.datalength:
14652+            end = offset + self.data.get_size()
14653+            self.end_segment = mathutil.div_ceil(end,
14654+                                                 segment_size)
14655+            self.end_segment -= 1
14656+        self.log("got start segment %d" % self.starting_segment)
14657+        self.log("got end segment %d" % self.end_segment)
14658 
14659 
14660     def _push(self, ignored=None):
14661hunk ./src/allmydata/mutable/publish.py 618
14662         if self.num_segments == 0 and self._version == SDMF_VERSION:
14663             self._add_dummy_salts()
14664 
14665-        if segnum == self.num_segments:
14666+        if segnum > self.end_segment:
14667             # We don't have any more segments to push.
14668             self._state = PUSHING_EVERYTHING_ELSE_STATE
14669             return self._push()
14670hunk ./src/allmydata/mutable/publish.py 715
14671             else:
14672                 hashed = sharedata
14673             block_hash = hashutil.block_hash(hashed)
14674-            self.blockhashes[shareid].append(block_hash)
14675+            self.blockhashes[shareid][segnum] = block_hash
14676 
14677             # find the writer for this share
14678             writer = self.writers[shareid]
14679hunk ./src/allmydata/mutable/publish.py 1227
14680         assert isinstance(s, str)
14681 
14682         MutableFileHandle.__init__(self, StringIO(s))
14683+
14684+
14685+class TransformingUploadable:
14686+    """
14687+    I am an IMutableUploadable that wraps another IMutableUploadable,
14688+    and some segments that are already on the grid. When I am called to
14689+    read, I handle merging of boundary segments.
14690+    """
14691+    implements(IMutableUploadable)
14692+
14693+
14694+    def __init__(self, data, offset, segment_size, start, end):
14695+        assert IMutableUploadable.providedBy(data)
14696+        # offset == data.get_size() means that we're appending.
14697+        assert offset <= data.get_size()
14698+
14699+        self._newdata = data
14700+        self._offset = offset
14701+        self._segment_size = segment_size
14702+        self._start = start
14703+        self._end = end
14704+
14705+        self._read_marker = 0
14706+        self._first_segment_offset = offset % segment_size
14707+
14708+
14709+    def get_size(self):
14710+        # TODO
14711+        pass
14712+
14713+
14714+    def read(self, length):
14715+        # We can be in three states here:
14716+        #   0. In the first segment of data. In this segment, we need to
14717+        #      return the original data until we get to the new data.
14718+        #      This is so that replacing data in the middle of an
14719+        #      existing segment works as expected.
14720+        #   1. After the first segment, before the last segment. In this
14721+        #      state, we delegate all reads to the underlying
14722+        #      IMutableUploadable.
14723+        #   2. Reading the last segment of data. If our replacement ends
14724+        #      in the middle of an existing segment, we need to pad the
14725+        #      replacement data with enough data from the end of segment
14726+        #      so that the replacement can happen.
14727+
14728+        # are we in state 0?
14729+        if self._read_marker < self._first_segment_offset:
14730+            # We need to read at least some data from the first segment
14731+            # to satisfy this read.
14732+            old_data_length = self._first_segment_offset - self._read_marker
14733+            if old_data_length > length:
14734+                old_data_length = length
14735+
14736+            new_data_length = length - old_data_length
14737+            old_data_end = old_data_length + self._read_marker
14738+            old_data = self._start[self._read_marker:old_data_end]
14739+            new_data = self._newdata.read(new_data_length)
14740+            new_data = "".join(new_data)
14741+            data = old_data + new_data
14742+
14743+        # are we in state 3?
14744+        elif self._read_marker + self._length > \
14745+                self._offset + self._newdata.get_size():
14746+            # We need to pad this read (for the last segment) with an
14747+            # appropriate amount of data from the old segment.
14748+            new_data_length = self._newdata.get_size() - self._read_marker
14749+            new_data = self._newdata.read(new_data_length)
14750+            new_data = "".join(data)
14751+            old_data_length = length - new_data_length
14752+            old_data_offset = new_data_length
14753+            old_data = self._end[old_data_offset:old_data_offset +
14754+                                 old_data_length]
14755+            data = new_data + old_data
14756+        else:
14757+            data = self._newdata.read(length)
14758+            data = "".join(data)
14759+
14760+        assert len(data) == length
14761+        self._read_marker += length
14762+        return data
14763+
14764+
14765+    def close(self):
14766+        pass
14767}
14768[mutable/retrieve.py: expose decoding and decrypting methods to callers
14769Kevan Carstensen <kevan@isnotajoke.com>**20100730234141
14770 Ignore-this: db66ee9019755f49388fe74049f26982
14771] hunk ./src/allmydata/mutable/retrieve.py 291
14772         return self._done_deferred
14773 
14774 
14775+    def decode(blocks_and_salts, segnum):
14776+        """
14777+        I am a helper method that the mutable file update process uses
14778+        as a shortcut to decode and decrypt the segments that it needs
14779+        to fetch in order to perform a file update. I take in a
14780+        collection of blocks and salts, and pick some of those to make a
14781+        segment with. I return the plaintext associated with that
14782+        segment.
14783+        """
14784+        self._setup_encoding_parameters()
14785+        d = self._decode_blocks(blocks_and_salts, segnum)
14786+        d.addCallback(self._decrypt_segment)
14787+        return d
14788+
14789+
14790     def _setup_encoding_parameters(self):
14791         """
14792         I set up the encoding parameters, including k, n, the number
14793[mutable/servermap.py: lay some groundwork for IWritable
14794Kevan Carstensen <kevan@isnotajoke.com>**20100730234229
14795 Ignore-this: c9ef2cace79db1e15937d030bcad20d9
14796] {
14797hunk ./src/allmydata/mutable/servermap.py 9
14798 from twisted.python import failure
14799 from foolscap.api import DeadReferenceError, RemoteException, eventually, \
14800                          fireEventually
14801-from allmydata.util import base32, hashutil, idlib, log
14802+from allmydata.util import base32, hashutil, idlib, log, deferredutil
14803 from allmydata.storage.server import si_b2a
14804 from allmydata.interfaces import IServermapUpdaterStatus
14805 from pycryptopp.publickey import rsa
14806hunk ./src/allmydata/mutable/servermap.py 124
14807         self.bad_shares = {} # maps (peerid,shnum) to old checkstring
14808         self.last_update_mode = None
14809         self.last_update_time = 0
14810+        self.update_data = {} # (verinfo,shnum) => data
14811 
14812     def copy(self):
14813         s = ServerMap()
14814hunk ./src/allmydata/mutable/servermap.py 340
14815         return False
14816 
14817 
14818+    def get_update_data_for_share_and_verinfo(self, shnum, verinfo):
14819+        """
14820+        I return the update data for the given shnum
14821+        """
14822+        update_data = self.update_data[shnum]
14823+        update_datum = [i[1] for i in update_data if i[0] == verinfo][0]
14824+        return update_datum
14825+
14826+
14827+    def set_update_data_for_share_and_verinfo(self, shnum, verinfo, data):
14828+        """
14829+        I record the block hash tree for the given shnum.
14830+        """
14831+        self.update_data.setdefault(shnum , []).append((verinfo, data))
14832+
14833+
14834 class ServermapUpdater:
14835     def __init__(self, filenode, storage_broker, monitor, servermap,
14836hunk ./src/allmydata/mutable/servermap.py 358
14837-                 mode=MODE_READ, add_lease=False):
14838+                 mode=MODE_READ, add_lease=False, update_range=None):
14839         """I update a servermap, locating a sufficient number of useful
14840         shares and remembering where they are located.
14841 
14842hunk ./src/allmydata/mutable/servermap.py 405
14843             # we use unpack_prefix_and_signature, so we need 1k
14844             self._read_size = 1000
14845         self._need_privkey = False
14846+
14847         if mode == MODE_WRITE and not self._node.get_privkey():
14848             self._need_privkey = True
14849         # check+repair: repair requires the privkey, so if we didn't happen
14850hunk ./src/allmydata/mutable/servermap.py 412
14851         # to ask for it during the check, we'll have problems doing the
14852         # publish.
14853 
14854+        self.fetch_update_data = False
14855+        if mode == MODE_WRITE and update_range:
14856+            # We're updating the servermap in preparation for an
14857+            # in-place file update, so we need to fetch some additional
14858+            # data from each share that we find.
14859+            assert len(update_range) == 2
14860+
14861+            self.start_segment = update_range[0]
14862+            self.end_segment = update_range[1]
14863+            self.fetch_update_data = True
14864+
14865         prefix = si_b2a(self._storage_index)[:5]
14866         self._log_number = log.msg(format="SharemapUpdater(%(si)s): starting (%(mode)s)",
14867                                    si=prefix, mode=mode)
14868hunk ./src/allmydata/mutable/servermap.py 643
14869                       level=log.NOISY)
14870         now = time.time()
14871         elapsed = now - started
14872-        self._queries_outstanding.discard(peerid)
14873-        self._servermap.reachable_peers.add(peerid)
14874-        self._must_query.discard(peerid)
14875-        self._queries_completed += 1
14876+        def _done_processing(ignored=None):
14877+            self._queries_outstanding.discard(peerid)
14878+            self._servermap.reachable_peers.add(peerid)
14879+            self._must_query.discard(peerid)
14880+            self._queries_completed += 1
14881         if not self._running:
14882             self.log("but we're not running, so we'll ignore it", parent=lp,
14883                      level=log.NOISY)
14884hunk ./src/allmydata/mutable/servermap.py 651
14885+            _done_processing()
14886             self._status.add_per_server_time(peerid, "late", started, elapsed)
14887             return
14888         self._status.add_per_server_time(peerid, "query", started, elapsed)
14889hunk ./src/allmydata/mutable/servermap.py 679
14890             #     public key. We use this to validate the signature.
14891             if not self._node.get_pubkey():
14892                 # fetch and set the public key.
14893-                d = reader.get_verification_key()
14894+                d = reader.get_verification_key(queue=True)
14895                 d.addCallback(lambda results, shnum=shnum, peerid=peerid:
14896                     self._try_to_set_pubkey(results, peerid, shnum, lp))
14897                 # XXX: Make self._pubkey_query_failed?
14898hunk ./src/allmydata/mutable/servermap.py 688
14899             else:
14900                 # we already have the public key.
14901                 d = defer.succeed(None)
14902+
14903             # Neither of these two branches return anything of
14904             # consequence, so the first entry in our deferredlist will
14905             # be None.
14906hunk ./src/allmydata/mutable/servermap.py 705
14907             #   to get the version information. In MDMF, this lives at
14908             #   the end of the share, so unless the file is quite small,
14909             #   we'll need to do a remote fetch to get it.
14910-            d3 = reader.get_signature()
14911+            d3 = reader.get_signature(queue=True)
14912             d3.addErrback(lambda error, shnum=shnum, peerid=peerid:
14913                 self._got_corrupt_share(error, shnum, peerid, data, lp))
14914             #  Once we have all three of these responses, we can move on
14915hunk ./src/allmydata/mutable/servermap.py 714
14916             # Does the node already have a privkey? If not, we'll try to
14917             # fetch it here.
14918             if self._need_privkey:
14919-                d4 = reader.get_encprivkey()
14920+                d4 = reader.get_encprivkey(queue=True)
14921                 d4.addCallback(lambda results, shnum=shnum, peerid=peerid:
14922                     self._try_to_validate_privkey(results, peerid, shnum, lp))
14923                 d4.addErrback(lambda error, shnum=shnum, peerid=peerid:
14924hunk ./src/allmydata/mutable/servermap.py 722
14925             else:
14926                 d4 = defer.succeed(None)
14927 
14928-            dl = defer.DeferredList([d, d2, d3, d4])
14929+
14930+            if self.fetch_update_data:
14931+                # fetch the block hash tree and first + last segment, as
14932+                # configured earlier.
14933+                # Then set them in wherever we happen to want to set
14934+                # them.
14935+                ds = []
14936+                # XXX: We do this above, too. Is there a good way to
14937+                # make the two routines share the value without
14938+                # introducing more roundtrips?
14939+                ds.append(reader.get_verinfo())
14940+                ds.append(reader.get_blockhashes(queue=True))
14941+                ds.append(reader.get_block_and_salt(self.start_segment,
14942+                                                    queue=True))
14943+                ds.append(reader.get_block_and_salt(self.end_segment,
14944+                                                    queue=True))
14945+                d5 = deferredutil.gatherResults(ds)
14946+                d5.addCallback(self._got_update_results_one_share, shnum)
14947+            else:
14948+                d5 = defer.succeed(None)
14949+
14950+            dl = defer.DeferredList([d, d2, d3, d4, d5])
14951+            reader.flush()
14952             dl.addCallback(lambda results, shnum=shnum, peerid=peerid:
14953                 self._got_signature_one_share(results, shnum, peerid, lp))
14954             dl.addErrback(lambda error, shnum=shnum, data=data:
14955hunk ./src/allmydata/mutable/servermap.py 764
14956         # that we returned to our caller to fire, which tells them that
14957         # they have a complete servermap, and that we won't be touching
14958         # the servermap anymore.
14959+        dl.addCallback(_done_processing)
14960         dl.addCallback(self._check_for_done)
14961         dl.addErrback(self._fatal_error)
14962         # all done!
14963hunk ./src/allmydata/mutable/servermap.py 799
14964                  peerid=idlib.shortnodeid_b2a(peerid),
14965                  level=log.NOISY,
14966                  parent=lp)
14967-        _, verinfo, signature, __ = results
14968+        _, verinfo, signature, __, ___ = results
14969         (seqnum,
14970          root_hash,
14971          saltish,
14972hunk ./src/allmydata/mutable/servermap.py 864
14973         return verinfo
14974 
14975 
14976+    def _got_update_results_one_share(self, results, share):
14977+        """
14978+        I record the update results in results.
14979+        """
14980+        assert len(results) == 4
14981+        verinfo, blockhashes, start, end = results
14982+        update_data = (blockhashes, start, end)
14983+        self._servermap.set_update_data_for_share_and_verinfo(share,
14984+                                                              verinfo,
14985+                                                              update_data)
14986+
14987+
14988     def _deserialize_pubkey(self, pubkey_s):
14989         verifier = rsa.create_verifying_key_from_string(pubkey_s)
14990         return verifier
14991}
14992[mutable: fix bugs that prevented the update tests from working
14993Kevan Carstensen <kevan@isnotajoke.com>**20100802224821
14994 Ignore-this: e87a1c81f7ecf9643248554a820dc62d
14995] {
14996hunk ./src/allmydata/mutable/filenode.py 10
14997 from allmydata.interfaces import IMutableFileNode, ICheckable, ICheckResults, \
14998      NotEnoughSharesError, MDMF_VERSION, SDMF_VERSION, IMutableUploadable, \
14999      IMutableFileVersion, IWritable
15000-from allmydata.util import hashutil, log, consumer, deferredutil
15001+from allmydata.util import hashutil, log, consumer, deferredutil, mathutil
15002 from allmydata.util.assertutil import precondition
15003 from allmydata.uri import WriteableSSKFileURI, ReadonlySSKFileURI
15004 from allmydata.monitor import Monitor
15005hunk ./src/allmydata/mutable/filenode.py 961
15006         self._size = size
15007         return res
15008 
15009-    def _update(self, data, offset):
15010+    def update(self, data, offset):
15011         """
15012         Do an update of this mutable file version by inserting data at
15013         offset within the file. If offset is the EOF, this is an append
15014hunk ./src/allmydata/mutable/filenode.py 986
15015         completed.
15016         """
15017         d = self._do_update_update(data, offset)
15018-        d.addCallback(self._decode_and_decrypt_segments)
15019-        d.addCallback(self._build_uploadable_and_finish)
15020+        d.addCallback(self._decode_and_decrypt_segments, data, offset)
15021+        d.addCallback(self._build_uploadable_and_finish, data, offset)
15022         return d
15023 
15024 
15025hunk ./src/allmydata/mutable/filenode.py 1016
15026             end_data = offset + data.get_size()
15027             end_segment = mathutil.div_ceil(end_data, DEFAULT_MAX_SEGMENT_SIZE)
15028             end_segment -= 1
15029+        self._start_segment = start_segment
15030+        self._end_segment = end_segment
15031 
15032         # Now ask for the servermap to be updated in MODE_WRITE with
15033         # this update range.
15034hunk ./src/allmydata/mutable/filenode.py 1021
15035-        u = ServermapUpdater(self, self._storage_broker, Monitor(),
15036+        u = ServermapUpdater(self._node, self._storage_broker, Monitor(),
15037                              self._servermap,
15038                              mode=MODE_WRITE,
15039                              update_range=(start_segment, end_segment))
15040hunk ./src/allmydata/mutable/filenode.py 1036
15041         used by the new uploadable. I return a Deferred that fires with
15042         the segments.
15043         """
15044-        r = Retrieve(self._node, self._servermap, self._version, fetch_privkey)
15045+        r = Retrieve(self._node, self._servermap, self._version)
15046         # decode: takes in our blocks and salts from the servermap,
15047         # returns a Deferred that fires with the corresponding plaintext
15048         # segments. Does not download -- simply takes advantage of
15049hunk ./src/allmydata/mutable/filenode.py 1046
15050         # XXX: If the methods in the servermap don't work as
15051         # abstractions, you should rewrite them instead of going around
15052         # them.
15053-        data = sm.update_data
15054-        data = [data[1] for i in update_data if i[0] == self._verinfo]
15055-        start_seg_data = [d[0] for d in data]
15056-        end_seg_data = [d[1] for d in data]
15057-        d1 = r.decode(start_seg_data)
15058-        d2 = r.decode(end_seg_data)
15059-        return deferredutil.gatherResults([d1, d2])
15060+        update_data = sm.update_data
15061+        start_segments = {} # shnum -> start segment
15062+        end_segments = {} # shnum -> end segment
15063+        blockhashes = {} # shnum -> blockhash tree
15064+        for (shnum, data) in update_data.iteritems():
15065+            data = [d[1] for d in data if d[0] == self._version]
15066+
15067+            # Every data entry in our list should now be share shnum for
15068+            # a particular version of the mutable file, so all of the
15069+            # entries should be identical.
15070+            datum = data[0]
15071+            assert filter(lambda x: x != datum, data) == []
15072+
15073+            blockhashes[shnum] = datum[0]
15074+            start_segments[shnum] = datum[1]
15075+            end_segments[shnum] = datum[2]
15076+
15077+        d1 = r.decode(start_segments, self._start_segment)
15078+        d2 = r.decode(end_segments, self._end_segment)
15079+        d3 = defer.succeed(blockhashes)
15080+        return deferredutil.gatherResults([d1, d2, d3])
15081 
15082 
15083hunk ./src/allmydata/mutable/filenode.py 1069
15084-    def _build_uploadable_and_finish(self, segments, data, offset):
15085+    def _build_uploadable_and_finish(self, segments_and_bht, data, offset):
15086         """
15087         After the process has the plaintext segments, I build the
15088         TransformingUploadable that the publisher will eventually
15089hunk ./src/allmydata/mutable/filenode.py 1079
15090         """
15091         u = TransformingUploadable(data, offset,
15092                                    DEFAULT_MAX_SEGMENT_SIZE,
15093-                                   segments[0],
15094-                                   segments[1])
15095+                                   segments_and_bht[0],
15096+                                   segments_and_bht[1])
15097         p = Publish(self._node, self._storage_broker, self._servermap)
15098hunk ./src/allmydata/mutable/filenode.py 1082
15099-        return p.update(data, offset, blockhashes)
15100+        return p.update(data, offset, segments_and_bht[2])
15101hunk ./src/allmydata/mutable/publish.py 168
15102 
15103         self.log("starting update")
15104         self.log("adding new data of length %d at offset %d" % \
15105-                    data.get_size(), offset)
15106+                    (data.get_size(), offset))
15107         self.log("new data length is %d" % self.datalength)
15108         self._status.set_size(self.datalength)
15109         self._status.set_status("Started")
15110hunk ./src/allmydata/mutable/publish.py 1254
15111 
15112 
15113     def get_size(self):
15114-        # TODO
15115-        pass
15116+        return self._offset + self._newdata.get_size()
15117 
15118 
15119     def read(self, length):
15120hunk ./src/allmydata/mutable/retrieve.py 145
15121         self.readers = {}
15122         self._paused = False
15123         self._paused_deferred = None
15124+        self._offset = None
15125+        self._read_length = None
15126 
15127 
15128     def get_status(self):
15129hunk ./src/allmydata/mutable/retrieve.py 293
15130         return self._done_deferred
15131 
15132 
15133-    def decode(blocks_and_salts, segnum):
15134+    def decode(self, blocks_and_salts, segnum):
15135         """
15136         I am a helper method that the mutable file update process uses
15137         as a shortcut to decode and decrypt the segments that it needs
15138hunk ./src/allmydata/mutable/retrieve.py 302
15139         segment with. I return the plaintext associated with that
15140         segment.
15141         """
15142+        # shnum => block hash tree. Unusued, but setup_encoding_parameters will
15143+        # want to set this.
15144+        # XXX: Make it so that it won't set this if we're just decoding.
15145+        self._block_hash_trees = {}
15146         self._setup_encoding_parameters()
15147hunk ./src/allmydata/mutable/retrieve.py 307
15148+        # This is the form expected by decode.
15149+        blocks_and_salts = blocks_and_salts.items()
15150+        blocks_and_salts = [(True, [d]) for d in blocks_and_salts]
15151+
15152         d = self._decode_blocks(blocks_and_salts, segnum)
15153         d.addCallback(self._decrypt_segment)
15154         return d
15155hunk ./src/allmydata/mutable/servermap.py 870
15156         """
15157         assert len(results) == 4
15158         verinfo, blockhashes, start, end = results
15159+        (seqnum,
15160+         root_hash,
15161+         saltish,
15162+         segsize,
15163+         datalen,
15164+         k,
15165+         n,
15166+         prefix,
15167+         offsets) = verinfo
15168+        offsets_tuple = tuple( [(key,value) for key,value in offsets.items()] )
15169+
15170+        # XXX: This should be done for us in the method, so
15171+        # presumably you can go in there and fix it.
15172+        verinfo = (seqnum,
15173+                   root_hash,
15174+                   saltish,
15175+                   segsize,
15176+                   datalen,
15177+                   k,
15178+                   n,
15179+                   prefix,
15180+                   offsets_tuple)
15181+
15182         update_data = (blockhashes, start, end)
15183         self._servermap.set_update_data_for_share_and_verinfo(share,
15184                                                               verinfo,
15185}
15186[mutable: fix some bugs in update logic.
15187Kevan Carstensen <kevan@isnotajoke.com>**20100806000338
15188 Ignore-this: 6fd3040efd6de8b2b80e7bb9f90f8003
15189] {
15190hunk ./src/allmydata/mutable/filenode.py 10
15191 from allmydata.interfaces import IMutableFileNode, ICheckable, ICheckResults, \
15192      NotEnoughSharesError, MDMF_VERSION, SDMF_VERSION, IMutableUploadable, \
15193      IMutableFileVersion, IWritable
15194+from allmydata import hashtree
15195 from allmydata.util import hashutil, log, consumer, deferredutil, mathutil
15196 from allmydata.util.assertutil import precondition
15197 from allmydata.uri import WriteableSSKFileURI, ReadonlySSKFileURI
15198hunk ./src/allmydata/mutable/filenode.py 986
15199         offset. I return a Deferred that fires when this has been
15200         completed.
15201         """
15202+        # We have two cases here:
15203+        # 1. The new data will add few enough segments so that it does
15204+        #    not cross into the next power-of-two boundary.
15205+        # 2. It doesn't.
15206+        #
15207+        # In the former case, we can modify the file in place. In the
15208+        # latter case, we need to re-encode the file.
15209+        new_size = data.get_size() + offset
15210+        old_size = self.get_size()
15211+        segment_size = self._version[3]
15212+        num_old_segments = mathutil.div_ceil(old_size,
15213+                                             segment_size)
15214+        num_new_segments = mathutil.div_ceil(new_size,
15215+                                             segment_size)
15216+        log.msg("got %d old segments, %d new segments" % \
15217+                        (num_old_segments, num_new_segments))
15218+
15219+        # We also do a whole file re-encode if the file is an SDMF file.
15220+        if self._version[2]: # version[2] == SDMF salt, which MDMF lacks
15221+            log.msg("doing re-encode instead of in-place update")
15222+            return self._do_modify_update(data, offset)
15223+
15224+        log.msg("updating in place")
15225         d = self._do_update_update(data, offset)
15226         d.addCallback(self._decode_and_decrypt_segments, data, offset)
15227         d.addCallback(self._build_uploadable_and_finish, data, offset)
15228hunk ./src/allmydata/mutable/filenode.py 1015
15229         return d
15230 
15231 
15232+    def _do_modify_update(self, data, offset):
15233+        """
15234+        I perform a file update by modifying the contents of the file
15235+        after downloading it, then reuploading it. I am less efficient
15236+        than _do_update_update, but am necessary for certain updates.
15237+        """
15238+        def m(old, servermap, first_time):
15239+            start = offset
15240+            rest = offset + data.get_size()
15241+            new = old[:start]
15242+            new += "".join(data.read(data.get_size()))
15243+            new += old[rest:]
15244+            return MutableData(new)
15245+        return self._modify(m, None)
15246+
15247+
15248     def _do_update_update(self, data, offset):
15249         """
15250         I start the Servermap update that gets us the data we need to
15251hunk ./src/allmydata/mutable/filenode.py 1118
15252         completed without issue.
15253         """
15254         u = TransformingUploadable(data, offset,
15255-                                   DEFAULT_MAX_SEGMENT_SIZE,
15256+                                   self._version[3],
15257                                    segments_and_bht[0],
15258                                    segments_and_bht[1])
15259         p = Publish(self._node, self._storage_broker, self._servermap)
15260hunk ./src/allmydata/mutable/filenode.py 1122
15261-        return p.update(data, offset, segments_and_bht[2])
15262+        return p.update(u, offset, segments_and_bht[2], self._version)
15263hunk ./src/allmydata/mutable/publish.py 133
15264         return log.msg(*args, **kwargs)
15265 
15266 
15267-    def update(self, data, offset, blockhashes):
15268+    def update(self, data, offset, blockhashes, version):
15269         """
15270         I replace the contents of this file with the contents of data,
15271         starting at offset. I return a Deferred that fires with None
15272hunk ./src/allmydata/mutable/publish.py 158
15273 
15274         # XXX: Use the MutableFileVersion instead.
15275         self.datalength = self._node.get_size()
15276-        if offset + data.get_size() > self.datalength:
15277-            self.datalength = offset + data.get_size()
15278+        if data.get_size() > self.datalength:
15279+            self.datalength = data.get_size()
15280 
15281         if self.datalength >= DEFAULT_MAX_SEGMENT_SIZE:
15282             self._version = MDMF_VERSION
15283hunk ./src/allmydata/mutable/publish.py 310
15284         # them in place as publishing happens.
15285         self.blockhashes = {} # (shnum, [blochashes])
15286         for (i, bht) in blockhashes.iteritems():
15287-            self.blockhashes[i] = bht
15288+            # We need to extract the leaves from our old hash tree.
15289+            old_segcount = mathutil.div_ceil(version[4],
15290+                                             version[3])
15291+            h = hashtree.IncompleteHashTree(old_segcount)
15292+            bht = dict(enumerate(bht))
15293+            h.set_hashes(bht)
15294+            leaves = h[h.get_leaf_index(0):]
15295+            for j in xrange(self.num_segments - len(leaves)):
15296+                leaves.append(None)
15297+
15298+            assert len(leaves) >= self.num_segments
15299+            self.blockhashes[i] = leaves
15300+            # This list will now be the leaves that were set during the
15301+            # initial upload + enough empty hashes to make it a
15302+            # power-of-two. If we exceed a power of two boundary, we
15303+            # should be encoding the file over again, and should not be
15304+            # here. So, we have
15305+            #assert len(self.blockhashes[i]) == \
15306+            #    hashtree.roundup_pow2(self.num_segments), \
15307+            #        len(self.blockhashes[i])
15308+            # XXX: Except this doesn't work. Figure out why.
15309 
15310         # These are filled in later, after we've modified the block hash
15311         # tree suitably.
15312hunk ./src/allmydata/mutable/publish.py 555
15313                                                   segment_size)
15314             self.starting_segment = mathutil.div_ceil(offset,
15315                                                       segment_size)
15316+            self.starting_segment -= 1
15317             if offset == 0:
15318                 self.starting_segment = 0
15319 
15320hunk ./src/allmydata/mutable/publish.py 574
15321 
15322         if segment_size and self.datalength:
15323             self.tail_segment_size = self.datalength % segment_size
15324+            self.log("got tail segment size %d" % self.tail_segment_size)
15325         else:
15326             self.tail_segment_size = 0
15327 
15328hunk ./src/allmydata/mutable/publish.py 602
15329         self.end_segment = self.num_segments - 1
15330         # Now figure out where the last segment should be.
15331         if self.data.get_size() != self.datalength:
15332-            end = offset + self.data.get_size()
15333+            end = self.data.get_size()
15334             self.end_segment = mathutil.div_ceil(end,
15335                                                  segment_size)
15336             self.end_segment -= 1
15337hunk ./src/allmydata/mutable/publish.py 685
15338         # XXX: This is dumb. Why return a list?
15339         data = "".join(data)
15340 
15341-        assert len(data) == segsize
15342+        assert len(data) == segsize, len(data)
15343 
15344         salt = os.urandom(16)
15345 
15346hunk ./src/allmydata/mutable/publish.py 737
15347             else:
15348                 hashed = sharedata
15349             block_hash = hashutil.block_hash(hashed)
15350+            old_hash = self.blockhashes[shareid][segnum]
15351             self.blockhashes[shareid][segnum] = block_hash
15352hunk ./src/allmydata/mutable/publish.py 739
15353-
15354             # find the writer for this share
15355             writer = self.writers[shareid]
15356             writer.put_block(sharedata, segnum, salt)
15357hunk ./src/allmydata/mutable/publish.py 776
15358             self.blockhashes[shnum] = list(t)
15359             # set the leaf for future use.
15360             self.sharehash_leaves[shnum] = t[0]
15361+
15362             writer = self.writers[shnum]
15363             writer.put_blockhashes(self.blockhashes[shnum])
15364 
15365hunk ./src/allmydata/mutable/publish.py 1200
15366         # reported to the uploader.
15367         self._filehandle.seek(0)
15368 
15369+        # We have not yet read anything, so our position is 0.
15370+        self._marker = 0
15371+
15372 
15373     def get_size(self):
15374         """
15375hunk ./src/allmydata/mutable/publish.py 1223
15376         return self._size
15377 
15378 
15379+    def pos(self):
15380+        """
15381+        I return the position of my read marker -- i.e., how much data I
15382+        have already read and returned to callers.
15383+        """
15384+        return self._marker
15385+
15386+
15387     def read(self, length):
15388         """
15389         I return some data (up to length bytes) from my filehandle.
15390hunk ./src/allmydata/mutable/publish.py 1239
15391         for example, if I am asked to read beyond the end of a file, or
15392         an error occurs.
15393         """
15394-        return [self._filehandle.read(length)]
15395+        results = self._filehandle.read(length)
15396+        self._marker += len(results)
15397+        return [results]
15398 
15399 
15400     def close(self):
15401hunk ./src/allmydata/mutable/publish.py 1276
15402 
15403     def __init__(self, data, offset, segment_size, start, end):
15404         assert IMutableUploadable.providedBy(data)
15405-        # offset == data.get_size() means that we're appending.
15406-        assert offset <= data.get_size()
15407 
15408         self._newdata = data
15409         self._offset = offset
15410hunk ./src/allmydata/mutable/publish.py 1284
15411         self._end = end
15412 
15413         self._read_marker = 0
15414+
15415         self._first_segment_offset = offset % segment_size
15416 
15417hunk ./src/allmydata/mutable/publish.py 1287
15418+        num = self.log("TransformingUploadable: starting", parent=None)
15419+        self._log_number = num
15420+        self.log("got fso: %d" % self._first_segment_offset)
15421+        self.log("got offset: %d" % self._offset)
15422+
15423+
15424+    def log(self, *args, **kwargs):
15425+        if 'parent' not in kwargs:
15426+            kwargs['parent'] = self._log_number
15427+        if "facility" not in kwargs:
15428+            kwargs["facility"] = "tahoe.mutable.transforminguploadable"
15429+        return log.msg(*args, **kwargs)
15430+
15431 
15432     def get_size(self):
15433         return self._offset + self._newdata.get_size()
15434hunk ./src/allmydata/mutable/publish.py 1306
15435 
15436 
15437     def read(self, length):
15438-        # We can be in three states here:
15439-        #   0. In the first segment of data. In this segment, we need to
15440-        #      return the original data until we get to the new data.
15441-        #      This is so that replacing data in the middle of an
15442-        #      existing segment works as expected.
15443-        #   1. After the first segment, before the last segment. In this
15444-        #      state, we delegate all reads to the underlying
15445-        #      IMutableUploadable.
15446-        #   2. Reading the last segment of data. If our replacement ends
15447-        #      in the middle of an existing segment, we need to pad the
15448-        #      replacement data with enough data from the end of segment
15449-        #      so that the replacement can happen.
15450+        # We can get data from 3 sources here.
15451+        #   1. The first of the segments provided to us.
15452+        #   2. The data that we're replacing things with.
15453+        #   3. The last of the segments provided to us.
15454 
15455         # are we in state 0?
15456hunk ./src/allmydata/mutable/publish.py 1312
15457-        if self._read_marker < self._first_segment_offset:
15458-            # We need to read at least some data from the first segment
15459-            # to satisfy this read.
15460-            old_data_length = self._first_segment_offset - self._read_marker
15461+        self.log("reading %d bytes" % length)
15462+
15463+        old_start_data = ""
15464+        old_data_length = self._first_segment_offset - self._read_marker
15465+        if old_data_length > 0:
15466             if old_data_length > length:
15467                 old_data_length = length
15468hunk ./src/allmydata/mutable/publish.py 1319
15469+            self.log("returning %d bytes of old start data" % old_data_length)
15470 
15471hunk ./src/allmydata/mutable/publish.py 1321
15472-            new_data_length = length - old_data_length
15473             old_data_end = old_data_length + self._read_marker
15474hunk ./src/allmydata/mutable/publish.py 1322
15475-            old_data = self._start[self._read_marker:old_data_end]
15476-            new_data = self._newdata.read(new_data_length)
15477-            new_data = "".join(new_data)
15478-            data = old_data + new_data
15479+            old_start_data = self._start[self._read_marker:old_data_end]
15480+            length -= old_data_length
15481+        else:
15482+            # otherwise calculations later get screwed up.
15483+            old_data_length = 0
15484 
15485hunk ./src/allmydata/mutable/publish.py 1328
15486-        # are we in state 3?
15487-        elif self._read_marker + self._length > \
15488-                self._offset + self._newdata.get_size():
15489-            # We need to pad this read (for the last segment) with an
15490-            # appropriate amount of data from the old segment.
15491-            new_data_length = self._newdata.get_size() - self._read_marker
15492-            new_data = self._newdata.read(new_data_length)
15493-            new_data = "".join(data)
15494-            old_data_length = length - new_data_length
15495-            old_data_offset = new_data_length
15496-            old_data = self._end[old_data_offset:old_data_offset +
15497-                                 old_data_length]
15498-            data = new_data + old_data
15499-        else:
15500-            data = self._newdata.read(length)
15501-            data = "".join(data)
15502+        # Is there enough new data to satisfy this read? If not, we need
15503+        # to pad the end of the data with data from our last segment.
15504+        old_end_length = length - \
15505+            (self._newdata.get_size() - self._newdata.pos())
15506+        old_end_data = ""
15507+        if old_end_length > 0:
15508+            self.log("reading %d bytes of old end data" % old_end_length)
15509+
15510+            # TODO: We're not explicitly checking for tail segment size
15511+            # here. Is that a problem?
15512+            old_data_offset = (length - old_end_length + \
15513+                               old_data_length) % self._segment_size
15514+            self.log("reading at offset %d" % old_data_offset)
15515+            old_end = old_data_offset + old_end_length
15516+            old_end_data = self._end[old_data_offset:old_end]
15517+            length -= old_end_length
15518+            assert length == self._newdata.get_size() - self._newdata.pos()
15519+
15520+        self.log("reading %d bytes of new data" % length)
15521+        new_data = self._newdata.read(length)
15522+        new_data = "".join(new_data)
15523 
15524hunk ./src/allmydata/mutable/publish.py 1350
15525-        assert len(data) == length
15526-        self._read_marker += length
15527-        return data
15528+        self._read_marker += len(old_start_data + new_data + old_end_data)
15529 
15530hunk ./src/allmydata/mutable/publish.py 1352
15531+        return old_start_data + new_data + old_end_data
15532 
15533     def close(self):
15534         pass
15535}
15536[test/test_mutable.py: add tests for updating behavior
15537Kevan Carstensen <kevan@isnotajoke.com>**20100806000408
15538 Ignore-this: d7860e9b44744f95ea20a706b921ec54
15539] {
15540hunk ./src/allmydata/test/test_mutable.py 8
15541 from twisted.internet import defer, reactor
15542 from allmydata import uri, client
15543 from allmydata.nodemaker import NodeMaker
15544-from allmydata.util import base32, consumer
15545+from allmydata.util import base32, consumer, mathutil
15546 from allmydata.util.hashutil import tagged_hash, ssk_writekey_hash, \
15547      ssk_pubkey_fingerprint_hash
15548 from allmydata.util.deferredutil import gatherResults
15549hunk ./src/allmydata/test/test_mutable.py 28
15550      NotEnoughServersError, CorruptShareError
15551 from allmydata.mutable.retrieve import Retrieve
15552 from allmydata.mutable.publish import Publish, MutableFileHandle, \
15553-                                      MutableData
15554+                                      MutableData, \
15555+                                      DEFAULT_MAX_SEGMENT_SIZE
15556 from allmydata.mutable.servermap import ServerMap, ServermapUpdater
15557 from allmydata.mutable.layout import unpack_header, unpack_share, \
15558                                      MDMFSlotReadProxy
15559hunk ./src/allmydata/test/test_mutable.py 2868
15560         d.addCallback(lambda data:
15561             self.failUnlessEqual(data, self.small_data))
15562         return d
15563+
15564+
15565+class Update(GridTestMixin, unittest.TestCase, testutil.ShouldFailMixin):
15566+    def setUp(self):
15567+        GridTestMixin.setUp(self)
15568+        self.basedir = self.mktemp()
15569+        self.set_up_grid()
15570+        self.c = self.g.clients[0]
15571+        self.nm = self.c.nodemaker
15572+        self.data = "test data" * 100000 # about 900 KiB; MDMF
15573+        self.small_data = "test data" * 10 # about 90 B; SDMF
15574+        return self.do_upload()
15575+
15576+
15577+    def do_upload(self):
15578+        d1 = self.nm.create_mutable_file(MutableData(self.data),
15579+                                         version=MDMF_VERSION)
15580+        d2 = self.nm.create_mutable_file(MutableData(self.small_data))
15581+        dl = gatherResults([d1, d2])
15582+        def _then((n1, n2)):
15583+            assert isinstance(n1, MutableFileNode)
15584+            assert isinstance(n2, MutableFileNode)
15585+
15586+            self.mdmf_node = n1
15587+            self.sdmf_node = n2
15588+        dl.addCallback(_then)
15589+        return dl
15590+
15591+
15592+    def test_append(self):
15593+        # We should be able to append data to the middle of a mutable
15594+        # file and get what we expect.
15595+        new_data = self.data + "appended"
15596+        d = self.mdmf_node.get_best_mutable_version()
15597+        d.addCallback(lambda mv:
15598+            mv.update(MutableData("appended"), len(self.data)))
15599+        d.addCallback(lambda ignored:
15600+            self.mdmf_node.download_best_version())
15601+        d.addCallback(lambda results:
15602+            self.failUnlessEqual(results, new_data))
15603+        return d
15604+
15605+
15606+    def test_replace(self):
15607+        # We should be able to replace data in the middle of a mutable
15608+        # file and get what we expect back.
15609+        new_data = self.data[:100]
15610+        new_data += "appended"
15611+        new_data += self.data[108:]
15612+        d = self.mdmf_node.get_best_mutable_version()
15613+        d.addCallback(lambda mv:
15614+            mv.update(MutableData("appended"), 100))
15615+        d.addCallback(lambda ignored:
15616+            self.mdmf_node.download_best_version())
15617+        d.addCallback(lambda results:
15618+            self.failUnlessEqual(results, new_data))
15619+        return d
15620+
15621+
15622+    def test_replace_and_extend(self):
15623+        # We should be able to replace data in the middle of a mutable
15624+        # file and extend that mutable file and get what we expect.
15625+        new_data = self.data[:100]
15626+        new_data += "modified " * 100000
15627+        d = self.mdmf_node.get_best_mutable_version()
15628+        d.addCallback(lambda mv:
15629+            mv.update(MutableData("modified " * 100000), 100))
15630+        d.addCallback(lambda ignored:
15631+            self.mdmf_node.download_best_version())
15632+        d.addCallback(lambda results:
15633+            self.failUnlessEqual(results, new_data))
15634+        return d
15635+
15636+
15637+    def test_append_power_of_two(self):
15638+        # If we attempt to extend a mutable file so that its segment
15639+        # count crosses a power-of-two boundary, the update operation
15640+        # should know how to reencode the file.
15641+
15642+        # Note that the data populating self.mdmf_node is about 900 KiB
15643+        # long -- this is 7 segments in the default segment size. So we
15644+        # need to add 2 segments worth of data to push it over a
15645+        # power-of-two boundary.
15646+        segment = "a" * DEFAULT_MAX_SEGMENT_SIZE
15647+        new_data = self.data + (segment * 2)
15648+        d = self.mdmf_node.get_best_mutable_version()
15649+        d.addCallback(lambda mv:
15650+            mv.update(MutableData(segment * 2), len(self.data)))
15651+        d.addCallback(lambda ignored:
15652+            self.mdmf_node.download_best_version())
15653+        d.addCallback(lambda results:
15654+            self.failUnlessEqual(results, new_data))
15655+        return d
15656+
15657+
15658+    def test_update_sdmf(self):
15659+        # Running update on a single-segment file should still work.
15660+        new_data = self.small_data + "appended"
15661+        d = self.sdmf_node.get_best_mutable_version()
15662+        d.addCallback(lambda mv:
15663+            mv.update(MutableData("appended"), len(self.small_data)))
15664+        d.addCallback(lambda ignored:
15665+            self.sdmf_node.download_best_version())
15666+        d.addCallback(lambda results:
15667+            self.failUnlessEqual(results, new_data))
15668+        return d
15669+
15670+    def test_replace_in_last_segment(self):
15671+        # The wrapper should know how to handle the tail segment
15672+        # appropriately.
15673+        replace_offset = len(self.data) - 100
15674+        new_data = self.data[:replace_offset] + "replaced"
15675+        rest_offset = replace_offset + len("replaced")
15676+        new_data += self.data[rest_offset:]
15677+        d = self.mdmf_node.get_best_mutable_version()
15678+        d.addCallback(lambda mv:
15679+            mv.update(MutableData("replaced"), replace_offset))
15680+        d.addCallback(lambda ignored:
15681+            self.mdmf_node.download_best_version())
15682+        d.addCallback(lambda results:
15683+            self.failUnlessEqual(results, new_data))
15684+        return d
15685+
15686+
15687+    def test_multiple_segment_replace(self):
15688+        replace_offset = 2 * DEFAULT_MAX_SEGMENT_SIZE
15689+        new_data = self.data[:replace_offset]
15690+        new_segment = "a" * DEFAULT_MAX_SEGMENT_SIZE
15691+        new_data += 2 * new_segment
15692+        new_data += "replaced"
15693+        rest_offset = len(new_data)
15694+        new_data += self.data[rest_offset:]
15695+        d = self.mdmf_node.get_best_mutable_version()
15696+        d.addCallback(lambda mv:
15697+            mv.update(MutableData((2 * new_segment) + "replaced"),
15698+                      replace_offset))
15699+        d.addCallback(lambda ignored:
15700+            self.mdmf_node.download_best_version())
15701+        d.addCallback(lambda results:
15702+            self.failUnlessEqual(results, new_data))
15703+        return d
15704}
15705[web: add a handler to PUT that can update MDMF files
15706Kevan Carstensen <kevan@isnotajoke.com>**20100806234849
15707 Ignore-this: 3d4cd8a36b66802236c007dc52260fab
15708] {
15709hunk ./src/allmydata/web/common.py 34
15710     else:
15711         return boolean_of_arg(replace)
15712 
15713+
15714+def parse_offset_arg(offset):
15715+    # XXX: This will raise a ValueError when invoked on something that
15716+    # is not an integer. Is that okay? Or do we want a better error
15717+    # message? Since this call is going to be used by programmers and
15718+    # their tools rather than users (through the wui), it is not
15719+    # inconsistent to return that, I guess.
15720+    offset = int(offset)
15721+    return offset
15722+
15723+
15724 def get_root(ctx_or_req):
15725     req = IRequest(ctx_or_req)
15726     # the addSlash=True gives us one extra (empty) segment
15727hunk ./src/allmydata/web/filenode.py 17
15728 
15729 from allmydata.web.common import text_plain, WebError, RenderMixin, \
15730      boolean_of_arg, get_arg, should_create_intermediate_directories, \
15731-     MyExceptionHandler, parse_replace_arg
15732+     MyExceptionHandler, parse_replace_arg, parse_offset_arg
15733 from allmydata.web.check_results import CheckResults, \
15734      CheckAndRepairResults, LiteralCheckResults
15735 from allmydata.web.info import MoreInfo
15736hunk ./src/allmydata/web/filenode.py 199
15737         req = IRequest(ctx)
15738         t = get_arg(req, "t", "").strip()
15739         replace = parse_replace_arg(get_arg(req, "replace", "true"))
15740+        offset = parse_offset_arg(get_arg(req, "offset", -1))
15741 
15742         if not t:
15743hunk ./src/allmydata/web/filenode.py 202
15744-            if self.node.is_mutable():
15745+            if self.node.is_mutable() and offset >= 0:
15746+                return self.update_my_contents(req, offset)
15747+
15748+            elif self.node.is_mutable():
15749                 return self.replace_my_contents(req)
15750             if not replace:
15751                 # this is the early trap: if someone else modifies the
15752hunk ./src/allmydata/web/filenode.py 212
15753                 # directory while we're uploading, the add_file(overwrite=)
15754                 # call in replace_me_with_a_child will do the late trap.
15755                 raise ExistingChildError()
15756+            if offset >= 0:
15757+                raise WebError("PUT to a file: append operation invoked "
15758+                               "on an immutable cap")
15759+
15760+
15761             assert self.parentnode and self.name
15762             return self.replace_me_with_a_child(req, self.client, replace)
15763         if t == "uri":
15764hunk ./src/allmydata/web/filenode.py 284
15765         d.addCallback(lambda res: self.node.get_uri())
15766         return d
15767 
15768+
15769+    def update_my_contents(self, req, offset):
15770+        req.content.seek(0)
15771+        added_contents = MutableFileHandle(req.content)
15772+
15773+        d = self.node.get_best_mutable_version()
15774+        d.addCallback(lambda mv:
15775+            mv.update(added_contents, offset))
15776+        d.addCallback(lambda ignored:
15777+            self.node.get_uri())
15778+        return d
15779+
15780+
15781     def replace_my_contents_with_a_formpost(self, req):
15782         # we have a mutable file. Get the data from the formpost, and replace
15783         # the mutable file's contents with it.
15784}
15785[test: add tests for new web update behavior
15786Kevan Carstensen <kevan@isnotajoke.com>**20100806234936
15787 Ignore-this: ebc9a33046134de4838764fa082add49
15788] {
15789hunk ./src/allmydata/test/common.py 354
15790         return defer.succeed(self)
15791 
15792 
15793+    def get_best_mutable_version(self):
15794+        return defer.succeed(self)
15795+
15796+    # Ditto for this, which is an implementation of IWritable.
15797+    # XXX: Declare that the same is implemented.
15798+    def update(self, data, offset):
15799+        assert not self.is_readonly()
15800+        def modifier(old, servermap, first_time):
15801+            new = old[:offset] + "".join(data.read(data.get_size()))
15802+            new += old[len(new):]
15803+            return MutableData(new)
15804+        return self.modify(modifier)
15805+
15806+
15807     def read(self, consumer, offset=0, size=None):
15808         data = self._download_best_version()
15809         if size:
15810hunk ./src/allmydata/test/test_web.py 2931
15811         d.addCallback(_done)
15812         return d
15813 
15814+
15815+    def test_PUT_update_at_offset(self):
15816+        file_contents = "test file" * 100000 # about 900 KiB
15817+        d = self.PUT("/uri?mutable=true", file_contents)
15818+        def _then(filecap):
15819+            self.filecap = filecap
15820+            new_data = file_contents[:100]
15821+            new = "replaced and so on"
15822+            new_data += new
15823+            new_data += file_contents[len(new_data):]
15824+            assert len(new_data) == len(file_contents)
15825+            self.new_data = new_data
15826+        d.addCallback(_then)
15827+        d.addCallback(lambda ignored:
15828+            self.PUT("/uri/%s?replace=True&offset=100" % self.filecap,
15829+                     "replaced and so on"))
15830+        def _get_data(filecap):
15831+            n = self.s.create_node_from_uri(filecap)
15832+            return n.download_best_version()
15833+        d.addCallback(_get_data)
15834+        d.addCallback(lambda results:
15835+            self.failUnlessEqual(results, self.new_data))
15836+        # Now try appending things to the file
15837+        d.addCallback(lambda ignored:
15838+            self.PUT("/uri/%s?offset=%d" % (self.filecap, len(self.new_data)),
15839+                     "puppies" * 100))
15840+        d.addCallback(_get_data)
15841+        d.addCallback(lambda results:
15842+            self.failUnlessEqual(results, self.new_data + ("puppies" * 100)))
15843+        return d
15844+
15845+
15846+    def test_PUT_update_at_offset_immutable(self):
15847+        file_contents = "Test file" * 100000
15848+        d = self.PUT("/uri", file_contents)
15849+        def _then(filecap):
15850+            self.filecap = filecap
15851+        d.addCallback(_then)
15852+        d.addCallback(lambda ignored:
15853+            self.shouldHTTPError("test immutable update",
15854+                                 400, "Bad Request",
15855+                                 "immutable",
15856+                                 self.PUT,
15857+                                 "/uri/%s?offset=50" % self.filecap,
15858+                                 "foo"))
15859+        return d
15860+
15861+
15862     def test_bad_method(self):
15863         url = self.webish_url + self.public_url + "/foo/bar.txt"
15864         d = self.shouldHTTPError("test_bad_method",
15865}
15866
15867Context:
15868
15869[misc/build_helpers/run-with-pythonpath.py: fix stale comment, and remove 'trial' example that is not the right way to run trial.
15870david-sarah@jacaranda.org**20100726225729
15871 Ignore-this: a61f55557ad69a1633bfb2b8172cce97
15872] 
15873[docs/specifications/dirnodes.txt: 'mesh'->'grid'.
15874david-sarah@jacaranda.org**20100723061616
15875 Ignore-this: 887bcf921ef00afba8e05e9239035bca
15876] 
15877[docs/specifications/dirnodes.txt: bring layer terminology up-to-date with architecture.txt, and a few other updates (e.g. note that the MAC is no longer verified, and that URIs can be unknown). Also 'Tahoe'->'Tahoe-LAFS'.
15878david-sarah@jacaranda.org**20100723054703
15879 Ignore-this: f3b98183e7d0a0f391225b8b93ac6c37
15880] 
15881[docs: use current cap to Zooko's wiki page in example text
15882zooko@zooko.com**20100721010543
15883 Ignore-this: 4f36f36758f9fdbaf9eb73eac23b6652
15884 fixes #1134
15885] 
15886[__init__.py: silence DeprecationWarning about BaseException.message globally. fixes #1129
15887david-sarah@jacaranda.org**20100720011939
15888 Ignore-this: 38808986ba79cb2786b010504a22f89
15889] 
15890[test_runner: test that 'tahoe --version' outputs no noise (e.g. DeprecationWarnings).
15891david-sarah@jacaranda.org**20100720011345
15892 Ignore-this: dd358b7b2e5d57282cbe133e8069702e
15893] 
15894[TAG allmydata-tahoe-1.7.1
15895zooko@zooko.com**20100719131352
15896 Ignore-this: 6942056548433dc653a746703819ad8c
15897] 
15898[relnotes.txt: updated for v1.7.1 release!
15899zooko@zooko.com**20100719083059
15900 Ignore-this: 9f10eb19b65a39d652b546c57481da45
15901] 
15902[immutable: add test case of #1128, fix test case of #1118
15903zooko@zooko.com**20100719081612
15904 Ignore-this: 8f9f742e7dac2bd9b49c19bd10f1c204
15905] 
15906[NEWS: add #1118 and reflow
15907zooko@zooko.com**20100719081248
15908 Ignore-this: 37a2e39d58c7b584b3c7f193bc1b30df
15909] 
15910[immutable: fix bug in which preexisting_shares and merged were shallowly referencing the same sets
15911zooko@zooko.com**20100719075426
15912 Ignore-this: 90827f8ce7ff0fc0c3c7f819399b8cf0
15913 This bug had the effect of making uploads sometimes (rarely) appear to succeed when they had actually not distributed the shares well enough to achieve the desired servers-of-happiness level.
15914] 
15915[upload.py: fix #1118 by aborting newly-homeless buckets when reassignment runs. This makes a previously failing assert correct. This version refactors 'abort' into two methods, rather than using a default argument.
15916david-sarah@jacaranda.org**20100719044655
15917 Ignore-this: 142d182c0739986812140bb8387077d5
15918] 
15919[docs/known_issues.txt: update release version and date.
15920david-sarah@jacaranda.org**20100718235940
15921 Ignore-this: dbbb42dbfa6c0d205a0b8e6e58eee9c7
15922] 
15923[relnotes.txt, docs/quickstart.html: prepare for 1.7.1 release. Don't claim to work on Cygwin (this might work but is untested).
15924david-sarah@jacaranda.org**20100718235437
15925 Ignore-this: dfc7334ee4bb76c04ee19304a7f1024b
15926] 
15927[immutable: extend the tests to check that the shares that got uploaded really do make a sufficiently Happy distribution
15928zooko@zooko.com**20100719045047
15929 Ignore-this: 89c33a7b795e23018667351045a8d5d0
15930 This patch also renames some instances of "find_shares()" to "find_all_shares()" and other instances to "find_uri_shares()" as appropriate -- the conflation between those names confused me at first when writing these tests.
15931] 
15932[immutable: test for #1118
15933zooko@zooko.com**20100718221537
15934 Ignore-this: 8882aabe2aaec6a0148c87e735d817ad
15935] 
15936[immutable: test for #1124
15937zooko@zooko.com**20100718222907
15938 Ignore-this: 1766e3cbab92ea2a9e246f40eb6e770b
15939] 
15940[docs/logging.txt: document that _trial_temp/test.log does not receive messages below level=OPERATIONAL, due to <http://foolscap.lothar.com/trac/ticket/154>.
15941david-sarah@jacaranda.org**20100718230420
15942 Ignore-this: aef40f2e74ddeabee5e122e8d80893a1
15943] 
15944[trivial: fix unused import (sorry about that, pyflakes)
15945zooko@zooko.com**20100718215133
15946 Ignore-this: c2414e443405072b51d552295f2c0e8c
15947] 
15948[tests, NEWS, CREDITS re: #1117
15949zooko@zooko.com**20100718203225
15950 Ignore-this: 1f08be2c692fb72cc0dd023259f11354
15951 Give Brian and Kevan promotions, move release date in NEWS to the 18th, commit Brian's test for #1117.
15952 fixes #1117
15953] 
15954[test/test_upload.py: test to see that aborted buckets are ignored by the storage server
15955Kevan Carstensen <kevan@isnotajoke.com>**20100716001046
15956 Ignore-this: cc075c24b1c86d737f3199af894cc780
15957] 
15958[test/test_storage.py: test for the new remote_abort semantics.
15959Kevan Carstensen <kevan@isnotajoke.com>**20100715232148
15960 Ignore-this: d3d6491f17bf670e770ca4b385007515
15961] 
15962[storage/immutable.py: make remote_abort btell the storage server about aborted buckets.
15963Kevan Carstensen <kevan@isnotajoke.com>**20100715232105
15964 Ignore-this: 16ab0090676355abdd5600ed44ff19c9
15965] 
15966[test/test_upload.py: changes to test plumbing for #1117 tests
15967Kevan Carstensen <kevan@isnotajoke.com>**20100715231820
15968 Ignore-this: 78a6d359d7bf8529d283e2815bf1e2de
15969 
15970     - Add a callRemoteOnly method to FakeBucketWriter.
15971     - Change the abort method in FakeBucketWriter to not return a
15972       RuntimeError.
15973] 
15974[immutable/upload.py: abort buckets if peer selection fails
15975Kevan Carstensen <kevan@isnotajoke.com>**20100715231714
15976 Ignore-this: 2a0b643a22284df292d8ed9d91b1fd37
15977] 
15978[test_encodingutil: correct an error in the previous patch to StdlibUnicode.test_open_representable.
15979david-sarah@jacaranda.org**20100718151420
15980 Ignore-this: af050955f623fbc0e4d78e15a0a8a144
15981] 
15982[NEWS: Forward-compatibility improvements for non-ASCII caps (#1051).
15983david-sarah@jacaranda.org**20100718143622
15984 Ignore-this: 1edfebc4bd38a3b5c35e75c99588153f
15985] 
15986[test_dirnode and test_web: don't use failUnlessReallyEqual in cases where the return type from simplejson.loads can vary between unicode and str. Use to_str when comparing URIs parsed from JSON.
15987david-sarah@jacaranda.org**20100718142915
15988 Ignore-this: c4e78ef4b1478dd400da71cf077ffa4a
15989] 
15990[test_encodingutil: StdlibUnicode.test_open_representable no longer uses a mock.
15991david-sarah@jacaranda.org**20100718125412
15992 Ignore-this: 4bf373a5e2dfe4209e5e364124af29a3
15993] 
15994[docs: add comment clarifying #1051
15995zooko@zooko.com**20100718053250
15996 Ignore-this: 6cfc0930434cbdbbc262dabb58f1505d
15997] 
15998[docs: update NEWS
15999zooko@zooko.com**20100718053225
16000 Ignore-this: 63d5c782ef84812e6d010f0590866831
16001] 
16002[Add tests of caps from the future that have non-ASCII characters in them (encoded as UTF-8). The changes to test_uri.py, test_client.py, and test_dirnode.py add tests of non-ASCII future caps in addition to the current tests. The changes to test_web.py just replace the tests of all-ASCII future caps with tests of non-ASCII future caps. We also change uses of failUnlessEqual to failUnlessReallyEqual, in order to catch cases where the type of a string is not as expected.
16003david-sarah@jacaranda.org**20100711200252
16004 Ignore-this: c2f193352369d32e06865f8f3e951894
16005] 
16006[Debian documentation update
16007jacob@appelbaum.net**20100305003004] 
16008[debian-docs-patch-final
16009jacob@appelbaum.net**20100304085955] 
16010[M-x whitespace-cleanup
16011zooko@zooko.com**20100718032739
16012 Ignore-this: babfd4af6ad2fc885c957fd5c8b10c3f
16013] 
16014[docs: tidy up NEWS a little
16015zooko@zooko.com**20100718032434
16016 Ignore-this: 54f2820fd1a37c8967609f6bfc4e5e18
16017] 
16018[benchmarking: update bench_dirnode.py to reflect the new directory interfaces
16019zooko@zooko.com**20100718031710
16020 Ignore-this: 368ba523dd3de80d9da29cd58afbe827
16021] 
16022[test_encodingutil: fix test_open_representable, which is only valid when run on a platform for which we know an unrepresentable filename.
16023david-sarah@jacaranda.org**20100718030333
16024 Ignore-this: c114d92c17714a5d4ae005c15267d60c
16025] 
16026[iputil.py: Add support for FreeBSD 7,8 and 9
16027francois@ctrlaltdel.ch**20100718022832
16028 Ignore-this: 1829b4cf4b91107f4cf87841e6167e99
16029 committed by: zooko@zooko.com
16030 date: 2010-07-17
16031 and I also patched: NEWS and CREDITS
16032] 
16033[NEWS: add snippet about #1083
16034zooko@zooko.com**20100718020653
16035 Ignore-this: d353a9d93cbc5a5e6ba4671f78d1e22b
16036] 
16037[fileutil: docstrings for non-obvious usage restrictions on methods of EncryptedTemporaryFile.
16038david-sarah@jacaranda.org**20100717054647
16039 Ignore-this: 46d8fc10782fa8ec2b6c5b168c841943
16040] 
16041[Move EncryptedTemporaryFile from SFTP frontend to allmydata.util.fileutil, and make the FTP frontend also use it (fixing #1083).
16042david-sarah@jacaranda.org**20100711213721
16043 Ignore-this: e452e8ca66391aa2a1a49afe0114f317
16044] 
16045[NEWS: reorder NEWS snippets to be in descending order of interestingness
16046zooko@zooko.com**20100718015929
16047 Ignore-this: 146c42e88a9555a868a04a69dd0e5326
16048] 
16049[Correct stringutils->encodingutil patch to be the newer version, rather than the old version that was committed in error.
16050david-sarah@jacaranda.org**20100718013435
16051 Ignore-this: c8940c4e1aa2e9acc80cd4fe54753cd8
16052] 
16053[test_cli.py: fix error that crept in when rebasing the patch for #1072.
16054david-sarah@jacaranda.org**20100718000123
16055 Ignore-this: 3e8f6cc3a27b747c708221dd581934f4
16056] 
16057[stringutils: add test for when sys.stdout has no encoding attribute (fixes #1099).
16058david-sarah@jacaranda.org**20100717045816
16059 Ignore-this: f28dce6940e909f12f354086d17db54f
16060] 
16061[CLI: add 'tahoe unlink' as an alias to 'tahoe rm', for forward-compatibility.
16062david-sarah@jacaranda.org**20100717220411
16063 Ignore-this: 3ecdde7f2d0498514cef32e118e0b855
16064] 
16065[minor code clean-up in dirnode.py
16066zooko@zooko.com**20100714060255
16067 Ignore-this: bb0ab2783203e605024b3e2f798256a1
16068 Impose micro-POLA by passing only the writekey instead of the whole node object to {{{_encrypt_rw_uri()}}}. Remove DummyImmutableFileNode in nodemaker.py, which is obviated by this. Add micro-optimization by precomputing the netstring of the empty string and branching on whether the writekey is present or not outside of {{{_encrypt_rw_uri()}}}. Add doc about writekey to docstring.
16069 fixes #967
16070] 
16071[Rename stringutils to encodingutil, and drop listdir_unicode and open_unicode (since the Python stdlib functions work fine with Unicode paths). Also move some utility functions to fileutil.
16072david-sarah@jacaranda.org**20100712003015
16073 Ignore-this: 103b809d180df17a7283077c3104c7be
16074] 
16075[Allow URIs passed in the initial JSON for t=mkdir-with-children, t=mkdir-immutable to be Unicode. Also pass the name of each child into nodemaker.create_from_cap for error reporting.
16076david-sarah@jacaranda.org**20100711195525
16077 Ignore-this: deac32d8b91ba26ede18905d3f7d2b93
16078] 
16079[docs: CREDITS and NEWS
16080zooko@zooko.com**20100714060150
16081 Ignore-this: dc83e612f77d69e50ee975f07f6b16fe
16082] 
16083[CREDITS: more creds for Kevan, plus utf-8 BOM
16084zooko@zooko.com**20100619045503
16085 Ignore-this: 72d02bdd7a0f324f1cee8cd399c7c6de
16086] 
16087[cli.py: make command descriptions consistently end with a full stop.
16088david-sarah@jacaranda.org**20100714014538
16089 Ignore-this: 9ee7fa29ca2d1631db4049c2a389a97a
16090] 
16091[SFTP: address some of the comments in zooko's review (#1106).
16092david-sarah@jacaranda.org**20100712025537
16093 Ignore-this: c3921638a2d4f1de2a776ae78e4dc37e
16094] 
16095[docs/logging.txt: note that setting flogging vars might affect tests with race conditions.
16096david-sarah@jacaranda.org**20100712050721
16097 Ignore-this: fc1609d215fcd5561a57fd1226206f27
16098] 
16099[test_storage.py: potential fix for failures when logging is enabled.
16100david-sarah@jacaranda.org**19700713040546
16101 Ignore-this: 5815693a0df3e64c52c3c6b7be2846c7
16102] 
16103[upcase_since_on_welcome
16104terrellrussell@gmail.com**20100708193903] 
16105[server_version_on_welcome_page.dpatch.txt
16106freestorm77@gmail.com**20100605191721
16107 Ignore-this: b450c76dc875f5ac8cca229a666cbd0a
16108 
16109 
16110 - The storage server version is 0 for all storage nodes in the Welcome Page
16111 
16112 
16113] 
16114[NEWS: add NEWS snippets about two recent patches
16115zooko@zooko.com**20100708162058
16116 Ignore-this: 6c9da6a0ad7351a960bdd60f81532899
16117] 
16118[directory_html_top_banner.dpatch
16119freestorm77@gmail.com**20100622205301
16120 Ignore-this: 1d770d975e0c414c996564774f049bca
16121 
16122 The div tag with the link "Return to Welcome page" on the directory.xhtml page is not correct
16123 
16124] 
16125[tahoe_css_toolbar.dpatch
16126freestorm77@gmail.com**20100622210046
16127 Ignore-this: 5b3ebb2e0f52bbba718a932f80c246c0
16128 
16129 CSS modification to be correctly diplayed with Internet Explorer 8
16130 
16131 The links on the top of page directory.xhtml are not diplayed in the same line as display with Firefox.
16132 
16133] 
16134[runnin_test_tahoe_css.dpatch
16135freestorm77@gmail.com**20100622214714
16136 Ignore-this: e0db73d68740aad09a7b9ae60a08c05c
16137 
16138 Runnin test for changes in tahoe.css file
16139 
16140] 
16141[runnin_test_directory_xhtml.dpatch
16142freestorm77@gmail.com**20100622201403
16143 Ignore-this: f8962463fce50b9466405cb59fe11d43
16144 
16145 Runnin test for diretory.xhtml top banner
16146 
16147] 
16148[stringutils.py: tolerate sys.stdout having no 'encoding' attribute.
16149david-sarah@jacaranda.org**20100626040817
16150 Ignore-this: f42cad81cef645ee38ac1df4660cc850
16151] 
16152[quickstart.html: python 2.5 -> 2.6 as recommended version
16153david-sarah@jacaranda.org**20100705175858
16154 Ignore-this: bc3a14645ea1d5435002966ae903199f
16155] 
16156[SFTP: don't call .stopProducing on the producer registered with OverwriteableFileConsumer (which breaks with warner's new downloader).
16157david-sarah@jacaranda.org**20100628231926
16158 Ignore-this: 131b7a5787bc85a9a356b5740d9d996f
16159] 
16160[docs/how_to_make_a_tahoe-lafs_release.txt: trivial correction, install.html should now be quickstart.html.
16161david-sarah@jacaranda.org**20100625223929
16162 Ignore-this: 99a5459cac51bd867cc11ad06927ff30
16163] 
16164[setup: in the Makefile, refuse to upload tarballs unless someone has passed the environment variable "BB_BRANCH" with value "trunk"
16165zooko@zooko.com**20100619034928
16166 Ignore-this: 276ddf9b6ad7ec79e27474862e0f7d6
16167] 
16168[trivial: tiny update to in-line comment
16169zooko@zooko.com**20100614045715
16170 Ignore-this: 10851b0ed2abfed542c97749e5d280bc
16171 (I'm actually committing this patch as a test of the new eager-annotation-computation of trac-darcs.)
16172] 
16173[docs: about.html link to home page early on, and be decentralized storage instead of cloud storage this time around
16174zooko@zooko.com**20100619065318
16175 Ignore-this: dc6db03f696e5b6d2848699e754d8053
16176] 
16177[docs: update about.html, especially to have a non-broken link to quickstart.html, and also to comment out the broken links to "for Paranoids" and "for Corporates"
16178zooko@zooko.com**20100619065124
16179 Ignore-this: e292c7f51c337a84ebfeb366fbd24d6c
16180] 
16181[TAG allmydata-tahoe-1.7.0
16182zooko@zooko.com**20100619052631
16183 Ignore-this: d21e27afe6d85e2e3ba6a3292ba2be1
16184] 
16185Patch bundle hash:
1618631df89b0d56eb5b492577ed8eca6ab0f95436b05