Clarify behavior of overwriting in immutable uploads

[itamarst]

This is related but not quite the same as #3793.

1. Current interactions from Fooslcap when uploading an immutable basically upload in order. With HTTP, uploads can happen out of order. How to handle holes?
2. Immutables are supposed to be immutable. As such, the data changing between different write attempts to the same in-progress immutable suggests something went very wrong.
3. Current behavior is "last write wins". This could potentially lead to unexpected outcomes if there are bugs.

Alternative behaviors to current behavior include:

1. Disallow overlapping writes.
2. Allow overlapping writes so long as data is unchanged.

In either case the need to deal with out-of-order writes remains due to HTTP. Current code might work, but would result in holes that then get overwritten. Policies that explicitly forbid/validate overlapping writes would need to deal with holes.

[itamarst]

See also discussion in ​https://github.com/tahoe-lafs/tahoe-lafs/pull/1122#discussion_r706234564

[itamarst]

If I were designing this from scratch, one design would be:

1. Upload chunks of, say, 1MB.
2. Only allow uploads in order.
3. Chunks are written atomically, so e.g. write of less than 1MB due to server crash is truncated.
4. Conflicting writes are not allowed, which is fine given atomicity, chunks, and ordering.

The use of relatively large chunks mitigates the latency issues of ordered writes except in cases of very high bandwidth combination very high latency, which is not common.

Problems with the above approach with current code:

1. The uploaded file combines metadata with data in a single file. Which means semantically you can't have same-sized chunks. Maybe? Atomicity seems like it'd also be harder too.
2. Would have to add limits on overwrites, and maybe hard code the append only which doesn't exist at the moment.
3. Chunk sizes are smaller due to historical reasons I don't understand.

---

The current data format could be changed (it's a temporary form of storage, so it is easier to evolve), and given flexibility in format maybe the above design is unnecessary; if it's not just one big file, keeping out of order chunks is a lot easier e.g. in separate files or in sqlite or something.

Supposedly there's a branch somewhere with sqlite implementation...

[itamarst]

Maybe this branch, not actually SQLite: ​https://github.com/tahoe-lafs/tahoe-lafs/tree/1836-use-leasedb-for-share-count

LMDB is probably more appropriate than SQLite, thinking it about more for 30 seconds.

[itamarst]

After further discussion, and learning that storage servers wipe in-progress uploads on startup:

1. We will change semantics such that out-of-order uploads are explicitly allowed, but overlapping writes are explicitly disallowed.
2. In order to prevent overlapping writes, metadata tracking what ranges have already been uploaded will be tracked in-memory.
3. Later, and out of scope for this ticket, interaction with IStorageServer may need some tweaks to maximize HTTP's ability to do parallel uploads.

[exarkun]

Citation for servers wiping in-progress uploads on startup:

• ​https://github.com/tahoe-lafs/tahoe-lafs/blob/2f742d8c1b04583544d9a178ef722e7d30fa1384/src/allmydata/storage/server.py#L89-L90
• ​https://github.com/tahoe-lafs/tahoe-lafs/blob/2f742d8c1b04583544d9a178ef722e7d30fa1384/src/allmydata/storage/server.py#L192-L193

[itamarst]

Switch from model where Foolscap connection going away causes disconnect to a timeout-based mechanism where BucketWriter? stays alive until timeout, refreshed on new upload. This addresses #3793 somewhat in that there is only one BucketWRiter per sotrage index so less worries about concurrency.

[itamarst]

Timeout of 30 minutes before cleanup, in case uploads are slow.

[itamarst]

Things that are changed semantically:

1. Two clients uploading at once is possible. If methods are synchronous this is probably fine, but test. Can address with DeferredLock?.
2. Client that disconnected would previously start from scratch. Now it won't. This would break re-uploads of failed uploads.

[itamarst]

To solve problem 2 above, instead of rejecting overlapping writes outright, allow overlapping writes if and only if the data being written matches existing data.

[itamarst]

In order to reduce scope of PRs, persistent BucketWriter will happen in #3793, and this will just do the "prevent overlapping writes that don't match", with state in the ... BucketWriter? Or ShareFile, ugh.

[itamarst]

BucketWriter stores a lease, so need to figure out what happens with a second lease...

[GitHub <noreply@…>]

In fadfbca/trunk:

Merge pull request #1130 from tahoe-lafs/3801-no-overlapping-writes-immutable-upload

Disallow conflicting overlapping writes when doing an immutable upload.

Fixes ticket:3801