Uploading a 20MB mutable fails in HTTP, but succeeds in Foolscap

[itamarst]

To reproduce: use benchmarks/upload_download.py from ​https://github.com/tahoe-lafs/tahoe-lafs/tree/3952-benchmarks, with a 20MB file. It'll run with Foolscap, but in HTTP:

>               result = await self.clients[0].create_mutable_file(MutableData(DATA))
E               allmydata.mutable.common.NotEnoughServersError: ("Publish ran out of good servers, last failure was: [Failure instance: Traceback: <class 'allmydata.storage.http_client.ClientException'>: (413, b'')\n/home/itamarst/devel/tahoe-lafs/venv/lib/python3.10/site-packages/twisted/internet/defer.py:734:errback\n/home/itamarst/devel/tahoe-lafs/venv/lib/python3.10/site-packages/twisted/internet/defer.py:797:_startRunCallbacks\n/home/itamarst/devel/tahoe-lafs/venv/lib/python3.10/site-packages/twisted/internet/defer.py:891:_runCallbacks\n/home/itamarst/devel/tahoe-lafs/venv/lib/python3.10/site-packages/twisted/internet/defer.py:1791:gotResult\n--- <exception caught here> ---\n/home/itamarst/devel/tahoe-lafs/venv/lib/python3.10/site-packages/twisted/internet/defer.py:1692:_inlineCallbacks\n/home/itamarst/devel/tahoe-lafs/venv/lib/python3.10/site-packages/twisted/python/failure.py:518:throwExceptionIntoGenerator\n/home/itamarst/devel/tahoe-lafs/src/allmydata/storage_client.py:1445:slot_testv_and_readv_and_writev\n/home/itamarst/devel/tahoe-lafs/venv/lib/python3.10/site-packages/twisted/internet/defer.py:1696:_inlineCallbacks\n/home/itamarst/devel/tahoe-lafs/src/allmydata/storage/http_client.py:868:read_test_write_chunks\n]", None)

That is, we're hitting a hard-coded size limit in the client code which presumably shouldn't be enforced given this works for Foolscap?

[itamarst]

Mutable uploads get sent as CBOR, and we limit CBOR messages to 1MB cause we don't have streaming parsing or streaming validation at the moment.

Solutions might be:

1. Figure out streaming parsing + validation.
2. Break up uploads into chunks in the HTTP client, which is... semantically dangerous possibly.
3. Change HTTP protocol so it doesn't use CBOR for the actual data.
4. ...

[itamarst]

cbor2 can parse from file, at least.

In theory one could do terrible things with mmap() such that the Rust CDDL (which requiers a byte slice) can validate a file.

But first... does the very flexible Foolscap mutable writing API actually use all that flexibility? If not, we might be able to simplify the HTTP protocol a lot.

[itamarst]

The test vectors are "check strings", there's only one, and it looks like check strings are always short. But it's hard to be quite sure :( But plausibly saying "validation checks need to fit in an HTTP header" is actually a reasonable thing to try.

[itamarst]

MDMF has multiple entries in the write vector. So options are:

1. Coalesce them into a single write (likely to be possible, but need to investigate).
2. Continue to do body as CBOR, and fudge the schema validation.
3. Use the actual relevant HTTP feature... which isn't that different than option 2, really.

Next I will look into coalescing.

[itamarst]

Coalescing is mostly OK, except! It's possible to do a write at an offset. So there will be holes, and when first creating a file a hole means null bytes (zeros), but later on it means "don't overwrite".

[itamarst]

Given all the above, best bet is just:

1. Increasing size limit of CBOR for the mutable upload case.
2. Do our best to not use too much memory for CBOR parsing and CDDL validation.

[GitHub <noreply@…>]

In 7ef1c020/trunk:

Merge pull request #1244 from tahoe-lafs/3956-mutable-uploads

Fix mutable uploads over HTTP above a certain size

Fixes ticket:3956