need test for advertisement of large share support

[zooko]

per mailing list message ​http://allmydata.org/pipermail/tahoe-dev/2009-January/000962.html , storage servers should advertise that they support large shares.

Once they do that, clients (>= v1.3.0) will be able to use them to upload large files.

[warner]

Zooko points out that we should just calculate the available disk space on this server, and advertise that as the maximum share size. We only make the advertisement once, at startup time, so this will have the slightly unfortunate property that a server which is completely full at boot time, but which gets more space later (because old files are deleted), will continue to advertise a small value until it is restarted. That's probably ok.

[warner]

c7cd3f38e7b8768d adds the new advertisement, using the same 'df' calculation that the "reserved space" configuration knob uses (which means on windows, we advertise 264).

The calculation will actually be made each time a client connects to the storage server, so when clients restart and reconnect, they'll get the latest advertisement.

Now all that's left to close this ticket (or perhaps reopen+close #346) is to do a manual test of a large file, with shares that are larger than 4GiB, so probably we should try a 15GiB file.

[zooko]

Hm, I think we could also have an automated test along the lines of test_storage, but initiated by the client instead of the server. Such an automated test should be marked skip on systems that don't have efficient sparse files and plenty of available filesystem capacity.

[warner]

well, I'll do a test on my local box, but I expect it to be a CPU problem, not efficient-sparse-files and lots-of-disk. Our development efforts are being hampered enough by having a 20-minute unit test cycle time.. adding five hours to that for a 15GiB upload would make them unusable.

[zooko]

See how the test in test_storage that I mentioned doesn't store the entire large share -- just the part of the large share that would fall past the 2³² cutoff. Likewise, the test that I am proposing would upload just a few bytes at a large offset. This wouldn't take hours to compute, but it would (momentarily) allocate more than 4 GiB of disk space, and if the OS doesn't support efficient sparse files then it would also fill in 4 GiB of disk space with zeroes.

[warner]

Hm, interesting. I guess that would let us prove that the storage server is ready, but it wouldn't expose any other bugs that would prevent users from uploading large files.

It looks like my 15GiB test failed, since we aren't yet using the larger WriteBucketProxy_v2 class during the upload. I suppose the best fix will be to use these if we're uploading a large file, so that smaller files will remain backwards compatible with tahoe-1.2.0, but larger files will still work.

[zooko]

Hm, I'm pretty sure this was fixed way back in v1.4.1...

[zooko]

I think this is probably covered by 8c37b8e3af2f4d1b, and if it isn't then oh well--the large-share feature has been working in practice for a long time and I no longer care if there isn't a specific unit test of it...