Copying directories containing caps from the future

[davidsarah]

#708 left the following forward-compatibility issue unresolved:

As I understand it, the fact that we can't add unknown caps into a directory means that we can't copy directories which contain caps from the future. (If we do copy such a directory then the entries in it which had new-style caps will be omitted from the newly created copy of the directory). In theory it should be possible to do that safely just by copying the write-cap field from the entry in the source dir into the write-cap field of the newly created entry in the target dir, and likewise copying the read-cap.

[...]

I don't know how important it would be for clients from the past to be able to copy your new-style caps.

I think it's important. If we add a completely new cap format, then will be quite possible to end up with a mixture of new and old caps in a directory, especially if multiple people are using it. It would be nice for old clients to be able to copy such a directory, at least for immutable files (where copying is equivalent to referencing). Where a new cap references a mutable file, it's less clear what to do.

Continuing the discussion from #708:

The internal 'move' method does just that, and the JSON representation of a directory includes all the information we have about the unknown object (i.e. both the writecap and the readcap). What I don't know is how the CLI-side "tahoe cp" works, specifically if the put-lots-of-caps-at-once dirnode webapi operation will accept the same "unknown cap" structure that the JSON representation hands down. Also, I wanted to discourage people from adding new unknown caps to a directory (because they might just be adding complete junk, or a typo, or a blank string, and it'd be nice to detect that early), so the current code is liberal in what it accepts from the backend, but strict in what it accepts from the frontend, and this might prevent the frontend-based tools from doing this sort of copy.

So yes, I think that approach would be safe, and it might already work. (of course we have no way to tell if the unknown-cap is a file or a directory, or something even more exotic, so we might be creating a hardlink to a mutable directoryish-thing when the rest of the copy operation was trying to make a deepcopy of individual files).

The test would need to go in test_cli.py where it tests the "tahoe cp" operation. grep around the test suite for UnknownNode, you have to be a bit sneaky to get the cap-from-the-future into a directory to start with.

To close this issue:

• find out whether copying caps-from-the-future already works from the CLI
• decide whether it should work
• if it should work and doesn't, then make it work
• add tests.

[davidsarah]

Perhaps we need to be able to distinguish whether caps-from-the-future are mutable or immutable. That seems doable, by establishing a convention for the JSON encoding (which shouldn't constrain the future design significantly, I think).

[davidsarah]

If we add a "copyByReference" flag, which controls whether a directory entry with an unknown cap format is copied just by copying its representation, then we can defer the decision as to which caps are copied in that way to when the new cap formats are designed.

It would also be possible to add a field giving the lowest Tahoe version that fully supports that cap format. So when you operate on a directory with unknown caps, it would find the maximum unknown version, x.y.z, and print something like "Some entries in this directory are not supported by the current version of Tahoe. The creator of one or more of these entries claimed that Tahoe version x.y.z is needed."

[zooko]

I think we are out of time to do this for v1.7.0.

By the way, I think we should be careful not to enable users to accidentally shallow-copy mutable objects when they don't mean to. Immutables are always a good idea to shallow-copy. The idea of shallow-copying a mutable which is "from the future" -- your Tahoe-LAFS gateway isn't even capable of recognizing its type -- sounds scary to me.

[davidsarah]

There are some cases where it is possible to copy future caps without introducing any risk that we haven't already accepted. For instance, copying a future cap prefixed with {{imm.}}}) (including the case of copying from an immutable directory) introduces no new risk: it will never be treated as a valid mutable cap by future clients.

[zooko]

Oh, right. Thanks for the reminder.

[davidsarah]

I've been looking at the code for tahoe cp and think I know what to change to make this work. The default behaviour IMHO should be to skip unknown things that can't be copied by reference (and also local special files), with a warning for each, but at the end print a count of the number of the number of things that have been skipped. The cases mentioned in comment:5 don't need to be skipped.

[davidsarah]

This is an enhancement, so it has missed the 1.9 feature freeze.