[tahoe-dev] use case request for accounting/leasedb

[David-Sarah Hopwood]

On 18/10/12 20:48, Zooko Wilcox-O'Hearn wrote:
> On Fri, Oct 12, 2012 at 4:29 PM, David-Sarah Hopwood
> <david-sarah at jacaranda.org> wrote:
>>
>>> The only difference between this and the current scheme is that the
>>> storage server will never do that on its own — it only does it when
>>> you tell it that it is okay to do it.
>>
>> Okay, so I think this means that lease expiry -- the operation of removing
>> leases that are past their expiration date -- needs to be triggered separately
>> from deletion of unleased shares.
>>
>> If lease expiry can be triggered independently per account, then that
>> becomes the "okay to expire" operation. The accounting crawler will only
>> delete shares with no lease on any account, and will not itself expire
>> leases.
>>
>> Does this seem like it supports what you want?
> 
> Almost, but I don't think that will be good enough to implement
> everything we'll need here. I think it probably *would* be sufficient
> if you could scan and update all the shares (to remove Alice's lease)
> instantaneously.
> 
> But, the process of scanning all shares could take a long time, it
> could even take weeks to finish, and is expensive in terms of disk
> usage (seeks, not bandwidth), or cloud storage API requests (number of
> requests, not bulk upload/download). So the first issue with the
> proposal above is, what happens if I tell you "Okay now you can remove
> any shares of mine which I haven't marked.", and you start doing that,
> and then you crash and restart before you finish doing it?

Note that, although the marking process may take a long time, the
request to delete unmarked leases from a given server's leasedb does
not -- it can be a single SQL update, potentially.
(It's also idempotent, so if it fails due to a network error, the
client could retry.)

So for your desired use case, the protocol would be (with each step
done for each server):
1. Ask the server for its current time, and remember that.
2. Trace non-garbage files, renewing leases for their shares.
3. Ask the server to cancel all leases for the account with
   renewal timestamps before the remembered time.
4. If the previous step cancelled the last lease for a share, the
   server will delete that share when the accounting crawler gets
   around to it.

For the current design the accounting crawler is crawling over all
shares and so it may take a while for it to get around to all the
ones that should be deleted, but that part of the design isn't critical.
The server could instead query the leasedb for all shares that should
be deleted after step 3, and put them in a queue.

Note that the accounting crawler doesn't have to read any shares
(except, possibly, for discovered ones that were not in the leasedb,
to check the header).

> It isn't
> okay for you to leave my shares in place and keep charging me for
> them, so you'd better write it down that you intend to do that so
> you'll remember to get back to it.

That's written down by removing the lease entries.

> Second, what if you are in the process of looking for shares that are
> expired (for me), and I say "Okay! I'm about to mark all my *current*
> documents and then I want you to stop charging me for all others.
> Ready? Here I go!".

Again, the process of removing lease entries is quick and can be
synchronous, in which case it is not be interruptible by marking.
This is a big advantage of making the leasedb local to the server.

-- 
David-Sarah Hopwood ⚥

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 554 bytes
Desc: OpenPGP digital signature
URL: <http://tahoe-lafs.org/pipermail/tahoe-dev/attachments/20121018/c522a91a/attachment.pgp>