drop-upload: don't perform redundant uploads when a file is quickly modified three or more times

[davidsarah]

When a file is uploaded by the drop-upload frontend, events that would cause further uploads of the same file are queued. This queue only needs to record one event for a given file, e.g. if a file is uploading and gets two more modification events before the first upload completes, the middle event can be dropped.

[daira]

See also #2220.

[dawuud]

I thought of a solution. Would something along these lines be acceptable? :

We use an explicit dequeue implementation and we utilize a helping hash map to track currently queued files/dirs. Each time we add a file to the queue we first check if that file is already in the hashmap. We only add a file to the queue when it is not already in the hashmap. Likewise we must remove it from the hashmap once that queue item has been processed.

[daira]

Replying to dawuud:

We use an explicit dequeue implementation and we utilize a helping hash map to track currently queued files/dirs. Each time we add a file to the queue we first check if that file is already in the hashmap. We only add a file to the queue when it is not already in the hashmap. Likewise we must remove it from the hashmap once that queue item has been processed.

Yes, this is basically the same as the pending_delay code in ​allmydata/windows/inotify.py. We should add something similar to drop_upload.py, and then we can probably remove the Windows-specific implementation.

[dawuud]

ok here's some code that does that: ​https://github.com/david415/tahoe-lafs/tree/david-1440-1

although this code doesn't work because i branched from my code for ticket #1449

[dawuud]

here's my latest... i've used the explicit deque from the other branch and i've added a pending set of files... this allows deduplication: ​https://github.com/david415/tahoe-lafs/tree/dropupload-redundant-uploads-1

I had to comment out part of the drop uploader unit test to get it to pass. We need a unit test for this deduplicating of upload events.

[dawuud]

OK I pushed more code into the same branch. I worked on the unit test for deduplicating uploads but so far only the mock test works ;-( Maybe Daira can help me figure this out?

--- This is indeed incorrect.. and further more after some reflection I think it falls under the category of beginners mistake with regards to being a async interleave concurrency programming design error. ;-)

I'm very interested in know about the advantages and disadvantages of a deferred-based-upload-queue versus an explicit queue like this one.

[dawuud]

latest working code here: ​https://github.com/david415/tahoe-lafs/tree/2406.otf-objective-2.1-bugfixes-1

[daira]

Closing this and using ticket #2406 for any further review comments.

[warner]

Milestone renamed