#2635 closed defect (fixed)

Magic Folder: periodically do a full uploader scan

Reported by: daira Owned by: daira
Priority: normal Milestone: undecided
Component: code-frontend-magic-folder Version: n/a
Keywords: magic-folder reliability upload Cc:
Launchpad Bug:

Description

This would help to compensate for:

  • missed events, due to unreliability of the filesystem notification mechanism or queue overflows (#1430),
  • failed uploads.

Change History (22)

comment:1 Changed at 2015-12-14T15:59:22Z by daira

  • Owner changed from daira to dawuud

Work in progress on branch 2635.periodic-uploader-scan.0 (I hadn't got very far with it).

comment:2 Changed at 2015-12-15T16:25:23Z by dawuud

in my dev branch i taught the uploader's _scan method to use the deque instead of calling _process directly. this breaks all our magic-folder tests... so the next is to make the tests pass by firing the hook deferred when a directory and all it's children are done being uploaded.

https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.0

comment:3 Changed at 2015-12-16T15:08:02Z by dawuud

after teaching the uploader to use the deque to process directories i've been trying to get the tests to pass... and i'm not sure how to do it but i pushed my latest WIP comment anyways.

https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.0

maybe daira or meejah could help me with this?

comment:4 Changed at 2015-12-16T15:08:17Z by dawuud

  • Owner changed from dawuud to daira

comment:5 Changed at 2015-12-16T16:04:13Z by daira

The current design is wrong because it assumes that _scan finishes processing all of the descendant directories under the directory it is called with, when actually it only adds callbacks to scan all of its direct children. This affects both the initial scan triggered by start_scanning (which assumes that all desendants have been removed from the _pending set), and the intent of the tests that move subtrees.

comment:6 Changed at 2015-12-17T11:06:16Z by dawuud

It's almost as if we want nested DeferredList?`s representing nested directories such that we can determine when a given directory and all it's children have been uploaded.

comment:7 Changed at 2015-12-17T16:23:19Z by dawuud

here's the dev branch from our pairing session today: https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.1

comment:8 Changed at 2015-12-17T21:50:20Z by dawuud

ok... all the tests are passing again with this commit: https://github.com/david415/tahoe-lafs/commit/4610a7f97d4823548129b4d483d7e50e4c5c18ee

next i need to periodic scan... should be easy.

comment:9 Changed at 2015-12-17T23:23:27Z by dawuud

here's a naive periodic full scan: https://github.com/david415/tahoe-lafs/commit/d1eef67f7632ee96d8e3e30b1c8b5f9d084d8973

actually i think it's an open question how we should determine the full scan is complete and we should therefore perform another full scan now because our duration has elapsed.

just think how bad it would be if the full scan didn't complete before the next one started!

perhaps we can just check the pending set every X minutes and if it's empty then do the full scan?

Last edited at 2015-12-17T23:27:29Z by dawuud (previous) (diff)

comment:10 Changed at 2015-12-18T00:37:03Z by dawuud

ok i made it only do the full scan if the pending set is empty. i also added a unit test... it's a MockTest? similar to test_move_tree but upon file creation inside the magicfolder dir we do not emit a fake inotify event. instead we wait for the periodic scan and upload... however i am trying to use clock advances to speed up this test and it's not yet passing.

comment:11 Changed at 2015-12-21T17:57:49Z by daira

  • Status changed from new to assigned

Reviewing progress so far.

comment:13 Changed at 2015-12-22T16:14:30Z by dawuud

work in progress from out pairing session: https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.2

includes minor fixes and more debug print statements

comment:14 Changed at 2016-01-01T00:39:49Z by daira

Version 0, edited at 2016-01-01T00:39:49Z by daira (next)

comment:15 Changed at 2016-01-25T14:52:59Z by dawuud

the current state of this ticket is that we cannot get the new unit test to pass.

comment:17 Changed at 2016-01-27T13:00:23Z by dawuud

here's the latest dev branch from the pairing session with Daira: https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.4

we did not get the unit test passing... there wasn't enough time. we thought we had the solution but seeing is believing ;-p i tried adding a callback to the upload_d deferred to do another clock advance but the test still hangs.

comment:18 Changed at 2016-01-27T16:04:00Z by daira

We fixed this in the pairing today :-)

comment:19 Changed at 2016-01-27T16:06:59Z by dawuud

Fixed in this dev branch: https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.4

here's the pull request into stable.10: https://github.com/tahoe-lafs/tahoe-lafs/pull/231/files

if we want to squash history then i can do that... but anyway we can use this as a place for code review.

comment:20 Changed at 2016-01-27T16:08:21Z by dawuud

  • Status changed from assigned to new

comment:21 Changed at 2016-02-03T11:34:05Z by dawuud

daira fixed the periodic uploader unit test: https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.4

i think we can review and then merge this into our latest stable branch soon.

comment:22 Changed at 2016-02-08T13:29:27Z by dawuud

  • Resolution set to fixed
  • Status changed from new to closed

it looks like we can close this ticket because the code has already been merged into our stable.12 branch;

https://github.com/tahoe-lafs/tahoe-lafs/pull/237/files

Note: See TracTickets for help on using tickets.