tahoe-lafs/trac

Magic Folder: periodically do a full uploader scan #2635

New issue

Closed

opened 2015-12-10 15:27:42 +00:00 by daira · 20 comments

daira commented

2015-12-10 15:27:42 +00:00

This would help to compensate for:

missed events, due to unreliability of the filesystem notification mechanism or queue overflows (#1430),
failed uploads.

This would help to compensate for: * missed events, due to unreliability of the filesystem notification mechanism or queue overflows (#1430), * failed uploads.

daira added the

c/code-frontend-magic-folder

p/normal

t/defect

v/unknown

labels 2015-12-10 15:27:42 +00:00

daira added this to the undecided milestone 2015-12-10 15:27:42 +00:00

daira self-assigned this 2015-12-10 15:27:42 +00:00

daira commented

2015-12-14 15:59:22 +00:00

Author

Work in progress on branch 2635.periodic-uploader-scan.0 (I hadn't got very far with it).

daira removed their assignment 2015-12-14 15:59:22 +00:00

david415 was assigned by daira

2015-12-14 15:59:22 +00:00

david415 commented

2015-12-15 16:25:23 +00:00

in my dev branch i taught the uploader's _scan method to use the deque instead of calling _process directly. this breaks all our magic-folder tests... so the next is to make the tests pass by firing the hook deferred when a directory and all it's children are done being uploaded.

https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.0

david415 commented

2015-12-16 15:08:02 +00:00

after teaching the uploader to use the deque to process directories i've been trying to get the tests to pass... and i'm not sure how to do it but i pushed my latest WIP comment anyways.

https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.0

maybe daira or meejah could help me with this?

after teaching the uploader to use the deque to process directories i've been trying to get the tests to pass... and i'm not sure how to do it but i pushed my latest WIP comment anyways. <https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.0> maybe daira or meejah could help me with this?

david415 removed their assignment 2015-12-16 15:08:17 +00:00

daira was assigned by david415

2015-12-16 15:08:17 +00:00

daira commented

2015-12-16 16:04:13 +00:00

Author

The current design is wrong because it assumes that _scan finishes processing all of the descendant directories under the directory it is called with, when actually it only adds callbacks to scan all of its direct children. This affects both the initial scan triggered by start_scanning (which assumes that all desendants have been removed from the _pending set), and the intent of the tests that move subtrees.

The current design is wrong because it assumes that `_scan` finishes processing all of the descendant directories under the directory it is called with, when actually it only adds callbacks to scan all of its direct children. This affects both the initial scan triggered by `start_scanning` (which assumes that all desendants have been removed from the `_pending` set), and the intent of the tests that move subtrees.

david415 commented

2015-12-17 11:06:16 +00:00

It's almost as if we want nested DeferredList`s representing nested directories such that we can determine when a given directory and all it's children have been uploaded.

It's almost as if we want nested [DeferredList](wiki/DeferredList)`s representing nested directories such that we can determine when a given directory and all it's children have been uploaded.

david415 commented

2015-12-17 16:23:19 +00:00

here's the dev branch from our pairing session today:
https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.1

here's the dev branch from our pairing session today: <https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.1>

david415 commented

2015-12-17 21:50:20 +00:00

ok... all the tests are passing again with this commit:
https://github.com/david415/tahoe-lafs/commit/4610a7f97d4823548129b4d483d7e50e4c5c18ee

next i need to periodic scan... should be easy.

ok... all the tests are passing again with this commit: <https://github.com/david415/tahoe-lafs/commit/4610a7f97d4823548129b4d483d7e50e4c5c18ee> next i need to periodic scan... should be easy.

david415 commented

2015-12-17 23:23:27 +00:00

here's a naive periodic full scan:
https://github.com/david415/tahoe-lafs/commit/d1eef67f7632ee96d8e3e30b1c8b5f9d084d8973

actually i think it's an open question how we should determine the full scan is complete and we should therefore perform another full scan now because our duration has elapsed.

just think how bad it would be if the full scan didn't complete before the next one started!

perhaps we can just check the pending set every X minutes and if it's empty then
do the full scan?

here's a naive periodic full scan: <https://github.com/david415/tahoe-lafs/commit/d1eef67f7632ee96d8e3e30b1c8b5f9d084d8973> actually i think it's an open question how we should determine the full scan is complete and we should therefore perform another full scan now because our duration has elapsed. just think how bad it would be if the full scan didn't complete before the next one started! perhaps we can just check the pending set every X minutes and if it's empty then do the full scan?

david415 commented

2015-12-18 00:37:03 +00:00

ok i made it only do the full scan if the pending set is empty.
i also added a unit test... it's a MockTest similar to test_move_tree but upon file creation inside the magicfolder dir we do not emit a fake inotify event. instead we wait for the periodic scan and upload... however i am trying to use clock advances to speed up this test and it's not yet passing.

ok i made it only do the full scan if the pending set is empty. i also added a unit test... it's a [MockTest](wiki/MockTest) similar to test_move_tree but upon file creation inside the magicfolder dir we do not emit a fake inotify event. instead we wait for the periodic scan and upload... however i am trying to use clock advances to speed up this test and it's not yet passing.

daira commented

2015-12-21 17:57:49 +00:00

Author

Reviewing progress so far.

daira commented

2015-12-21 22:30:43 +00:00

Author

Rebased to https://github.com/tahoe-lafs/tahoe-lafs/commits/2635.periodic-uploader-scan.2.

Rebased to <https://github.com/tahoe-lafs/tahoe-lafs/commits/2635.periodic-uploader-scan.2>.

david415 commented

2015-12-22 16:14:30 +00:00

work in progress from out pairing session:
https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.2

includes minor fixes and more debug print statements

work in progress from out pairing session: <https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.2> includes minor fixes and more debug print statements

daira commented

2016-01-01 00:39:49 +00:00

Author

The current rebased version of this branch is https://github.com/tahoe-lafs/tahoe-lafs/tree/2635.periodic-uploader-scan.3. It doesn't include https://github.com/david415/tahoe-lafs/commit/b729c9ae5ee70eca36cc6b4c304f1b0a808d137d but you can cherry-pick that.

The current rebased version of this branch is <https://github.com/tahoe-lafs/tahoe-lafs/tree/2635.periodic-uploader-scan.3>. It doesn't include <https://github.com/david415/tahoe-lafs/commit/b729c9ae5ee70eca36cc6b4c304f1b0a808d137d> but you can cherry-pick that.

david415 commented

2016-01-25 14:52:59 +00:00

the current state of this ticket is that we cannot get the new unit test to pass.

david415 commented

2016-01-25 15:21:40 +00:00

(https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.3)

david415 commented

2016-01-27 13:00:23 +00:00

here's the latest dev branch from the pairing session with Daira:
https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.4

we did not get the unit test passing... there wasn't enough time. we thought we had the solution but seeing is believing ;-p i tried adding a callback to the upload_d deferred to do another clock advance but the test still hangs.

here's the latest dev branch from the pairing session with Daira: <https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.4> we did not get the unit test passing... there wasn't enough time. we thought we had the solution but seeing is believing ;-p i tried adding a callback to the upload_d deferred to do another clock advance but the test still hangs.

daira commented

2016-01-27 16:04:00 +00:00

Author

We fixed this in the pairing today :-)

david415 commented

2016-01-27 16:06:59 +00:00

Fixed in this dev branch:
https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.4

here's the pull request into stable.10:
https://github.com/tahoe-lafs/tahoe-lafs/pull/231/files

if we want to squash history then i can do that... but anyway we can use this as a place for code review.

Fixed in this dev branch: <https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.4> here's the pull request into stable.10: <https://github.com/tahoe-lafs/tahoe-lafs/pull/231/files> if we want to squash history then i can do that... but anyway we can use this as a place for code review.

david415 commented

2016-02-03 11:34:05 +00:00

daira fixed the periodic uploader unit test:
https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.4

i think we can review and then merge this into our latest stable branch soon.

daira fixed the periodic uploader unit test: <https://github.com/david415/tahoe-lafs/tree/2635.periodic-uploader-scan.4> i think we can review and then merge this into our latest stable branch soon.

david415 commented

2016-02-08 13:29:27 +00:00

it looks like we can close this ticket because the code has already been merged into our stable.12 branch;

https://github.com/tahoe-lafs/tahoe-lafs/pull/237/files

it looks like we can close this ticket because the code has already been merged into our stable.12 branch; <https://github.com/tahoe-lafs/tahoe-lafs/pull/237/files>

Rows
Columns