tahoe-lafs/trac

Magic Folder on Linux: handle case when directory is removed from monitored directory #2427

New issue

Closed

opened 2015-05-21 18:00:09 +00:00 by david415 · 12 comments

david415 commented

2015-05-21 18:00:09 +00:00

We need to handle the case where a directory is removed or moved out of the monitored local directory.

Specifically this case needs to be specially handled only when the sub-directory was moved into the drop uploader directory and a inotify watch for that sub-directory was added. Further investigation is needed to determine the proper way to remove this inotify watch.

~~So far it seems that we will respond to either the IN_DELETE_SELF or IN_MOVE_SELF events for each of these sub-directory inotify "watches".~~

We need to handle the case where a directory is removed or moved out of the monitored local directory. ~~Specifically this case needs to be specially handled only when the sub-directory was moved into the drop uploader directory and a inotify watch for that sub-directory was added. Further investigation is needed to determine the proper way to remove this inotify watch.~~ ~~So far it seems that we will respond to either the `IN_DELETE_SELF` or `IN_MOVE_SELF` events for each of these sub-directory inotify "watches".~~

david415 added the

labels 2015-05-21 18:00:09 +00:00

david415 added this to the undecided milestone 2015-05-21 18:00:09 +00:00

daira was assigned by david415

2015-05-21 18:00:09 +00:00

daira commented

2015-05-23 18:49:33 +00:00

It seems as though the IN_EXCL_UNLINK flag would help with this:

   IN_EXCL_UNLINK (since Linux 2.6.36)
          By default, when watching events on the children of a
          directory, events are generated for children even after
          they have been unlinked from the directory.  This can
          result in large numbers of uninteresting events for some
          applications (e.g., if watching /tmp, in which many
          applications create temporary files whose names are
          immediately unlinked).  Specifying IN_EXCL_UNLINK changes
          the default behavior, so that events are not generated for
          children after they have been unlinked from the watched
          directory.

It seems as though the `IN_EXCL_UNLINK` flag would help with this: ``` IN_EXCL_UNLINK (since Linux 2.6.36) By default, when watching events on the children of a directory, events are generated for children even after they have been unlinked from the directory. This can result in large numbers of uninteresting events for some applications (e.g., if watching /tmp, in which many applications create temporary files whose names are immediately unlinked). Specifying IN_EXCL_UNLINK changes the default behavior, so that events are not generated for children after they have been unlinked from the watched directory. ```

daira commented

2015-05-24 14:37:45 +00:00

The constant for IN_EXCL_UNLINK isn't defined by Twisted (https://twistedmatrix.com/trac/ticket/7789) but that's okay, we can define it ourselves, since Twisted passes through unknown flags.

The constant for IN_EXCL_UNLINK isn't defined by Twisted (<https://twistedmatrix.com/trac/ticket/7789>) but that's okay, we can define it ourselves, since Twisted passes through unknown flags.

daira commented

2015-05-24 16:55:10 +00:00

Note that this is not an issue on Windows, because ReadDirectoryChangesW only reads events that occur for paths that are still under the recursively-watched directory.

Note that this is not an issue on Windows, because `ReadDirectoryChangesW` only reads events that occur for paths that are still under the recursively-watched directory.

daira removed the

c/unknown

label 2015-05-24 16:55:29 +00:00

daira modified the milestone from undecided to 1.11.0

2015-05-24 16:55:29 +00:00

daira changed title from ~~handle case when directory is removed from drop upload directory~~ to drop-upload on Linux: handle case when directory is removed from drop upload directory

2015-05-24 16:56:21 +00:00

daira removed their assignment 2015-05-26 09:54:26 +00:00

david415 was assigned by daira

2015-05-26 09:54:26 +00:00

david415 commented

2015-05-26 18:28:03 +00:00

Author

Furthermore these additional tests should also demonstrate that a directory which has been moved into the drop upload directory will have the IN_EXCL_UNLINK behavior just like the upload directory itself.

daira commented

2015-06-01 16:11:09 +00:00

Add magic-folder keyword to all drop-upload tickets.

daira changed title from ~~drop-upload on Linux: handle case when directory is removed from drop upload directory~~ to Magic Folder on Linux: handle case when directory is removed from monitored directory

2015-06-01 16:54:54 +00:00

david415 commented

2015-06-03 20:04:10 +00:00

Author

I'm a little tiny bit frustrated trying to debug the unit tests. They seem to be written correctly but are producing unexpected results. Perhaps we've overlooked a simple and silly mistake somewhere.

I've pushed a commit to my dev branch which demonstrates some broken behavior; a directory with a file (small_tree) is moved into the upload directory. The directory and file are uploaded... however when an additional file is written to the subdir... it is not uploaded. So confusing!

https://github.com/david415/tahoe-lafs/tree/2427.remove-uploader-watch.1

I'm a little tiny bit frustrated trying to debug the unit tests. They seem to be written correctly but are producing unexpected results. Perhaps we've overlooked a simple and silly mistake somewhere. I've pushed a commit to my dev branch which demonstrates some broken behavior; a directory with a file (small_tree) is moved into the upload directory. The directory and file are uploaded... however when an additional file is written to the subdir... it is not uploaded. So confusing! <https://github.com/david415/tahoe-lafs/tree/2427.remove-uploader-watch.1>

david415 removed their assignment 2015-06-03 20:04:24 +00:00

daira was assigned by david415

2015-06-03 20:04:24 +00:00

daira commented

2015-06-03 22:01:53 +00:00

Remember that we removed the autoAdd=True argument from the initial call to watch, in https://github.com/david415/tahoe-lafs/commit/e57645a9f15a7223eb33984cdd495a55b246e474. We did this because we believed that adding the extra explicit call to watch in _add_dir (here) was the correct way to handle directories being moved into the monitored directory tree, since we had found that autoAdd does not implicitly add a watch in that case.

It's likely that the problem is in that code; I would suggest adding a print to make sure that watch is being called for the moved-in directory, and with the correct arguments.

Remember that we removed the `autoAdd=True` argument from the initial call to `watch`, in <https://github.com/david415/tahoe-lafs/commit/e57645a9f15a7223eb33984cdd495a55b246e474>. We did this because we believed that adding the extra explicit call to `watch` in `_add_dir` ([here](https://github.com/david415/tahoe-lafs/commit/e57645a9f15a7223eb33984cdd495a55b246e474#diff-d8c61dbeca89240ad731932115de07d7R185)) was the correct way to handle directories being moved into the monitored directory tree, since we had found that `autoAdd` does not implicitly add a watch in that case. It's likely that the problem is in that code; I would suggest adding a `print` to make sure that `watch` is being called for the moved-in directory, and with the correct arguments.

daira removed their assignment 2015-06-03 22:01:53 +00:00

david415 was assigned by daira

2015-06-03 22:01:53 +00:00

david415 commented

2015-06-03 23:31:38 +00:00

Author

yess! i got the real tests passing in my dev branch but i'll have to fix the mock tests later tomorrow.

daira commented

2015-06-04 10:15:23 +00:00

\o/. Looking forward to seeing what the problem was.

david415 commented

2015-06-04 23:45:59 +00:00

Author

the problem was that i accidentally had some important code bits commented out...
and the original problem was solved by our not using the autoadd feature.
fixed here:
https://github.com/david415/tahoe-lafs/commits/2427.remove-uploader-watch.1
https://github.com/david415/tahoe-lafs/commit/fe797eb76b88d3e31da842551b2abe2d5084b6e6

the problem was that i accidentally had some important code bits commented out... and the original problem was solved by our not using the autoadd feature. fixed here: <https://github.com/david415/tahoe-lafs/commits/2427.remove-uploader-watch.1> <https://github.com/david415/tahoe-lafs/commit/fe797eb76b88d3e31da842551b2abe2d5084b6e6>