Magic Folder on Mac OS X and other BSDs #1432

New issue

Closed

opened 2011-07-17 02:13:59 +00:00 by daira · 45 comments

daira commented

2011-07-17 02:13:59 +00:00

The Magic Folder feature currently only works on Linux (where it uses inotify to detect filesystem changes), and Windows (where it uses ReadDirectoryChangesW). This ticket is about supporting the same feature on Mac OS X and possibly other BSD-based operating systems.

Mac OS X has the fsevents API, and all BSDs including Mac OS X have kqueue/kevent. If I understand correctly, neither fsevents nor kqueue/kevent give notifications of which files have changed, so the implementation will have to scan the directory and look at last-modified times to determine that. (This is occasionally also necessary for correctness on Linux and Windows; see #1430.)

The Magic Folder feature currently only works on Linux (where it uses inotify to detect filesystem changes), and Windows (where it uses `ReadDirectoryChangesW`). This ticket is about supporting the same feature on Mac OS X and possibly other BSD-based operating systems. Mac OS X has the [fsevents](http://en.wikipedia.org/wiki/FSEvents) API, and all BSDs including Mac OS X have kqueue/kevent. If I understand correctly, neither fsevents nor kqueue/kevent give notifications of which files have changed, so the implementation will have to scan the directory and look at last-modified times to determine that. (This is occasionally also necessary for correctness on Linux and Windows; see #1430.)

daira added the

labels 2011-07-17 02:13:59 +00:00

daira added this to the undecided milestone 2011-07-17 02:13:59 +00:00

daira changed title from ~~drop-upload on Mac OS X~~ to drop-upload on Mac OS X and other BSDs

2011-07-17 02:21:50 +00:00

daira added

t/enhancement

and removed

t/defect

labels 2011-07-25 12:26:43 +00:00

daira commented

2011-08-15 19:38:42 +00:00

Author

Python's select module supports kqueue, which is a lot easier and less error-prone than accessing it via ctypes or an extension module. Note that a critical bug in that support was fixed in Python 2.6.5. (2.7 was released after 2.6.5 and has this fix.)

Python's [select](http://docs.python.org/library/select.html) module supports kqueue, which is a lot easier and less error-prone than accessing it via ctypes or an extension module. Note that a [critical bug](http://bugs.python.org/issue5910) in that support was fixed in Python 2.6.5. (2.7 was released after 2.6.5 and has this fix.)

daira added

p/normal

and removed

p/major

labels 2012-03-29 19:51:21 +00:00

malaparte commented

2012-11-15 06:07:59 +00:00

Owner

I thought I'd just save this thought here, because I don't know if I have time to look at it closely. Have you considered 'watchdog' http://packages.python.org/watchdog/ to watch for changes on a folder across platforms?

I thought I'd just save this thought here, because I don't know if I have time to look at it closely. Have you considered 'watchdog' <http://packages.python.org/watchdog/> to watch for changes on a folder across platforms?

daira commented

2012-11-15 23:39:14 +00:00

Author

watchdog uses a C extension module to access the fsevents API on OS X. I would strongly prefer to use ctypes for this (if needed, i.e. if we don't use kqueue instead) and not add any more native code dependencies.

watchdog uses a C [extension module](https://github.com/gorakhargosh/watchdog/blob/master/src/watchdog_fsevents.c) to access the fsevents API on OS X. I would strongly prefer to use ctypes for this (if needed, i.e. if we don't use kqueue instead) and not add any more native code dependencies.

daira commented

2012-11-16 00:22:50 +00:00

Author

Also,

davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet watchdog

http://cloc.sourceforge.net v 1.09  T=4.0 s (15.0 files/s, 2563.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          36           1286           2238           4010
C                                3            168            473            738
Bourne Shell                    13             71            188            273
make                             3             46              5            193
DOS Batch                        2             27              1            172
XML                              1              0              0            155
C/C++ Header                     1             28             68             93
YAML                             1              0              0             21
-------------------------------------------------------------------------------
SUM:                            60           1626           2973           5655
-------------------------------------------------------------------------------

plus watchdog's PyYaml, argh and pathtools dependencies

compared to:

davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet trunk/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-x86_64.egg/twisted/internet/inotify.py

http://cloc.sourceforge.net v 1.09  T=0.5 s (2.0 files/s, 810.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           1             84            140            181
-------------------------------------------------------------------------------
davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet trunk/src/allmydata/frontends/drop_upload.py

http://cloc.sourceforge.net v 1.09  T=0.5 s (2.0 files/s, 246.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           1             25             10             88
-------------------------------------------------------------------------------

plus how ever much code it takes to support Windows (about 200-300 code lines), and BSD/OS X.

Also, ``` davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet watchdog http://cloc.sourceforge.net v 1.09 T=4.0 s (15.0 files/s, 2563.5 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Python 36 1286 2238 4010 C 3 168 473 738 Bourne Shell 13 71 188 273 make 3 46 5 193 DOS Batch 2 27 1 172 XML 1 0 0 155 C/C++ Header 1 28 68 93 YAML 1 0 0 21 ------------------------------------------------------------------------------- SUM: 60 1626 2973 5655 ------------------------------------------------------------------------------- ``` plus watchdog's [PyYaml, argh and pathtools dependencies](https://github.com/gorakhargosh/watchdog/blob/937d16b0be1495b1563353014b865e1234dacc9b/setup.py#L91) compared to: ``` davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet trunk/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-x86_64.egg/twisted/internet/inotify.py http://cloc.sourceforge.net v 1.09 T=0.5 s (2.0 files/s, 810.0 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Python 1 84 140 181 ------------------------------------------------------------------------------- davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet trunk/src/allmydata/frontends/drop_upload.py http://cloc.sourceforge.net v 1.09 T=0.5 s (2.0 files/s, 246.0 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Python 1 25 10 88 ------------------------------------------------------------------------------- ``` plus how ever much code it takes to support Windows (about [200-300 code lines](https://tahoe-lafs.org/trac/tahoe-lafs/attachment/ticket/1431/drop-upload-including-windows.darcs.patch#L788)), and BSD/OS X.

daira commented

2012-11-16 00:29:27 +00:00

Author

I should mention that I don't care about lines of code per se; I'm just using it as a rough proxy measure for complexity and opportunity-for-bugs.

I should mention that I don't care about lines of code *per se*; I'm just using it as a rough proxy measure for complexity and opportunity-for-bugs.

warner removed the

c/code-frontend

label 2014-12-02 19:48:43 +00:00

daira commented

2015-06-01 16:11:09 +00:00

Author

Add magic-folder keyword to all drop-upload tickets.

daira changed title from ~~drop-upload on Mac OS X and other BSDs~~ to Magic Folder on Mac OS X and other BSDs

2015-10-28 23:18:23 +00:00

daira commented

2015-12-04 21:14:33 +00:00

Author

SpiderOak has an fsevents-based OS X filesystem watcher which is open-source (GPL): https://github.com/SpiderOak/spideroak_osx_fsevents

[I did not mean that we should necessarily use this as-is.]Edit:

SpiderOak has an fsevents-based OS X filesystem watcher which is open-source (GPL): <https://github.com/SpiderOak/spideroak_osx_fsevents> [I did not mean that we should necessarily use this as-is.]Edit:

david415 commented

2016-01-27 16:09:39 +00:00

step number one seems to be: find an OSX VPS provider!

zooko commented

2016-04-07 14:03:34 +00:00

I think we should start with https://github.com/SpiderOak/spideroak_osx_fsevents and then consider changing it or switching to other techniques or whatever after we have it running.

I think we should start with <https://github.com/SpiderOak/spideroak_osx_fsevents> and then consider changing it or switching to other techniques or whatever after we have it running.

zooko commented

2016-04-07 16:50:18 +00:00

Okay I looked at https://github.com/SpiderOak/spideroak_osx_fsevents/blob/master/main.c and other code in watchdog and so on and I changed my mind, because https://github.com/SpiderOak/spideroak_osx_fsevents doesn't have a Python API.

We already have a ctypes-based API to Linux and Windows, so I guess our code would be more consistent, and easy for us to write, if we also wrote a ctypes-based API to OSX?

Okay I looked at <https://github.com/SpiderOak/spideroak_osx_fsevents/blob/master/main.c> and other code in watchdog and so on and I changed my mind, because <https://github.com/SpiderOak/spideroak_osx_fsevents> doesn't have a Python API. We already have a ctypes-based API to Linux and Windows, so I guess our code would be more consistent, and easy for us to write, if we also wrote a ctypes-based API to OSX?

david415 commented

2016-04-13 12:03:40 +00:00

Of the several steps needed to setup the Apple async filesystem events API we need to firstly call FSEventStreamCreate.

There's a snippet of code showing the FSEventStreamCreate being used:

https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/FSEvents_ProgGuide/UsingtheFSEventsFramework/UsingtheFSEventsFramework.html

It doesn't look like ctypes allows the creation of c structs or objective c objects unless you redefine them and create them the ctypes way. So initially I was thinking that maybe recreating all the necessary structs/objective-c-objects in python ctypes manually is kinda tricky... and requires us to have the Apple source code or at least the .h header files with the struct/object definitions.

Perhaps the pyobjc library can help us to compose these objective-c structures so that we can use
ctypes to pass them into API functions. Here need a python equivalent for composing the CFArray of CFString which holds the filesystems paths to watch:

    CFStringRef mypath = CFSTR("/path/to/scan");

    CFArrayRef pathsToWatch = CFArrayCreate(NULL, (const void **)&mypath, 1, NULL);

    void *callbackInfo = NULL; // could put stream-specific data here.

    FSEventStreamRef stream;

    CFAbsoluteTime latency = 3.0; /* Latency in seconds */

The pyobjc module requies xcode and when I pip installed it after properly installed xcode it downloaded many more modules; I wonder if this thing is too heavy weight. We probably want to avoid heavyweight dependencies?

I also discovered that google wrote a ctypes wrapper for Apple objective-c structures in this project:

https://github.com/google/grr
https://github.com/google/grr/blob/master/grr/client/osx/objc.py

oh and i also found this other ctypes wrapper API here:
https://github.com/wbond/oscrypto/blob/master/oscrypto/_osx/_core_foundation_ctypes.py

Of the several steps needed to setup the Apple async filesystem events API we need to firstly call *FSEventStreamCreate*. There's a snippet of code showing the *FSEventStreamCreate* being used: <https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/FSEvents_ProgGuide/UsingtheFSEventsFramework/UsingtheFSEventsFramework.html> It doesn't look like ctypes allows the creation of c structs or objective c objects unless you redefine them and create them the ctypes way. So initially I was thinking that maybe recreating all the necessary structs/objective-c-objects in python ctypes manually is kinda tricky... and requires us to have the Apple source code or at least the .h header files with the struct/object definitions. Perhaps the pyobjc library can help us to compose these objective-c structures so that we can use ctypes to pass them into API functions. Here need a python equivalent for composing the CFArray of CFString which holds the filesystems paths to watch: ``` CFStringRef mypath = CFSTR("/path/to/scan"); CFArrayRef pathsToWatch = CFArrayCreate(NULL, (const void **)&mypath, 1, NULL); void *callbackInfo = NULL; // could put stream-specific data here. FSEventStreamRef stream; CFAbsoluteTime latency = 3.0; /* Latency in seconds */ ``` The pyobjc module requies xcode and when I pip installed it after properly installed xcode it downloaded many more modules; I wonder if this thing is too heavy weight. We probably want to avoid heavyweight dependencies? I also discovered that google wrote a ctypes wrapper for Apple objective-c structures in this project: <https://github.com/google/grr> <https://github.com/google/grr/blob/master/grr/client/osx/objc.py> oh and i also found this other ctypes wrapper API here: <https://github.com/wbond/oscrypto/blob/master/oscrypto/_osx/_core_foundation_ctypes.py>

david415 commented

2016-04-14 14:24:33 +00:00

Currently I have a failed attempt at creating this CFArray of CFString`s; running it results in a segmentation fault... however the VM i'm using won't write a core file. What to do?

(virtenv-objc) xcloud114:~ Xcloud$ python test.py
Segmentation fault: 11


import ctypes
from ctypes import POINTER, byref, create_string_buffer, addressof
from ctypes import cdll, c_void_p
from ctypes.util import find_library

# kCFStringEncodingUTF8                                                                                                           
UTF8 = 134217984
# kCFAllocatorDefault                                                                                                             
CF_DEFAULT_ALLOCATOR = None

core_services = cdll.LoadLibrary(find_library('CoreServices'))
core_foundation = cdll.LoadLibrary(find_library('CoreFoundation'))

def PyStringToCFString(pystring):
    return core_foundation.CFStringCreateWithCString(CF_DEFAULT_ALLOCATOR,
                                  pystring.encode('utf8'),
                                  UTF8)

my_cfarray = core_foundation.CFArrayCreate(None, None, 0, ctypes.c_int.in_dll(core_services, "kCFTypeArrayCallBacks"))
my_cfstr = PyStringToCFString("/Users/Xcloud/magic-folder")

core_foundation.CFArrayAppendValue(my_cfarray, my_cfstr)

Currently I have a failed attempt at creating this CFArray of CFString`s; running it results in a segmentation fault... however the VM i'm using won't write a core file. What to do? ``` (virtenv-objc) xcloud114:~ Xcloud$ python test.py Segmentation fault: 11 ``` ```/usr/bin/env python import ctypes from ctypes import POINTER, byref, create_string_buffer, addressof from ctypes import cdll, c_void_p from ctypes.util import find_library # kCFStringEncodingUTF8 UTF8 = 134217984 # kCFAllocatorDefault CF_DEFAULT_ALLOCATOR = None core_services = cdll.LoadLibrary(find_library('CoreServices')) core_foundation = cdll.LoadLibrary(find_library('CoreFoundation')) def PyStringToCFString(pystring): return core_foundation.CFStringCreateWithCString(CF_DEFAULT_ALLOCATOR, pystring.encode('utf8'), UTF8) my_cfarray = core_foundation.CFArrayCreate(None, None, 0, ctypes.c_int.in_dll(core_services, "kCFTypeArrayCallBacks")) my_cfstr = PyStringToCFString("/Users/Xcloud/magic-folder") core_foundation.CFArrayAppendValue(my_cfarray, my_cfstr) ```

david415 commented

2016-04-15 12:49:10 +00:00

today i wrote a partial working prototype for Mac OS X using the python watchdog library, here in my dev branch:

https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog.0

many of the magic-folder tests pass. however it seems that test_magic_folder and test_move_tree fail due to watchdog not recursively watching directories or not adding a watch to newly added directories or some such thing.

anyway we should be able to replace this usage of the watchdog api with a ctypes equivalent for OSX if we want to do that.

today i wrote a partial working prototype for Mac OS X using the python watchdog library, here in my dev branch: <https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog.0> many of the magic-folder tests pass. however it seems that test_magic_folder and test_move_tree fail due to watchdog not recursively watching directories or not adding a watch to newly added directories or some such thing. anyway we should be able to replace this usage of the watchdog api with a ctypes equivalent for OSX if we want to do that.

zooko commented

2016-04-15 12:59:01 +00:00

Whoo-hoo! Way to go!

david415 commented

2016-04-16 15:46:29 +00:00

we could of course replace it with two ctypes based inotify APIs. one for kqueue and one for Mac OS X fs events api...

currently there are only two or three unit tests that fail and these failures are not so terrible... it fails because it's emitting more events than the unit tests expect.

this actually is a BSD compatibility change since getting magic-folder to work with this watchdog based inotify API actually makes it work on all the BSD operating systems using kqueue and it should also work on Mac OS X using the native apple filesystem events api. we could of course replace it with two ctypes based inotify APIs. one for kqueue and one for Mac OS X fs events api... currently there are only two or three unit tests that fail and these failures are not so terrible... it fails because it's emitting more events than the unit tests expect.

david415 commented

2016-05-02 13:56:31 +00:00

the latest watchdog inotify magic-folder branch is here:
https://github.com/tahoe-lafs/tahoe-lafs/tree/1432.osx-watchdog.0

you may note that the travis ci tests fail but this is due to changes in the dot travis yaml file as we attempt to get it working in a linux container and an osx container.
this difficulty with the travis configuration was due to the removal of the config options specifying python and instead telling travis to run in either a linux or osx container. then we are required to manually install many package dependencies so that pip and tahoe-lafs function properly and so on. i have previously gotten the travis configuration file to work properly for linux and osx... but this indeed involved changing other configuration options such as enabling sudo.

what's the next step to getting this merged into tahoe master or magic-folder stable?

the latest watchdog inotify magic-folder branch is here: <https://github.com/tahoe-lafs/tahoe-lafs/tree/1432.osx-watchdog.0> you may note that the travis ci tests fail but this is due to changes in the dot travis yaml file as we attempt to get it working in a linux container and an osx container. this difficulty with the travis configuration was due to the removal of the config options specifying python and instead telling travis to run in either a linux or osx container. then we are required to manually install many package dependencies so that pip and tahoe-lafs function properly and so on. i have previously gotten the travis configuration file to work properly for linux and osx... but this indeed involved changing other configuration options such as enabling sudo. what's the next step to getting this merged into tahoe master or magic-folder stable?

daira commented

2016-05-02 14:59:20 +00:00

Author

dawuud: please paste the current test failures.

david415 commented

2016-05-03 15:33:25 +00:00

[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 1104, in <lambda>
    d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 4))
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 6 != 4

allmydata.test.test_magic_folder.RealTest.test_move_tree
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 687, in <lambda>
    d.addCallback(lambda ign: self._check_uploader_count('objects_not_uploaded', 1, magic=self.bob_magicfolder))
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 211, in _check_uploader_count
    expected)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 2 != 1

allmydata.test.test_magic_folder.RealTestAliceBob.test_alice_bob
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 169, in _check_file
    os.unlink(path_u)
exceptions.OSError: [Errno 2] No such file or directory: '/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/MockTest/test_magic_fo
lder/gIRrj8/temp/local_dir/tempfile'

allmydata.test.test_magic_folder.MockTest.test_magic_folder
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 169, in _check_file
    os.unlink(path_u)
exceptions.OSError: [Errno 2] No such file or directory: '/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_magic_fo
lder/ff93bG/temp/local_dir/tempfile'

allmydata.test.test_magic_folder.RealTest.test_magic_folder
-------------------------------------------------------------------------------
Ran 35 tests in 171.881s

FAILED (failures=2, errors=2, successes=31)

``` [FAIL] Traceback (most recent call last): File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 1104, in <lambda> d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 4)) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual self.failUnlessEqual(a, b, msg=msg) File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 6 != 4 allmydata.test.test_magic_folder.RealTest.test_move_tree =============================================================================== [FAIL] Traceback (most recent call last): File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 687, in <lambda> d.addCallback(lambda ign: self._check_uploader_count('objects_not_uploaded', 1, magic=self.bob_magicfolder)) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 211, in _check_uploader_count expected) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual self.failUnlessEqual(a, b, msg=msg) File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 2 != 1 allmydata.test.test_magic_folder.RealTestAliceBob.test_alice_bob =============================================================================== [ERROR] Traceback (most recent call last): File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks result = g.send(result) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 169, in _check_file os.unlink(path_u) exceptions.OSError: [Errno 2] No such file or directory: '/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/MockTest/test_magic_fo lder/gIRrj8/temp/local_dir/tempfile' allmydata.test.test_magic_folder.MockTest.test_magic_folder =============================================================================== [ERROR] Traceback (most recent call last): File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks result = g.send(result) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 169, in _check_file os.unlink(path_u) exceptions.OSError: [Errno 2] No such file or directory: '/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_magic_fo lder/ff93bG/temp/local_dir/tempfile' allmydata.test.test_magic_folder.RealTest.test_magic_folder ------------------------------------------------------------------------------- Ran 35 tests in 171.881s FAILED (failures=2, errors=2, successes=31) ```

david415 commented

2016-05-09 14:54:06 +00:00

the above test results were actually from an earlier commit;
this commit id
8939fbf16baa2cbd9ede7b23037b61dfbe323cdd

seems to have fewer test errors and failures than the HEAD of that branch. This is of course because the code review "corrections" caused more tests to fail... I'll have to track down which changes cause the problems. I suspect they are unicode related changes.

Furthermore our inotify/watchdog API must implement two more features:

allow the user to set an events mask to select which events to get notifications for
event attributes. the event messages may contain information about the file object which caused the events such as: file type (e.g. file or directory).

HOWEVER, given that the watchdog API passes nearly all the magic-folder unit tests, this means either the watchdog api is good enough or our unit test coverage sucks.

the above test results were actually from an earlier commit; this commit id 8939fbf16baa2cbd9ede7b23037b61dfbe323cdd seems to have fewer test errors and failures than the HEAD of that branch. This is of course because the code review "corrections" caused more tests to fail... I'll have to track down which changes cause the problems. I suspect they are unicode related changes. Furthermore our inotify/watchdog API must implement two more features: 1. allow the user to set an events mask to select which events to get notifications for 2. event attributes. the event messages may contain information about the file object which caused the events such as: file type (e.g. file or directory). HOWEVER, given that the watchdog API passes nearly all the magic-folder unit tests, this means either the watchdog api is good enough or our unit test coverage sucks.

david415 commented

2016-05-24 12:24:13 +00:00

i made a new dev branch with the watchdogi inotify stable changes on top of the latest stable magic-folder branch (stable 13):
https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.1

pull request here:
https://github.com/tahoe-lafs/tahoe-lafs/pull/287

i made a new dev branch with the watchdogi inotify stable changes on top of the latest stable magic-folder branch (stable 13): <https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.1> pull request here: <https://github.com/tahoe-lafs/tahoe-lafs/pull/287>

david415 commented

2016-05-25 14:36:49 +00:00

new dev branch based off of meejah's magic-folder stable 18 branch here:
https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.3

no errors, two failures:

===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 183, in _check_file
    previously_disappeared + 1)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 0 != 1

allmydata.test.test_magic_folder.RealTest.test_magic_folder
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 1131, in <lambda>
    d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 4))
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 6 != 4

allmydata.test.test_magic_folder.RealTest.test_move_tree
-------------------------------------------------------------------------------
Ran 35 tests in 182.773s

FAILED (failures=2, successes=33)

new dev branch based off of meejah's magic-folder stable 18 branch here: <https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.3> no errors, two failures: ``` =============================================================================== [FAIL] Traceback (most recent call last): File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks result = g.send(result) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 183, in _check_file previously_disappeared + 1) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual self.failUnlessEqual(a, b, msg=msg) File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 0 != 1 allmydata.test.test_magic_folder.RealTest.test_magic_folder =============================================================================== [FAIL] Traceback (most recent call last): File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 1131, in <lambda> d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 4)) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual self.failUnlessEqual(a, b, msg=msg) File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 6 != 4 allmydata.test.test_magic_folder.RealTest.test_move_tree ------------------------------------------------------------------------------- Ran 35 tests in 182.773s FAILED (failures=2, successes=33) ```

david415 commented

2016-05-25 15:29:30 +00:00

however a simple change, ignore events outside the watched directory shows the score a bit closer to the target:

===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 183, in _check_file
    previously_disappeared + 1)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in 
assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in 
_baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 0 != 1

allmydata.test.test_magic_folder.RealTest.test_magic_folder
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 1131, in <lambda>
    d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 4))
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in 
assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in 
_baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 3 != 4

allmydata.test.test_magic_folder.RealTest.test_move_tree
-------------------------------------------------------------------------------
Ran 35 tests in 188.915s

FAILED (failures=2, successes=33)

however a simple change, ignore events outside the watched directory shows the score a bit closer to the target: ``` =============================================================================== [FAIL] Traceback (most recent call last): File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks result = g.send(result) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 183, in _check_file previously_disappeared + 1) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual self.failUnlessEqual(a, b, msg=msg) File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 0 != 1 allmydata.test.test_magic_folder.RealTest.test_magic_folder =============================================================================== [FAIL] Traceback (most recent call last): File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 1131, in <lambda> d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 4)) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual self.failUnlessEqual(a, b, msg=msg) File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 3 != 4 allmydata.test.test_magic_folder.RealTest.test_move_tree ------------------------------------------------------------------------------- Ran 35 tests in 188.915s FAILED (failures=2, successes=33) ```

david415 commented

2016-05-26 14:29:05 +00:00

new error output from tests show all tests pass except one:

    test_magic_folder ... set pending delay                                                                                                   [2/1850]
init INotifyEventHandler
START READING BEGIN
START READING END
PROCESS EVENT <FileCreatedEvent: src_path='/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_magic_folder/Q5TLUN/te$p/local_dir/short'>
PROCESS EVENT <DirModifiedEvent: src_path='/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_magic_folder/Q5TLUN/te$
p/local_dir'>
DO CALLBACKS
                                               [FAIL]
stopReading begin
stopReading end
wait until stopped

===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 188, in _check_file
    self.failUnlessReallyEqual(actual_data, data)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 'test' != 'different'

allmydata.test.test_magic_folder.RealTest.test_magic_folder
-------------------------------------------------------------------------------
Ran 2 tests in 5.686s

FAILED (failures=1, successes=1)

new error output from tests show all tests pass except one: ``` test_magic_folder ... set pending delay [2/1850] init INotifyEventHandler START READING BEGIN START READING END PROCESS EVENT <FileCreatedEvent: src_path='/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_magic_folder/Q5TLUN/te$p/local_dir/short'> PROCESS EVENT <DirModifiedEvent: src_path='/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_magic_folder/Q5TLUN/te$ p/local_dir'> DO CALLBACKS [FAIL] stopReading begin stopReading end wait until stopped =============================================================================== [FAIL] Traceback (most recent call last): File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks result = g.send(result) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 188, in _check_file self.failUnlessReallyEqual(actual_data, data) File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual self.failUnlessEqual(a, b, msg=msg) File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual raise self.failureException(msg) twisted.trial.unittest.FailTest: 'test' != 'different' allmydata.test.test_magic_folder.RealTest.test_magic_folder ------------------------------------------------------------------------------- Ran 2 tests in 5.686s FAILED (failures=1, successes=1) ```

david415 commented

2016-05-30 15:26:12 +00:00

in another dev branch i saved a commit with added debug print statements so i could track down the root cause of the above unit test failure:
https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.3.wip.1

what is happening is that this test case fails:
https://github.com/david415/tahoe-lafs/blob/e451e2eaa410d84aa3ae0b62a5aaf5a4d4aa786b/src/allmydata/test/test_magic_folder.py#L1240

_check_file is failing on the temporary file case; specifically here's the comparison that fails:

https://github.com/david415/tahoe-lafs/blob/e451e2eaa410d84aa3ae0b62a5aaf5a4d4aa786b/src/allmydata/test/test_magic_folder.py#L182

yield self.failUnlessReallyEqual(self._get_count('uploader.objects_disappeared'),
                                             previously_disappeared + 1)

uploader.objects_disappeared 0
previously_disappeared + 1 == 1

OK so it uploads the temporary file when it shouldn't. But is that really so bad? Seems better than not uploading enough.

in another dev branch i saved a commit with added debug print statements so i could track down the root cause of the above unit test failure: <https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.3.wip.1> what is happening is that this test case fails: <https://github.com/david415/tahoe-lafs/blob/e451e2eaa410d84aa3ae0b62a5aaf5a4d4aa786b/src/allmydata/test/test_magic_folder.py#L1240> _check_file is failing on the temporary file case; specifically here's the comparison that fails: <https://github.com/david415/tahoe-lafs/blob/e451e2eaa410d84aa3ae0b62a5aaf5a4d4aa786b/src/allmydata/test/test_magic_folder.py#L182> ``` yield self.failUnlessReallyEqual(self._get_count('uploader.objects_disappeared'), previously_disappeared + 1) ``` ``` uploader.objects_disappeared 0 previously_disappeared + 1 == 1 ``` OK so it uploads the temporary file when it shouldn't. But is that really so bad? Seems better than not uploading enough.

david415 commented

2016-06-01 07:33:51 +00:00

I have had two conversations about magic-folder unit test race conditions; one with Daira and one with Meejah. Currently the plan is for Meejah to fix/refactor all the uploader unit tests after figuring out a better API for uploader testing similar in spirit to how we refactored the downloader tests. Here's the ticket for that:

https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2790

david415 commented

2016-06-22 15:59:41 +00:00

here i've put the watchdog inotify changes on top of meejah's stable 18 branch:
https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.4

here i've put the watchdog inotify changes on top of meejah's stable 18 branch: <https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.4>

david415 commented

2016-06-22 16:10:16 +00:00

oh dear the linux build passes but the osx build fails with the current travis yaml configuration::

RuntimeError: You are linking against OpenSSL 0.9.8, which is no longer support by the OpenSSL
project. You need to upgrade to a newer version of OpenSSL.

there's a ticket regarding automated testing on OSX here:
https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2789

oh dear the linux build passes but the osx build fails with the current travis yaml configuration:: [RuntimeError](wiki/RuntimeError): You are linking against OpenSSL 0.9.8, which is no longer support by the OpenSSL project. You need to upgrade to a newer version of OpenSSL. there's a ticket regarding automated testing on OSX here: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2789>

warner commented

2016-08-16 17:55:57 +00:00

BTW, that openssl thing was fixed. The travis OS-X builder was using the wrong version of python, and failing to pick up the newer openssl library. If you update the PR, travis should run with the proper version.

david415 commented

2016-08-17 13:05:49 +00:00

the current dev branch is:
https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.5

and current pull request is:
https://github.com/tahoe-lafs/tahoe-lafs/pull/308

the current dev branch is: <https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.5> and current pull request is: <https://github.com/tahoe-lafs/tahoe-lafs/pull/308>

david415 commented

2016-08-29 13:31:04 +00:00

currently on the darwin platform fails at least this test:
allmydata.test.test_magic_folder.RealTest.test_delete

I don't understand why this fails. I added a print statement to display the platform filesystem encoding which is UTF-8

test_delete ... set pending delay

init INotifyEventHandler

START READING BEGIN

START READING END

PROCESS EVENT <FileCreatedEvent: src_path='/Users/travis/build/tahoe-lafs/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_delete/nvdp3V/temp/local_dir/foo'>

FILESYSTEM ENCODING: utf-8

Exception in thread Thread-8:

Traceback (most recent call last):

> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner

    self.run()

> File "/Users/travis/build/tahoe-lafs/tahoe-lafs/.tox/py27/lib/python2.7/site-packages/watchdog/observers/api.py", line 199, in run

    self.dispatch_events(self.event_queue, self.timeout)

> File "/Users/travis/build/tahoe-lafs/tahoe-lafs/.tox/py27/lib/python2.7/site-packages/watchdog/observers/api.py", line 368, in dispatch_events

    handler.dispatch(event)

> File "/Users/travis/build/tahoe-lafs/tahoe-lafs/.tox/py27/lib/python2.7/site-packages/watchdog/events.py", line 322, in dispatch

    self.on_any_event(event)

> File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/watchdog/inotify.py", line 73, in on_any_event

    self.process(event)

> File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/watchdog/inotify.py", line 47, in process

    event_filepath_u = abspath_expanduser_unicode(event_filepath_u, base=self._path)

> File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/util/fileutil.py", line 310, in abspath_expanduser_unicode

    precondition_abspath(base)

> File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/util/fileutil.py", line 276, in precondition_abspath

    raise [AssertionError](wiki/AssertionError)("an abspath must be a Unicode string")

[AssertionError](wiki/AssertionError): an abspath must be a Unicode string

currently on the darwin platform fails at least this test: ``allmydata.test.test_magic_folder.RealTest.test_delete`` I don't understand why this fails. I added a print statement to display the platform filesystem encoding which is UTF-8 ``` test_delete ... set pending delay init INotifyEventHandler START READING BEGIN START READING END PROCESS EVENT <FileCreatedEvent: src_path='/Users/travis/build/tahoe-lafs/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_delete/nvdp3V/temp/local_dir/foo'> FILESYSTEM ENCODING: utf-8 Exception in thread Thread-8: Traceback (most recent call last): > File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() > File "/Users/travis/build/tahoe-lafs/tahoe-lafs/.tox/py27/lib/python2.7/site-packages/watchdog/observers/api.py", line 199, in run self.dispatch_events(self.event_queue, self.timeout) > File "/Users/travis/build/tahoe-lafs/tahoe-lafs/.tox/py27/lib/python2.7/site-packages/watchdog/observers/api.py", line 368, in dispatch_events handler.dispatch(event) > File "/Users/travis/build/tahoe-lafs/tahoe-lafs/.tox/py27/lib/python2.7/site-packages/watchdog/events.py", line 322, in dispatch self.on_any_event(event) > File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/watchdog/inotify.py", line 73, in on_any_event self.process(event) > File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/watchdog/inotify.py", line 47, in process event_filepath_u = abspath_expanduser_unicode(event_filepath_u, base=self._path) > File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/util/fileutil.py", line 310, in abspath_expanduser_unicode precondition_abspath(base) > File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/util/fileutil.py", line 276, in precondition_abspath raise [AssertionError](wiki/AssertionError)("an abspath must be a Unicode string") [AssertionError](wiki/AssertionError): an abspath must be a Unicode string ```

exarkun commented

2016-11-21 21:28:32 +00:00

Some changes built on david's last branch - https://github.com/exarkun/tahoe-lafs/tree/1432.osx-watchdog-stable.9

In particular, this adds a direct inotify-level test suite (copied from Twisted!) and makes some of the tests pass - including the IN_CLOSE_WRITE notification.

Some changes built on david's last branch - <https://github.com/exarkun/tahoe-lafs/tree/1432.osx-watchdog-stable.9> In particular, this adds a direct inotify-level test suite (copied from Twisted!) and makes some of the tests pass - including the IN_CLOSE_WRITE notification.

david415 commented

2016-11-23 20:49:22 +00:00

more progress here in this dev branch.
i've changed the inotify test suite to skip many of the tests for inotify
features that we do not use or need. I also modified some of the tests so
that they now pass when using either the watchdog inotify and the linux native inotify.
not yet tested with our windows inotify implementation.

https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.9

more progress here in this dev branch. i've changed the inotify test suite to skip many of the tests for inotify features that we do not use or need. I also modified some of the tests so that they now pass when using either the watchdog inotify and the linux native inotify. not yet tested with our windows inotify implementation. <https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.9>

david415 commented

2016-11-23 22:45:46 +00:00

now that we have a inotify test suite perhaps we can remove the "real" inotify magic folder tests?

in the above mentioned dev branch i've removed the real tests in this commit here:
https://github.com/david415/tahoe-lafs/commit/451dcf093b26b475378ef588e2cf68b40697cc8c

now that we have a inotify test suite perhaps we can remove the "real" inotify magic folder tests? in the above mentioned dev branch i've removed the real tests in this commit here: <https://github.com/david415/tahoe-lafs/commit/451dcf093b26b475378ef588e2cf68b40697cc8c>

meejah commented

2016-11-25 00:32:38 +00:00

Is the reactor.callFromThread() because the "watchdog" library is running the on_any_event from its own thread(s)?

Otherwise looks plausible. And yes I'd like to get rid of some of the "long" magic-folder tests especially. If our test show that all 3 underlying inotify-like implementations are doing the right thing, then yes it would be great to get rid of the "Real" vs "Mock" dual tests in magic-folder as well! (i.e. just have the Mock ones).

Is the `reactor.callFromThread()` because the "watchdog" library is running the `on_any_event` from its own thread(s)? Otherwise looks plausible. And yes I'd like to get rid of some of the "long" magic-folder tests especially. If our test show that all 3 underlying inotify-like implementations are doing the right thing, then yes it would be great to get rid of the "Real" vs "Mock" dual tests in magic-folder as well! (i.e. just have the Mock ones).

david415 commented

2016-12-08 19:37:30 +00:00

this latest dev branch is currently in a pull-request
https://github.com/tahoe-lafs/tahoe-lafs/pull/381

all tests pass except for the appveyor testing on windows.
currently i can see that the inotify tests are failing for windows;
the test_attrib fails and test_closeWrite timesout.

meejah and i plan to look at this soon.

this latest dev branch is currently in a pull-request <https://github.com/tahoe-lafs/tahoe-lafs/pull/381> all tests pass except for the appveyor testing on windows. currently i can see that the inotify tests are failing for windows; the test_attrib fails and test_closeWrite timesout. meejah and i plan to look at this soon.

david415 commented

2017-01-09 16:02:08 +00:00

i'm trying to make this branch pass all the tests.
in my latest commit i'm trying to make the test_inotify tests be skipped on windows...
we can make the tests pass on windows later.
for now i'd like to get the bsd changes merged if possible.

i'm trying to make this branch pass all the tests. in my latest commit i'm trying to make the test_inotify tests be skipped on windows... we can make the tests pass on windows later. for now i'd like to get the bsd changes merged if possible.

david415 commented

2017-02-14 23:53:08 +00:00

my latest dev branch:
https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.10

all tests pass. the magic-folder "real inotify" tests have been disabled.
the watch inotify implementation has it's own test suite that was borrowed and inspired from twisted's inotify test suite.

my latest dev branch: <https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.10> all tests pass. the magic-folder "real inotify" tests have been disabled. the watch inotify implementation has it's own test suite that was borrowed and inspired from twisted's inotify test suite.

david415 commented

2017-02-15 18:13:55 +00:00

No description provided.

david415 commented

2017-02-20 18:31:25 +00:00

attempting to skip the inotify tests on windows:
https://github.com/tahoe-lafs/tahoe-lafs/pull/401

the tests seem to be skipped but prints dirty reactor errors like this:

test_closeWrite ... SKIPPED
Traceback (most recent call last):
Failure: twisted.trial.util.DirtyReactorAggregateError: Reactor was unclean.
DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug)
<DelayedCall 0x4af3eb8 [0.00999999046326s] called=0 cancelled=0 LoopingCall<0.01>(INotify._poll, *(<function at 0x069DE770>, 1487535067.863), **{})()>
ERROR

why!? how to fix?

attempting to skip the inotify tests on windows: <https://github.com/tahoe-lafs/tahoe-lafs/pull/401> the tests seem to be skipped but prints dirty reactor errors like this: test_closeWrite ... SKIPPED Traceback (most recent call last): Failure: twisted.trial.util.DirtyReactorAggregateError: Reactor was unclean. [DelayedCalls](wiki/DelayedCalls): (set twisted.internet.base.DelayedCall.debug = True to debug) <DelayedCall 0x4af3eb8 [0.00999999046326s] called=0 cancelled=0 [LoopingCall](wiki/LoopingCall)<0.01>(INotify._poll, *(<function <lambda> at 0x069DE770>, 1487535067.863), **{})()> ERROR why!? how to fix?

david415 commented

2017-02-20 20:50:29 +00:00

fixed it

crwood commented

2018-07-26 01:02:01 +00:00

Freshly resurrected in PR 516.

Freshly resurrected in [PR 516](https://github.com/tahoe-lafs/tahoe-lafs/pull/516).

exarkun commented

2019-01-31 19:47:50 +00:00

Bad news. Watchdog upstream have decided to stop supporting Python 2.

https://github.com/gorakhargosh/watchdog/issues/516

Bad news. Watchdog upstream have decided to stop supporting Python 2. <https://github.com/gorakhargosh/watchdog/issues/516>

exarkun commented

2019-02-22 19:21:54 +00:00

They changed their mind. There are some other issues, though, like events being missed with the fsevents backend on macOS.

GitHub <noreply@github.com> commented

2019-03-14 17:02:13 +00:00

Owner

In c1e6f08/trunk:

Merge pull request #558 from tahoe-lafs/1432.watchdog-magic-folder-with-eliot

Add support for macOS to Magic-Folders.

Fixes: ticket:1432

In [c1e6f08/trunk](/tahoe-lafs/trac/commit/c1e6f0881331b6549337af651e87e2f1521c8945): ``` Merge pull request #558 from tahoe-lafs/1432.watchdog-magic-folder-with-eliot Add support for macOS to Magic-Folders. Fixes: ticket:1432 ```