Magic Folder: tests fail on Travis-CI with UnicodeEncodeError from INotify.watch

[daira]

​https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/65770833

allmydata.test.test_drop_upload.RealTest.test_drop_upload ... Traceback (most recent call last):
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/test/test_drop_upload.py", line 57, in _create_uploader
    dbfile, inotify=self.inotify, pending_delay=0.2)
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/frontends/drop_upload.py", line 75, in __init__
    recursive=True)
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-15.2.1-py2.7-linux-x86_64.egg/twisted/internet/inotify.py", line 360, in watch
    recursive=False)
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-15.2.1-py2.7-linux-x86_64.egg/twisted/internet/inotify.py", line 368, in watch
    return self._addWatch(path, mask, autoAdd, callbacks)
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-15.2.1-py2.7-linux-x86_64.egg/twisted/internet/inotify.py", line 190, in _addWatch
    wd = self._inotify.add(self._fd, path.path, mask)
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-15.2.1-py2.7-linux-x86_64.egg/twisted/python/_inotify.py", line 40, in add
    wd = libc.inotify_add_watch(fd, path, mask)
ctypes.ArgumentError: argument 2: <type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode character u'\u0101' in position 126: ordinal not in range(128)
[ERROR]

[daira]

This needs a call to NonASCIIPathMixin.unicode_or_fallback​ to construct the filename. It would also be good if we could get Travis to accept Unicode filenames, since that provides better test coverage.

[daira]

See #2268.

[daira]

Oh wait, I think this is a different error than I originally thought: it could be due to the FilePath's path attribute holding a Unicode string rather than a str.

This is strictly speaking a Twisted bug: in Twisted 15.2.1 which is the version used in this test, FilePath is supposed to accept Unicode paths (and so inotify.py should be using path._asBytesPath() instead of path.path ​here). Nevertheless, constructing it with a str path may be the best workaround.

[daira]

Filed against Twisted as ​https://twistedmatrix.com/trac/ticket/7928.

[daira]

There are actually two bugs here, one in Tahoe-LAFS (we should not be attempting to test Unicode filenames that the filesystem encoding can't represent), and the one in Twisted. The latter only causes a test failure on the branches of Tahoe-LAFS that have Windows inotify-emulation support enabled (because only those branches use Unicode-mode FilePaths).

[daira]

Replying to daira:

There are actually two bugs here, one in Tahoe-LAFS (we should not be attempting to test Unicode filenames that the filesystem encoding can't represent), and the one in Twisted. The latter only causes a test failure on the branches of Tahoe-LAFS that have Windows inotify-emulation support enabled (because only those branches use Unicode-mode FilePaths).

Correction: I had already fixed the first bug on those branches. The remaining problem is that we have this code in encodingutil.py which attempts to determine whether a Unicode-mode FilePath should be used:

use_unicode_filepath = sys.platform == "win32" or hasattr(FilePath, '_asTextPath')

Because of the Twisted bug, for now this needs to be just:

use_unicode_filepath = sys.platform == "win32"

(We won't hit the Twisted bug on Windows because on that platform we use our own INotify emulation which *does* accept Unicode FilePaths, and which even works on versions of Twisted before the fix to ​https://twistedmatrix.com/trac/ticket/7928.)