PyFilesystem backend

[zooko]

PyFilesystem is a python library to abstract various sorts of filesystemish things:

​http://code.google.com/p/pyfilesystem/

It just got included in Debian:

​http://packages.debian.org/sid/python-fs

This ticket is a wish that someone would make a Tahoe-LAFS backend for PyFilesystem and contribute it to PyFilesystem.

[warner]

Looks pretty cool. From what I can tell, PyFilesystem is entirely synchronous, so the tahoe backend would really be a wrapper around the httplib-based webapi interfaces like those used in tahoe ls and tahoe get (as opposed to something you might use inside the Tahoe client node process). When you instantiate a tahoe-backed FS object, the arguments would probably be a nodeurl and a rootcap.

Neat: PyFilesystem has code to expose the FS you create to FUSE and SFTP, among other frontends.

[zooko]

Hey look! slush (marek@…) did this!

​http://code.google.com/p/pyfilesystem/issues/detail?id=28

I haven't had time to read through the whole ticket or the code yet. This is cool! Has anyone used it for anything?

[slush]

Has anyone used it for anything?

Ehm, me :-). I'm using it with Dokan (virtual drive for Windows) as my main interface for TahoeLAFS.

Those four lines does the trick:

from fs.contrib.tahoefs import TahoeFS
from fs.expose import dokan
fs = TahoeFS('URI:DIR2:<your dircap>')
mp = dokan.mount(fs, "f", foreground=True)

[zooko]

Awesome.

[davidsarah]

This is way cool! But it needs to be reviewed for security, correctness, test coverage, etc., so I'm reopening the ticket for that.

[zooko]

I imagine that it is the responsibility of the maintainers of pyfilesystem to ensure that. However, I would be happy to contribute some help. Maybe we should ask them if they want such help, and if so open a ticket on the pyfilesystem issue tracker to track whether the patch has tests and review?

[slush]

TahoeFS is using standard PyFS test coverage (​http://code.google.com/p/pyfilesystem/source/browse/trunk/fs/contrib/tahoefs/test_tahoefs.py) and pass it without any troubles on VolunteerGrid. I switched off only those tests which require atomic file operations, which is not available on TahoeLAFS. I'm open to any code review and improvements. I enjoyed hacking TahoeLAFS and PyFS together during my summer holiday in Italy, so maybe good wine and beautiful Italian women affected quality of final code ;).

[slush]

If PyFS need some help, it is better RemoteFileBuffer?, which manage loading and storing files to remote filesystems (in general, not only TahoeLAFS). Opening large remotte files directly from applications (movie) is still with troubles, because seeking in remote file needs loading full content before. I have improvements of this in my head, but time is my problem now.

[davidsarah]

Replying to slush:

If PyFS need some help, it is better RemoteFileBuffer?, which manage loading and storing files to remote filesystems (in general, not only TahoeLAFS). Opening large remotte files directly from applications (movie) is still with troubles, because seeking in remote file needs loading full content before. I have improvements of this in my head, but time is my problem now.

(RemoteFileBuffer source is ​here.)

The SFTP frontend has OverwriteableFileConsumer which is doing essentially the same thing. Like RemoteFileBuffer, reading a chunk waits until all of the data up to the end of that chunk has been downloaded; segments are never downloaded out-of-order. Unlike RemoteFileBuffer, it supports writing chunks of the file that haven't been read downloaded yet.

Out-of-order downloads would require use of HTTP range requests, if the connection to the web-API gateway is over HTTP.

[zooko]

What's the status of this ticket? I say we close this as "fixed", or perhaps as "this was never Tahoe-LAFS's job in the first place". :-)

If there are other changes that we could make in Tahoe-LAFS to improve the behavior or performance or testability of the PyFilesystem backend, we can open a new ticket for that. If there are changes that we could make in PyFilesystem to improve things then we could open a ticket in PyFilesystem's issue tracker.

[zooko]

Well, Tahoe-LAFS appears to be compatible with pyfilesystem. Closing this ticket as invalid because it didn't really require any changes to Tahoe-LAFS as such. If anybody wants to cooperate with us on improving Tahoe-LAFS+pyfilesystem integration (for example, setting up a buildbot to run automated tests!), just get in touch!