investigate best FUSE+sshfs options to use for performance and correctness of SFTP via sshfs

[davidsarah]

It looks as though at least direct_io and big_writes may be beneficial, so that writes are not limited to 4 KiB blocks.

[bj0]

I started doing a couple random tests I could think of. I didn't repeat many of the tests, since they took a bit of time and it's all manual, but it sort of gives an idea:

Most of these tests were copying a large file (728MB .iso file) to and from a tahoe introducer/storage client running inside a VirtualBox? VM on the same host (both guest and host running ubuntu). Most of the copying was done with "time rsync -rhPa ", and when copying a large file to tahoe, after the transfer the command hangs for another minute or so. I checked the flog during this time and there was activity (read i think) so it may be that rsync tries to checksum the file after transfer, i'm not sure.

To verify the file was being transfered correctly, I also did "time md5sum mnt/iso". I would expect this to be similar to simply reading the file, but for some reason it performed differently...

with it mounted as: "sshfs -p PORT server:/ mnt/"

• rsync .iso TO tahoe: 4m51s to transfer ( 2.39 MB/s ), 6m14s total
• rsync .iso FROM tahoe: 5m42s ( 2.13 MB/s )
• cp .iso FROM tahoe: 5m31s
• time md5sum iso:
  • first try: 7m46s
  • second try: 5m13s

with it mounted as: "sshfs -p PORT -o direct_io,big_writes server:/ mnt/"

• rsync .iso TO tahoe: 3m37s to transfer ( 3.19 MB/s ), 5m1s total
• rsync .iso FROM tahoe: 5m15s ( 2.31 MB/s )
• cp .iso FROM tahoe: 5m31s
• time md5sum iso:
  • first try: 10m40s
  • second try: 9m25s

obviously I expect the values to fluctuate a bit, but it seems like direct_io,big_writes bumps up the write speed a bit, without really affecting the read speed. I'm not really sure why it hit md5sum time so bad...

I also tried to rsync a large directory of source files (4mb, 981 files) to tahoe, but it seems to be acting odd, and stalls a lot, resulting in a very long transfer time (120 - 140 minutes). This happened with and without the options.

[zooko]

Dear bj0: thanks for the report!

What version(s) of Tahoe-LAFS were you using? If you have just been tracking the official trunk repo at ​http://tahoe-lafs.org/source/tahoe-lafs/trunk and haven't applied any other patches, then you can find out by running make make-version.

[davidsarah]

Thanks bj0.

big_writes should only affect writes, and I can't immediately see why it would have anything but a beneficial effect. direct_io might affect both reads and writes, and could cause some loss of performance for applications that are relying for performance on kernel caching. Can you try the same tests with -o big_writes only?

[I initially suggested both because ​http://xtreemfs.blogspot.com/2008/08/fuse-performance.html said that direct_io was needed (at least for some version of sshfs and Linux tested in 2008) to support writing in blocks greater than 4 KiB. However, point 2 in ​http://article.gmane.org/gmane.comp.file-systems.fuse.devel/5292 suggests that this restriction might have been lifted.]

What Linux kernel version and sshfs version did you use?

[bj0]

make make-version returned: setup.py darcsver: wrote '1.8.0c2-r4702' into src/allmydata/_version.py

sshfs Version: 2.2-1build1 (from ubuntu repo)

client uname -a: Linux nazgul 2.6.32-020632-generic #020632 SMP Thu Dec 3 10:09:58 UTC 2009 x86_64 GNU/Linux server uname -a: Linux testbuntu 2.6.32-24-generic #41-Ubuntu SMP Thu Aug 19 01:12:52 UTC 2010 i686 GNU/Linux

I was going to try the tests without direct_io, but I seem to be having trouble with my vm...