Fix for mutable files with FTP

[frozenfire]

FTP hangs when a directory list contains mutable files, as their filesize is the string "?", and Twisted's ftpd expects an integer. This patch by warner should fix it.

[frozenfire]

The patch, in diff/patch format

[zooko]

Actually it is the mutable files whose size shows up as "?". (This is because their size could change, and it would be expensive to examine each child to find out its current size when listing the parent directory.)

If only the code changed by this patch were covered by automated tests, I would consider including this patch in Tahoe-1.4.0. But in the interests of stability I don't want to apply patches to untested code just before the 1.4.0 release.

[warner]

I feel pretty comfortable about this patch. I'll try to apply it tonight. If 1.4.0 happens before then, I'd be ok with that too.

[frozenfire]

Hmm, this seems to misbehave with curlftpfs, will test with normal FTP client too. With curlftpfs it simply sees them as files of zero length, as the FTP server says. So they're unusable there.

[frozenfire]

Unusable in normal FTP client too D=

[warner]

Drat.. maybe the client is stubbornly remembering the "0" size, and refuses to download more than 0 bytes of it later on. A simple test of this would be to change that "if not a number, return 0" to "if not a number, return 1", and then see what the client does when you try to download the file.

Anyways, now I'm thinking we should *not* apply this patch before 1.4.0 .. better to keep the broken behavior the same, rather than switch it to a different broken behavior.

[zooko]

Should we add it to docs/known_issues.txt, with a suggested workaround saying "Don't use mutable files."?

[davidsarah]

FTP is a horrible protocol, and grokking its specification (RFC 959) is not helped by the fact that it has never been streamlined and updated for a world where bytes are always 8 bits and files are flat.

In any case, the relevant part of RFC 959 seems to be section 3.4, which says

The following transmission modes are defined in FTP:

3.4.1. STREAM MODE

If the structure is a file structure, the EOF is indicated by the sending host closing the data connection and all bytes are data bytes.

It is very unlikely that the FTP server library is using anything other than STREAM transmission mode with a file structure ( ​http://cr.yp.to/ftp/type.html says the other options are all obsolete). If so, then the tested clients are nonconformant in treating anything other than a close of the data connection as end-of-file. ​http://cr.yp.to/ftp/browsing.html seems to suggest that browser ftp clients do actually look for connection close to signal EOF -- so I'm stumped.

[davidsarah]

SFTP has a similar problem -- just writing up a new ticket about that. I think blackmatch fuse also gets this wrong.

[davidsarah]

Replying to zooko:

Should we add it to docs/known_issues.txt, with a suggested workaround saying "Don't use mutable files."?

Yes (more precisely, don't use mutable files with FTP). We should leave this ticket open since it might be fixable, even though I don't currently see how to do so reliably.

[zooko]

I guess this doesn't rise to the level of a Known Issue to be added to docs/known_issues.txt. (That file is more for things which might badly burn you if you don't know about them.) Let's add a note about this issue to the documentation of the (S)FTP interfaces.

[zooko]

Do we need to document this problem with mutable files in FTP? I don't think it is serious enough to deserve an entry in docs/known_issues.txt, but maybe in docs/frontends/FTP-and-SFTP.txt? Oh, I see there is already a note in there:

Mutable files are not supported by the FTP frontend.

(from docs/frontends/FTP-and-SFTP.txt@4439#L201) Hm, this documentation seems incomplete. What happens if I put a mutable file into a directory with tahoe put or tahoe ln or the wui, and then I list that directory using my FTP client?

[davidsarah]

Replying to zooko:

Do we need to document this problem with mutable files in FTP? I don't think it is serious enough to deserve an entry in docs/known_issues.txt, but maybe in docs/frontends/FTP-and-SFTP.txt? Oh, I see there is already a note in there:

Mutable files are not supported by the FTP frontend.

(from docs/frontends/FTP-and-SFTP.txt@4439#L201) Hm, this documentation seems incomplete. What happens if I put a mutable file into a directory with tahoe put or tahoe ln or the wui, and then I list that directory using my FTP client?

Well, according to the bug description it hangs (it's not clear whether the client hangs or the FTP server, probably the former).

The patch has not been applied because of the problem in comment:3. On the other hand, hanging on a directory listing is a more serious problem than failing to download the file correctly in some (broken) clients, so perhaps the patch should be applied anyway. Not for 1.7.0, though.