webapi doesnt handle Range header correctly

[spreitzer]

The web-API hangs if a GET request with Range header is sent and the end byteRange is equal the size or size-1 if cap is a unterminated string.

given that filecap contains 'hi':

130 sspreitz@sspreitz:~/workspace/tahoefuse/src$ curl -v -r 0-0 http://127.0.0.1:3456/uri/URI:SSK:rei5f5wqbqycmq2mwtjygglyii:ho4xwwvgbvhwp6xyleafclcjikszuvilb2yzyap4slwgsnmz6joa                                                             
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 3456 (#0)
> GET /uri/URI:SSK:rei5f5wqbqycmq2mwtjygglyii:ho4xwwvgbvhwp6xyleafclcjikszuvilb2yzyap4slwgsnmz6joa HTTP/1.1
> Range: bytes=0-0
> User-Agent: curl/7.37.0
> Host: 127.0.0.1:3456
> Accept: */*
> 
< HTTP/1.1 206 Partial Content
< Content-Length: 1
< Accept-Ranges: bytes
* Server TwistedWeb/14.0.2 is not blacklisted
< Server: TwistedWeb/14.0.2
< Content-Range: bytes 0-0/2
< Date: Sat, 04 Jul 2015 15:02:39 GMT
< Content-Type: text/plain
< 
* Connection #0 to host 127.0.0.1 left intact
hsspreitz@sspreitz:~/workspace/tahoefuse/src$ curl -v -r 0-1 http://127.0.0.1:3456/uri/URI:SSK:rei5f5wqbqycmq2mwtjygglyii:ho4xwwvgbvhwp6xyleafclcjikszuvilb2yzyap4slwgsnmz6joa                                                                
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 3456 (#0)
> GET /uri/URI:SSK:rei5f5wqbqycmq2mwtjygglyii:ho4xwwvgbvhwp6xyleafclcjikszuvilb2yzyap4slwgsnmz6joa HTTP/1.1
> Range: bytes=0-1
> User-Agent: curl/7.37.0
> Host: 127.0.0.1:3456
> Accept: */*
> 
< HTTP/1.1 206 Partial Content
< Content-Length: 2
< Accept-Ranges: bytes
* Server TwistedWeb/14.0.2 is not blacklisted
< Server: TwistedWeb/14.0.2
< Content-Range: bytes 0-1/2
< Date: Sat, 04 Jul 2015 15:02:50 GMT
< Content-Type: text/plain
< 
^C
130 sspreitz@sspreitz:~/workspace/tahoefuse/src$ curl -v -r 0-2 http://127.0.0.1:3456/uri/URI:SSK:rei5f5wqbqycmq2mwtjygglyii:ho4xwwvgbvhwp6xyleafclcjikszuvilb2yzyap4slwgsnmz6joa                                                             
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 3456 (#0)
> GET /uri/URI:SSK:rei5f5wqbqycmq2mwtjygglyii:ho4xwwvgbvhwp6xyleafclcjikszuvilb2yzyap4slwgsnmz6joa HTTP/1.1
> Range: bytes=0-2
> User-Agent: curl/7.37.0
> Host: 127.0.0.1:3456
> Accept: */*
> 
< HTTP/1.1 206 Partial Content
< Content-Length: 2
< Accept-Ranges: bytes
* Server TwistedWeb/14.0.2 is not blacklisted
< Server: TwistedWeb/14.0.2
< Content-Range: bytes 0-1/2
< Date: Sat, 04 Jul 2015 15:02:56 GMT
< Content-Type: text/plain
< 
^C

Apache httpd returns:

sspreitz@sspreitz:~/workspace/tahoefuse/src$ curl -v -r 0-0 http://people.redhat.com/sspreitz/hi                                                                                                                                            
* Hostname was NOT found in DNS cache
*   Trying 10.5.19.28...
* Connected to people.redhat.com (10.5.19.28) port 80 (#0)
> GET /sspreitz/hi HTTP/1.1
> Range: bytes=0-0
> User-Agent: curl/7.37.0
> Host: people.redhat.com
> Accept: */*
> 
< HTTP/1.1 206 Partial Content
< Date: Sat, 04 Jul 2015 15:06:32 GMT
* Server Apache is not blacklisted
< Server: Apache
< Last-Modified: Sat, 04 Jul 2015 15:06:06 GMT
< ETag: "2-51a0e030688e9"
< Accept-Ranges: bytes
< Content-Length: 1
< Content-Range: bytes 0-0/2
< Connection: close
< 
* Closing connection 0
hsspreitz@sspreitz:~/workspace/tahoefuse/src$ curl -v -r 0-1 http://people.redhat.com/sspreitz/hi                                                                                                                                             
* Hostname was NOT found in DNS cache
*   Trying 10.5.19.28...
* Connected to people.redhat.com (10.5.19.28) port 80 (#0)
> GET /sspreitz/hi HTTP/1.1
> Range: bytes=0-1
> User-Agent: curl/7.37.0
> Host: people.redhat.com
> Accept: */*
> 
< HTTP/1.1 206 Partial Content
< Date: Sat, 04 Jul 2015 15:06:36 GMT
* Server Apache is not blacklisted
< Server: Apache
< Last-Modified: Sat, 04 Jul 2015 15:06:06 GMT
< ETag: "2-51a0e030688e9"
< Accept-Ranges: bytes
< Content-Length: 2
< Content-Range: bytes 0-1/2
< Connection: close
< 
* Closing connection 0
hisspreitz@sspreitz:~/workspace/tahoefuse/src$ curl -v -r 0-2 http://people.redhat.com/sspreitz/hi
* Hostname was NOT found in DNS cache
*   Trying 10.5.19.28...
* Connected to people.redhat.com (10.5.19.28) port 80 (#0)
> GET /sspreitz/hi HTTP/1.1
> Range: bytes=0-2
> User-Agent: curl/7.37.0
> Host: people.redhat.com
> Accept: */*
> 
< HTTP/1.1 206 Partial Content
< Date: Sat, 04 Jul 2015 15:06:42 GMT
* Server Apache is not blacklisted
< Server: Apache
< Last-Modified: Sat, 04 Jul 2015 15:06:06 GMT
< ETag: "2-51a0e030688e9"
< Accept-Ranges: bytes
< Content-Length: 2
< Content-Range: bytes 0-1/2
< Connection: close
< 
* Closing connection 0

[daira]

We can reproduce this problem with cURL and HTTPie. The command line for the latter is:

http -v http://127.0.0.1:3456/uri/${URI} 'Range:bytes=0-1'

for example. (This suggests that it is not a cURL-specific problem.)

[daira]

This seems to be specific to SDMF, which is strange. (Also tried LIT, MDMF, and 100-byte immutable files.)

[daira]

See ​https://tools.ietf.org/html/rfc2616#section-14.16 for the HTTP/1.1 spec. The correct response for the Range: bytes=0-2 case is the whole file. (A '416 Requested Range not satisfiable' error should not be returned in that case, because the requested range does have an overlap with the file contents. The web-API seems to correctly return a 416 error if the starting byte offset is past the end of the file.)

There is currently no test for Range requests for SDMF in test_mutable.py; only for MDMF. I have added one for SDMF on the ​https://github.com/tahoe-lafs/tahoe-lafs/commits/2659.test-sdmf-version-partial-read.0 branch.

The behaviour of this test (Version.test_partial_read_sdmf_*) is confusing; it works when the data is 100 bytes or 2 bytes, but not if it is 90 bytes. Perhaps there is an off-by-one error that only triggers when the size of the data is a multiple of k bytes? (See 2462#comment:3 for why that might happen.) But the case that hangs in this ticket is 2 bytes, which suggests that the test is not finding the same bug.