Opened at 2011-10-25T11:02:37Z
Closed at 2020-10-30T12:35:44Z
#1570 closed defect (wontfix)
S3 backend: support streaming writes to immutable shares
| Reported by: | davidsarah | Owned by: | |
|---|---|---|---|
| Priority: | major | Milestone: | undecided |
| Component: | code-storage | Version: | 1.9.0b1 |
| Keywords: | security anti-censorship streaming performance memory s3 cloud-backend storage | Cc: | |
| Launchpad Bug: |
Description (last modified by daira)
For immutable shares, the current S3 backend implementation writes data to a StringIO on each call to write_share_data (corresponding to remote_write), and only PUTs the share to S3 when the share is closed (corresponding to remote_close). This increases latency, and requires memory usage in the storage server proportional to the size of the share.
Sending the data over HTTP as it is received by calls to remote_write is not difficult in principle, although it isn't currently supported by txaws. However, how would we calculate the MD5 hash of the share file? The Content-MD5 header is optional, but without it there is an easy replay attack against the S3 authentication, allowing an attacker to replace share files without knowing the S3 secret key. (They cannot break integrity, but they can arbitrarily delete data.) A possible workaround is to require TLS for PUTs to S3, so that the Authentication header is secret and not available for replay. Note that this doesn't require the storage server to be authenticated at the TLS level, but it would require validating Amazon's certificate (or that of an S3 lookalike services) in order to prevent MITM attacks.
Change History (2)
comment:1 Changed at 2014-03-18T18:11:35Z by daira
- Description modified (diff)
- Keywords s3 cloud-backend added; s3-backend removed
comment:2 Changed at 2020-10-30T12:35:44Z by exarkun
- Resolution set to wontfix
- Status changed from new to closed
The established line of development on the "cloud backend" branch has been abandoned. This ticket is being closed as part of a batch-ticket cleanup for "cloud backend"-related tickets.
If this is a bug, it is probably genuinely no longer relevant. The "cloud backend" branch is too large and unwieldy to ever be merged into the main line of development (particularly now that the Python 3 porting effort is significantly underway).
If this is a feature, it may be relevant to some future efforts - if they are sufficiently similar to the "cloud backend" effort - but I am still closing it because there are no immediate plans for a new development effort in such a direction.
Tickets related to the "leasedb" are included in this set because the "leasedb" code is in the "cloud backend" branch and fairly well intertwined with the "cloud backend". If there is interest in lease implementation change at some future time then that effort will essentially have to be restarted as well.