Opened at 2011-12-16T23:02:30Z

Closed at 2012-08-06T18:56:57Z

Last modified at 2015-04-12T22:20:39Z

#1638 closed defect (fixed)

S3 backend: Upload of large files consumes memory > twice the size of the file

Reported by: zancas Owned by:
Priority: major Milestone: eventually
Component: code-storage Version: cloud-branch
Keywords: reliability availability scalability usability performance memory large s3-backend Cc:
Launchpad Bug:

Description (last modified by daira)

During upload greater than 2 (~2.5) times the size of shares (both mutable and immutable) are consumed on the storage server. For storageservers running with tight memory constraints and/or large shares this is problematic. Attached is a 1-second-resolution time-series of memory usage generated by ps for a 200 MB immutable file upload to an amazon EC2 t1.micro.

The exact command: ps -Ovsize,rss -C tahoe

Attachments (2)

upload200MB_im_mutablelogfile.txt​ (92.6 KB) - added by zancas at 2011-12-16T23:03:50Z.
uploadmutablelogfile.txt​ (29.2 KB) - added by zancas at 2011-12-16T23:06:45Z.
This is for a 100MB SDMF share

Download all attachments as: .zip

Change History (7)

Changed at 2011-12-16T23:03:50Z by zancas

Changed at 2011-12-16T23:06:45Z by zancas

This is for a 100MB SDMF share

comment:1 Changed at 2011-12-17T01:44:36Z by zooko

  • Summary changed from Upload of large files consumes >2 the size of the file memory to Upload of large files with S3 backend consumes >2 the size of the file memory

This is true only of the S3 backend (ticket #999), not of the standard disk backend.

comment:2 Changed at 2011-12-17T01:44:46Z by zooko

  • Keywords s3 added

comment:3 Changed at 2011-12-17T05:32:53Z by davidsarah

  • Keywords s3-backend added; s3 removed
  • Milestone changed from undecided to 1.10.0
  • Summary changed from Upload of large files with S3 backend consumes >2 the size of the file memory to S3 backend: Upload of large files consumes memory > twice the size of the file

comment:4 Changed at 2012-08-06T18:56:57Z by davidsarah

  • Resolution set to fixed
  • Status changed from new to closed

This is fixed in the cloud backend. Since it is the cloud backend that will be merged to trunk, this ticket doesn't need to stay open.

comment:5 Changed at 2015-04-12T22:20:39Z by daira

  • Description modified (diff)
  • Milestone changed from 1.10.1 to eventually
  • Version changed from 1.9.0 to cloud-branch
Note: See TracTickets for help on using tickets.