uploading too-large mutable file causes huge memory usage, 100% CPU

[warner]

Jan-Benedict Glaw noticed that trying to upload a large (32MB) mutable file caused his client node to start using 100% CPU and a very large amount of memory.

We have a limit of 3.5MB on mutable files, since we have not yet impemented MDMF (see #393 for details). But uploading a file larger than this limit should cause a sensible error or exception, not cause the node to explode.

I've reproduced this locally. A 4MB file causes memory usage to peak at about 136MB (on 'monolith', my laptop). An 8MB file causes a peak usage of 234MB. A 16MB file causes peak usage of 430MB.

I suspect this is another case of #379, in which the flattened Failure instance contains multiple copies of the generated shares, causing brief memory allocation equal to some multiple of the file size. Hopefully the fix will be to check the filesize earlier, to reduce the number of stack frames that reference the large string.

[warner]

Note that this isn't actually an infinite loop: it's just a very large memory allocation and a lot of string copies, frequently large enough to bring the rest of the system down to a crawl. With enough time and swap, the operation should error out correctly.

[warner]

Hm, so my math suggests that we've got 25 copies of this string ( (234MB-136MB) / (8MB-4MB) ~= 25 , as does (430MB-234MB) / (16MB-8MB) ).

The mutable file's new contents are passed around (in a single string) as one of the arguments in a series of function calls. I'm guessing that between the arguments and the locals, this string must show up in the Failure's stack frames about 25 times.

Lacking a fix in Failures, the easiest workaround I can think of is to wrap the mutable file's contents in a small class (maybe called StringContainer, or Contents), which has a useful (but abbreviated!) repr, and then have the mutable-file publish code extract the real string at the very last moment. If the generated shares get passed as method arguments too, they may need to be wrapped also.

[zooko]

Brian fixed #379, which might have fixed this issue. The newly opened issue #694 (remove hard limit on mutable file size) is another case of someone hitting this problem, and that person -- sigmonsays -- didn't complain of excessive memory usage. In any case, I think the right way to proceed is to remove the hard limit on mutable file size entirely, as decribed in #694, so I'm closing this ticket.