Opened at 2022-11-28T16:09:02Z
Closed at 2023-02-21T14:37:35Z
#3946 closed enhancement (fixed)
Reduce HTTP protocol chattiness for downloads
Reported by: | itamarst | Owned by: | GitHub <noreply@…> |
---|---|---|---|
Priority: | normal | Milestone: | HTTP Storage Protocol |
Component: | unknown | Version: | n/a |
Keywords: | Cc: | ||
Launchpad Bug: |
Description
#3939 reduced upload chattiness a lot; there is still likely many improvements that could be done when downloading.
Change History (7)
comment:1 Changed at 2022-12-05T19:25:02Z by itamarst
comment:2 Changed at 2022-12-13T17:00:47Z by itamarst
And there's a comment: # TODO: quantize to reasonably-large blocks YES YES YOU SHOULD
comment:3 Changed at 2023-01-23T17:02:08Z by itamarst
Increasing segment size would help, but:
- Some code actually relies on it (e.g. https://github.com/tahoe-lafs/tahoe-lafs/blob/master/src/allmydata/immutable/downloader/node.py#L74) though _hopefully_ that's just an optimization. Would need testing of all relevant places that rely on it, at least.
- It doesn't help with existing downloads.
comment:4 Changed at 2023-01-24T15:15:44Z by itamarst
(Deleted incorrect comment).
comment:5 Changed at 2023-01-24T16:30:27Z by itamarst
The segment fetching code is complex enough that I don't think it's worth messing with at this juncture. So I'm just going to increase segment size, which at least will help with new uploads. Let's say... 1MB, from current 128KB.
Required testing and checks:
- An upload with 128KB segment can be downloaded by client with 1MB expected max segment.
- An upload with 1MB segment can be downloaded by client with 128KB expected max segment.
- Validate all uses of DEFAULT_MAX_SEGMENT_SIZE are advisory, if not add more bullet points here.
comment:6 Changed at 2023-01-24T16:41:55Z by itamarst
With 40ms round-trip latency, and downloading a 20MB immutable file:
- 128KB segment size means a 15 second download.
- 1MB segment size means a 3.5 second download.
So that's still kinda terrible (there's no bandwidth bottleneck!) but much much better.
comment:7 Changed at 2023-02-21T14:37:35Z by GitHub <noreply@…>
- Owner set to GitHub <noreply@…>
- Resolution set to fixed
- Status changed from new to closed
In 0b0af62/trunk:
For immutables, allmydata/immutable/downloader/share.py has a bunch of code that devolves to downloading blocks.