Opened at 2023-12-08T15:54:13Z
#4080 new enhancement
Increase chunk sizes to improve performance and parallelism
Reported by: | itamarst | Owned by: | |
---|---|---|---|
Priority: | normal | Milestone: | Performance and Benchmarking |
Component: | unknown | Version: | n/a |
Keywords: | Cc: | ||
Launchpad Bug: |
Description
- Processing data in larger chunk sizes can reduce overhead of all the little Python function calls for each chunk.
- Larger chunk sizes for functions that release GIL add additional opportunities for utilizing parallelism by shifting to a thread pool. Larger chunks of work are more efficient for this scheme.
Note: See
TracTickets for help on using
tickets.