Opened at 2019-07-01T14:57:18Z
Last modified at 2019-07-25T12:40:55Z
#3138 new defect
Run the test suite with some concurrency on CircleCI
Reported by: | exarkun | Owned by: | exarkun |
---|---|---|---|
Priority: | normal | Milestone: | undecided |
Component: | dev-infrastructure | Version: | n/a |
Keywords: | Cc: | ||
Launchpad Bug: |
Description (last modified by exarkun)
CircleCI provides runners with two vCPUs. If a single CircleCI job can split its work across two Python processes we can take better advantage of the runner configuration and hopefully get a much faster CI run.
trial supports spreading test execution over multiple local processes with --jobs. We can already pass extra arguments to trial through tox thanks to #2894.
There are some problems with coverage collection's interactions with --jobs. There is a Twisted ticket to deal with this, https://twistedmatrix.com/trac/ticket/9663, but there is also https://pypi.org/project/coverage_enable_subprocess/ which basically solves the problem externally.
So, add coverage_enable_subprocess and --jobs 2 to our CircleCI configuration and get a ~50% reduction in CircleCI runtime.
Change History (5)
comment:1 Changed at 2019-07-01T14:57:32Z by exarkun
- Description modified (diff)
comment:2 Changed at 2019-07-01T17:05:46Z by exarkun
comment:3 Changed at 2019-07-01T17:17:07Z by exarkun
Also, before the hang, it doesn't look like it is on its way to being *fast*. It seems to take about 6x as long to run the test suite this way (~18 minutes until it hung, seemingly pretty close to the end, compared to 3 minutes for coverage run -m twisted.trial --jobs 2 allmydata or 6 minutes for coverage run -m twisted.trial allmydata on master - which of course misses a lot of coverage measurement).
The result is arguably more correct but it can't be said that it's any faster...
comment:4 Changed at 2019-07-01T17:35:11Z by exarkun
Another option would be to apply this change only to the integration tests where it may be more useful to collect the additional information. As I understand it, the integration tests essentially _only_ provide coverage via subprocesses so all of their value goes unmeasured. The "unit test" suite should, perhaps, be penalized for providing coverage in child processes since these are hardly "units" (but this is an opinionated stance that probably needs to be evaluated in the context of tahoe-lafs and the goals of the different parts of the automate test suite).
comment:5 Changed at 2019-07-25T12:40:55Z by exarkun
- Summary changed from Run the test suite concurrency on CircleCI to Run the test suite with some concurrency on CircleCI
Unfortunately the test suite hangs when run under coverage with coverage_enable_subprocess installed. Presumably this is fixable but the goal isn't achieved just by setting up the test environment as described above ...