Opened at 2016-11-29T14:10:29Z

Closed at 2017-08-07T18:29:16Z

#2845 closed enhancement (fixed)

Run tests in parallel

Reported by: rvs Owned by: rvs
Priority: normal Milestone: soon
Component: code Version: n/a
Keywords: tox test easy Cc:
Launchpad Bug:

Description

Currently tox takes quite a lot of time, even on beefy machine with SSD & many CPUs.

Looks like xdist plugin can run tests in paralel which could speed things up quite a bit.

See e.g. ​https://qxf2.com/blog/xdist-run-tests-parallel-pytest/

Change History (6)

comment:1 Changed at 2016-12-23T13:46:30Z by rvs

  • Component changed from unknown to code
  • Owner set to rvs
  • Status changed from new to assigned

comment:2 Changed at 2017-01-10T17:38:13Z by daira

  • Summary changed from Run tests in parrallel to Run tests in parallel

comment:4 Changed at 2017-01-13T00:04:06Z by warner

Huh.. I had no idea that trial could parallelize like that. Let's see how well travis handles the PR, but then yeah, let's land that.

Any idea if we could use this on tox -e coverage too? I don't know how the .coverage files are written, or if the multiple workers would clobber each other's data.

comment:5 Changed at 2017-01-13T02:46:41Z by warner

The github PR is showing failures on the integration tests: I think they're more sensitive to concurrency, so we probably can't enable it for those.

Also it looks like trial -j causes _trial_temp/test.log to be split into several pieces (one per worker, in files like _trial_temp/0/test.log and _trial_temp/1/test.log). I believe our buildbot reads from test.log to count various things, so we may need to rewrite some tooling to handle that. But the speedup is so impressive, I think it's worth the effort.

comment:6 Changed at 2017-08-07T18:29:16Z by exarkun

  • Keywords review-needed removed
  • Resolution set to fixed
  • Status changed from assigned to closed

Sorry, ​https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2894 was a duplicate of this.

Note: See TracTickets for help on using tickets.