CI test_system fails inconsistently

[hacklschorsch]

CI reactors under test.test_system on CircleCI fail inconsistently ONLY in the Tahoe-lafs Circle CI org. Cannot reproduce locally on Nixos nor on GitHub? CI (inside similar docker images).

Possible root cause discussed in ​https://github.com/tahoe-lafs/tahoe-lafs/pull/1381#issuecomment-2476885548 meejah writes:

The unclean-reactor errors may be simply a downstream symptom of the real errors that also happen in that run (e.g. several tests time out).

My own tests suggest that indeed, raising the SystemTests? timeout make ​a couple of flaky tests much more stable:

Failure count	Test name
1	allmydata.test.test_system.HTTPSystemTest.test_mutable_mdmf
3	allmydata.test.test_system.HTTPSystemTest.test_mutable_sdmf
30	allmydata.test.test_system.HTTPSystemTest.test_upload_and_download_convergent
11	allmydata.test.test_system.HTTPSystemTest.test_upload_and_download_random_key

This ticket is similar but not equal to ticket:4085, ticket:4022, ticket:2994 .

[hacklschorsch]

A better fix to the dirty reactors might be to clean them up - and indeed, SystemTests does some reactor-cleanup dance in tearDown(), which ​*should* also be called if a test fails - but it seems that does not always happen or always help.

This is called even if the test method raised an exception [...] This method will only be called if the setUp() succeeds

Also see the discussion in ​https://stackoverflow.com/questions/39883058/teardown-not-called-after-timeout-in-twisted-trial

[hacklschorsch]

Raising timeout in ​https://github.com/tahoe-lafs/tahoe-lafs/pull/1387

[btlogy]

If I'm not mistaken, those reactor