Flaky hypothesis test in test_storage_http

[itamarst]

[ERROR]
Traceback (most recent call last):
Failure: testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/nix/store/qqsbr90v1ag6maa1pn0qnfzr2v6lx1p4-python3-3.7.11-env/lib/python3.7/site-packages/twisted/internet/defer.py", line 191, in maybeDeferred
    result = f(*args, **kwargs)
  File "/nix/store/qqsbr90v1ag6maa1pn0qnfzr2v6lx1p4-python3-3.7.11-env/lib/python3.7/site-packages/testtools/testcase.py", line 702, in _run_test_method
    return self._get_test_method()()
  File "/nix/store/qqsbr90v1ag6maa1pn0qnfzr2v6lx1p4-python3-3.7.11-env/lib/python3.7/site-packages/allmydata/test/eliotutil.py", line 171, in run_with_logging
    return test_method(*args, **kwargs)
  File "/nix/store/qqsbr90v1ag6maa1pn0qnfzr2v6lx1p4-python3-3.7.11-env/lib/python3.7/site-packages/allmydata/test/test_storage_http.py", line 852, in test_read_with_no_range
    def test_read_with_no_range(self, data_length):
  File "/nix/store/qqsbr90v1ag6maa1pn0qnfzr2v6lx1p4-python3-3.7.11-env/lib/python3.7/site-packages/hypothesis/core.py", line 1190, in wrapped_test
    raise the_error_hypothesis_found
  File "/nix/store/qqsbr90v1ag6maa1pn0qnfzr2v6lx1p4-python3-3.7.11-env/lib/python3.7/site-packages/hypothesis/core.py", line 886, in __flaky
    raise Flaky(message)
hypothesis.errors.Flaky: Hypothesis test_read_with_no_range(self=<allmydata.test.test_storage_http.ImmutableHTTPAPITests.test_read_with_no_range id=0x7fcaf2949550>, data_length=77788) produces unreliable results: Falsified on the first call but did not on a subsequent one


allmydata.test.test_storage_http.ImmutableHTTPAPITests.test_read_with_no_range

Update: After some digging (see comments) the issue seems to be that the CI Hypothesis profile is not being enabled in the Nix builders. This may be an issue in other builders as well.

[itamarst]

This specifically happened on NixOS 21.05 builder, and nowhere else: ​https://app.circleci.com/pipelines/github/tahoe-lafs/tahoe-lafs/3452/workflows/e4143c58-6d85-4b87-af06-d9a403e037cb/jobs/68124

[itamarst]

It seems like on large lengths, the first call can be quite slow, so it hits hypothesis timeout and is marked as failed. Later calls (strangely) are not as slow, so they pass, thus the flaky error warning.

[itamarst]

Running directly on large lengths does _not_ cause it to hit the timeout. So... maybe urandom() is running out randomness if enough examples are called?

[itamarst]

Supposedly urandom() doesn't block? And switching to Python's random generator causes issues, strangely (not random enough when run in bulk?).

[itamarst]

Going to take a break, but for future reference, just adding @hypothesis.settings(max_examples=50000) to the test makes it fail fairly consistently on my computer.

[itamarst]

Maybe it's GC?

[itamarst]

• Running a manual test of just urandom(), I can't make it block (which matches documented behavior), so that suggests it probably is not the issue.
• Adding @hypothesis.settings(max_examples=5000) makes tests fail pretty consistently. Disabling GC makes them start passing again.

So it's probably GC.

[itamarst]

All that being said: in theory Tahoe is disabling the 200ms deadline that GC is hitting in CI profile. So maybe it's that CI profile isn't being enabled in all cases.