Opened at 2020-08-22T12:10:16Z
Closed at 2020-09-15T21:33:59Z
#3391 closed enhancement (fixed)
Change codecov default settings
| Reported by: | sajith | Owned by: | sajith |
|---|---|---|---|
| Priority: | normal | Milestone: | undecided |
| Component: | dev-infrastructure | Version: | n/a |
| Keywords: | Cc: | ||
| Launchpad Bug: |
Description
Codecov.io has been a source of unhappiness lately, with their test coverage checks leaving red check marks in CI status in a rather arbitrary manner: example, example.
It is not clear why "91.47% (+-0.06%) compared to base commit" or "91.49% (+-0.03%) compared to base commit" is a bad thing. My thinking is that these are not large enough changes in coverage to warrant CI turning red, and maybe this is happening because computers are not very good with floating point numbers. We will ask codecov to drop the decimals and turn a few other knobs by adding a codecov.yml file (this is quite well documented) to the repository and see how that works for us.
Change History (6)
comment:1 Changed at 2020-08-22T21:56:36Z by sajith
- Keywords review-needed added
comment:2 Changed at 2020-08-25T15:56:20Z by exarkun
comment:3 Changed at 2020-09-02T19:39:59Z by meejah
- Keywords review-needed removed
comment:4 Changed at 2020-09-15T21:17:13Z by sajith
Those coverage reports look odd to me. In this specific instance:
- 222735010 (GitHub CI) has coverage.xml: OK.
- 222735010 (GitHub CI) has coverage.xml: OK.
- 27078.0 (CircleCI ubuntu-16.04) has coverage.el, but no coverage.xml: dubious?
- 27076.0 (CircleCI c-locale): ditto
- 27075.0 (CircleCI deprecations): ditto
- 27079.0 (CircleCI ubuntu-18.04): ditto
- 27082.0 (CircleCI debian-8): ditto
- 27080.0 (CircleCI fedora-28): ditto
I'm not sure if this happens because coverage.xml wasn't generated on CircleCI runs, or because of any issues with codecov uploader itself.
comment:5 Changed at 2020-09-15T21:31:42Z by exarkun
Maybe we should add some more logging to the CI configuration to show us what coverage does get generated by each job - before it is supposed to be uploaded to codecov?
comment:6 Changed at 2020-09-15T21:33:59Z by GitHub <noreply@…>
- Resolution set to fixed
- Status changed from new to closed
In 1b207d8/trunk:
Just sharing, the codecov docs at https://docs.codecov.io/docs/troubleshooting-guide say you can get the actual coverage reports its operating on from the "Builds" tab (note it's a tab on the commits page). This gives you a page with a bunch of download links, eg https://codecov.io/gh/tahoe-lafs/tahoe-lafs/commit/bce701d5bc810b71c07d2b7142a59aaac28a45e4/build, which could then presumably be locally inspected to see if codecov is wrong or the coverage reports are actually different.