coverage: compute+display differential coverage

[warner]

It would be great if the buildbot produced, for each new build, a description of lines of code that gained or lost test coverage.

I just made a change to the Makefile to include the raw figleaf pickle in the directory that gets uploaded to the webserver, so from here on out we should have machine-parsable historical data. (for the builds until this point, we only have rendered HTML for each build).

I suppose the right approach is to have a 'render-figleaf-delta' target, which starts by trying to download the pickle file from the previous build, then have a script to construct the deltas, then render those deltas into more HTML, then include the delta HTML (as well as a machine-readable delta pickle) in the next batch of uploaded data.

[zooko]

If you like this ticket, you may also like #810 (where did figleaf_htmlizer.py come from?), #109 (current upstream figleaf doesn't do the right thing), and #387 (figleaf confused by python2.5).

[warner]

FYI, I had figleaf deltas working for a little while, but the results weren't very helpful: they were too easily tricked by insertion/removal of lines. If I were going to reimplement this code again now, I would try to be more clever. Maybe compare the contents of lines (if you think a line has lost coverage, compare it against all lines which have gained coverage, and allow identical lines to cancel each other out). Or maybe find a clever 'diff' algorithm that spots insertions.

Also, the report should only show files which have had their coverage change. We don't need yet another giant report causing users to have to scroll a lot to find the interesting parts.

[zooko]

Maybe use the darcs patch instead of diffing the before- and after- versions of the file.

[zooko]

Oh, in fact maybe what you really want to know is several different things:

• lines removed that were covered
• lines removed that were uncovered :-)
• lines added and covered
• lines added and uncovered :-(
• lines not touched but went from covered to uncovered :-(
• lines not touched but went from uncovered to covered :-)

[Daira Hopwood <daira@…>]

In 61371a31df3c89afe1f7b51ac6fd7e94f3fc49f4/trunk:

Support coverage reporting via coverage.io. refs #623

Signed-off-by: Daira Hopwood <daira@…>

[daira]

Actually coveralls.io: ​https://coveralls.io/r/tahoe-lafs/tahoe-lafs

This isn't using buildbot but I think it fulfills the spirit of this ticket.

[daira]

Hmm, actually coveralls.io reports display the number of lines by which coverage changed for each file, but they don't display a diff of the coverage change. Perhaps we should file an enhancement request for that.

[daira]

Filed ​https://github.com/lemurheavy/coveralls-public/issues/359.

Also see ​https://github.com/lemurheavy/coveralls-public/issues/31 for display of branch coverage. (We collect the per-line branch coverage information but it's not displayed on coveralls.)

[daira]

Reopening until coveralls supports differential coverage reports (or we find another way of doing this).

[daira]

​http://nedbatchelder.com/code/coverage/branch.html says:

The XML report produced by coverage xml also includes branch information, including separate statement and branch coverage percentages.

​Here is an example of how the branch data is represented in the XML format; it looks quite comprehensive.

So, the lack of branch coverage in the coveralls reports is purely a problem with the rendering done by coveralls.io.

[daira]

Aside -- this is how detailed code coverage *should* be: ​Haskell example