coverage: compute+display differential coverage

[warner]

It would be great if the buildbot produced, for each new build, a description of lines of code that gained or lost test coverage.

I just made a change to the Makefile to include the raw figleaf pickle in the directory that gets uploaded to the webserver, so from here on out we should have machine-parsable historical data. (for the builds until this point, we only have rendered HTML for each build).

I suppose the right approach is to have a 'render-figleaf-delta' target, which starts by trying to download the pickle file from the previous build, then have a script to construct the deltas, then render those deltas into more HTML, then include the delta HTML (as well as a machine-readable delta pickle) in the next batch of uploaded data.

[zooko]

If you like this ticket, you may also like #810 (where did figleaf_htmlizer.py come from?), #109 (current upstream figleaf doesn't do the right thing), and #387 (figleaf confused by python2.5).

[warner]

FYI, I had figleaf deltas working for a little while, but the results weren't very helpful: they were too easily tricked by insertion/removal of lines. If I were going to reimplement this code again now, I would try to be more clever. Maybe compare the contents of lines (if you think a line has lost coverage, compare it against all lines which have gained coverage, and allow identical lines to cancel each other out). Or maybe find a clever 'diff' algorithm that spots insertions.

Also, the report should only show files which have had their coverage change. We don't need yet another giant report causing users to have to scroll a lot to find the interesting parts.

[zooko]

Maybe use the darcs patch instead of diffing the before- and after- versions of the file.

[zooko]

Oh, in fact maybe what you really want to know is several different things:

• lines removed that were covered
• lines removed that were uncovered :-)
• lines added and covered
• lines added and uncovered :-(
• lines not touched but went from covered to uncovered :-(
• lines not touched but went from uncovered to covered :-)

[Daira Hopwood <daira@…>]

In 61371a31df3c89afe1f7b51ac6fd7e94f3fc49f4/trunk:

Support coverage reporting via coverage.io. refs #623

Signed-off-by: Daira Hopwood <daira@…>

[daira]

Actually coveralls.io: ​https://coveralls.io/r/tahoe-lafs/tahoe-lafs

This isn't using buildbot but I think it fulfills the spirit of this ticket.