How soon do CPAN Testers start testing your releases?

CPAN TesterschartsQA hackathon Thu 13 March 2014

Today three of us were wondering what the pattern of CPAN Testers report submissions was. I guessed that most tests come in within the first week. The people I needed to talk to, to get and plot this data, are all in the same room as me today, so here are the charts.

This first chart shows the number of CPAN Testers reports submitted in the days following the most recent stable release of HTML-Summary:

So things really drop off after 8 days. Next, here's the daily graph for JSON-Parse-0.29_02, a developer release:

These are pretty similar. If you want to get the sense of the quality of a release: wait 8 days.

This next chart shows the number of test submissions by hour, for HTML-Summary:

CPAN Testers latency (hours)

And again, for JSON-Parse:

That early spike surprised me — on one dist I looked at, the spike was in the first hour following the upload to CPAN. I asked BARBIE about it, thinking maybe my date calculations were wrong. "oh yeah, that's ANDK and BINGOS, and possibly others. They have smokers grabbing stuff and running tests sometimes within minutes".

Just another reminder of how lucky we are to have not only CPAN Testers, but people like Andreas and Chris who have dedicated resources running smoke testers in this way.

Next I'll need to look at coverage of operating systems and versions of Perl. I suspect you might have to wait a bit longer for good coverage.

comments powered by Disqus