I broke Perl::MinimumVersion, sorry

CPANdependenciesprocesslessons Mon 12 May 2014

Two days ago I released Perl::MinimumVersion 1.36 to CPAN; there were two main commits that added checks for 3 changes in Perl. Both of those changes had bugs, and in doing so I broke things for other people, something I really didn't want to do. As a result I had to do a release tonight, to fix one of the bugs and back out the other until I can do it properly. I'm going to talk through how I've been working on Perl::MinimumVersion, the bugs, and what I'm going to change. Hopefully people might point out other ways I can improve how I work.

When I got co-maint for Perl::MinimumVersion ('PMV' from here) I was somewhat nervous; it may 'only' have 15 dists dependent on it, but some of those are serious dists, and overall there are more than 8000 dists indirectly dependent on Perl::MinimumVersion. I didn't want to break any of them.

My process for Perl::MinimumVersion, to date

Here's the process and philosophy I started working to, when making changes to PMV:

I decided I should only go a couple of times round that loop per release, at least until I've built some confidence. The first release I did had more stuff because there was a backlog of changes from multiple people.

In addition to the RT buglist, I'm putting together a todo list of missing checks, based on reading the perl delta files from past releases.

The first bug

One of the features I picked to support was "stacked labels". Before Perl 5.14.0 you couldn't have two labels attached to the same statement

USER:
OUTERLOOP:
while (my $user = $iterator->next) {
    ...
}

Having written my scripts, here's what I distilled the test cases down to; I've added comments to show my thinking:

$x = 0;                          # single statement, no label
LABEL1: $x = 0;                  # single statement, one label
LABEL1: LABEL2: $x = 0;          # stacked labels, separated by whitespace
LABEL1:LABEL2: $x = 0;           # stacked labels, no whitespace
LABEL1: $x = 0; LABEL2: $y = 0;  # nearby labels on separate statements

So the 3rd and 4th tests should require 5.14.0, but not the others.

That's nowhere near exhaustive enough. I didn't have full coverage of the perl constructs that could have labels attached. As a result I wasn't seeing all the different PPI cases I needed to code for, so even though my check got all the above cases right, there is at least one it gets wrong.

# put example here

I'll add a minimal example once I understand it enough to add it.

The second bug

The other change I made was looking for the special code blocks (in the style of BEGIN { }) that have been added more recently. Here are my tests:

my %examples=(
    q/ BEGIN { }     /       => '5.004',
    q/ INIT { }      /       => '5.006',
    q/ CHECK { }     /       => '5.006002',
    q/ UNITCHECK { } /       => '5.010',
);

I get the code in each key of the hash, work out the minimum version, and compare it to the corresponding value in the hash. Do you know what the problem is?

It's the trailing zero on 5.010. On Perl 5.010 and later (coincidence that it's the same version as appears in the test), the version returned by PMV matches 5.010 (as a result of this change, I think). But on earlier versions it's 5.01, which doesn't match, not the way I was doing the comparison anyway.

I'd been bitten by this on an earlier release too, and both times been put straight by Paul Howarth. Sorry Paul, you shouldn't have to do that twice.

What should I do differently?

What could I have done differently in my process to have prevented these?

Developer releases

I should be doing developer releases, so that CPAN Testers can help me find things like the version number issue, instead of end users. When I adopt a new module I often do one or more developer releases, partly because it's often been a long time since the module was last released, and I want to see what I get back from CPAN Testers. I also do developer releases if the change I've made feels at all hairy.

For some reason I felt like that wasn't really needed here, because I thought I was being pretty cautious and thorough here. But those are different things from the coverage that CPAN Testers gives you.

I also need to be patient: as I previously learned, I should wait a few days (up to 8) to see what failures, if any, I get.

Multiple versions of Perl

When someone pointed out that my stacked labels test was wrongly asserting a 5.14 dependency on some code, my first thought was that I'd misunderstood what the perldelta was telling me. If I'd had the relevant perl versions installed locally, I could have run my test scripts past various versions of perl and confirmed my understanding of what I was checking for was correct.

I'd previously decided to install multiple versions of Perl, but got stuck trying to install 5.8.x (on my mac).

I've recently installed Strawberry Perl on a cheap windows laptop to help me investigate CPAN Testers failures from Windows. I need to get a Linux machine set up with key versions from 5.6.1 onwards, I think.

Better coverage of Perl syntax in my test cases

I need some kind of concise reference for Perl syntax, so that when I'm writing my example scripts and test cases, I can have more confidence that I'm hitting all the cases.

Are there any tools that can help me with this?

Test the modules that are dependent on PMV

CPAN Testers and multiple versions of Perl may not have revealed the "stacked labels" bug -- it was only running someone else's code against this dist that revealed it. Dave Rolsky's Test::DependentModules can be used to test your reverse dependencies. I should be using it.

Look at Travis?

I'm only vaguely aware of Travis. I need to learn whether it will give me anything above and beyond the previous points?

Summary

If you take on a critical module like Perl::MinimumVersion, then you should follow a more rigorous development, test and release process than you might for some modules. There are many modules that I rely on, that get regularly updated, and never break. I'm thankful for that, and that we have many reliable shoulders to stand on.

Releasing early and often can be good, but based on analysis of your reverse dependencies, early and often developer releases may be what's appropriate.

My apologies to Tux and Paul Howarth for breaking things, and thanks for the positive way you reported it. Thanks to Paul for testing a tarball for me, so I could get a release out and unbreak Tux.

Also sorry to ADAMK; I nagged him to give me co-maint, and don't feel like I've done him proud thus far.

If you've seen Kent Beck's excellent Ease at work talk, you may recognise that I'm not feeling particularly at ease as I write this. But I recognise that the pendulum has swung further to the left than is really called for (you may disagree :-)

I'm going to do something about that, as in general I take great pleasure, and generally feel at ease, when working with Perl, and want to maximise the chances of that continuing. Thus I'm happy to hear any suggestions for additional improvements to my way of working.

comments powered by Disqus