Coding Is Like Cooking

A while back, Gojko Adzic published this article “Redefining Software Quality” and I think it’s pretty insightful, pointing out that we often expend a lot of effort ensuring quality at lower levels of the pyramid, when we should perhaps be investing higher up.

I wanted to work out what testing activities you’d do to ensure quality at different levels of Gojko’s Quality Hierarchy, so I began by thinking about the software testing quadrants. The testing quadrants were originally documented by Brian Marick in his blog, and later developed by Lisa Crispin and Janet Gregory in their book “Agile Testing“. (Here is a slide deck by Janet Gregory that gives a summary of the quandrants).

I like the agile testing quadrants because they help you to reason about different testing activities and why you’re doing them. They make it clear that in agile, testing has this big role in supporting the team – spreading knowledge about what’s being built and enabling the team to be agile about feature changes. In more traditional projects, testing focuses almost exclusively on the right side of the quadrants, missing this role entirely.

Anyway, if you put the agile testing quadrants alongside Gojko’s quality hierarchy, I think you get something like this:

Clearly, Q1 and Q2 are all about ensuring functionality basically works, and usually Q2 tests will run against deployed software, (deployed in a test environment). Q4 tests cover things like performance and security, and usability falls under Q3. I think things get a little more tricky for the higher levels though. There’s a distinct danger we’ve just run out of quadrants!

Beyond the testing quadrants

To test a piece of software is useful or successful, you’ll need to look at ideas that are relatively new in Agile. In his article, Gojko of course points out his book on Impact Mapping, and mentions Feature Injection and Lean Startup. Lisa and Janet published their book in 2009, “Lean Startup” by Eric Ries came out in 2011, Gojko’s “Impact Mapping” book is from 2012, and correct me if I’m wrong, but I don’t think there is a book on Feature Injection yet.

Agile ideas are moving on, especially compared with where they were when methodologies like Scrum and XP were originally documented a decade or more ago. I think it’s clear testers need to embrace new ideas and practices too.

If you look at Lean Startup, one of the concrete ideas is to do A/B testing of new features – that is, you divide your users into an “A” group, who see the new feature and a “B” group who don’t. Then you measure how each group behaves, and from that draw conclusions about whether the feature was any good. If the “A” group buys more stuff, spends more time on the site, generally acts like they are happier than the “B” group, you keep the feature, otherwise it gets dropped. It’s a bit like a trial of a new medicine – you compare the patients who get it with a control group who don’t, before you decide whether to approve it.

It seems to me that testers should be involved in designing and performing A/B tests – they’re well positioned with critical thinking skills, knowledge of the user, and technical automation skills. The results of such tests should tell us about whether users find a new feature useful. So that should get us up another level on Gojko’s pyramid:

The last level is fairly dependent on how you define success, but for a lot of software products, success means lots of users. Lean startup has another interesting idea here – the “Net Promoter Score”. Basically you ask a small group of initial users if they’d recommend your product, and make a simple calculation to predict if your user base is going to grow when you release it more widely. It’s an idea for what to put at the top level:

Conclusions

Of course, for many teams, it’s enough of a struggle to test for quality at the lower levels of the pyramid, without worrying about A/B tests or Net Promoter score! James Shore and Diana Larsen have come up with a model of “Agile Fluency” which I think is relevant here. Basically it outlines how, as teams get better at agile, their practices change. The three star level of fluency seems to contain a lot of ideas from Lean Startup, and optimizing for quality at the top two levels of Gojko’s pyramid. At one and two star fluency, delivering business value on the market cadence, just the testing quadrants get you a long way.

Changing testing activities also implies changes for testers. The role of tester has already changed with the advent of agile methods, and I predict it’s going to continue changing. I see a technical tester role appearing that is pretty close to business analyst, doing things like supporting the team with test automation and data analysis of A/B tests. So testers: get a head start and find out about Lean Startup!