Kent Beck is one of the original authors of JUnit, an open source tool that kick-started a huge shift in programmer behaviour that I and others have found immensely valuable. In 2019 he published his “Test Desiderata” with advice about the properties he values in tests. I have found this list useful, but at the same time confusing and incomplete. In this post I’m going to propose an update – Test Desiderata 2.0.
Programmer Tests
Over the course of 25 years or so I’ve seen programmers writing tests for their code go from a niche interest to something that is more or less standard. I was lucky to be an early adopter, I wrote my first unit test with JUnit back in 2000 and I haven’t looked back since. Particularly in the early years I had to do a lot of persuading of the people around me that programmers writing tests was a good idea at all. These days as it’s often my job as a technical coach to help programmers to write better tests, but the fundamental debate about whether they should even spend time on this has largely been settled. Yes, you should write tests for your code!
Ok, so I should write tests. What is a good test? What kinds of tests should I have?
It’s a bit like asking what is a good pair of shoes. There are good tests like there are good shoes. At some level, all shoes provide some protection from weather and dirt as you move around. All tests should protect you from mistakes as you develop software. In the same way as you can’t handle all situations with only one pair of shoes, you don’t expect one kind of tests to find all mistakes. The contents of the rest of your wardrobe is relevant when you assess the benefits of one pair of shoes.
This is the fundamental difficulty I’ve found with Beck’s Test Desiderata. It is both a description of what you value in individual shoes as well as what you value about a whole wardrobe of shoes. His ‘macro’ desiderata, (Inspiring and Predictive), refer to the whole test suite, and the three other areas containing ten other desiderata all refer to properties of individual tests.
Four Macro Desiderata
I would like to propose four different ‘macro’ desiderata. These are properties of the whole suite of tests that programmers use in their development work:
- Predict success in production
- Fast to get feedback
- Support ongoing code design change
- Low total cost of ownership
The first one is equivalent to ‘Predictive’ in Beck’s desiderata, and the second one maps to ‘Fast’. I was surprised to realize that Beck, with all his work on Test-Driven Development, didn’t seem to have explicitly mentioned the macro benefits you get from tests while you develop them. Many other authors highlight this – the idea that the tests are an active partner in design work and support a programmer’s need to continually change code design.
The last point – low cost – seems to me to always be a tradeoff in any kind of engineering work. Likewise the obvious benefit of fast feedback. Programmers want to know as quickly as possible if their current design is likely to work in production, for minimal effort.
A note on Inspiring – my view is that whether your tests inspire confidence depends on whether they are predictive of success in production, and it’s not a separate macro property of the suite.
Desiderata for Individual Tests
I believe that all of Beck’s 9 other desiderata are useful for assessing individual test cases. I would also include additional desiderata that I found other authors have written about. In the sections below I will go through each of the four macro test suite properties and suggest the main properties of individual tests which support them. Of course some of these support more than one, but I have aimed to allocate each to the one they support most clearly. I have also changed the names of some of Beck’s desiderata where I felt either I or other authors could describe them more clearly. (See below for links to additional authors).
Test Desiderata that lead to “Predict Success in Production”
- Sensitive to Behaviour (Beck’s “Behavioral”)
- Sensitive to Execution Qualities such as performance
Test Desiderata that lead to “Fast to get Feedback”
- Minimal data
- Run in any Order (Beck’s “Isolated”)
- Run in Parallel
Test Desiderata that lead to “Minimize Total Cost of Ownership”
- Automated (Beck)
- Deterministic (Beck)
- Diagnosable (Bazuzi, Beck’s “Specific”)
- Easy to Read (Beck’s “Readable”)
- Easy to Update
- Easy to Write (Beck’s “Writeable”)
- Insensitive to Code Structure (Beck’s “Structure Insensitive”)
Test Desiderata the lead to “Support Ongoing Code Design Changes”
- Composable (Beck)
- Documents Intent (Shore)
- Durable, remains valuable throughout the product lifetime (Cunningham)
- Necessary, Guide Development Choices (Farley)
- Organized, easy to locate corresponding test and code (Bazuzi, Rainsberger)
- Positive Design Pressure
Influences and Credits
In coming up with my list I have been inspired by many different authors. In addition to Beck’s desiderata and my own list “Principles for Agile Test Automation“, these are the sources I have studied in this process:
- Arlo Belshee “What makes a good test suite?”
- Jay Bazuzi “What makes a great test?”
- Lisa Crispin “What makes a good test suite?“
- Ward Cunningham “What makes a good test suite?“
- Kent C. Dodds “How to know what to test“
- Urs Enzler (private communication)
- Markus Gaertner “What makes a good test suite?“
- Corey Haines “What makes a good test suite?“
- Elisabeth Hendrickson “What makes a good test suite?“
- Llewellyn Falco “What makes a good test suite?“
- David Farley “Properties of Good Automated Tests” (pdf from his video training course on TDD/BDD)
- Michael Feathers “Working Effectively with Legacy Code” book from 2004, specifically the chapter “What is unit testing?”
- Steve Freeman “What makes a good test suite?“
- Kevlin Henney “Program with GUTs” from “97 things every Java Programmer Should Know”
- Jim Humelsine “Attributes of Effective Unit Tests“
- Ron Jeffries “What makes a good test suite?“
- Vladimir Khorikov “4 pillars of a good unit test” from his book “Unit Testing Principles, Practices and Patterns”
- Lasse Koskela “Effective Unit Testing” book, chapter 2
- Roy Osherove “Art of Unit Testing” book, table 10.1
- Ottinger & Schuchert “FIRST properties of unit tests“
- J. B. Rainsberger “What makes a good test suite?“
- James Shore “What Does a Good Test Suite Look Like?“
- Milan Todorov “About them automated tests“
Research Method
I studied as many lists of good test properties as I could find, trying to understand where they overlap and where they differ. I began with Beck’s 12 desiderata as columns in a table, with rows for each other author’s list. I tried to map each item onto one or more of Beck’s properties, and if I failed, put them in a new ‘desiderata’ column at the end. I wanted to see if any desiderata were rarely mentioned, and what additional ones there were. I also wanted to see if other people had better names for some of these desiderata.
I started with lists I knew from people like Farley and Khorikov, and added ones others people suggested to me when I asked on social media. Not all of these people are answering the same question as Beck, and I have undoubtedly misunderstood some of them. Almost all these lists are referring to unit tests written with a framework like JUnit. Most do not mention macro properties of all the tests a programmer has in their suite.
I have used my own judgement in creating a new list of test desiderata, influenced by all these sources.
Specific Desiderata Explanations
Many of the new desiderata I am proposing can be directly traced back to authors I have studied, although I have invented shorter names for some of them. This is a summary of the additional desiderata:
- Sensitive to Execution Qualities – Tests should not only be sensitive to functional behaviour changes but also changes in performance and other non-functional execution qualities in the system under test. Beck may have intended that ‘behaviour’ includes these qualities but I thought it was better to be explicit.
- Minimal Data – less test data means faster test execution. It also improves test readability and updatability which are related to minimizing costs.
- Easy to Update – When the system under test changes and the test case becomes out of date, it should be easy to update it to match the new requirements.
- Positive Design Pressure – this is an effect I observe in Test Driven Development, where writing the test pushes the developer towards a better design – clearer APIs, better encapsulation, smaller units.
- Run in Parallel – Run a test in parallel with other tests, or other copies of itself, without that influencing the result. This is a higher bar than being able to run your tests in any order.
How to use this Test Desiderata 2.0
I have similar goals to Beck in that I want to help developers to write better tests. I hope to follow up this article with some more specific analysis of particular kinds of tests.
Updates to the model
This model is a work in progress and after I hit publish I realized I wanted to change a couple of things.
2025-12-09 – Added ‘Run in parallel’ and moved ‘Insensitive to code structure’ from ‘support ongoing code design changes’ to ‘Minimize cost of ownership’.
