This is a companion post to my video with the same title.
Whether you’re using an AI tool to write them or not, you need tests. If you’re doing Continuous Delivery, the tests in your pipeline are essential to determine releasability. Unfortunately, it’s often a source of strife. I’m going to go through 7 common misunderstandings and myths about developer testing, and explain how we do it in Modern Software Engineering.
#1 Testers and Developers have an adversarial relationship and this is necessary
This attitude seems to be really old and ingrained – testers have to be independent of developers so they have the necessary distance to critique the software. Otherwise it’s like the developers are marking their own homework – there is no independent validation of correctness.
This adversarial relationship unfortunately seems to breed contempt – “testers are trying to break my code and disgrace me” or “those lazy developers are just throwing stuff over the wall and they don’t care”. In continuous delivery and devops we need to acknowledge that we’re on the same team – we are working towards the same goal. The testers are not trying to break your code, they are trying to break your illusions about the code. If the software doesn’t work, we all benefit if we find that out sooner rather than later.
The reason for the different roles is that it gives us different perspectives on the product. I’m in favour of testers and developers collaborating during software development. A process like Behaviour Driven Development gives us a concrete structure for that collaboration and ensures both perspectives are taken into account without things getting adversarial.
#2 Developers are responsible for unit testing, and specialists do the other kinds
This is pretty common – that developers do unit testing and specialists create integration testing and system testing and performance testing and so on. Now, I’m well aware that those different kinds of testing require specialist skillsets and other tools that developers don’t necessarily have. I absolutely want you to have excellent skilled testers working with you.
What I think is a mistake, is if the developers are only responsible for unit tests and don’t have to care about the rest. This can create really bad incentives for developers – well the unit tests pass, I did my part, I moved my ticket to the ‘done’ column. All those other tests are someone else’s responsibility until you file a ticket in my backlog.
If developers are too divorced from the tests beyond the unit level then everyone loses. That’s why there is such an emphasis in Continuous Delivery on definitive feedback in pipelines with short time horizons. All the tests together form the specification, whoever happened to write each kind. Even if developers don’t write the other tests, they need to be using that test feedback to actively inform their development decisions. Again – we are a team with the same goal – great software.
#3 Business people specify acceptance tests and developers ensure they pass
An acceptance test by definition is a test from the perspective of a user or customer of the software, that ensures the software is acceptable to them. So in some sense, it is the business people who decide what should be in the acceptance tests. In practice I’ve seen people with a Product Owner or Product Manager job title say things like “I specify what, the developers decide how”. Total power in the hands of one person.
That’s not a healthy relationship. In Continuous Delivery we are discovering together what the software should do. Developers are engaged with the same questions as the business focussed people – everyone cares about creating high quality software and is involved in the details of what that means.
Yes, business people are more likely to understand the subtleties of the user’s needs, and will be able to outline business requirements, scenarios and realistic examples. They will discuss this with developers, who will bring in their important perspectives about what’s feasible, what’s possible, and ideas for improvements based on their understanding. We also need other perspectives in the mix to make truly great product decisions – testers, UX designers, performance specialists, data scientists. If you have one person or even a small team of business people specifying acceptance tests, then you’re risking far too much on a single perspective.
#4 – When we test units in isolation, we are doing unit testing.
Ok, so in order to test in isolation, we need to draw a box around a unit in the code, and mock out all the dependencies. This unfortunately often leads to very brittle tests that break every time you change the way you distribute responsibilities between classes and dependencies.
Actually, what we mean when we say ‘isolation’ in modern unit testing with a tool like JUnit is different from the traditional definition. It’s a fundamental misinterpretation where we apply an old word ‘unit testing’ to a new context – continuous delivery – and miss that the situation and purpose changed in the meantime.
We do want isolated tests- but It’s not about isolating a chunk of code, it’s about having a test that is isolated from other tests, so you can run them in any order locally or in the pipeline.
It’s also about isolating that test in time – we isolate each batch of changes from the others by running the tests in between. That’s fundamentally why Continuous Delivery is so different from what came before. We work in really small batches – the tests were passing the last time the pipeline ran, so if they are not passing now, I’ve isolated the batch of changes that caused the failure.
That’s what isolation means. It’s not about the size of the piece of code we’re testing, it’s the size of the batch of changes.
#5 The ‘unit’ in unit testing refers to a unit of code
Actually no, this is closely related to the previous point. The unit here is not one class or function, it’s not a box you can draw on a code listing around a piece of code structure.
A unit is a testable slice of behaviour. This is a key skill in TDD and in agentic development generally – that you develop a whole feature by slicing it thinly, using focused examples and partitioning the problem to restrict the context you need to keep in mind at once.
#6 A goal of unit testing is 100% coverage
A lot of people measure the coverage of their unit tests and have a goal of perhaps 80% or even higher for safety critical code. I’m not saying that code coverage is a bad thing – but Goodhart’s law comes in here. As soon as a measure becomes a target, it ceases to be a good measure.
It is useful to know if your coverage is bad, because there is almost never a good reason for that. It’s useful to know if your coverage went down unexpectedly – these AI agents have a nasty habit of deleting failing tests rather than fixing them. Coverage unexpectedly dipping is one way to detect that.
I’m not against measuring coverage, far from it, but I am against having a specific target. High coverage should be a consequence of other good practices, like doing TDD. Good coverage is a side effect of good engineering, and it’s really counterproductive to try to force good engineering by forcing good coverage. It’s too easy to game this metric.
#7 Writing the test first is always better than writing it afterwards
This is perhaps the most surprising myth – at least for me, a staunch proponent of Test Driven Development! TDD means you write the test first and testing becomes a key part of the development process. This has lots of benefits, and you miss out on that if the tests come afterwards. For example:
- It is better to write the test first if you want help limiting the scope of the current development episode – just make this test pass
- It’s better to write the test first if you want help pushing you towards a better quality design with testability for a start, also modularity, separation of concerns…
- It’s better to write the test first if you want help documenting your intent and identifying necessary behaviour
This activity – writing tests first – has so many valuable consequences and ripple effects that I usually want to do it, but there are situations where you don’t need it.
TDD is not always better. Prototyping, exploring the design landscape, evaluating options, or simply adding regression tests for code you know already works. Often the case for legacy code.
Conclusions
That was seven myths about testing, and explanations of how we do things differently in Moden Software Engineering and Continuous Delivery. Good testing is hugely important for success and fundamentally it’s part of everyone’s job to get the tests specified and running and supporting all the decisions we make.
Happy Coding!
