This post is also available as a video on the Modern Software Engineering channel.
It’s a really scary time to be a software developer. So much of our expertise feels like it’s irrelevant now. This last year has changed everything about the way we write code. Particularly when it comes to the skill of Test-Driven Development, TDD. This has been a cornerstone of the way I worked these past 20 years or more. A recent article on Martin Fowler’s site says that asking agentic AI to do TDD is more theater than actual value. Is TDD worthless now? Actually, I think TDD with agentic AI could still be a good idea. Let me explain.
The article on Martin Fowler’s site was from Birgitta Böckeler – “TDD inside the agent loop – theater or actual value?”. She’s done an evaluation, and concluded “there was no discernable difference based on TDD workflow vs no TDD”. Birgitta was kind enough to share an early review copy of this article with me before it was published, although I don’t think I had much in the way of useful comments at that time. I was really surprised by this conclusion, and I took some time to study Birgitta’s work carefully before coming up with this response.
Firstly I do recommend reading the article – it contains a lot more interesting information than only the headline conclusion, particularly about the benefits you’d like to get from TDD. The conclusion that TDD isn’t worth the tokens it costs is logical based on the study she’s done. I’ve not seen any other formal evaluations of TDD for agents, whether it works, and how many tokens it costs. Kudos to Birgitta for doing this.
I’m not convinced though. A lot of people I trust are saying something else. Several issues caught my eye when I read through the study more carefully.
Birgitta’s study
Birgitta’s study was designed to measure the effectiveness of prompting an AI agent to do TDD. The use case is, you’re having the agent run in a loop to implement a whole feature without any human intervention. Do you tell it to do TDD or not? Does that help?
Birgitta has come up with several specs for various features to build, and she has different agents implement them. In the study she varies the starting prompts. Some are told to do TDD, and some are not. Then she analyses the various solutions for compliance with the spec, test coverage, code quality, and judges whether the ones doing TDD did better overall. Apparently, they didn’t.
Well, were the ones that had a TDD prompt actually doing TDD? The prompt is pretty brief, and the difference between the TDD case and the non-TDD case is only a couple of paragraphs of extra text. It looks a lot less detailed than the TDD descriptions I’ve seen other practitioners using with their agents.
(In the image above, the TDD prompt is on the right, non-TDD on the left).
I was reassured at first that Birgitta had analysed the session logs and concluded the agents had been doing TDD when prompted. Tests were being written before implementation code, in small steps iteratively. Yes, good. And what about the rest of TDD?
Missing Test List
In canon TDD by Kent Beck – the inventor of TDD – he says the first step is to make a test list. Every time you go round a TDD loop, you should update this list with what you learnt about the problem. Birgitta’s prompt doesn’t specify to do that step. Missing this out means that the agents may be correctly writing the tests before the implementation, working in a small loop, but they are not keeping track of the whole task and which tests should be written.
In fact, in one of the experiments when the TDD solution came out particularly badly, Birgitta did extra analysis which shows the biggest factor causing the poor outcome. It was that the agent treated the tests as the only spec, and the tests didn’t cover a lot of things that were actually needed. The analysis says “unlisted-but-required behaviour simply never got written”. It seems to me that including a test list step in the TDD process could have prevented that.
Prompting for Refactoring
The other big problem for me with claiming that the agent really was doing TDD, is that the refactor step is not prompted very well.
In fact, half way through the study Birgitta changes the TDD prompt to improve it. She adds more instructions, including strengthening the refactor step. The prompt now includes some specific code smells to watch out for. Then in the next run, the code quality improves considerably compared with the control. That’s good, TDD really needs a strong refactoring step to get good results.
(In the image above we see a diff between the original TDD prompt on the left vs Updated TDD prompt on the right. Note the increased text in step 5 – refactor).
Stating in general terms a list of code smells to watch out for is not really enough though. In my previous article with Ivett Ördög we discussed Habit Hooks. You include a deterministic sensor in the agent harness that detects code smells, (not just a prompt – something like a linter), and then when it identifies a smell, you give the agent a prompt with specific guidance about how to fix that code smell. Birgitta’s TDD harness didn’t have anything like that.
I speculate that even stronger refactoring guidance would have even further improved the code quality.
What we can learn from this study
I admire that Birgitta tried to do this study but I don’t share her conclusion that “the role of TDD is significantly smaller than pre-GenAI”. Her TDD prompt and setup is just not good enough to show that.
What Birgitta’s article does do well is to enumerate the benefits of TDD which we would like to still have when using GenAI for coding. She has designed a study to find out if we can get them through a simple prompt. That is valuable framing, and she doesn’t find good evidence for it. Birgitta is clear on this point – she has not proved TDD gives better or worse results, although it does seem to cost more tokens.
Well. There’s a challenge. Any researchers out there with a good token budget want to take on this evaluation? You could try out some different TDD prompts, like with a test list or stronger refactoring hooks?
Understand what practitioners are doing first
I’m not personally offering to do that study. At least not right now. My focus at the moment is to first understand what people are doing before we try to evaluate or compare or judge. Then when we design a study we can evaluate realistic TDD prompts and harnesses that people are already recommending.
A couple of months back, I proposed a code kata to the community of Samman technical coaches that I lead. I know there are a lot of people in that group who were previously experts with TDD, much like me, and have recently adopted agentic AI.
I got 8 fully worked solutions showing people’s harnesses and TDD setup. I’ve written about two of them – from Nizar Salander and Ivett Ördög – which I thought were each worth talking about. The other six are also really interesting. All of them are doing something more sophisticated than two paragraphs in a prompt at the start of a coding episode. I’m going to be talking about more of them in future, and I expect the state of the art to continue to move on rapidly.
All these practitioners are coding daily with their setup and adjusting it frequently. It’s going to take some time for everyone to try stuff out and learn from each other and cross-pollinate. There is so much innovation going on in this space.
Conclusions
Thanks Birgitta for your study. The question you posed – “Is TDD inside the agent loop theater or actual value” – is still open. On this channel and my blog and in my community, I’m sharing what I’ve tried and what I’ve seen working. The last year has changed everything I knew about coding, and I’m going to keep looking for how to get those benefits of TDD with Agentic AI.
Happy Coding!
