Coding Is Like Cooking

I’m at Agile Testing Days this week, and yesterday I attended a tutorial with Lisa Crispin. I’ve never actually met her before, although I have read many of her articles, and the book she wrote together with Janet Gregory, “Agile Testing”. It’s an important book, covering in some detail the role of testers in agile teams, with lots of practical advice and anecdotes from real life.

So I was very interested to meet Lisa and hear what she had to say. She had lots of general advice and war stories about what kinds of tools are useful in an agile setting. From all the new insights and thoughts I gathered from the tutorial, there were two stories that I’d like to share with you here.

The Whole team decides
Lisa’s main message was that automated testing tool choice is a whole team decision, since everyone on an agile team will be affected. She told a story about when she joined a team of Java programmers and persuaded them to try Watir for web testing – a tool that lets you write tests in Ruby. Although the developers agreed to this and were initially keen to learn Ruby, it became clear after a while that they just weren’t comfortable with it, and Lisa found she just didn’t get the help she needed when extending and maintaining the tests. They switched to a tool where the tests were written in Java and things worked much better.

That depressed me a little, I have to say. I’d personally much rather be writing Ruby than Java! Someone else chipped in with another story though. On his team they were also developing a system in Java, with the difference that the developers were very keen to learn Groovy. They had started writing tests using it, and it was working very well. It made writing tests more fun, since they got to learn a language they were interested in. The test automation work was less effort than expected, since they felt much more productive in Groovy than Java. I guess the difference is that the developers were motivated to learn the new language because they had chosen it.

You can do ATDD with GUI testing tools
Lisa told an interesting story about GUI testing. She said she was working on some new features and realized that the only way to test them was was via a GUI testing tool. She was at first very skeptical that they would be able to do Acceptance Test Driven Development with a GUI tool – the GUI hadn’t been written yet, so how could they use this tool to write tests?

In the end she said it turned out really well. They worked from GUI mockups of the new features, and wrote tests with placeholders. When the test scripts needed to interact with GUI elements that didn’t exist yet, she just wrote them in terms of what she’d like there to be there, based on the mockups. When the programmers came to implement the GUI, they could fill in the placeholders and quickly get the new tests running.

This was encouraging since it’s basically the way you work with PyUseCase too. A criticism we get sometimes is that since PyUseCase is a capture-replay tool, you can’t use it to define the tests before the GUI exists – a problem if you’re trying to do Acceptance Test Driven Development. Our experience matches Lisa’s though – you can define the test in general terms, with placeholders, and parts that won’t execute at all at first. Some parts of the test of course can be recorded from the existing GUI. As the GUI is extended for the new feature, gradually you replace the placeholders with executable statements until the whole test passes.

And now for the rest of the conference…
Geoff is giving a talk this morning about PyUseCase and TextTest as part of the main Agile Testing Days conference. We’d really like to get some feedback from experienced testers. It’s a different approach to capture-replay that most people here, like Lisa, will not have seen before. There are lots of other interesting talks and things going on too, and I hope to find time to blog a little more about the conference later in the week.

I had fun for a few months programming Ruby at eLabs, but now I’m moving on*. What exactly I’ll be doing next is not entirely clear. My plan, at least initially, is to work as an independent consultant specializing in automated testing and agile coaching. I’d also like to do some contract Ruby or Python programming. Hopefully I will find customers who are willing to hire me to do those things on a part-time basis, or with short term contracts, so I can do a mixture.

I have a long term dream to build some kind of product around PyUseCase and TextTest. I really believe in the approach Geoff has built for testing rich client python GUIs, and I’d like to see if it could be adapted for testing web applications. I have some ideas I’d like to try out, but I’ll need to find real customers with real applications and problems to try them out on. I’m hoping that will be possible through the agile testing consulting that I’ll be doing.

I’d also like to develop the idea of the coding dojo as a forum for teaching Test Driven Development and related agile engineering practices. I’m certain there is more that could be done to help people to get going with these skills. I’m looking into what courses are currently offered by local training providers, and hoping to both teach and develop those courses. I’m also working on my own formal training course based around the JDojo@Gbg meetings I led last year.

As you may have noticed from previous posts, I really enjoy going to conferences, and often speak at them. I don’t expect that to change much 🙂 Geoff and I are both scheduled to speak at Agile Testing Days in Berlin in October, which I am really looking forward to. It’ll be a chance for us to learn from some of the best in our industry, and share some of our ideas. Geoff’s testing tools are developing all the time, and I’ll be talking about what we’ve learnt from the many dojo meetings going on in Göteborg. I’ll be speaking together with my friend and former colleague Fredrik Wendt, a stalwart member of GothPy and assistant leader of JDojo@Gbg.

So it’s an exciting time for me, and I have several activities lined up to get me going with my new business. I’m also hoping to find a bit more time to spend writing articles for this blog. We’ll see if I succeed!

* eLabs is a very young company with only about 8 employees, and after I agreed to join CJ and his team back in January the company strategy changed a little. After I started in March, my role didn’t work out the way I’d anticipated. CJ and I had a good talk about it, and I think it’s with no hard feelings on either side that I left the company at the end of June.

I’ve just heard that two of my proposals for XP2010 have been accepted, which means I will definitely be off to Trondheim in early June. I’ve heard Trondheim is very beautiful, and the XP conference it usually excellent, so I’m really looking forward to it. It will actually be my 8th XP conference!

I’m going to be running a half day workshop “Test Driven Development: Performing Art”, which will be similar to the one I ran at XP2009, (which I blogged about here). I’ve put up a call for proposals on the codingdojo wiki, so do write to me if you’re interested in taking part.

The other thing I’ll be doing is a lightning talk “Making GUI testing productive and agile”. This will basically be a brief introduction to PyUseCase with a little demo. Hopefully it will raise interest in this kind of approach.

Perhaps I’ll see you there?

At GothPy yesterday, Geoff talked about code coverage and tests. Geoff has spent a lot of his evenings lately working on PyUseCase, and getting the test coverage up to 100%, (statement coverage), a feat which he achieved last week. The evidence for this is available for all to see on the texttest site, (which is updated daily, btw, so if it is not green and 100% the day you read this post, then clearly Geoff had a bad day yesterday).

I have limited experience of using coverage statistics to evaluate my tests, so it was interesting to hear Geoff summarize his findings. He thought it had been well worth the effort to get coverage to 100%, he’d found some bugs, some dead code, and improved his design along the way. Actually, saying he has 100% coverage is a statement that needs qualification. The tool he’s been using – coverage.py – has a feature whereby you can mark lines of code as # pragma: no cover, ie I don’t want this line counted for coverage purposes. So he’s marked 37 of 3242 lines like this.

The reason for excluding these lines is mostly practical – due to the nature of the tool you can’t test it automatically when it is in “interactive” mode without physically pressing the buttons yourself – so automated tests for that part are impossible. Some excluded lines are for error cases which should never occur, but for which it would be useful to have a good error message if they ever did.

Overall, Geoff thinks coverage is very useful to help you to identify

• poorly tested areas of your code
• mistakes in your tests
• dead code
• refactoring opportunities

The first one is obvious, but the others might take more explaination. Generally, each test is for a specific feature. If you think you have a test for a feature, but the code coverage shows the implementation of that feature not to be covered, then there is probably a mistake in your test.

Similarly, if your tests cover all your features and some code is not covered, maybe it’s not that important code at all, and could be safely removed. Geoff’s tests are not unit tests, they are testing the whole of PyUseCase, and that maybe makes a difference with this particular point. If I just had unit tests, and a piece of code wasn’t covered, I’m not sure I could as easily infer that it wasn’t needed as a part of a larger feature.

Refactoring opportunities can be identified from gaps in coverage too. The idea is that poorly tested code is a clue that it has other problems too. Perhaps you find two pieces of code are similar, and one copy has a gap in coverage. This could indicate they originate from copy-paste programming, and could be combined into one routine, with full test coverage.

Geoff had some tips for people who wanted to use coverage statistics to improve their tests.

• Don’t design your tests around coverage. Write appropriate tests, and then measure coverage.
• This applies even when working with coverage results. See the coverage report as containing clues for new tests, not commands.
• Use “#pragma : no cover” in your code to be explicit about code that you decide not to try and cover. Review these periodically.
• Don’t be fanatical about absolute numbers. Commands like “Aim for at least 85% coverage” are counterproductive. (You get what you measure).
• It’s always good to increase feasible coverage. It’s sometimes better to spend your limited time on other things. But if you don’t measure, you can’t make that decision effectively.

These last points are mostly also made in an article by Brian Marick which is quite old (1997). Geoff found the article when he was researching the talk for GothPy, and thought it was very good, and fits his experience.

Inspired by Geoff’s talk, I spent some time today trying to get some coverage numbers for the code and tests I’m working on at present. Unfortunately it seemed to be a bit tricky to get the coverage tool to work. It’s not python, of course, and that may have something to do with it. Hopefully I’ll sort it out and be able to write a new blog post about my own experiences with coverage statistics sometime soon.

One of the things I like about GothPy is that lots of the people in the group enjoy writing code in their spare time, and like to share with us what they’re up to.

A little while back, Olof came up with this little tool, PyTDDmon, which is to help you when you’re doing TDD. Instead of your running your tests by hand, PyTDDmon sits in the background and runs them for you. When they are green, it has a little green blob, when they are red, it tells you how many are failing, and clicking on it gives you the stacktrace. Olof put together a screencast to show how it works.

Following Geoff’s presentation of PyUseCase at GothPy in December, he and Olof started to discuss whether it could be used to test PyTDDmon. Since it’s using the GUI library Tkinter instead of pyGTK, it didn’t look all that straightforward.

Of course, Olof developed the app using TDD in the first place, (definite own dog food eating going on!), and his unit tests have about 48% statement coverage. The parts with least coverage are GUI-related, so there seemed to be scope to improve matters by adding tests via the GUI.

Geoff sat down a few days ago with a branch of the PyTDDmon code, started trying to generalize PyUseCase to work with Tkinter, and to write some tests for PyTDDmon. (Geoff also likes writing code in his spare time 🙂

As I write, he’s just checking in his changes and a new suite of tests for PyTDDmon that use the new, improved PyUseCase, with (basic) Tkinter support! He says the hardest part was understanding Tkinter, and after a spot of refactoring, only an extra couple of hundred lines of code were needed to support it.

Subsequently adding tests for PyTDDmon was very easy (Geoff wrote most of them while I made biscuits with the children this afternoon 🙂 All he had to do to Olof’s code was assign names to some of the widgets, and sort out a problem with closing a window. (There was a listener for a mouse click that would destroy the window, before other listeners got a chance to do anything, such as record the click in a use case log…)

So now the statement coverage of the PyTDDmon tests is at 98%. I look forward to future, equally productive GothPy meetings and discussions!