Coding Is Like Cooking

Last week I was in Oxford at “Iverson College”, which is a conference on the topic of Array Language Programming. There were about 25 programmers there, most of whom are expert in one or more of APL, J, K, or Q. It’s not my usual comfort zone, put it that way! I’m fairly competent with a number of programming languages, notably Python and Java, but nothing I know is really much like these array languages. It’s been a huge culture shock, but in a good way, I think.

My main discoveries are that Array Programming is different again from Object Oriented Programming and Functional Programming, (although it has a lot in common with functional programming), and that this community contains some exceptional programmers. The total number of array language programmers is however extremely small and their work seems to be pretty much unknown to the wider programming community.

Array Programming Languages
I mentioned before four languages, APL, J, K and Q. They are similar to each other, kind of like Ruby and Python are similar to each other. I’ve gone through an introductory training in each language this week, largely given by the language designers themselves. I’d like to relate a little of what I’ve discovered about them.

APL
This is the oldest of the array languages, invented by Ken Iverson in the 1960s. It’s notorious for using an alphabet of funny-looking symbols to represent the built-in functions. You can try it out at http://tryapl.org – an interactive REPL (Read-Evaluate-Print-Loop) where you can put in snippets of code and see what the symbols do.

I thought at first that APL looked really intimidating and unnecessarily weird. Now having got to know it a little, I can see the benefits to the little symbols. They make the code really concise and unambiguous, and it doesn’t take long to learn their names. Once you can pronounce each symbol in your head as you read the code, it’s not much different from writing out the names in full in the editor.

The variant of APL that most of the conference attendees use is produced by the company Dyalog. I first met the CTO, Morten Kromberg, at an XP conference in 2006. He’s shown me some APL before, but this time I really got a chance to sit down with him and look at how he writes code. Dyalog APL has a powerful IDE including a REPL, where Morten showed me how he plays around with data and code, in order to come up with some useful APL expressions. When he’d got something working, he transfers code from the REPL into a file, to make it re-usable and shareable. It’s a familiar way of working to me, many Python programmers code this way, flipping between the REPL and a script file. It was a real pleasure to code with Morten – he is an extremely skilled programmer. Dyalog APL looks nice too, it has a fully-fledged IDE, and interfaces with .Net, Excel spreadsheets, ASP.net and more. It would fit nicely into the technology stack of many IT departments basically.

J
This was Ken Iverson’s next language, created together with Roger Hui, who now continues development of it. J is similar to APL in many ways, but is open source, and uses only ASCII characters. They’ve made an effort to make it open and less intimidating to newcomers, and probably for that reason, it’s the one I chose to download and try to learn before the conference.

I met Roger at breakfast on the first day of the conference, knowing nothing about who he was, he just said he was a programmer. I confessed that I’d downloaded J and made some joke about hoping I’d get on better with it than Ron Jeffries. (Ron wrote articles in his blog, about his efforts to learn J, and later gave up, finding it too hard!). Roger genuinely didn’t know who Ron Jeffries is, although he did know of the agile manifesto. He was very kind and concerned to help me to understand J though, (and Ron, if he wants!)

Despite my head start with J, by the end of the conference I found APL code easier to grasp – J seems more extreme to me. Roger calls J “executable mathematical notation”, and I’ve always been a bit more of an engineer than a mathematician.

K
K was invented by Arthur Whitney, who was also at the conference. I didn’t really get a feel for how the language works, more than that it’s extremely terse.

Arthur gave a talk at the conference, about his new project, KOS. He and two other guys are writing an operating system pretty much from scratch, using K, C, and bits of the linux kernel, (although they’d like to remove those). He showed us how you write applications for this new OS in K, by demonstrating building a text editor. He began from the alpha version of the OS with just a window manager, and a plain new window canvas that didn’t respond to any keyboard or mouse events.

Arthur added a line of K code to let you enter text into the window – a listener to key presses. Then a line of code to move the caret around with the arrow keys. Then a line of code for changing the font size. Then scrolling. Code to handle Ctrl-C and Ctrl-V to copy and paste text…. in the space of less than half an hour, he had an equivalent to notepad working. No compilation, no reloading. And the code was…. phenomenal. You can see a version of it here. I can’t read it really, it looks mostly like line noise to me. All his variable and function names are one or two characters, and K just seems pathologically terse.

I raised my hand and asked Arthur if he thought his code would be more readable if he used longer variable names? He thought for a moment, looking surprised and a little bewildered by the question. Then shook his head and said slowly “No. no. I don’t think I need that. I want to see all my code on the screen at once”. Needless to say, that was a big culture shock moment for me!

The size of the codebase is something all the array language programmers seem really concerned about, even if Arthur’s code is considered extreme even in that community. One thing I did later that week was take a piece of code that is in Robert C. Martin’s book “Clean Code”  (Args.java), as an example of clean Java code, and showed it to the group. There were general exclamations of “aargghh! that hurts my eyes!” but after a little while as I explained the structure of the code they seemed to appreciate it a little better. What they did say that intrigued me though, was that they automatically scanned the page looking for the symbols – the >, !, = signs – the parts that do something, as they put it. The other text they said obscures the structure, it distracts the eye. Yes, that’s right. Having names for the functions and variables makes the code less readable.

KDB+ and Q
KDB+ is a very small and fast commercial database largely used by financial institutions, also originally created by Arthur Whitney. Q is a kind of domain specific language built on top of K, that you use to query the data in a KDB+ instance.

I sat down with Attila Vrabecz, an experienced Q and KDB+ programmer, and we coded together for a couple of hours. We tackled a problem I’d previously coded with Morten in APL, to help me see what was different. There were many similarities – the workflow was the same for example – experimenting in the REPL before transferring the code into a more permanent, reusable form. I noticed Q has many more English words in it, fewer strange symbols, and Attila made more use of library functions than Morten did. It seems Q is designed to be approachable for a former SQL programmer, although once you scratch the surface, it’s much more like APL than SQL.

Test-Driven Development
I gave a talk at the conference about TDD. My aim was to provoke discussion, and argued that writing automated tests using TDD is the best approach. I was definitely successful at sparking a discussion! Actually, it didn’t seem the idea that programmers should write automated tests for their code was all that controversial, especially amongst the more seasoned developers present. We got way more hung up on how large a chunk of code counts as a “unit”, for your unit test, and what clean code looks like in an array language. To my eye, their units are large and their clean code is terse.

A challenge for the future
Dave Thomas, former lead developer for the Eclipse project, and general software visionary, is also an APL and K programmer. He flew in for just one day of the conference, and his talk functioned as the keynote address for the week – it was a clear challenge to the Array Languages community.

Dave painted a vision for the future where people will be living in a sea of big data they don’t understand, and lack adequate tools to query. He sees a great opportunity for array languages, which are generally very good at handling large amounts of data.

He ended his talk with an ambitious challenge to this community to get its act together, start being seen as a credible alternative, and grow. I could only applaud and agree – I found his advice insightful, and I hope the array languages community will do as he suggests.

I spent one very pleasant evening chatting with a woman who is about to embark on her PhD in atmospheric science. She’s hoping to use array languages to help her create software models that will execute quickly on huge arrays of multi-dimensional climate data. Her work sounds fascinating, and I hope it’s a sign of array languages starting to be used beyond their traditional niche in finance.

So I’m leaving the conference carrying a huge tome entitled “A complete introduction to Dyalog APL”, some pieces of code I’ve written, and good intentions to study further. I do find it fascinating that even with the little I know of it, APL allows me to think about and solve a problem differently than I do in Python. I anticipate I’ll find plenty of people in the Array Languages community willing to help me if I do continue to try to learn it. They’re an opinionated, quirky, mature, gentle, yet small bunch of extremely skilled programmers, and I’m glad to have met and coded with them.

I’ve been working on this Kata “Gilded Rose” at a few different coding dojos lately. There is even a video of a session I did at the “Tampere Goes Agile” conference recently. In the video, you can see me talking about my Principles of Agile Test Automation, which I have just written about, and updated in my last blog post.

I think these test automation principles are useful to think about when you’re doing the Gilded Rose kata. The basic plot of the Kata is that you’ve just been hired to look after an existing system, and the customer wants a new feature. Having a look at the code, you can see you’re going to want to refactor it a little before adding the new feature, and before you do that, you’re going to want some automated tests.

So the first part of the Kata is to add automated tests to the existing code. You’ve got a requirements document the customer has given you, and you can use it to identify test cases. You’ve also got the code which you can read and execute and work out what it does. The customer is happily using the code in production right now, so you can assume that the behaviour it has is the behaviour they want to keep, whatever it says in the requirements document. (hint!)

Warning – spoilers lie ahead! You should probably try the Gilded Rose kata for yourself before reading on!

When I’ve done this exercise with various groups, I’ve spent a lot of time discussing with people how to make their test cases really readable, and express the requirements clearly, and at the same time useful as regression protection when refactoring the code later.

When you design a test suite you have two main aims – to help you understand what the code should do, (and what it does now), and protection from regression failures when you update it. It can be a bit tricky to do both with the same test suite. If you focus solely on describing the requirements in an executable way, you tend to miss edge cases and there are gaps in the regression protection. If you focus only on regression protection, you’ll spend time analysing the edge cases, and measuring code coverage to see how well you’re doing, but the test cases can become quite hard to read and understand.

You can see for yourself by comparing this test case by Bobby Johnson with this text-based approval test. (It was written by several people at a GothPy meeting). Bobby’s test case is extremely readable and expresses the requirements clearly. He’s done pretty well on the edge cases, but I think he’s missing one or two*. With the text-based approval tests, it’s not so easy to understand what the underlying business rules are, although the regression protection is very good.

When I do this kata with a group, we spend some time discussing the various test cases we’ve come up with, and showing them on the projector. When we did this last week at the Booster Conference, people commented that showing these different test cases had given them a better understanding of “readability” and “regression protection”, and many went on to improve their test suites.

Once you’re reasonably happy with your test suite, the next task is to do the refactoring and add the new feature. How useful are your test cases for regression protection? It’s very easy to make refactoring mistakes in this kata, and you will be testing your tests! You may discover while refactoring that there are more test cases that you want to add. Version control can be pretty useful, so you can run the new test cases against the original code.

There’s also an interesting restriction on your refactoring options – the “Item” class is owned by a nasty-sounding goblin and he doesn’t want you to change his code, so if you do, you have to be prepared for some serious consequences! When comparing refactored solutions at the end of the dojo, this is often an interesting discussion point – did you change the Item class? Is your new design so great that you’re prepared to argue with the goblin for it?!

I havn’t tried this, but I would actually like to try running the text-based approval test against all the refactored solutions at the end of the coding dojo, as input to the retrospective. I think this test covers all the edge cases very well, and would reveal any refactoring mistakes that were not caught by the tests people had developed themselves. That would be interesting feedback to have!

If you havn’t tried the Gilded Rose kata yourself, I do recommend it for practicing writing good test cases. I’d be happy to get a pull request from you if you want to translate the exercise into your favourite programming language, or you can do it in the original C#, as Bobby suggests.

If you’re interested in taking part in a coding dojo with me, I’ll be at several conferences later this year: ACCU in Bristol, XP2013 in Vienna and Test Automation Day in the Netherlands.

* I believe he’s missing a check that the quality of backstage passes doesn’t increase past 50

One of the great privileges of being the programme chair for Scandinavian Developer Conference is getting to choose the keynote speakers. This year, I’m delighted to present Dan North and Janice Fraser, both thought leaders in the field of software development. Although from different backgrounds and perspectives, they’re both accomplished at building software that delivers great business outcomes. I’d like to tell you a little about each person, and why I’ve invited them to Göteborg for SDC2013.

Dan North – a man full of intriguing ideas

I first came accross Dan North at a conference in 2007, talking about a topic I was very familiar with – unit testing – but using a whole new set of words. Behaviour Driven Development (BDD) intrigued me then, and still does now. How can switching the word “Test” for “Behaviour” and “AssertEqual” to “Should be” make such a difference to the way you end up designing your code?

In his famous article from 2006, “Introducing BDD” Dan explains that he found when he stopped talking about “Testing” and started instead used the word “Behaviour”, “… a whole category of coaching problems disappeared”. People understand more easily that defining the behaviour of the software is an important activity for the whole team, not just testers. It also changes the way you as a programmer think about your code, and helps you focus on what’s important.

BDD as an approach to software development is still being actively developed and written about, although Dan himself has largely stepped aside in favour of other thought leaders like Elizabeth Keogh, Chris Matts, Olav Maasen, and of course Gojko Adzic. Gojko wrote the hugely influential book “Specification by Example” which is all about having useful conversations about software behaviour, and expressing that in terms of executable examples – ie a lot like BDD. At about the time that book was being written, Dan himself chose a different road. He actually stepped out of the consultant life entirely for about two years, taking up a full time position developing software at a financial trading firm.

His latest ideas around “Accelerated Agile” to a large extent come from his experiences working in that high-powered trading environment. He wrote in his blog: “This team was the most insanely effective delivery machine I’ve ever been a part of”, and I find that particularly intriguing. He says that standard agile practices like Continuous Integration and maintaining a Product Backlog weren’t being used! So what exactly did they do in order to be so effective?

Dan has also famously opined that “Programming is not a Craft”, and argues that the “Software Craftsmanship Manifesto … [is] a spectacularly easy bandwagon to jump on”. He says he’d rather see, “…a call to arms to stop navel-gazing and treat programming as the skilled trade that it is.” So there. Dan certainly has some strong opinions, and when I’ve met him, he always seems to express himself with wit and intelligence.

I’m really looking forward to Dan’s keynote address on Tuesday 5th March. I’m intrigued to find out what “Patterns of Effective Delivery” is all about, and to hear his latest opinions on the practice of software development.

Janice Fraser – a pioneer of Lean User Experience

To a large extent, Silicon Valley is the epicenter of our whole industry. Many of the biggest and most influential software companies in the world are based there, and as an incubator with a friendly climate for startup companies, it is unparalleled, despite the efforts of many other regions around the globe to imitate their success.

Janice Fraser has been working in Silicon Valley for over 15 years, and she says in her CV “I’ve seen a lot — bubbles, bursts, and fantastic acts of collaboration that have transformed literally billions of lives.” Yes, that’s right, billions of lives.

The latest trend coming out of the Valley is “Lean Startup”, a term coined by Eric Ries, and documented in his bestselling book “The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Business”.  I first heard about it in 2011, when Joshua Kerievsky, an early adopter of eXtreme Programming and successful entrepreneur, published an article “Agile vs. Lean Startup”. He says, “[Lean Startup] rocks. It rocks far more than Agile.” If it rocks far more than agile, then I find that pretty intriguing!

Janice Fraser is of course also an early adopter of “Lean Startup”, and has pointed out that the ideas in it are not all new. She says “The Lean Startup, is a rediscovery of user centered design… [it] gives UX teams an unqualified mandate to make products customers love.”

Janice herself is a serial entrepreneur, having led several startup companies. She says in her CV, “My proudest success is Adaptive Path, a leading product design firm. I was a founder and served as the company’s first CEO”. Adaptive Path is still successfully in business.

Janice is not shy about recounting her failures either, she’s written a candid report of how she started “Emmet Labs” in 2007 intending to change the world, right through to when she laid off all the staff in 2009. I think her article “7 things I did right with Emmet Labs” shows how much courage and determination it takes to build a company, and how resilient and clear-thinking Janice herself can be in a crisis.

Janice’s business these days is helping other startup companies to succeed: “Before you build anything, find customers, learn their needs & goals, and measure your progress towards your vision” – an extract from the marketing materials for her company, Luxr. She’s also just about to publish a book “The Lean Product Book: How Smart Teams Work Better”, which I guess will document the kind of advice she gives to her clients – all about Lean User Experience.

All this talk of startups and product development in Silicon Valley might seem a long way from chilly Göteborg and our IT industry, dominated by a few huge corporations. I think it’s just the kind of thing we need to hear about, though. Companies of any size need to renew themselves and develop great new products in order to flourish, and this is clearly an area where Janice is innovating and leading the world. I’m really looking forward to hear what she’s going to say in her keynote “Lean Startup Product Teams: Principles of Success”, on Monday 4th March.

Do you automatically get better design with TDD? Does an otherwise average software developer produce superior designs if they write the tests first rather than afterwards? Does it make a difference what style of TDD you use?

incident #1

I was at a session at XP2012 with J.B. Rainsberger called “Architecture without Trying”. He demonstrated how he could develop a software system for Point-of-Sale terminals using TDD, and how the design naturally tended towards an MVC pattern as he did it. He claimed that purely by doing TDD, and focussing on two things, (removing duplication and improving names) that a good design would naturally emerge.

incident #2

I heard a talk by Luca Minudel at Agile Testing Days 2011 called “TDD with Mock Objects: Design Principles and Emergent Properties”. He was talking about a study he had done where he got people with varying levels of experience at TDD to do four short exercises. He also got them to answer a questionnaire about their knowledge of SOLID principles, and TDD. He then evaluated how well the designs they came up with in the exercises adhered to SOLID principles, and tried to correlate that with their TDD skill. He found that the people skilled in TDD did better in the exercises than those who only knew the theory of SOLID principles. The practice of TDD seems to help people with design. Luca also found that those more experienced with the London School of TDD did even better than other TDDers.

incident #3

I was working at a client recently when I met a developer from a different department. He came to see me several times over a period of a couple of weeks, and asked for advice about TDD. On about his fourth visit he told me he had written some code and now it was basically working, he wanted to write tests for it. He said he was having difficulty since he’d written a lot of static “helper” methods. I advised him that static methods make code quite hard to test, and can often be a sign of a not very good object oriented design.

He suggested we should invest in a fancy mocking tool that would enable him to easily replace these static methods in the tests. I told him a better investment would be for him to learn to write the tests first, get better at OO design, and not use static methods in the first place. I was probably a bit blunt, and he was quite polite, all things considered. He protested that he shouldn’t have to change the production code in order to get it under test, then left. That was the last time he came to me for advice.

Discussion

So does doing TDD guarantee better design? Well it should certainly help. I’ve presented before about the way TDD gives you early feedback on your design and plenty of opportunities to refactor. It’s less help though if you don’t know what a good design looks like in the first place. I think J.B. goes too far in his claims – if you don’t know MVC or SOLID principles then I’d be surprised if they started turning up in your code with any consistency.

“No tool nor technique can survive inadequately trained developers” 

(A quote attributed to Steve Freeman). I think you do need to invest in learning good design techniques independently of TDD. If you lack basic OO design skills you probably won’t be able to do TDD in the first place, London School or otherwise.

I’ve been learning and improving my practice of TDD, including the London School, for many years now, and I was intrigued by Luca’s claims that it led to better adherence to SOLID principles than classic TDD. The London School involves an outside-in approach to design, that makes heavy use of mocks to check interactions between objects. This is in contrast to a more classic TDD style that prefers to verify the code works by checking the state of an object after an interaction.  I wouldn’t claim to be an expert in the London School of TDD, but I think I understand the basics and can adopt this style when I feel the problem is appropriate for it.

I tried out Luca’s four problems, (here on github) to see how I did. Luca very kindly gave me some feedback on my code, and I found hadn’t done as well as I had hoped to in adhering to SOLID principles. I’d got the code under test, but in a few places I could have improved the design more. I also slightly misunderstood the requirements for two of the problems, which led me to fork the repo and improve the instructions 🙂

I think in the cases where I could have done better with the design, it’s possible using the London School of TDD would have led to the improvements. I’m feeling there might be something in Luca’s conjecture. On the other hand, these problems might be so small and abstract, that I didn’t behave the same as I would in a real codebase. Certainly in one case I felt it wasn’t worth extracting an interface when there was only one implementation for it. In a real system maybe it would be more obvious that more implementations were likely, and that adding the interface would lead to a more decoupled design. Or then again maybe I’m just too used to python where expicit interface classes don’t tend to be used. Or maybe I’m just making excuses! In any case, doing these exercises has made me more interested to improve my knowledge and practice of the London School TDD style. 

I think these exercises are interesting little code katas in their own right, quite apart from Luca’s study on TDD. I think you can use them to learn about the SOLID principles, and practice some of the refactorings you often have to do to get badly designed code under test.

I’m working on a python translation of the exercises so we can try them out at the Gothenburg Python User Group meeting next week. Feel free to fork the repo and have a go at them yourself.

Scandinavian Developer Conference will be held in Göteborg in April, and last week we launched the detailed programme on the website. I’ve been involved with the conference ever since the first event in 2009, but this year I’ve taken on increased responsibilities, acting as Programme Chair.

When P-A Freiholtz, the Managing Director at Apper Systems*, (the company behind the conference), approached me about taking this role, I jumped at the chance. My business is all about professional development for software developers, and a conference in Göteborg is a really good opportunity for a lot of local people to hear about what’s going on in the world outside. Many of the developers I meet in my work don’t often get away from their desks, spending the majority of their time caught up in the intrigues and deadlines of a single project and company environment. SDC2012 will be a chance for a lot of local people to broaden their horizons without it costing their boss a fortune, or disrupting their usual schedule too much.

I wrote an editorial on the front page of the conference website which explains more about what’s on the programme and who should come. Everyone involved in software development, basically 🙂

These days, we compete in a global marketplace for software development. I’m hopeful about the future, I’m enthusiastic about Swedish working culture, and I think Scandinavian Developer Conference is helping Göteborg grow as a center of competence. Please do join us at the conference.

* formerly named Iptor – now under new ownership.