Coding Is Like Cooking

Continuous Integration and Code Review are strongly correlated with success. Many use Pull Requests for code review, but for co-located teams this can be an obstacle for CI. Is there a better way?

There are three developers on the (fictitious) team: Annika, Boris and Carol. Annika is a recent hire, fresh from university, Boris is the team lead, and Carol has been around the longest. Each of them is working on a different task. They all synchronized their work with their shared master branch when they arrived at the office today, and now it’s approaching morning coffee time. They have all made some changes in the code which they’d like to share with the rest of the team.

Annika is working on a local branch called ‘red’. She checks it’s up-to-date with master and pushes it to a remote branch named ‘ready/red’. It’s similar for Boris and Carol. They are on blue and orange branches respectively and push their changes to ready/blue and ready/orange.

The Build Server is set up so that it detects new branches on the Version Control Server that follow a naming convention. Any branch beginning with ‘ready/’ is scheduled for integration, and only one of these integration builds runs at a time. The Build Server delegates builds to one or more agents,and since the ready-job agent is idle, it picks up the ‘ready/red’ change straight away and leaves the other two ready-branch builds in the queue.

The build job has several steps. First, the agent merges the ready-branch into a local copy of the master branch. Annika’s changes get a simple fast-forward. The agent performs a full build, static analysis, code style check, and unit test. Everything goes well, so the agent pushes the merge result up to the Version Control Server and posts a message on the team message board.

Things start out similarly for Boris’ ready/blue branch. The build agent takes a copy of master from the git server and merges in the ready-branch. This isn’t a fast-forward merge, since there is a new commit in master for the ‘red’ changes, but it’s still ok. So long as the agent can do the merge without finding any conflicts the build can continue.

The agent then proceeds to the next build steps. Unfortunately, Boris hadn’t noticed that one of his changes caused a test failure. The team has previously agreed that the code in master should always pass the tests, so this means Boris’ changes shouldn’t be shared. The build agent sends a message to Boris telling him about the failed tests, discards its merged branch, and moves on. Carol’s ready/orange branch is up next. The build agent starts again with a fresh copy of the latest master taken from the git server. Carol’s changes also merge without difficulty and this time both build and tests pass. The build agent pushes the merge commit to the server and notifies the team.

Boris and Carol are having a cup of coffee while they wait for the build server to integrate their changes. Annika is chatting with the Product Owner about the new feature she plans to work on next, ‘cyan’. When they get back to their desks they see the messages from the build server.

Annika is happy to see her changes integrated successfully. She fetches the latest master from the remote git server. She’s now completed the work on the ‘red’ task, and her changes should undergo a code review. She marks the ‘red’ task as finished in the issue tracker and adds an agenda item to the team’s next scheduled code review meeting, which is later that week. Annika selects a new task to work on and checks out a local branch from master called ‘cyan’. Boris sees the message about his failed tests and realizes immediately what he missed. He’s a little embarrassed about his mistake, but happy his teammates are not affected. They may not even notice what’s happened. Boris takes the opportunity to merge the latest changes from master into his ‘blue’ branch. He is quickly able to address the problem with the tests and pushes an update to ready/blue. The build agent gets to work straight away.

Carol is not finished with the ‘orange’ task, but is happy to see her initial changes integrated successfully. She fetches master and merges it into ‘orange’ before continuing work there. She’s noticed a design change that would make her task easier. She plans the refactoring in steps so she can push small changes frequently as she completes the re-design. Sharing her changes with the team often will make it easier for everyone to avoid costly merges. Later that week, in the code review meeting, the team looks at Annika’s changes for the ‘red’ task. It represents a couple of days’ work. The code review tool presents a summary of all the commits involved and they discuss all the changes in the development of the ‘red’ feature.

Unfortunately, Boris and Carol are not happy with a part of the design Annika has made and the code formatting needs improving in places. The outcome of the meeting is that they agree to pair program with Annika on a refactoring of the design, and encourage her to initiate informal design discussions more often during development. The idea is that the more experienced developers, Boris and Carol, should help Annika to learn better design skills. The team finds the code-formatting issues a bit annoying since this kind of detail shouldn’t be in focus for a code review meeting. They create a task to improve the code-style checker in the pre-tested integration build to catch any similar code formatting problem in future.

Commentary

This development process is working really well for Annika, Boris and Carol, and pre-tested integration is a small but important piece. They are not using pull requests, but they have checks on what code is allowed into the master branch, and they have good code-review culture. Integration to master happens at a faster cadence than the flow of work-items. That’s important. Integration is less painful the more often you do it and you might not want to break your work items down to the same small granularity that would be best for code changes. You also don’t necessarily want to delay your integration by waiting for a teammate to review your pull request.

Strictly speaking, this process is not Trunk-Based development since there are more branches involved than just trunk, but so long as the integration is frequent in practice it’s indistinguishable. The benefit of this over Trunk-Based development is, of course, that Boris or any other developer can’t unwittingly break master for the rest of the team.

If you’re using Jenkins you can easily automate the integration process with our Pretested Integration plugin. It’s not difficult to implement this functionality yourself for other build servers. Whichever approach your team chooses, I recommend you settle on a process that results in frequent integration together with collaborative and constructive code-reviews.

Continuous Integration is now synonymous with having a server set up to build and test any change submitted to a central repository. But this isn’t the only way, or even how CI used to work. What did we do before DVCSs and Jenkins?

What is the connection between cricket and pre-tested integration? Pre-tested integration is about the way you set up your Continuous Integration server to metaphorically catch the balls you drop and the tests that fail, just like in cricket.

When a team first sets up a build server, this is the typical way to do it – you have it build any change in the mainline branch, and notify you if it fails. Developers are supposed to run the build and test before they push the code; the build server is supposed to be a kind of back-stop, catching any balls you drop. Any developer who manages to break the build is singled out by having their name displayed on the ‘information radiator’ screen in the corner. Just a little ridicule, some good-natured teasing, enough so that they are shamed into remembering to run the tests next time.

The thing is, it’s quite easy to make a mistake that causes the build to fail. People can get upset with you for breaking the build, and it can be embarrassing to have your name in lights. You can get into a bad cycle of stress, making mistakes more likely, causing you to break the build more often, making you more stressed…

Even a short length of time with a broken master can disrupt anyone else who is trying to integrate their changes, and discourage them from doing so. You can get into a bad cycle of people mistrusting the master and hoarding their changes. This causes bigger commits, more likely failed builds, and less trust… if it gets really bad you can find yourself spending several miserable days resolving three-way merges. Believe me, you don’t want to go there!

I think we can learn something from how they used to do Continuous Integration in the days before CI servers became widespread. I think they had a more humane process that was less stressful, and led to more virtuous cycles than downward spirals.

The whole point of Continuous Integration is to ensure that all the developers are working on essentially the same version of the source code. If you compared my working copy with those of my colleagues you should not be able to see any more differences than have accumulated through work either I or they have done today. Ideally, only changes we’ve made in the last couple of hours. The point of this is to make integrating our work a trivial task that takes seconds and is hard to mess up.

If you’re interested, you can read Jame Shore’s description of CI without a build server, but I will summarize it below:

When a developer was ready to integrate their changes they would physically walk to a designated ‘integration machine’, load their changes there, and perform a full build and test. No-one else was allowed to begin their integration until they were done. If the integration build passed they would simply say in a loud voice to the other developers to pull their changes from that machine. If there was a problem with the integration, and they couldn’t fix it quickly, they would revert their changes on the integration machine, and return to their workstation to fix the problem.

It’s a process that clearly only works if everyone is sitting in the same room. On the other hand, it works fine even when there are no atomic commits, no build servers, and no automated merges. Let me highlight some aspects of this process:

• Integration to mainline is serialized – one set of changes is integrated and verified at a time.
• Developers are notified when there are changes in mainline, and are expected to integrate them locally soon after.
• If the integration fails the rest of the team is not affected, mainline is not broken, no-one else need know.
• Developers can’t start working on the next task until the integration is completed because they sit at the integration machine, not their workstation. Developers can supervise the integration, fixing small problems as they arise.

I think those points have been lost in the way I see most teams set up their Continuous Integration build server. Basically – too much shaming, and not enough collaboration!

What we at Praqma have done with several of our customers is use a technique we’re calling ‘pre-tested integration’. I think it’s closer to the original CI process. You might have heard it called ‘validated merges’ or ‘pre-verified commits’.

When a developer has local changes that are ready to be integrated to the mainline they first push them to a remote branch named ‘ready/xxx’ (replace xxx with whatever you like, a Jira task number, a feature name, random string…) The build server is triggered whenever it detects a new remote branch that obeys this naming convention. It puts them in a queue and works on integrating them one at a time.

The build server takes a copy of the latest mainline from the central version control repository, and updates it with the changes from the ready-branch. For this to succeed any merges must be straightforward – i.e. possible for it to be done automatically. The build server then runs an automated build and test. If everything succeeds the server pushes the integrated changes up to the central code repository. The build server also updates a webpage or slack channel, notifying the team that new code is available in mainline. The ready-branch is now integrated, finished with, and can be deleted.

On the other hand, if the merge or the build fails, the developer has more work to do – their work wasn’t ready to integrate after all. The build server notifies them so they can go back and fix the problem and submit a new ready-branch. Crucially, the rest of the team is completely unaffected. Someone else is free to integrate their changes instead.

This pre-tested integration process is very similar to the CI process without a build server described earlier. We’ve added a little automation, we’re making use of some features of modern version control systems, but on the whole I think it’s closer to the old manual process than the usual way CI servers are set up.

• We ensure integration is serialized through build server configuration rather than a physical token.
• The build server notifies other developers when new code is successfully integrated – I think a post in a slack channel works much like talking in a loud voice in the team space.
• If an integration fails the shared mainline is not broken, only the developer who submitted the change is affected.

The other two points do differ though:

• Nothing forces developers to wait for the integration step before starting on a new task, but it is usually more convenient to do so. You wait for the build server chat room message, then pull your integrated change from the remote master before continuing work.
• Developers can’t fix problems in the integration step while it is ongoing.

During the integration step the build server is busy, but your developer machine is not. You could be tempted to move on to your next task before the previous one is finished and integrated. If the integration fails you’ll have to context-switch back, which may take more time than the idling you avoided.

There are upsides to having an integration process that actually can’t be supervised though. You get into a virtuous cycle where you are more successful if you submit small changes that can be integrated automatically, and you are more successful when you integrate more often. In short, you have put some automation in place that will draw you in the right direction. Smaller, more frequent commits, is exactly the behaviour we’d like to encourage.

If you’d like to find out more please take a look at Praqma’s pretested integration plugin for Jenkins. It’s one way to implement the ideas described in this post. It’s a free and open-source project, and we’re currently working on something of an overhaul. This development is funded through our CoDe Alliance, “where ambitious customers meet around continuous delivery”.

The latest version has support for Jenkins pipeline. This will bring the plugin up to date with modern usage – many people use Pipeline and freestyle jobs these days. The new version will also allow for much more flexibility in job design, with matrix and job combinations. It’s exactly the kind of tool I need for the team I’m working with right now.

Fundamentally though, succeeding with Continuous Integration is not really about tools, it’s about collaboration. You should choose tools that let developers feel safe, tools which encourage them to synchronize their work often, integrating small pieces continually in their work together.