Coding Is Like Cooking

Posts tagged ‘continuous delivery’

Pre-tested Integration: Back to the basis of CI

This post was originally published on Praqma’s blog

Continuous Integration is now synonymous with having a server set up to build and test any change submitted to a central repository. But this isn’t the only way, or even how CI used to work. What did we do before DVCSs and Jenkins?

What is the connection between cricket and pre-tested integration? Pre-tested integration is about the way you set up your Continuous Integration server to metaphorically catch the balls you drop and the tests that fail, just like in cricket.

When a team first sets up a build server, this is the typical way to do it – you have it build any change in the mainline branch, and notify you if it fails. Developers are supposed to run the build and test before they push the code; the build server is supposed to be a kind of back-stop, catching any balls you drop. Any developer who manages to break the build is singled out by having their name displayed on the ‘information radiator’ screen in the corner. Just a little ridicule, some good-natured teasing, enough so that they are shamed into remembering to run the tests next time.

The thing is, it’s quite easy to make a mistake that causes the build to fail. People can get upset with you for breaking the build, and it can be embarrassing to have your name in lights. You can get into a bad cycle of stress, making mistakes more likely, causing you to break the build more often, making you more stressed…

Even a short length of time with a broken master can disrupt anyone else who is trying to integrate their changes, and discourage them from doing so. You can get into a bad cycle of people mistrusting the master and hoarding their changes. This causes bigger commits, more likely failed builds, and less trust… if it gets really bad you can find yourself spending several miserable days resolving three-way merges. Believe me, you don’t want to go there!

I think we can learn something from how they used to do Continuous Integration in the days before CI servers became widespread. I think they had a more humane process that was less stressful, and led to more virtuous cycles than downward spirals.

The whole point of Continuous Integration is to ensure that all the developers are working on essentially the same version of the source code. If you compared my working copy with those of my colleagues you should not be able to see any more differences than have accumulated through work either I or they have done today. Ideally, only changes we’ve made in the last couple of hours. The point of this is to make integrating our work a trivial task that takes seconds and is hard to mess up.

If you’re interested, you can read Jame Shore’s description of CI without a build server, but I will summarize it below:

When a developer was ready to integrate their changes they would physically walk to a designated ‘integration machine’, load their changes there, and perform a full build and test. No-one else was allowed to begin their integration until they were done. If the integration build passed they would simply say in a loud voice to the other developers to pull their changes from that machine. If there was a problem with the integration, and they couldn’t fix it quickly, they would revert their changes on the integration machine, and return to their workstation to fix the problem.

It’s a process that clearly only works if everyone is sitting in the same room. On the other hand, it works fine even when there are no atomic commits, no build servers, and no automated merges. Let me highlight some aspects of this process:

Integration to mainline is serialized – one set of changes is integrated and verified at a time.
Developers are notified when there are changes in mainline, and are expected to integrate them locally soon after.
If the integration fails the rest of the team is not affected, mainline is not broken, no-one else need know.
Developers can’t start working on the next task until the integration is completed because they sit at the integration machine, not their workstation. Developers can supervise the integration, fixing small problems as they arise.

I think those points have been lost in the way I see most teams set up their Continuous Integration build server. Basically – too much shaming, and not enough collaboration!

What we at Praqma have done with several of our customers is use a technique we’re calling ‘pre-tested integration’. I think it’s closer to the original CI process. You might have heard it called ‘validated merges’ or ‘pre-verified commits’.

When a developer has local changes that are ready to be integrated to the mainline they first push them to a remote branch named ‘ready/xxx’ (replace xxx with whatever you like, a Jira task number, a feature name, random string…) The build server is triggered whenever it detects a new remote branch that obeys this naming convention. It puts them in a queue and works on integrating them one at a time.

The build server takes a copy of the latest mainline from the central version control repository, and updates it with the changes from the ready-branch. For this to succeed any merges must be straightforward – i.e. possible for it to be done automatically. The build server then runs an automated build and test. If everything succeeds the server pushes the integrated changes up to the central code repository. The build server also updates a webpage or slack channel, notifying the team that new code is available in mainline. The ready-branch is now integrated, finished with, and can be deleted.

On the other hand, if the merge or the build fails, the developer has more work to do – their work wasn’t ready to integrate after all. The build server notifies them so they can go back and fix the problem and submit a new ready-branch. Crucially, the rest of the team is completely unaffected. Someone else is free to integrate their changes instead.

This pre-tested integration process is very similar to the CI process without a build server described earlier. We’ve added a little automation, we’re making use of some features of modern version control systems, but on the whole I think it’s closer to the old manual process than the usual way CI servers are set up.

We ensure integration is serialized through build server configuration rather than a physical token.
The build server notifies other developers when new code is successfully integrated – I think a post in a slack channel works much like talking in a loud voice in the team space.
If an integration fails the shared mainline is not broken, only the developer who submitted the change is affected.

The other two points do differ though:

Nothing forces developers to wait for the integration step before starting on a new task, but it is usually more convenient to do so. You wait for the build server chat room message, then pull your integrated change from the remote master before continuing work.
Developers can’t fix problems in the integration step while it is ongoing.

During the integration step the build server is busy, but your developer machine is not. You could be tempted to move on to your next task before the previous one is finished and integrated. If the integration fails you’ll have to context-switch back, which may take more time than the idling you avoided.

There are upsides to having an integration process that actually can’t be supervised though. You get into a virtuous cycle where you are more successful if you submit small changes that can be integrated automatically, and you are more successful when you integrate more often. In short, you have put some automation in place that will draw you in the right direction. Smaller, more frequent commits, is exactly the behaviour we’d like to encourage.

If you’d like to find out more please take a look at Praqma’s pretested integration plugin for Jenkins. It’s one way to implement the ideas described in this post. It’s a free and open-source project, and we’re currently working on something of an overhaul. This development is funded through our CoDe Alliance, “where ambitious customers meet around continuous delivery”.

The latest version has support for Jenkins pipeline. This will bring the plugin up to date with modern usage – many people use Pipeline and freestyle jobs these days. The new version will also allow for much more flexibility in job design, with matrix and job combinations. It’s exactly the kind of tool I need for the team I’m working with right now.

Fundamentally though, succeeding with Continuous Integration is not really about tools, it’s about collaboration. You should choose tools that let developers feel safe, tools which encourage them to synchronize their work often, integrating small pieces continually in their work together.

Posted by Emily Bache on 2018-04-08 at 20:24 under Coding Skills.
Tags: continuous delivery, continuous integration
Comments Off on Pre-tested Integration: Back to the basis of CI.

From Monolith to Microservices

Please note: this article was originally published on Pagero’s site.

At Pagero we are very proud of the technical architecture of our flagship product, Pagero Online. We’re successfully handling more document transactions than ever, as we see an ever increasing demand for e-document services. In this article I’d like to tell you a little about the journey we’ve taken, from humble beginnings almost ten years ago, to the present day and beyond. I’ll be talking a little about the technology stack we’ve chosen, including the business and technical reasoning behind our choices. If you’ve ever worked on a high availability, cloud based platform handling millions of events, or aspire to do so, you could be interested in our story.

This spring I was at the Craft conference in Budapest, which I thoroughly recommend by the way. There was a full program, with lot of great sessions, and interesting speakers. I did notice, browsing the program beforehand, that there were a lot of talks about Microservices, and Docker. Everyone seemed to have an opinion on the best deployment options, how to manage distributed data, building, testing, logging… This is clearly the hip and trendy way to build systems these days. I found this quite gratifying, since at Pagero we’ve been using a Microservices architecture for some time now, and have been using Docker in production since early 2014. It’s become our everyday life, not some hyped trend that we just heard about. Our reasons for going with Docker and Microservices are firmly rooted in the needs of our business.

Let me explain. Pagero Online is a cloud-based platform for exchange of electronic documents between businesses, for example invoices and orders. The point is, our customers can send their documents to us in whatever format their internal system produces, and we will deliver it in the format the recipient finds easiest to process in their internal systems. It’s clearly a valuable service, since we have an impressive year on year growth in document transactions.

The growth illustrates the challenge we’ve been meeting successfully for several years now – to scale our cloud system to handle ever increasing traffic. It’s of course a great problem to have, and we in the R&D department have worked hard to keep everything running smoothly throughout. The architecture we had when we started, is not the architecture we have now.

IN THE BEGINNING

Back in 2007, Enterprise Java Beans were the thing to do, and we felt confident we were building a future-proof, scalable system, using a JBoss container talking to a PostgreSQL database. Moore’s law meant that we could initially scale just by buying a bigger machine now and then. As time went by, we needed more, and started using the clustering capabilities built into the J2EE platform – i.e. several instances of the same code, receiving requests via a load balancer. At some point in about 2012 we realized this approach could no longer handle the increase in traffic that we were experiencing. We could no longer just add new instances of the same code, the slowdown from the communication overhead between them would be greater than the speedup from the increased CPU power. We needed to give more CPU power to just a few parts of the code that were doing the most intensive processing, without also hitting the communication bottlenecks.

ENTER MICROSERVICES AND DOCKER

Everything was pointing to a need to break apart our monolith into more manageable pieces. Microservices and Docker seemed the perfect match to our problems, so we spent the next year or so building the infrastructure needed. In February 2014 we deployed our monolith, packaged in a Docker container, together with some essential services for monitoring, service discovery, and message passing, (with protobuf over Rabbit MQ). Over the following months, the whole of our R&D department completed a course in the Scala programming language, and we built and deployed several more services for new features in the system. It worked! Since 2014 we have been able to quickly grow to about twenty services a year later, and sixty today.

We’ve realized the Microservices architecture enabled organizational streamlining too. Over the years our development team has grown from a handful of developers in the same room, to about 30 people split across three time zones. By breaking up the codebase, we can also divide up the development work more efficiently. We now have half a dozen ‘devops’ teams each responsible for a handful of Microservices. Both new and seasoned developers are more productive when working in these smaller codebases.

SCALING THE DATABASE

It was in around mid-2015, however, we started to see where the bottleneck had moved to, now the application code was performing better. Our trusty PostgreSQL database was handling a good many more gigabytes of data than ever before, and some transactions were getting a little slow. We concocted a plan to split it up too, just like we were doing with the monolith of code. We settled upon Cassandra and worked out how we were going to safely migrate all the document data out of Postgres, and into this distributed data store. The rest of the data will remain where it is, but just taking out the documents should free up a good deal of space, and release the main bottleneck. We of course need to do this without disrupting our service in any way, so one way to reduce the risk is to run the new Cassandra database in parallel with the existing Postgres, duplicating all the data. Only once we’ve done extensive testing, and we can see it’s working ok, will we remove the redundant copy

That’s kind of where we are now, we have just started this parallel running, and initial results are looking good.

THE BIG BREAK-UP

The next challenge is to continue to break apart our monolith of code, and create new services out of the pieces. Although all our new features are being built in Microservices, we still have the heart of the system in the former monolith. We’ve seen so many benefits to having Microservices, we’d like all our code to look like that. In some ways it’s a more daunting prospect than breaking up the database. This is a large quantity of tried and tested code that has been running in production for many years – breaking it up is not something you can do over a weekend!

We have to make this big change without any interruption to our production service, and we’ve thought carefully about what our strategy should be. One way to do a big risky change is to split it into a series of less-risky, smaller changes. The idea is that after every step in the break up, to run a battery of automated regression tests. The shorter the time the tests take to run, the smaller increments we can work with, and less risk of breaking anything. I’m personally pretty excited by this prospect. We’ve spent several years now building and improving our automated tests for Pagero Online, to the point where we feel pretty confident in taking on this challenge.

The other part of the strategy is to do the same as we have with the database migration. We’ll run both the old and new versions of the service in production for a while before we cut over to the new one. This should find any issues missed by the automated tests, without affecting any of our production traffic.

It’s going to be a real proof of how good our testing and deployment routines are. What kind of tests and deployment tools we’ve built, now that’s a topic for another blog post. If I’m lucky, I might even be telling you about the hip and trendy hot technologies that will be all over the agenda of the next Craft conference :-).

Posted by Emily Bache on 2016-11-16 at 07:48 under Experience Report.
Tags: continuous delivery, microservices
Comments Off on From Monolith to Microservices.

About me…

I am an independent consultant with Bache Consulting and chair of the Samman Technical Coaching Society. As a technical coach I work with software development organizations who want to get better at the technical practices needed to be agile, including Test-Driven Development, Refactoring and Incremental Design. I live in Gothenburg, Sweden, although I am originally from the UK. I’ve written two books: “Technical Agile Coaching with the Samman method” and “The Coding Dojo Handbook”. I teach for both O’Reilly and Pluralsight. I’m married to Geoff Bache, creator of TextTest. I am also on Mastodon as emilybache@sw-development-is.social.

Posts tagged ‘continuous delivery’

Pre-tested Integration: Back to the basis of CI

From Monolith to Microservices

IN THE BEGINNING

ENTER MICROSERVICES AND DOCKER

SCALING THE DATABASE

THE BIG BREAK-UP

Emily Bache

About me…

Posts by Category

Post Archives

Tags

Meta