Ruth Malan re-stated Conway’s law like this: “If the architecture of the system and the architecture of the organization are at odds, the architecture of the organization wins”. As a technical leader, that really caught my attention. The sphere of things I should be able to influence in order to do my job well just grew!
I read that quote in the book “Team Topologies” by Matthew Skelton and Manuel Pais. I think it’s a really interesting book and introduces several ideas that were new to me at least. They deserve a wide audience within the software industry. In this article I’ll give you some highlights, and hopefully tempt you to read the book for yourself.
This first idea perhaps isn’t so novel, but it is the basis for the rest, so worth mentioning first. You should use the team as the fundamental building block for your organization. Rather than thinking about hierarchies or departments that get re-organized from time to time, think about building close-knit, long-lived teams. Adopt a team structure that is aligned to the business and adapts and evolves over time.
Conway’s law
When designing a team structure for your organization you should take into account Conway’s law. Your team structure and your software architecture are strongly linked. You should design team responsibilities and communication pathways. Skelton and Pais suggest there are four fundamental team topologies, ie kinds of team. The main one is ‘business stream-aligned’, the other three have supporting roles. The way they describe a stream-aligned team seems familiar to me. It’s like a DevOps team or a feature team. I think the name is better though. It’s a team that’s aligned to a single, valuable stream of work. They do everything needed to deliver valuable software to customers, gather feedback from how it’s used, and improve the way they work.
The other three kinds of team exist to reduce the burden on those stream-aligned teams. Skelton and Pais introduce the idea that good organization design requires restricting the amount of ‘cognitive load’ each team is expected to handle. Software systems can become large and complex and can require an overwhelming amount of detailed knowledge to work on them. This really resonated with me. I’ve worked on teams where the amount of software we had to look after was just too much to cope with. Important work was delayed or missed because we were overburdened.
The other three types of team are:
Platform
Complicated Subsystem
Enabling
There is a fuller description of each in the book of course, but to summarize – platform teams provide a self-service tool, API or software service. Enabling teams support other teams with specialized knowledge in particular techniques and business domains. Complicated subsystem teams look after a particular component of the software that needs specialized knowledge, like an optimizer or a machine learning algorithm.
Following from the idea of Conway’s law in particular, is the idea that you should have only three ‘interaction modes’ between teams. Restrict the communication pathways to get the architecture you want, and to avoid unnecessary team cognitive load. Skelton and Pais suggest teams should work together in one of three ways:
Collaborate
Facilitate
Provide X-as-a-Service
Collaboration means two teams work closely together, have a common goal, and only need to work this closely for a limited time. (Otherwise the teams would merge!) Facilitation is about one team clearing impediments that are holding back the other team. X-as-a-service is a much looser collaboration, where one team provides something like a software library or api to another team. The way teams interact with one another will evolve over time, and consequently your organization will also evolve.
I thought it was a good sign that I could imagine how my own work would fit into an organization designed this way. I think I would fit well into an Enabling team that would support stream-aligned teams through facilitation. We would work with several teams over time. My particular role is to facilitate a team to clear impediments around technical debt and code quality, and learn skills like Test-Driven Development.
Team Topologies really does make organizational design feel like you’re doing architecture. Skelton and Pais have pretty diagrams throughout their book with colours and patterns and lines and boxes describing various organizational designs. It’s all very attractive to a software developer like me. I think the intended audience is managers though. People who are designing organizations today. I really hope some of them read this book and are inspired to involve technical people in important organizational decisions.
A new card game to design Continuous Delivery pipelines
Note: This article was originally published on Praqma’s blog
What testing steps should you include in your Continuous Delivery pipeline? Don’t just string together existing manual processes – use simple, collaborative tools to design something better!
Creating a Continuous Delivery (CD) pipeline is a key development in an organization’s transformation to DevOps. A CD Pipeline covers all the activities needed to transform a code change made by a developer into updated software bringing value to users. Steps in the pipeline include building a new version of the software as well as testing and deploying it. Exactly what kinds of build, test and deployment steps will depend on many factors and there is no ‘one perfect pipeline’ which will suit all situations.
I created the card game ‘Pipeline’ as a quick and fun way to explore alternatives for a CD pipeline without actually building anything. The goal of the game is to design a pipeline for a given scenario and optimize the deployment lead time. You work in a small group and get to discuss what steps are needed and which order you want to do them in. You will run into design tradeoff decisions and may discover people have different risk tolerances. If you play the game a second time with a different scenario, or compare notes with another group, you can learn more about how different scenarios drive different decisions.
Playing the game should help you to build a real pipeline for the real software system you are working on. Building a CD pipeline needs specialist knowledge of particular tools and could be many weeks of work. Playing this game should help you to avoid some costly misunderstandings. It’s also a fun way to engage a small group for an hour or two while you think through the issues you’re facing.
Scenario cards
To play the game each group begins with a sketch of a guiding scenario. For example, imagine you work for a startup aiming to displace an established business, like a social media website or an online bank. The reason for the scenario is that it makes the discussions more concrete and productive. You can discuss some specific risks you need to mitigate through testing. With that in mind it’s easier to reason about how long to spend on each testing step, and also assess the potential impact of excluding a step entirely.
Knowing the deployment lead time of the competitor in the scenario helps guide the group to discussions about tradeoffs and where quality comes from. Can we leave out this slow testing step? Can we mitigate that risk another way?
Pipeline step cards
The ‘pipeline step’ cards form the majority of the cards in the deck. These are the ones you lay out on a table to define the structure of your pipeline. You can choose which steps are important and which can be optional or omitted entirely. You can put several steps in parallel to reduce overall lead time. You can choose to perform a step twice, both before and after building the deployable component for example, or to do some steps before the beginning of the pipeline. The deck includes a few blank cards so that you can invent completely new steps. Just put a new sticky note on one with a text explaining what the new step involves.
Game rule cards
You have a lot of freedom about how to design your pipeline, but in reality there are a few rules about how you must order the steps. For example, you can’t deploy to production before you’ve built a release candidate. The “Game Rule” cards help you to keep track of these rules. One of the cards reminds you to add sticky notes to each step in the pipeline with an expected execution time. You should estimate this – either how long you think it will take, or a timebox for how long it is allowed to take. If it’s a manual step this estimate should include the time spent waiting for the person to be available.
Review
It might take a group of 4-6 people an hour or so to design a pipeline for their scenario and estimate the overall deployment lead time. When this is done (or the time for this part of the exercise is used up!) you should bring out the ‘review’ card. Each group can discuss what the implications of having this pipeline would be for their startup business. How does your lead time compare with your competitor’s? You might discuss how deployment lead time relates to time to market, or whether you would be able to do Lean Product Development, or A/B testing.
Some of the fictitious competitors have rather short deployment lead times. That’s quite deliberate on my part. I’d like people to think about how the competitor is achieving that. What must their pipeline look like? What testing must they be doing differently? You may think they are taking too many risks, and they will be easy to out-compete. Or perhaps they have found a better way, and you’d better imitate them.
Some of the fictitious competitors have much longer deployment lead times. I’d like people to think about whether that is really necessary. Perhaps they have included heavyweight change processes that take time without providing any benefits.
In any case, it’s interesting to think about, and might help you make more informed choices when you come to design a pipeline for your production system. Playing this game could alert you to some of the real life pitfalls that often arise.
Notes for a facilitator
If you want to play this game with your team I hope it will be straightforward for you to organize. The instructions are on the box. Allocate something between 1-2 hours for the exercise, split people into groups of about 3-6 people, with one deck of cards per group. You will also need a block of small sticky notes and pens for each group.
Your job as facilitator is to make sure everyone understands the point of the exercise, follows the pipeline game rules, and correctly calculates their deployment lead time. You should keep time and answer questions as they come up.
One thing a group might do is to decide on a microservices architecture, or front and back ends that are deployed separately. They might want to design a different pipeline for each service or each deployable component. That’s harder to do with only one deck of cards. For the purposes of this exercise, ask them to pick only one pipeline to design to start with. If they complete it quickly, you could give them another deck of cards to design another one later on.
As facilitator you may want to be quite active in the ‘review’ part of the game. Ask leading questions about the pipeline design to get people to explain their reasoning. If there are several groups you’ll want to have them look at one another’s work. This will help people to understand that different scenarios might call for different pipeline design choices. Everyone should come away with a better understanding of how to design a CD Pipeline and the hard choices that sometimes have to be made. Remind them that what they are learning in this game could help you all to avoid costly mistakes when designing a CD pipeline for real.
Throughout, try to keep these goals in mind:
People should have fun
Each group should come up with a feasible Continuous Delivery pipeline for their scenario, and they should be able to explain why they designed it that way
People understand what deployment lead time is, and that it could be significant when it comes to business competitiveness
Acknowledgements and Thanks
The first idea for this game came from a fun exercise at a workshop at the European Testing Conference in February 2018, led by Lisa Crispin and Abby Bangser. They divided us into groups, then handed each a pile of cards showing names of steps you could include in a pipeline. For example, “automated test: unit test”, “manual test: exploratory test”, or “deploy step: deploy to performance testing environment”. We had to collaborate to lay out the cards on a table in the form of a pipeline, and estimate the overall lead time.
I really enjoyed the exercise. There is something about having cards to move about and wave at people that encourages discussion and group engagement. Some cards provoked lively controversy – I remember discussing whether to include a manual code review step before merging to master, with someone who was quite convinced it was essential.
Iterative improvements to the exercise
I liked the exercise so much I decided to adapt it for a workshop I lead about Testing in Continuous Delivery. As I did this exercise with different groups I tweaked it a little each time. I also did it once with Llewellyn Falco and incorporated some of his suggestions too.
As the game gradually grew more sophisticated; I realized it would be worth getting some custom cards printed, so I started working with Phable on a layout and design. My colleagues at Praqma did some play-testing of an early version, and contributed some useful ideas. I’d like to thank everyone who played earlier versions of this game with me along the way. It’s a better game because of your feedback!
The second version of the cards came about six months after the first print run, following many successful CD workshops. Some of the ‘scenario’ cards have a lead time measured in weeks, and I wanted to make it possible to model those pipelines too. I added a few cards with more traditional release activities like a ‘Go/No Go meeting’ and ‘Decide Release Version number’. I also took on board some feedback from Steve Smith that ‘Constraint’ cards were poorly named – it reminds too much of ‘Theory of Constraints’, which has nothing to do with this game. I now call them ‘Game Rule’ cards, which I hope you’ll agree is more accurate.
I’d like to thank everyone who played earlier versions of this game with me along the way. It’s a better game because of your feedback!
In the Accelerate book, researchers explain several metrics which they have shown will measure the performance of a DevOps organization, and crucially, drive performance of the organization as a whole. I will explain why this is important, using an analogy with your risk of a heart attack.
In 2018 Nicole Forsgren, Jez Humble and Gene Kim released Accelerate: The Science of Lean Software and DevOps to detail their research into DevOps. They identified a causal link between DevOps organizations which score well on a number of metrics and success in the marketplace. Good scores on those metrics are in turn driven by successful implementation of DevOps practices. The book explains which metrics their research has found to be particularly significant.
If you’ve been following the DevOps movement you won’t be surprised to learn that these practices include Continuous Integration and Deployment Automation. As the research proceeds they continue to refine our understanding of which practices are most significant and which metrics are most useful to measure. I think it is really encouraging that researchers are applying proper science to this problem. This relationship they’ve discovered, between DevOps practices, metrics and organizational performance, is really useful for helping leaders in all kinds of organizations to make more informed decisions about how to do their work.
Avoiding a heart attack
I like to compare it with the relationship between your risk of a heart attack, your cholesterol levels, and your lifestyle. A low risk of heart attack is what you’re trying to optimize for, just as an organization might try to optimize for creating profit, increasing shareholder value, or reducing suffering in the world. Each organization will have some kind of ultimate aim, but as an employee, or even a leader, it is probably not easy to influence directly.
Measuring your cholesterol levels is a way to assess your risk of a heart attack. In much the same way, measuring the DevOps metrics detailed in Accelerate is a way to assess your organization’s chances of successfully meeting its goals. The lifestyle choices you make will, of course, affect your cholesterol levels, and consequently your heart attack risk. Similarly, the practices you use for development and operations will affect the values you score on Accelerate’s DevOps metrics, and hence your chances of achieving your goals. Crucially, the feedback loop is much faster for the metric than for the ultimate goal.
Lifestyle matters
I watched a documentary recently where they had four couples each take up a new diet for one month. Before and after they measured all sorts of things: weight, fat percentage, and a host of biomarkers, including cholesterol. One of the couples changed to quite an extreme Low-Carb diet, where they ate lots of meat and dairy, no fruit, and only a restricted selection of vegetables. After only one month it was noticeable how much higher their cholesterol levels had become. They were still quite young, so their risk of heart attack was still relatively low, but you could see if they persisted with this diet for the long term, the risk was going to become significant.
I think you can use the Accelerate DevOps metrics in a similar way. You can measure them relatively easily for your organization, and get an idea of where you stand. If you have poor values it doesn’t necessarily mean you are in imminent danger of company failure, but it doesn’t look good for the long term. If you make changes to the way you build software, for example by introducing Continuous Integration or investing in deployment automation, you should be able to see changes in your metrics relatively quickly. Hopefully this will encourage you to press on with more productive practices and become more likely to achieve your organization’s aims.
Posted by Emily Bache on 2019-09-11 at 08:16 under Opinion. Tags: DevOps, metrics Comments Off on Winning with DevOps and Reducing Your Risk of a Heart Attack.
