On Cross-Functional Teams, DevOps and Spider Charts
##Part 1
From Toyota manufacturing we learn about the goal of 1:1 flow, that is, one input, and one output from a team such that we achieve the desired outcome in essentially one step. In order to acheive this, that step needs to be able to deliver the completed product without any other steps. One of the things we talk about often in Agile, and in DevOps, is the concept of a cross-functional team. One of the strengths of a cross-functional team is its ability to perform all the steps necessary to be able to produce their “product” without outside assistance.
In this post Adrian Cho argues that the best way to have a strong team that is able to respond to all their different challenges is to have a diversity of skills and experience, a mix of risk takers (dev) and the risk averse (ops). He argues this diversity is a key element in avoiding groupthink. I will go one step further and argue that it is precisely the ability to form teams in a manner that compensates for or provides the exact skill sets needed to achieve the target condition that makes them so successful.
Let’s look at an example of a cross-functional team. We will represent the team in a spider (or radar) chart. The closer a team member is to a discipline listed on the outside of the chart, the greater a degree of expertise they have in that particular discipline.
- Member 1 has solid knowledge of systems and development and some experience with network and QE, she would be the classic “DevOps engineer” that recruiters are always looking for, or what we’ve always just called, “a sysadmin who can code”.
- Member 2 has a deep knowledge of systems and networking but is not much of a coder and knows little about QE.
- Member 3 has strong QE and development skills but is lacking in systems and network.
- Member 4 would be your straight up network engineer, or what John Willis likes to call the next frontier for DevOps.
What is so special about this configuration? Each member of the team brings something to end to end delivery of the service. If you squint your eyes, you can almost see the outline of the diamond that covers all four functional areas (not that we are limited to 4 areas in any way). This team is also fully empowered– they can write the service to be delivered, test it, and have the ability to deploy and run it in production (assuming they follow DevOps practices like continuous deployment). This would actually be one representation of what many call “no-ops”. This does not mean we get rid of the operations capabilities of our organizations, it means that this responsibility is shared, and that there are members of the team who still have the skill sets needed to perform in ways traditionally expected of that role. No-ops really means we never throw anything “over the wall”. If our service is behaving very poorly and the TCP/IP stack needs to be tuned, we don’t call in outside help from another department. Instead member 4 steps up and uses their skill and experience to help deliver the service. Traditionally, we might have called that person operations, now, they are a valued skill set on a cross-functional team.
So does that mean that we’ve actually just invented a DevOps team? Of course not. Every one of our teams that operates a service is structured in a variant of this configuration based on the specific service they provide to the business. If all cross-functional teams are “DevOps” then everyone would be the “DevOps team”, which is silly, instead they are the search team, or the analytics team, etc. There could even be a team that consisted solely of DBAs. This is because the team composition is suited for the service they deliver. Our DBA team is practicing “infrastructure as code” principles so they can deliver a true “service” to the business, as opposed to being in service to the business. This means there are many ways to structure cross-functional teams in an organization that practices DevOps principles. We are simply trying to optimize the flow from creation of the service and into production and operation of that service. Does this mean these teams are “doing” DevOps? DevOps is a collection of principles and not practices. The fact that you can adhere to the principles of DevOps is what is important. You can’t “do” DevOps any more than you can “do” emergency preparedness walking down the street.
One thing that I’ve tried to do, that I hear others talk about doing, that unfortunately never really delivers the high performing cross-functional team, is “embeds”. This sounds like a great idea at first. We take a systems engineer from the ops team who is the “expert” on systems in production and drop her on a team of software engineers so that if they have any questions about operations, there will be someone there for them. She is still on the operations team, so that’s all fine in case there is a fire, but spends some time with the devs. Our poor SysEng is going to be constantly pulled off the team anytime there is a fire in operations. We’ve learned that these fires happen less frequently in organizations that truly practice DevOps from the DevOps Survey. This SysEng will also have to balance between the needs of two sides of the organization, probably to the detriment of both. With embeds there is no real opportunity to establish the true empathy required to bring a cross-functional team together.
This really became crystal clear to me listening to Jeff Patton give a keynote at Flowcon this past year. He was talking about really getting to know your customer and his example was that of Jane Goodall. It would have been more efficient for her to do “chimpanzees onsite” where the chimps were brought to the office for her to interview whenever she had a question about what it was like to be a chimp, rather than go spend time in the field with the chimps. You can imagine this would not have suited her research, and it does not suit our service teams either. The point is not to “get” ops or security involved “early in the process”, the point is to involve that input, natively, in the process itself, not as an add-on, bolt-on, or embed.
The nirvana of the cross-functional team, is that as members with different skill sets learn, grow, and collaborate with one another, each brings something to the table that helps the other members to go from the left to the right.
In this model, no one is throwing anything over the wall. At the very worst, in a highly regulated environment with poorly written compliance rules, you are handing things over the wall, and stopping to chat about tonight’s sportsball match or the newest restaurant each time you are standing and chatting at the wall. Jez Humble alluded to these regulated environments a few years ago.
Is this cross-functional model going to work for everyone? No. Each organization has to find the way to adhere to DevOps principles that works for them. But cross-functional teams have been promoted in Agile for years because they fit so well with the lean principles of eliminating waste in the system. If handoffs are the killers to flow, then the most well constructed true cross-functional teams will be best equiped to help an organization achieve the left to right flow optimization Gene Kim describes in his First Way, and achieve the really tight feedback loops described in the Third Way. Whether you succeed or fail at creating an environment that allows for all the advantages that DevOps can bring is up to you.
In part two, we’ll cover some evidence both in support of and contrary to this model of team composition.
Special thanks to Seth Katz and Alan Caudill for helping me sort much of this out and make it understandable. I hope I was able to make their efforts a reality.