urandom Mangot ideas

tech.mangot.com

On Cross-Functional Teams, DevOps and Spider Charts, Pt. 2

Part 2

As we saw in Part 1 of this essay, there are deliberate ways to organize our cross-functional teams for success in delivering our services in production. One of the most important ways is practicing infrastructure as code, which is critical to automation (the A in the CAMS, or Culture, Automation, Measurement and Sharing DevOps model), a core principle.

When working on that post, I was really excited to find 5 reasons everything you know about teamwork is wrong by Eric Barker that draws from Bronson and Merryman’s book Top Dog, while I was reading the Kates’ excellent Tech Leadership News.

There were two particular “rules” in that post that I felt were particularly relevant to our discussion of cross-functional teams.

Ninety percent of team success is determined before they start work

…60% of a team’s fate has been written before the team members even meet. Its destiny is decided by a combination of the team leader’s efficacy, whether the team’s goal is challenging yet attainable, and the ability level of the people recruited to the team. Thirty percent of a team’s fate is sealed with the initial launch of the team— how the teammates meet and, in those initial exchanges, how they split up the responsibilities and tasks before them. They need to agree on common codes of conduct and shared expectations. All told, 90% of a team’s fate has been decided before the team ever begins its real work.

I thought this was really fascinating given the emphasis Part 1 placed on the ability members of a cross-functional team have to help each other not only deliver, but to grow. It highlights the need to make smart choices about how the teams are formed like Chris Fry mentions in his description of the Twitter Engineering Culture. Does this mean that our cross-functional model doesn’t work? No. It just means that team composition has to take into account other elements (e.g. leadership, experience) than the simple axes we’d outlined in the description of our model. Just as Roy Rapoport said in his Flowcon presentation, “You can’t put smart people in a dumb organization to make the organization smarter”, the choices we make about how to compose our teams are critical to its success. Without the cross-functional aspect however, they are probably without any hope of having a successful deployment.

Defining roles may be the most important thing a team does

Clarifying who is going to do what— identifying distinct roles— is one of the most proven ways to increase the quality of teamwork. The egalitarian notion that team members should be equal in status and interchangeable in their roles is erroneous. Teams work best when participants know their roles, but not every role needs to be equal. Dr. Eduardo Salas, at the University of Central Florida, is one of the most widely cited scholars studying team efficiency. He has devoted his life to understanding the vast sea of team-building and team-training processes— analyzing teams used in the military, law enforcement, NASA, and numerous corporate settings. The only strategies that consistently deliver results are those that focus on role clarification: who’s going to do what when the pressure gets intense.

This quote really threw me for a loop. I had been making an argument that each team member should share responsibility for a variety of roles (dev, system, qe, etc.) on a cross-functional team and here it said “The egalitarian notion that team members should be equal in status and interchangeable in their roles is erroneous”. When reasoning about the possibilities of why this may or not be true, I looked at the teams they studied (“military, law-enforcement, NASA, and numerous corporate settings”). For the first group, it seemed to make sense. If I’m in the military and the guy next to me doesn’t know how to use the grenade launcher as well as I do, I sure want to be the guy using it in a firefight. But, how does that fit with the “numerous corporate settings” team? Our service delivery team is definitely in a corporate setting. Does this mean that the sysadmin should only ever do systems administration, and the developer likewise?

Then I re-read it with the last sentence “who’s going to do what when the pressure gets intense”. That’s when it actually made sense. When we have a production outage and we identify a problem, the developer is probably not going to be the one responsible for troubleshooting overruns in the TCP queuing if there are sysadmins or network engineers who have more experience and the goal is to get the site back up as fast as possible. That doesn’t mean that under normal circumstances the developer cannot come up to speed on network tuning (or that they must, they can contribute in their own way), just that when the “pressure gets intense”, the team needs to play to its strengths.

To use a sports analogy, in football, the cornerback genrally runs with the wide receiver, and the safety generally is his backup down the field. When the ball is snapped (our production outage situation), each plays the role they are assigned. However, that after each play is over they walk over and discuss with each other what they thought about the previous play and what they saw. Maybe after enough time playing together, they understand better and better how those roles are complementary to one another. That kind of open communication is essential for a team to function at its highest levels when the “pressure gets intense”. That is exactly how the team (be it sport team or service delivery team) gets better.

A cross-functional team is still necessary to achieve the 1:1 flow we’ve learned about from Toyota manufacturing. Agile emphasizes the advantage of cross-functional teams and we know that for a team to be effective in delivering a service in an infrastructure as code environment, they need to have all the skills required to deliver that service. We’ve also learned that the composition of the team goes beyond simply plugging in the correct functional area but there are other components that don’t just help, they actually determine the success of the team from the outset. In a high pressure situation, the cross-functional team is still effective, but team members are more likely to fall back to their core strengths in order to get past the stressor. In our corporate environments, cross-functional teams can be one of our most effective tools in helping us realize the benefits of practicing DevOps principles.

Special thanks to Seth Katz and Alan Caudill for helping me sort much of this out and make it understandable. I hope I was able to make their efforts a reality.

On Cross-Functional Teams, DevOps and Spider Charts

Part 1

From Toyota manufacturing we learn about the goal of 1:1 flow, that is, one input, and one output from a team such that we achieve the desired outcome in essentially one step. In order to acheive this, that step needs to be able to deliver the completed product without any other steps. One of the things we talk about often in Agile, and in DevOps, is the concept of a cross-functional team. One of the strengths of a cross-functional team is its ability to perform all the steps necessary to be able to produce their “product” without outside assistance.

In this post Adrian Cho argues that the best way to have a strong team that is able to respond to all their different challenges is to have a diversity of skills and experience, a mix of risk takers (dev) and the risk averse (ops). He argues this diversity is a key element in avoiding groupthink. I will go one step further and argue that it is precisely the ability to form teams in a manner that compensates for or provides the exact skill sets needed to achieve the target condition that makes them so successful.

Let’s look at an example of a cross-functional team. We will represent the team in a spider (or radar) chart. The closer a team member is to a discipline listed on the outside of the chart, the greater a degree of expertise they have in that particular discipline.

Spider chart

  • Member 1 has solid knowledge of systems and development and some experience with network and QE, she would be the classic “DevOps engineer” that recruiters are always looking for, or what we’ve always just called, “a sysadmin who can code”.
  • Member 2 has a deep knowledge of systems and networking but is not much of a coder and knows little about QE.
  • Member 3 has strong QE and development skills but is lacking in systems and network.
  • Member 4 would be your straight up network engineer, or what John Willis likes to call the next frontier for DevOps.

What is so special about this configuration? Each member of the team brings something to end to end delivery of the service. If you squint your eyes, you can almost see the outline of the diamond that covers all four functional areas (not that we are limited to 4 areas in any way). This team is also fully empowered– they can write the service to be delivered, test it, and have the ability to deploy and run it in production (assuming they follow DevOps practices like continuous deployment). This would actually be one representation of what many call “no-ops”. This does not mean we get rid of the operations capabilities of our organizations, it means that this responsibility is shared, and that there are members of the team who still have the skill sets needed to perform in ways traditionally expected of that role. No-ops really means we never throw anything “over the wall”. If our service is behaving very poorly and the TCP/IP stack needs to be tuned, we don’t call in outside help from another department. Instead member 4 steps up and uses their skill and experience to help deliver the service. Traditionally, we might have called that person operations, now, they are a valued skill set on a cross-functional team.

So does that mean that we’ve actually just invented a DevOps team? Of course not. Every one of our teams that operates a service is structured in a variant of this configuration based on the specific service they provide to the business. If all cross-functional teams are “DevOps” then everyone would be the “DevOps team”, which is silly, instead they are the search team, or the analytics team, etc. There could even be a team that consisted solely of DBAs. This is because the team composition is suited for the service they deliver. Our DBA team is practicing “infrastructure as code” principles so they can deliver a true “service” to the business, as opposed to being in service to the business. This means there are many ways to structure cross-functional teams in an organization that practices DevOps principles. We are simply trying to optimize the flow from creation of the service and into production and operation of that service. Does this mean these teams are “doing” DevOps? DevOps is a collection of principles and not practices. The fact that you can adhere to the principles of DevOps is what is important. You can’t “do” DevOps any more than you can “do” emergency preparedness walking down the street.

One thing that I’ve tried to do, that I hear others talk about doing, that unfortunately never really delivers the high performing cross-functional team, is “embeds”. This sounds like a great idea at first. We take a systems engineer from the ops team who is the “expert” on systems in production and drop her on a team of software engineers so that if they have any questions about operations, there will be someone there for them. She is still on the operations team, so that’s all fine in case there is a fire, but spends some time with the devs. Our poor SysEng is going to be constantly pulled off the team anytime there is a fire in operations. We’ve learned that these fires happen less frequently in organizations that truly practice DevOps from the DevOps Survey. This SysEng will also have to balance between the needs of two sides of the organization, probably to the detriment of both. With embeds there is no real opportunity to establish the true empathy required to bring a cross-functional team together.

This really became crystal clear to me listening to Jeff Patton give a keynote at Flowcon this past year. He was talking about really getting to know your customer and his example was that of Jane Goodall. It would have been more efficient for her to do “chimpanzees onsite” where the chimps were brought to the office for her to interview whenever she had a question about what it was like to be a chimp, rather than go spend time in the field with the chimps. You can imagine this would not have suited her research, and it does not suit our service teams either. The point is not to “get” ops or security involved “early in the process”, the point is to involve that input, natively, in the process itself, not as an add-on, bolt-on, or embed.

The nirvana of the cross-functional team, is that as members with different skill sets learn, grow, and collaborate with one another, each brings something to the table that helps the other members to go from the left to the right.

Cross-functional Progresson

In this model, no one is throwing anything over the wall. At the very worst, in a highly regulated environment with poorly written compliance rules, you are handing things over the wall, and stopping to chat about tonight’s sportsball match or the newest restaurant each time you are standing and chatting at the wall. Jez Humble alluded to these regulated environments a few years ago.

Is this cross-functional model going to work for everyone? No. Each organization has to find the way to adhere to DevOps principles that works for them. But cross-functional teams have been promoted in Agile for years because they fit so well with the lean principles of eliminating waste in the system. If handoffs are the killers to flow, then the most well constructed true cross-functional teams will be best equiped to help an organization achieve the left to right flow optimization Gene Kim describes in his First Way, and achieve the really tight feedback loops described in the Third Way. Whether you succeed or fail at creating an environment that allows for all the advantages that DevOps can bring is up to you.

In part two, we’ll cover some evidence both in support of and contrary to this model of team composition.

Special thanks to Seth Katz and Alan Caudill for helping me sort much of this out and make it understandable. I hope I was able to make their efforts a reality.

I’m Not a DevOps…Are You an Agile?

I was signing up for a new MeetUp group a few months ago, and as part of the process, I was supposed to answer a few questions. One of the questions was “Are you a DevOps?”. I was a little struck by this. I’m pretty sure I knew what the question was supposed to mean, but I also know that the question was nonsensical. Maybe a proper rephrasing would have been, “Are you an engineer who works in accordance with DevOps principles?”. Maybe that was just too long and loaded to ask as a question. I feel like we’re starting to lose control of the word “DevOps”. Is that just natural for our industry? Is it a “good thing” or a “bad thing”? Maybe this means that DevOps is gaining adoption, or maybe it means we’ve lost sight of why we’re doing this.

DevOps jobs

Taking a look at my inbox on any given day and it is filled with job opportunities. We are looking for “DevOps engineers”, or one of my personal favorites “Senior DevOps engineer” or “Lead DevOps engineer”. I feel like sometimes our industry will find some term and then it will get dragged through the mud for a few years until perhaps the proper usage will be found. Take the word “hacker”. When hacker first became popular it referred to someone who like to “hack” on code. Then the press got a hold of the word. After a while, a hacker was someone who tried to break into other’s computer systems. For years, in Hollywood movies, we learned about hackers and their nefarious telnet toner backdoor circuit electron overflows. I had a general counsel once explain to me the concept of “secondary meaning” in legal terms. This is when something that is generally known takes on “secondary meaning” as in the case of Apple. Everyone knew what an apple was when they were growing up, and they were delicious. Eventually, Apple the computer company became so big, that the word apple took on secondary meaning. I feel like this is what happened with the word hacker in our industry. Eventually it took on secondary meaning as a way of describing what folks on Bugtraq for years have been lamenting should have been called “cracker” (oh, the irony) instead. Now we have Hacker Dojos, Hacker Bees, IKEA hackers, and Hacker News. If you read news articles you will still see references to how hackers have infiltrated some government network (with their majicks!), but on the whole, our industry has either reclaimed the term, or allowed its dual usage based on context.

So then what about DevOps Engineers? Has Patrick Dubois, has our industry, already lost control of our own term? I’ve been going to DevOpsDays for years, I’ve been talking to people and giving talks about DevOps, I’ve been trying to help salesforce.com through a DevOps transformation. In none of those interactions have we ever talked about DevOps engineers. When I give my Introduction to DevOps talks, I usually tell the audience to substitute the word “collaboration” where they see the word DevOps and they will be much closer to an instant understanding of what we’re trying to accomplish with DevOps, even if they won’t understand the “why” quite as quickly.

If you’ve ever listened to Gene Kim talk about DevOps, he talks about making life better for thousands of IT professionals. DevOps is definitely about getting rid of the “throw it over the wall” mentality between development and operations, but it’s for the purpose of getting the business to focus on what is most important, being able to rapidly deliver value to the customer. In order to do that, Dev and Ops have to change too. Maybe those changes are a natural evolution. I remember the BOFH. I hope that as an industry, we’ve put that behind us already, we’ve evolved. The idea of a BOFH is antithecal to the DevOps movement. The BOFH said no to everyone, it’s no wonder people did not want to collaborate with him or her.

The BOFH was fine for the days when we used to think it was a good idea to outsource our IT departments, to cut costs. If your development wing is in the Phillipines, and your operations department is in India, and they are run by two completely different companies, it doesn’t matter all that much how well your devs and ops communicate, they don’t! You are already starting behind and will never catch up to companies practicing DevOps. DevOps is not about oursourcing, DevOps is about insourcing. The problem is, no matter how much the engineers in us try to escape it, and turn everything into an algorithm, it still all about people and how they communicate. The DevOps movement is a recognition of that fact. If you’ve ever read Crucial Conversations, a book about communication skills, one of the first concepts they teach you when you are going to have an imporant conversation is to Establish Mutual Purpose. That is DevOps in a nutshell. Development and Operations establishing mutual purpose. In DevOps, that purpose it to deliver the most amount of value to the business through streamlined processes, it’s about always seeking to increase flow. Once that purpose is recognized, you’re much more likely to have a successful conversation, and you’re much more likely to have a successful business.

So, I read with interest the recruiters who write me about the latest DevOps opportunity. I don’t remember a major called ‘recruiting’ in college. The closest I can think of would be sales or marketing. What are these recruiters trying to market or sell? Are they marketing a collaborative environment? Free from BOFHs? Where flow reigns supreme? If that is the case, isn’t every company looking for the exact same thing? Is the word “DevOps” redundant in this case? Is DevOps actually a descriptive differentiator? Should it just be “looking for systems administrator”, “looking for developer who likes to understand the entire architecture”? Or is it more than that? Sometimes I worry that “DevOps engineer”, actually means “sysadmin who understands writing code for automation”. But, after being involved in this industry for many years, that’s just what we’ve always called a “good sysadmin”. Is that what they are marketing? “Serious startup company seeks good systems adminstrator”. I guess it doesn’t have the same cache.

The problem, I suppose, is more than that. Where are these good sysadmins going to come from? Where are these systems thinking developers going to be taught? Traditionally developers might code and sysadmins might troubleshoot. But now, developers are responsible for their code in production (if you wrote it, you run it). Sysadmins write code. Where is the line? Where will the subject matter experts we’re accustomed to come from? I suspect that the industry will simply change. There will be people who naturally gravitate toward one or another aspect or specialization based on their interest or experience. It might be harder to find your traditional QE or Sysadmin. Maybe we’ll be looking for cloud engineers? I don’t want to get started on public vs. private clouds, etc.

So where does that leave us? Obviously, our industry is in transition. Everyone is looking for “DevOps Engineers”. Are you part of a movement that represents the fact that we can deliver more business value when people see delivery through lean principles where the empasis is on flow through the system, and short feedback loops, more than it is upon silos and politics? It’s people over process, a core tenent of the Agile Manifesto. Is that what these recruiters are asking me? Do I believe in people over process? Of course I do, that’s why I believe in Agile. I think the lessons that agile development process teaches us not only make us better developers, network admins, and sysadmins, I think they also make us better as people. The sprint retrospective may be about making the team better through a process of self improvement, but it’s also about a process of remembering to improve ourselves. Any strides that we make a person, as an individual, through better tooling, or better communication, make the team better, and benefit the business.

The sales and marketing folks are definitely having success using the DevOps term. When signing up for the O’Reilly Velocity conference I was asked “Do you primarily work in Web Operations, Web Performance or DevOps?”. Last week on Twitter, Puppetlabs asked “Lots of DevOps jobs out there — and more on the way. How do you become a DevOps engineer & get in on this trend?”. These are from two organizations that defintely “get” DevOps. Maybe the word has already acquired that “secondary meaning”. Maybe we’ll be calling all the jobs that are interesting “DevOps jobs”, until the whole industry is operating with that business model anyway. I just have to imagine that Toyota never advertised for “Kaizen Engineers” to work on the production line.

So when confronted with the question, “Are you a DevOps?” I answered the only way I knew how. Someone was asking me not whether I was qualified to fill a role, but whether I believed in a movement. Whether flow was of the utmost importance. Whether communication was more important than silos. “Of course not, are you an Agile?”

DevOps Recruiting http://www.build-doctor.com/wp-content/uploads/2011/07/devop.jpg