Home > Cloud Computing, Cloud Security, DevOps, Virtualization > Incomplete Thought: The DevOps Disconnect

Incomplete Thought: The DevOps Disconnect

DevOps — what it means and how it applies — is a fascinating topic that inspires all sorts of interesting reactions from people, polarized by their interpretation of what this term really means.

At CloudCamp Denver, adjacent to Gluecon, Aaron Peterson of OpsCode gave a lightning talk titled: “Operations as Code.”  I’ve seen this presentation on-line before, but listened intently as Aaron presented.  You can see John Willis’ version on Slideshare here.  Adrian Cole (@adrianfcole) of jClouds fame (and now Opscode) and I had an awesome hour-long discussion afterwards that was the genesis for this post.

“Operations as Code” (I’ve seen it described also as “Infrastructure as Code”) is really a fantastically sexy and intriguing phrase.  When boiled down, what I extract is that the DevOps “movement” is less about developers becoming operators, but rather the notion that developers can be part of the process whereby they help enable operations/operators to repeatably and with discipline, automate processes that are otherwise manual and prone to error.

[Ed: great feedback from Andrew Shafer: “DevOps isn’t so much about developers helping operations, it’s about operational concerns becoming more and more programmable, and operators becoming more and more comfortable and capable with that.  Further, John Allspaw (@allspaw) added some great commentary below – talking about DevOps really being about tools + culture + communication. Adam Jacobs from Opscode *really* banged out a great set of observations in the comments also. All good perspective.]

Automate, automate, automate.

While I find the message of DevOps totally agreeable, it’s the messaging that causes me concern, not because of the groups it includes, but those that it leaves out.  I find that the most provocative elements of the DevOps “manifesto” (sorry) are almost religious in nature.  That’s to be expected as most great ideas are.

In many presentations promoting DevOps, developers are shown to have evolved in practice and methodology, but operators (of all kinds) are described as being stuck in the dark ages. DevOps evangelists paint a picture that compares and contrasts the Agile-based, reusable componentized, source-controlled, team-based and integrated approach of “evolved” development environments with that of loosely-scripted, poorly-automated, inefficient, individually-contributed, undisciplined, non-source-controlled operations.

You can see how this might be just a tad off-putting to some people.

In Aaron’s presentation, the most interesting concept to me is the definition of “infrastructure.” Take the example to the right, wherein various “infrastructure” roles are described.  What should be evident is that to many — especially those in enterprise (virtualized or otherwise) or non-Cloud environments — is that these software-only components represent only a fraction of what makes up “infrastructure.”

The loadbalancer role, as an example makes total sense if you’re using HAproxy or Zeus ZXTM. What happens if it’s an F5 or Cisco appliance?

What about the routers, switches, firewalls, IDS/IPS, WAFs, SSL engines, storage, XML parsers, etc. that make up the underpinning of the typical datacenter?  The majority of these elements — as most of them exist today — do not present consistent interfaces for automation/integration. Most of them utilize proprietary/closed API’s for management that makes automation cumbersome if not impossible across a large environment.

Many will react to that statement by suggesting that this is why Cloud Computing is the great equalizer — that by abstracting the “complexity” of these components into a more “simplified” set of software resources versus hardware, it solves this problem and without the hardware-centric focus of infrastructure and the operations mess that revolves around it today, we’re free to focus on “building the business versus running the business.”

I’d agree.  The problem is that these are two different worlds.  The mass-market IaaS/PaaS providers who provide abstracted representations of infrastructure are still the corner-cases when compared to the majority of service providers who are entering the Cloud space specifically focused on serving the enterprise, and the enterprise — even those that are heavily virtualized — still very dependent upon hardware.

This is where the DevOps messaging miss comes — at least as it’s described today. DevOps is really targeted (today) toward the software-homogeneity of public, mass-market Cloud environments (such as Amazon Web Services) where infrastructure can be defined as abstract component, software-only roles, not the complex mish-mash of hardware-focused IT of the enterprise as it currently stands. This may be plainly obvious to some, but the messaging of DevOps is obscuring the message which is unfortunate.

DevOps is promoted today as a target operational end-state without explicitly defining that the requirements for success really do depend upon the level of abstraction in the environment; it’s really focused on public Cloud Computing.  In and of itself, that’s not a bad thing at all, but it’s a “marketing” miss when it comes to engaging with a huge audience who wants and needs to get the DevOps religion.

You can preach to the choir all day long, but that’s not going to move the needle.

My biggest gripe with the DevOps messaging is with the name itself. If you expect to truly automate “infrastructure as code,” we really should call it NetSecDevOps. Leaving the network and security teams — and the infrastructure they represent — out of the loop until they are either subsumed by software (won’t happen) or get religion (probable but a long-haul exercise) is counter-productive.

Take security, for example. By design, 95% of security technology/solutions are — by design — not easily automatable or are built to require human interaction given their mission and lack of intelligence/correlation with other tools.  How do you automate around that?  It’s really here that the statement I’ve made that “security doesn’t scale” is apropos. Believe me, I’m not making excuses for the security industry, nor am I suggesting this is how it ought to be, but it is how it currently exists.

Of course we’re seeing the next generation of datacenters in the enterprise become more abstract. With virtualization and cloud-like capabilities being delivered with automated provisioning, orchestration and governance by design for BOTH hardware and software and the vision of private/public cloud integration baked into enterprise architecture, we’re actually on a path where DevOps — at its core — makes total sense.

I only wish that (NetSec)DevOps evangelists — and companies such as Opscode — would  address this divide up-front and start to reach out to the enterprise world to help make DevOps a goal that these teams aspire to rather than something to rub their noses in.  Further, we need a way for the community to contribute things like Chef recipes that allow for flow-through role definition support for hardware-based solutions that do have exposed management interfaces (Ed: Adrian referred to these in a tweet as ‘device’ recipes)

/Hoff

Related articles by Zemanta

Enhanced by Zemanta
  1. May 31st, 2010 at 04:14 | #1

    Hey beaker,

    I fully support your idea that devops is a multi-disciplinary approach so indeed should include other fields such as security and network. In several companies they fall under operations group as well, not under the sysadmins group. Therefore I welcome posts such as yours to create an opening to other groups. Both in the puppet and the chef community people are looking to extend their recipes to non-server machines such as switches, firewalls and so on.

    Looks to me very similar to the dba's who slowly are getting integrated in 'Agile development'.

    Personally I think it would be even better to have business too in the name: making it busidevops. But somehow devops as a term catches on. But that's another story 🙂

  2. May 31st, 2010 at 05:10 | #2

    Let's hope this helps…

    I can see how the messaging of the devops thoughts would seem missing something, because the idea is not about automation. It's actually not about a bunch of things:

    It's not abstraction.

    It's not even "infrastructure as code".

    It's not any single tool.

    It's not about provisioning.

    It's not about deployment.

    It's not about a job description or position.

    It's also not about the cloud, except for the part where deployment and provisioning of infrastructure gets easier to understand for groups of people who historically wouldn't have touched that part of the business.

    It *is* about the collaborative and communicative culture and the tools and process that arise from that culture. Nothing more.

    But there's only so much that 140 characters can point out at a time, and what you're absorbing is snippets of a passionate group of people who are getting a glimpse of what happens when people actually talk to each other instead of framing the discussion as an "Us versus Them" position.

    How you're understanding that the ideas surrounding would leave *any* part of an organization out of the loop escapes me, actually. The idea(s) behind 'devops' is explicitly to include people who enable the business, and to capture what each group can bring to the table, not to lock anyone out.

    I don't care about the name, because frankly it could be ProdCareDesignBizNetSecDevEngOps as far as I'm concerned.

    One last point…this here:

    "In many presentations promoting DevOps, developers are shown to have evolved in practice and methodology, but operators (of all kinds) are described as being stuck in the dark ages."

    is not personally my position. My position is that both dev and ops has allowed contention, opposition, and stereotyped silo-ing to happen as an accepted situation, and that 'devops' or whatever you want to call it…is the adjustment of those cultures and attitudes.

    Successful engineering organizations bridge these gaps and work together. This means specifically not being fingerpointy, and not being exclusionary.

    But again, I have a feeling it'll be easier to talk about it over beer. You'll be at Velocity?

  3. May 31st, 2010 at 05:55 | #3

    John:

    Thanks for your well-thought out response.

    The thing that intrigues me is the part where you said:

    "How you’re understanding that the ideas surrounding would leave *any* part of an organization out of the loop escapes me, actually. The idea(s) behind ‘devops’ is explicitly to include people who enable the business, and to capture what each group can bring to the table, not to lock anyone out."

    ^^^ That's the whole point of my calling it a "disconnect." The post above builds what I think is a reasonable case of explaining it. I think there are definitely three camps who are preaching the DevOps mantra (or their version of it.)

    There are folks like you and Andrew (rational and communicative in nature,) those that are more "militant" or dogmatic in nature (us vs. them) and those who have something to sell.

    I'm not saying any of these groups is more or less right/wrong than the other, but it's not surprising to me that you can't grok why it's confusing it you're only viewing it from one perspective. I see it from all three…and I have my own opinions from the security perspective.

    Also, I have spent hours speaking with people about DevOps — the pros and cons…and the opinions are much more diverse than some may be comfortable with.

    I am very thankful that you took the time to explain YOUR position. It's a shame that it's not the prevailing experience.

    /Hoff

  4. May 31st, 2010 at 05:57 | #4

    I wouldn't assume that deeply committed devops orgs dont' include network device automation. I can't speak to the manual security processes, but certainly for things like network provisioning and basic policy the NASA Nebula project contemplates full layer 2 isolation orchestration using Chef.

    "There's a recipe for that" (esp if you are Joshua McKenty and just write it yourself)

  5. May 31st, 2010 at 06:05 | #5

    @James Watters

    Can you please expand upon what PHYSICAL "network devices" are provisioned to include "full layer 2 isolation" in your example above?

    If you say VLAN provisioning (in light of ALL the other examples I have) I am going to push you off another chair. 😉

  6. May 31st, 2010 at 07:18 | #6

    Chris,

    It was a 5-minute lightning talk on how our product "Opscode Chef" fits in the fabric of a new way to treat infrastructure. It was a targeted presentation for our niche audience. It was not meant to be a “Devops Manifesto”. Where I totally agree with a lot of your points about devops including security, network (and you forgot Storage), using our slides as the "Devops" manifesto example is waaaaay out of context. As for the load balancer example, it's just an example. We also didn't mention the entire network and storage infrastructure between those components. That doesn't mean that we don't believe they exist. In fact we have many examples were the load balancer is API based. The Chef product is completely API based and in fact we use many scenarios where the resources we address are API abstractions (e.g., EC2, EBS,DNS).

    Again, I agree that the Devops messaging needs to be fine-tuned in fact very similar to the NoSQL movement punted to “Not Only”. However, like NoSQl, I believe Devops” was a great stake in the ground to promote change. I think your voice is a much-needed one in this movement and you know I appreciate you being involved.

    One more point for the record, I do not believe “Operations as Code”/”Infrastructure as Code” is a zero sum gain. Nor do I agree that it is the single based definition of “Devops”. In my heart of hearts, I believe Devops is about a “change”. Doing things differently from the way the old legacy infrastructure and following some of the exciting things going on in the new “Internet” positioned enterprises.

    Hope that Helps

    John Willis

  7. May 31st, 2010 at 07:41 | #7

    John:

    I didn't call the lightning talk (or Opscode's presentation) a manifesto, I was referring to the overall messaging in general. Furthermore, it was the messaging, not necessarily the message. That's critical.

    Further, I'm not reacting to a SINGLE presentation or a SINGLE tweet, I'm reacting to the cumulative/aggregate of many conversations/tweets/presentations/discussions.

    If, after hours of homework, the disparate messaging of DevOps is unclear to me, think of what it's like to others…as for being a stake in the ground, I suppose that's true. My point/question is whether you want to put a stake in the sandbox or a much greater landscape.

    I'm giving you, companies like Opscode and the industry some feedback, that's all. I say I did a reasonable job of recognizing others' PoV's. I'm not pointing fingers, but so much of this is reacted to defensively or dismissively…there are 4-5 people who lead this pack and they do a good job of stating their cases, but often times without recognition of alternative points of view.

    That's OK, too…but recognize the impact this will have in the long term.

    /Hoff

    P.S. I did mention storage 😉

  8. May 31st, 2010 at 07:51 | #8

    @John Allspaw

    John,

    "It *is* about the collaborative and communicative culture and the tools and process that arise from that culture. Nothing more. "

    Totally agree with this as one of the best definitions of Devops. When I try to position Opscode Chef I am doing so as one of the really cool tools that is trying to help with these cultural process changes. Not as the "definition" of devops. I have also been guilty of creating slide decks that show some of the older non agile practices of operations. The point I am trying to make there is that sys-admins that hoard scripts and companies that don't embrace operations as code might not fit into a model of highly scalable operations.

    Thanks

    John W.

  9. neuroserve
    May 31st, 2010 at 08:15 | #9

    I like the term "system development" – that's what our team was called in one of my former lives. That team consisted of dbas, system engineers and various other types. And we lived on the same floor as our developers …

  10. May 31st, 2010 at 12:49 | #10

    I think it's important to separate out the two concepts at play here, that so

    often get interwined – especially when you're talking to people from Opscode,

    because we are so firmly entrenched in both. The first is the cultural and

    professional shift towards Webops culture becoming the dominant Operations

    culture, whcih people are calling DevOps. The second is the 'Infrastructure As

    Code' phenomen, which also comes out of Webops culture, which is about building

    and managing infrastructure programatically. While the two are often related,

    they aren't the same thing.

    I've always been uncomfortable with the word DevOps, for reasons similar to

    you. For me, it's a free-pass to wallow in the second-class citizen status

    many Systems Administrators have always felt. If I had a nickle for every

    person I've known in this industry who believed they were less skilled or

    valuable than a software developer, I would be a rich man. When I first

    started to see the word tossed around, one of the most common contexts was as a

    new job description.. and it drives me crazy. I'm proud to be a Systems

    Administrator, and I'm proud to have peers like John Allspaw and Theo

    Schlossnagle.

    The messaging problem you point out is a real one – you can either talk about

    what's happening culturally from a place of inclusion or a place of exclusion.

    The dominant cultural paradigm remains one of silo-ed specialization, and it's

    obvious from this post. As a Network and Security Administrator, you feel

    excluded by the focus on Systems Operations – because you see the disciplines

    are being separate. The shift here is that your specialization becomes

    subserviant to the overall business objectives and efficiency – rather than

    remaining separate from your peers, both organizationally and culturally, you

    become integrated into the whole, and valued for your expertise (rather than

    your organizational kingdom-building prowess).

    The inclusive way to talk about this is to say that all of us play an integral

    role in the success of our busineses. By coming together, communicating

    clearly, and helping each other understand exactly what it is that we do, we

    build more effective, efficient, and quite frankly *funner* teams and

    companies. We stop focusing on the things that make us different, and we start

    focusing on the way we compliment each other, and the ways we can help make

    each others lives easier. You can spot this camp because they spend time

    talking about how great their lives are, and how awesome what they build is.

    The exclusive way to talk about it is to make it a private club. There are the

    'enlightened' DevOps, and everyone else stuck living in the

    technological equivilant of Plato's Cave. This is the camp that wants a new

    job title – it sets you apart from the "others", who don't operate the way you

    do. The messages are deceptively similar – they both talk about integrated

    teams of Software Developers, and Systems/Network Administrators. You can

    spot this by tracking the negativity – if you're talking about how much

    something sucks, you're probably in this camp, rather than in the inclusive

    one.

    Devops, the cultural movement, is an awesome one. It's empowering and it makes

    the lives of the people it influences exponentially better. Let's cheer on the

    inclusionists and enlighten the exclusionists. 🙂

    Okay, lets move to Infrastructure as Code. Where Devops is a cultural change,

    this is a technological one – it's a shift in the way we think about assembling

    the infrastructures that run our businesses. In my contribution to the fine

    Mr. Allspaw's upcoming book, I define it through it's primary goal:

    "Enable the reconstruction of the business from nothing but a source code

    repository, an application data backup, and bare metal resources."

    Secondarily, there is another ideal here – the thing that should take the

    longest in this recovery process is the time it takes to restore your

    application data.

    This ideal isn't new – it tracks back at least as far as the
    http://www.infrastructures.org guys. The modern version draws on Service

    Oriented Architecture (and Web 2.0), Configuration Management, and Systems

    Integration for inspiration. The arrival of a new breed of infrastructure

    services (which in my mind started with EC2, honestly) lowered the barrier to

    actually achieving the goal (partially by narrowing your options at the

    physical layer, but primarily by giving us a real API to bootstrapping, which

    for most of us, was the thing that took longest during recovery.)

    Security, Networking, etc are all included here – if your goal is to

    re-constitute your business from scratch, you better be able to take it all the

    way. 🙂

    When people give presentations on building Infrastructure as Code, it's no

    suprise they focus on what can be easily automated. Vendors like Cisco and F5

    have not been building systems that are easy to automate – quite the opposite.

    When the APIs do exist, they are irritating to use, and difficult to integrate

    with. (In the case of Cisco, there is another problem – the automation often

    tries to abstract away the specialized knowledge embodied in every network

    engineer, which frustates the hell out of them.)

    We are still playing catch-up with physical infrastructure and proprietary

    devices. As the Devops cultural shift gains momentum, and more examples of

    fully-automated infrastructures exist in the world, we'll see the hurdles to

    this approach in the traditional infrastructure drop. Vendors will see the

    light, or the tradional enterprise will abandon their technology as fast as the

    big web shops have.

    I could go on all day, but I need to go for a run and burp my newborn baby. 🙂

    Thanks for the post, and I hope this clears something up – at the very least, I

    hope it underscores the dividing line between the cultural movement (Devops)

    and the technological one (Infrastructure as Code). Thanks for the post, and lets talk at Velocity about how we can get the Security and Network community involved in both the cultural and technological movements that are afoot.

  11. May 31st, 2010 at 13:11 | #11

    @Adam Jacob

    Totally f'ing awesome comment, Adam. Really.

    It really does open the discussion — one that needs to be had because the polarization the occurs due to whether someone's speaking from the inclusionary vs. exclusionary stance is really profound.

    Thanks for taking the time — as John and John did — to respond. Your perspective was really valuable and I'm sorry I had to bail from Glue because I really wanted to talk to Aaron after I heard him speak (and Jesse whom I missed @ Gov2.0 the next day) but didn't get a chance to. I appreciate the insight.

    I think DevOps really is a good thing. I think it can be a great thing. Just like Cloud 😉

    /Hoff

  12. neuroserve
    June 1st, 2010 at 06:53 | #12

    f5 -> devcentral, icontrol, tm shell

    cisco -> netomata config generator (ncg)

  13. June 2nd, 2010 at 19:33 | #13

    Great post and comment thread. To me the names don't matter – DevOps/OpsDev/WebOps etc. The message we need to be spreading is about organisational cultural change in technology business units – not Ops, Dev, SysAdmin people changing but organisations changing together. Working collaboratively and efficiently to deliver results customers want and return value to the company (shareholders, etc).

    I come from a largely non-WebOps background – large scale banking and finance ranging back to mainframe days – and we suffer probably more than any other technology businesses from the failings that DevOps is trying to combat. I, and others in our space, are working hard to spread the message outside the WebOps/Web 2.0 world, where a little of the light has already broken through, to people who are totally in the dark and really hurting because of a lack of collaboration and communication and a failure to embrace the needed cultural change.

    I've got a on what DevOps means to me and some slides I recently presented at DevOpsDownUnder talking about what DevOps means and how to start making some of the difficult and needed cultural and technological changes.

  14. Mark Bainter
    June 5th, 2010 at 12:24 | #14

    In spite of it not being about finger pointing, I think if we (operations) are honest with ourselves we'd have to admit that we have been somewhat stuck in the dark ages.

    But they *do* seem to have had more resources thrown at them in terms of figuring out how to improve that. These agile methodologies did come out of development practices after all. I'm frankly excited to see ops guys being honest about where we have been (or are), and more than that have ideas for how to make it better.

    Maybe it's just me, but I've never felt like it was finger pointing, but rather our finally talking about that stupid elephant in the room. Everyone was thinking it, but people are finally talking about it. Dev isn't this shining example on a hill, but the work they've done to look at things holistically, including everyone in the process, has inspired us to find our way forward down a similar path. You can't know where to go next if you don't understand where you are now – and maybe how you got here.

    I think those aspects of these presentations are valuable. We've lied to to ourselves for a long time, telling ourselves everything was fine, it was these constant changes from dev that were the problem. How many times have we said (even jokingly) "everything would be great if it wasn't for the (users|developers)". We can be so resistant to change that before we're shown the better path we really need to know why it's necessary.

  15. June 15th, 2010 at 03:19 | #15

    Sorry I'm coming to this party a little late… Some good points, a really good article and some really good comments. I also agree that the meaning of DevOps has been unclear and not communicated well. At Opscamp Austin, even among the "plugged in to DevOps" people there was clearly some differing expectations (It's automation! It's collaboration! It's ops folks becoming developers! It's a sandwich!). I personally think that the confusion is because DevOps is a rich term covering a lot of stuff, and its alternate name of "agile infrastructure" or "agile operations" is better because it breaks down into the same kinds of areas as agile development – I try to delineate the different areas of concern within DevOps here…

    I don't think that "DevOps" is an exclusionary term – Ops contains network, security, etc. just as Dev contains all the different varieties of coders (and testers and…). It just reflects that in every shop there are considered to be two "sides of the house" – the code/software people and the infrastructure/operations people, and that schism is harmful. Now, I do think that the term "Ops" is being used to cover two roles that are different in larger organizations, the "system admin/engineer" role that builds infrastructure and the "operations/support role" that maintains them, and that's an area to delve more into. Lack of cooperation among groups on that "side of the house" is a chronic problem. I don't think it's an either-or between devs understanding ops and writing their code to help them and automating operations, those are both parts of the collaboration and maturation process.

    Now, the cloud environment does make it EASIER to accomplish agile operations, for the reasons you note – less crufty hardware layers to deal with and more standard APIs means you can automate with less effort (and in fact, because of the dynamic nature of the cloud, you are FORCED to automate; in traditional environments automation is always that nice-to-have you maybe eventually get around to).

  16. Christian
    June 16th, 2010 at 21:07 | #16

    Hi,

    I totally agree that there is no clear or single definition of devops. First I thought a definition would be needed and I even thought about starting one in Wikipedia, but I didn’t. And from day to day I’m getting surer there is no definition needed at this time. Why should we limit devops with a definition? In my opinion devops just starts and by limiting it with a definition, it would not be as useful as it could be.

    The current discussion about a definition of devops seems a bit of a black or white discussion to me. But why do we need such a narrow definition and do not understand the different points of view as different parts or stages (people, technology, culture, …) of the same thought?

    Today devops seems to be mostly driven by webshops, but why limit it to them? Devops can also be applied to larger enterprises. It might get a bit trickier and take longer, because of more silos, more heterogeneous systems and legacy systems, but it is not impossible. This would be one more reason to not make the definition of devops to narrow. In enterprises it might be easier to apply devops one step after the other. The same happened with agile development. Most enterprises did a first small project with agile methods and than moved on. Why should we try a big-bang migration with devops?

    Even the discussion about the name devops is unnecessary. The name does not count, it is the idea behind it. The name is short, you can easily remember it and pronounce it the right way. What else do you want to have? You can’t put the description of a whole idea or movement in just a single word.

    Christian

  17. September 3rd, 2010 at 09:31 | #17

    Hey, some more recent news on this – I saw a great presentation recently on incorporating security into agile development teams that made me think of this article and "NetSecDevOps." I blogged it up and linked to the slides… http://theagileadmin.com/2010/09/02/devops-and-se

  1. May 31st, 2010 at 07:59 | #1
  2. May 31st, 2010 at 08:07 | #2
  3. June 1st, 2010 at 08:16 | #3
  4. June 3rd, 2010 at 14:49 | #4
  5. June 13th, 2010 at 01:02 | #5
  6. July 7th, 2010 at 08:23 | #6
  7. July 7th, 2010 at 09:03 | #7
  8. September 2nd, 2010 at 16:43 | #8
  9. March 30th, 2011 at 08:57 | #9
  10. April 8th, 2011 at 20:19 | #10
  11. December 10th, 2011 at 10:40 | #11