All For One, One For All? On Standardizing Virtual Appliance Operating Systems
- Image via Wikipedia
Hot on the tail of the announcement that VMware and Novell are entering into a deeper “strategic partnership” in order to deliver and support SUSE Linux Enterprise Server (SLES) for VMware vSphere environments, was an interesting blog post from Stu (@vinternals) titled “Enter the Appliance.”
Now, before we get to Stu’s post, let’s look at the language from the press release (the emphasis is mine):
VMware and Novell today announced an expansion to their strategic partnership with an original equipment manufacturer (OEM) agreement through which VMware will distribute and support the SUSE® Linux Enterprise Server operating system. Under the agreement, VMware also intends to standardize its virtual appliance-based product offerings on SUSE Linux Enterprise Server.
…
Customers who want to deploy SUSE Linux Enterprise Server for VMware® in VMware vSphere™ virtual machines will be entitled to receive a subscription to SUSE Linux Enterprise Server that includes patches and updates as part of their newly purchased qualifying VMware vSphere license and Support and Subscription. Under this agreement, VMware and its extensive network of solution provider partners will also be able to offer customers the option to purchase technical support for SUSE Linux Enterprise Server delivered directly by VMware for a seamless support experience. This expanded relationship between VMware and Novell benefits customers by reducing the cost and complexity of deploying and maintaining an enterprise operating system with VMware solutions.
As a result of this expanded collaboration, both companies intend to provide customers the ability to port their SUSE Linux-based workloads across clouds. Such portability will deliver choice and flexibility for VMware vSphere customers and is a significant step forward in delivering the benefits of seamless cloud computing.
Several VMware products are already distributed and deployed as virtual appliances. A virtual appliance is a pre-configured virtual machine that packages an operating system and application into a self-contained unit that is easy to deploy, manage and maintain. Standardizing virtual appliance-based VMware products on SUSE Linux Enterprise Server for VMware® will further simplify the deployment and ongoing management of these solutions, shortening the path to ROI.
What I read here is that VMware virtual appliances — those VMware products packaged as virtual appliances distributed by VMware — will utilized SLES as the underlying operating system of choice. I don’t see language or the inference that other virtual appliance ISVs will be required to do so
To that point, Stu’s blog post said:
VMware will be adopting SUSE Linux Enterprise Server, SLES, as the single platform for their virtual appliances.
I’ve ranted in the past about the problem with virtual appliances. Everything from the lack of a standard Linux platform even within a single vendor (let alone amongst multiple vendors), to the additional overhead such a model of software distribution would place upon software vendors, to the security needs of the Enterprise around patch response times etc. And today, every single one of those arguments has been nullified in one fell swoop. Hallelujah, someone was listening after all!
So far, so good. Seems pretty much in-line with what VMware said.
Here’s the interesting assertion Stu makes that inspired my commentary:
If you’re a software vendor looking to adopt the virtual appliance model to distribute your wares then I have some advice for you – if you’re not using SLES for the base of your appliance, start doing so. Now. This partnership will mean doors that were previously closed to virtual appliances will now be opened, but not to any old virtual appliance – it will need to be built on an Enterprise grade distro. And SLES is most certainly that.
Chris Wolf, Stu and I had a bit of banter on Twitter regarding this announcement wherein I suggested there’s a blurring of the lines and a conflation of messaging as well as a very unique perspective that’s not being discussed.
Specifically, I don’t see where it was implied that ISV’s would be forced to adopt SLES as their OS of choice for virtual appliances. I’m not suggesting it’s not compelling to do so for the support and distribution reasons stated above, but I suggest that the notion that “…doors that were previously closed to virtual appliances” from the perspective of support and uniformity of disto will also have and equal and opposite effect caused by a longer development lifecycle for many vendors.
Especially networking and security ISVs looking to move their products into a virtual appliance offering.
I summed up many of the issues associated with virtual security and networking appliances in my Four Horsemen of the Virtualization Security Apocalypse presentation, and given how the definition and capabilities of “the network” are (d)evolving (depending upon how you view abstraction) you might also find Where Are the Network Virtual Appliances? Hobbled By the Virtual Network, That’s Where… an interesting read:
What does this mean? It means that ultimately to ensure their own survival, virtualization and cloud providers will depend less upon virtual appliances and add more of the basic connectivity AND security capabilities into the VMMs themselves as its the only way to guarantee performance, scalability, resilience and satisfy the security requirements of customers. There will be new generations of protocols, APIs and control planes that will emerge to provide for this capability, but this will drive the same old integration battles we’re supposed to be absolved from with virtualization and Cloud.
Tell me that’s not what’s happening *right* now.
Unlike most user-facing or service-delivery applications that are not topology sensitive (that is, they simply expect to be able to speak to “the network” without knowing anything about it,) network and security ISVs do very interesting things with drivers and kernel-space code in order to deal with topology, where they sit in the stack, and how they improve performance and stability that are extremely dependent upon direct access to hardware or at the very least, customer drivers or extended/hacked kernels.
One of the reasons you see a slow trickle of network and security virtual appliances is because of these bespoke OS builds and what virtualization has done to how these services are delivered, scaled and deal with resilience. We’ve already seen the challenge of ISVs having to re-write code to fit the VMsafe fast/slow-path driver model.
You can imagine the consternation involved if what Stu alluded to is actually required — that you must build your virtual appliances on a specific OS. It’s going to slow down innovation and delivery of solutions if the ISV does not (for any number of valid reasons) use SLES. This is also one of the downsides of a JEOS approach.
Stu’s warnings about compliance to SLES development notwithstanding, this puts ISVs in a delicate position — one they’ve faced before but is now exacerbated by virtualization and Cloud. Security vendors generally minimize and harden OS stacks to fit their “application” and then tune the environment accordingly. We’re already introducing new monocultures and uniformity in attack surfaces with hypervisors. Are we going to do the same with the operating systems that power the virtual appliances/virtual machines that run atop them — especially those designed to protect these very systems?
Diversity is a good thing — at least when it comes to your networking and security infrastructure. While I happen to work for a networking vendor, we all recognize that uniformity brings huge benefits as well as the potential for nasty concerns. If you want an example, check out how a simple software error affected tens of millions of users of WordPress (WordPress and the dark side of multitenancy.) While we’re talking about a different layer in the stack, the issue is the same.
I totally grok the standardization argument for the cost control, support and manageability reasons Stu stated but I am also fearful of the extreme levels of lock-in and monoculture this approach can take.
/Hoff
Related articles by Zemanta
- VMware, Novell partner on Linux, virtualization (infoworld.com)
- Novell: The Microsoft-Taxed Fog Computing Solution (techrights.org)
- Where Are the Network Virtual Appliances? Hobbled By the Virtual Network, That’s Where… (rationalsurvivability.com)
- Virtualization & Cloud Don’t Offer An *Information* Security Renaissance… (rationalsurvivability.com)
- The Four Horsemen Of the Virtualization (and Cloud) Security Apocalypse… (rationalsurvivability.com)
- Security: In the Cloud, For the Cloud & By the Cloud… (rationalsurvivability.com)
- Incomplete Thought: The Other Side Of Cloud – Where The (Wild) Infrastructure Things Are… (rationalsurvivability.com)
- Virtual Networking/Nexus 1000v Virtual Switch Blogger Roundtable/WebEx Logistics – March 2nd. (rationalsurvivability.com)
- Patching the (Hypervisor) Platform: How Do You Manage Risk? (rationalsurvivability.com)
- The Hypervisor Platform Shuffle: Pushing The Networking & Security Envelope (rationalsurvivability.com)
I agree that diversity, at least in the case of virtual appliances, can be beneficial and that VMware has shown no signs of compelling SLES as an OS. VMware's traditional building momentum behind its SpringSource acquisition idicate continued support of a wide range of developmental preferences.
One of my favorite trivia questions about the history of virtualization is to ask people what HowNetWorks is. Nobody ever gets it right so instead of exercising your search box, I'll tell you: HowNetWorks was the winner of the first ever Virtual Appliance contest, which carried a pretty impressive prize of $100,000. Another interesting fact about HowNetWorks is that you can't get your hands on it any longer. It's gone. The software was never updated. It's not on the web. It's not on bittorrent, even if you can manage to find an increasingly rare torrent tracker to search for it.
How could this happen in an era where even the most trivial things we say on the Internet are cataloged forever? Obviously the reason is a lack of interest, but why? This lack of interest is a symptom of the many real problems people have using virtual appliances for anything more than just demoware. The difficulty takes many forms.
The key thing in Enterprise is accountability. When something crashes, some group or some person is responsible for fixing it. Same for updates and so on. You can't hire a person for each and every oddball Virtual Appliance stack that walks through your door, you need to establish critical mass on a known quantity.
This is the real barrier to innovation. Who knows what innovations we've missed out on simply because we don't yet have a vibrant ecosystem to support virtual appliances?
I often like to say that there is no approval committee, there is only a disapproval committee. A software vendor who tries to sell their virtual appliance on an unknown OS with unclear patching and audit mechanisms will quickly be introduced to the disapproval committee. A software vendor who builds their Virtual Appliance on SLES takes advantage of the relationship VMware has fought hard to establish.
This is not meant to be a lock-in, just a way of making life easy for everyone.
— Carter Shanklin (VMware)
@Carter Shanklin
This is a very compelling argument, but I dare say that it places a very narrow scope on the term 'innovation' and seems to use circular logic to establish your point.
The reality is that for the same reason you suggest an enterprise isn't going to deploy "…every oddball Virtual Appliance stack that walks through your door," reducing the OS to a common footprint still does nothing to ease the burden of having to patch/maintain/operate the application(s) in these virtual appliances.
A virtual appliance is marketed as not having to think in these terms in the first place.
If an enterprise isn't willing to deploy an "appliance" from a "no-name" company regardless of it's OS base, is it reasonable to assume that they will simply because it runs the same OS? No, of course not. Will it take one variable out of the equation? Sure. But in the case I talked about above (network and security appliances,) you introduce additional challenges.
Taking that analogy to its logical conclusion, a MATURE and "vibrant ecosystem" as it relates to SECURITY AND NETWORK virtual appliances isn't going to come from small no-name startups — at least not in volume. It's going to come from established players who take their physical appliances and find ways to create virtual appliances and have the two interact.
That being the case, how long has the "vibrant ecosystem" revolving around the VMsafe API been around? All the established vendors as well as the "innovative" startups who were yammering about how awesome this was going to be are merely crawling along. Most of the startups are either gone or acquired.
Those that are left are hooked to the platform and struggling to build their OWN ecosystems.
So by your illustration, we are expecting major ISVs who have huge stakes in supporting and developing physical implementations of their appliances across multiple, multi-year roadmaps, and you expect them to either:
a) Release a parallel appliance atop a different OS (SLES) that they may already run in order to run in VMware environments or,
b) Convert each and every product to run on SLES so they have roadmap parity.
Either of these choices is a tough one.
Again, I ask you to consider this from the perspective of network and security ISVs and not "regular" apps that little or no topological dependencies.
That was the point of the post; not to suggest that SLES for "virtual appliances" is a bad idea, but to present an argument that it ain't all sweet on the range.
I spent 3 years at Crossbeam dealing with ISV pain and compatability issues at the kernel/distro level where it would take 6+ months to struggle and get feature/functionality/performance parity…and we owned (and the ISV had direct access to) the hardware.
Would a common OS/distro have helped? Sure…but then what you find is that everybody does packet capture differently, everybody treats VLANs differently, everybody uses tweaked drivers/kernels or goes direct to hardware…messy.
I also work for a company that does a fair bit of engineering with VMware today, so we both know that entails a LOT of work.
I just don't want this brushed under the table like it's simple. It's not.
I applaud the effort and I think it will help the adoption of CERTAIN types of virtual appliances. I think network and security virtual appliances are not amongst them.
/Hoff
Cheers Hoff – now I see where the wheels have come off.
I never suggested that VMware would be recommending ISV's follow suit and use SLES as a base OS, and if I implied that then it was certainly unintentional. What I did say was that any ISV's *who are looking to leverage virtual appliances as a distribution model* (an important distinction in itself – personally I'm not sold on the idea) would do well to follow VMware's lead. This partnership will likely result in VMware providing much better support / configuration management / patching capabilities for VA's based on SLES, so rather than an ISV wasting resources on building OS expertise (which would be a requirement to adopting a VA distribution model as far as I'm concerned) they should just leverage what this agreement will provide on that layer.
I also think there is some confusion over the use of the word "appliance" – I too hope it will be a cold day in hell when long established security and networking hardware appliance vendors start using a general purpose OS. What I (and Carter) are referring to are ISV's who today would distribute their wares by way of an installer. These ISV's obviously see advantages in using virtual appliances as a distribution model, otherwise none of them would be providing them. The problem comes back to that domain expertise – they are generally not OS experts, and currently in the VA marketplace there is a random assortment of distro's being used for the base OS. This is not workable for the Enterprise for all the reasons Carter talked about.
Monoculture is nothing new in the Enterprise. You would struggle to find any operation of scale encouraging anything but the use of common components. That's certainly been the case at every place I have working in the last 10 years, all of which have been large corporates. And this extends to every conceivable component in Enterprise IT. In the datacenter, there's generally only 1 official Linux distro and 1 Windows (and UNIX is going the same way). HP or IBM x86 servers. Cisco or Juniper network infrastructure. EMC or NetApp storage. Oracle or SQL Server. Apache or IIS. JBOSS or Tomcat. Java or .NET. You get the idea – it's nothing new. Even Cisco themselves are going down this route I believe, by moving towards a consolidation of IOS, SAN-OS and NX-OS into just NX-OS.
I totally agree none of this is applicable to established hardware appliance vendors, not just networking and security but other things like storage too. But don't go playing the monoculture card with respect to a generic virtual appliance platform – it's pervasive in the Enterprise already (_especially_ the networking stack, only the desktop is in the same league). It may not be ideal from a security standpoint, but it's the only way to achieve economies of scale. Not just with regards to purchase price but operationally – the only meaningful "competitive advantage" that most large corporates can gain from IT is by having the lowest possible operational costs. And the only way to do that is via commoditisation – not specialisation. Again, I'm not saying it's ideal – it's just how it is.
I don't think you are disagreeing with any of that, just sayin'. I think your main point is that it would be stupid for a specialised appliance vendor to adopt a general purpose OS, and I 100% agree. But I don't think anyone is suggesting that – the whole virtual appliance push is more about the non-OS and non-appliance based ISV's, which is the vast majority of the software industry today.
@Stu
Gotcha. That wasn't immediately clear in your post — but then again, you also didn't specifically call out network/security "appliances," I did, so I see the divergence.
Besides suggesting that specialized vendors not adopt general purpose OS's for their appliances, what I'm really getting at is that if they expect to provide the same levels of performance, reliability, resilience, etc., they can't afford to…and the underlying VMMs are limiting (today) in terms of what networking capabilities exist (natively) that enable these solutions to work properly in the first place.
The more abstract and limiting the networking gets (number of interfaces, L2-> options, etc.) the more difficult engineering security becomes and forces us back up the "stack" to the point that we end up having to put more stuff in the guest/OS or play packet ping-pong with physical network appliances as the "big, flat and ugly" layer 2 virtual networks continue to bloat.
Thanks for clarifying your point. I figured 140 characters weren't enough to clarify mine 😉
/Hoff
I spend part of my time working on appliances (virtual and otherwise) for a small network security appliance vendor (I watch the World Cup the rest of the time these days).
The traditional switch/router is a unique beast in that the control and data plane and UI engines are co-located on the same hardware. Hence the ubiquitous CLI as the primary UI. But even Cisco and Juniper have plenty of reasons to introduce appliances on standard x64 hardware. Ironport comes to mind here. So does all the policy control stuff, triple-play/video/IMS control plane and many other software-based enhancements that don't need an NP to accelerate performance. IOS/JunOS etc don't have much of a role to play here. The design group (which may have been a startup at some point) is likely to pick Red Hat/SLES/MontaVista etc to get things started. Even guys building a stripped down OS are quite likely to start with a full distribution and start stripping.
So everybody has a few of these devices to ship and customers have to install and operate them and aren't always impressed by the choice of server OEM and the management capabilities on offer (redundancy, yet another LOM package or none whatsoever) and all the other details that goes into running a piece of hardware in your DC.
Hence the virtual appliance.
Now if you're vendor A that's on the Dow Jones Index or vendor B that's running on Series B fumes, you still have to pick an OS to base your appliance on. And while you're making that pick, you realize that the VMM provides optional-but-useful hooks and tools for you to install. For example, you might want the VmxNet3 virtual ethernet driver because you see packet loss with the standard E1000 one. Your marketing guys want to use the VMware-Ready tag, for which you need to go through the qualification program, which in turn nudges your OS patch strategy in a direction. All of these are ways in which the VMM vendor gets to influence the choices that VA-developers have to make. Further, your customers subscribe to Mitre CVEs, or use vuln-scanners on the appliances and want you to respond and the only scalable way to do this is to rely on a distro vendor. So going the independent route has it's own hazards.
VMware/SLES is very important and many have taken notice. There is value and conflict for platform people everywhere, including in the networking/security community.