Rational Survivability

There’s a little bit of serendipity floating about today and timing is everything.

Ed Skoudis (IntelGuardians) and I were chatting last week at ShmooCon regarding his previous research on VM guest escapes in hosted platforms and I raised a concern regarding my use of Parallel shared folders between my hosted XP installation and the underlying OSX host operating system.

I reckoned that this would be a very interesting vector for potential exploitation as it provides a direct pipeline to the underlying host OS and filesystem. 

While this bit of news isn’t about Parallels, it is about VMware’s comparable products (workstation, ACE, player, etc.) and it exploits the same vector.  From Computerworld:

February 24, 2008 (Computerworld)  A critical vulnerability in VMware Inc.’s virtualization software for Windows lets attackers escape the "guest" operating system and modify or add files to the underlying "host" OS, the company has acknowledged.

As of Sunday, there was no patch available for the flaw, which affects VMware’s Windows client virtualization programs, including Workstation, Player and ACE. The company’s virtual machine software for Windows servers, and for Mac- and Linux-based hosts, are not at risk.

The bug was reported by Core Security Technologies, makers of the penetration testing framework CORE IMPACT, said VMware in a security alert issued last Friday. "Exploitation of this vulnerability allows attackers to break out of an isolated Guest system to compromise the underlying Host system that controls it," claimed Core Security.

According to VMware, the bug is in the shared folder feature of its Windows client-based virtualization software. Shared folders lets users access certain files — typically documents and other application-generated files — from the host OS and any virtual machine on that physical system.

"On Windows hosts, if you have configured a VMware host-to-guest shared folder, it is possible for a program running in the guest to gain access to the host’s complete file system and create or modify executable files in sensitive locations," confirmed VMware.

There is currently no patch available.  The mitigation strategy is to disable shared folders.

It’s important to reiterate that this vulnerability does not affect VMware’s Type 1 (bare metal) virtualization platforms such as ESX.  However, on Friday, VMware released fixes for 5 vulnerabilities in ESX, some of which could be exploited to bypass security controls, gain access to data or result in denial of service.

/Hoff

{image from Anthony Martin Escapes}

UPDATE: Coverage of this is being hammed up quite a bit in the press to sound like it’s going to shake the very foundations of virtualization…not so much.  It’s an issue that is reasonably easy to address and represents what can be generally referred to as a relatively small attack surface.  Yes, it reinforces the need to think about VirtSec in the Type 2 (hosted) virtualization world, but as I said in the comments, it really depends upon how and why you’ve deployed client-side virtualization.

Last week I posted about a press release announcing a new product from Catbird called HypervisorShield.

I was having difficulty understanding some of the points raised in the press release/product brief, so I reached out to Michael Berman, Catbird’s CTO (also a blogger,) for a little clarification. 

Michael was kind enough to respond  to the points in my blog posting.

Rather than repost the entire blog entry, I have paraphrased the points Michael responded to and left his comments intact.  I think some of them invite further clarification and I’ll be following up with a Take5 interview shortly.  Some of the answers just beg for a little more digging…

Just to ground us all, here’s the skinny on HypervisorShield:

Catbird, provider of the only comprehensive security solution for virtual and physical networks, and developer of the V-Agent™ virtual appliance, today announced the launch of HypervisorShield™, the industry’s first dedicated comprehensive security solution specifically designed to guard against unauthorized hypervisor network access and attack.

Here are my points and Michael’s responses:

1. Hoff: The press release speaks to HypervisorShield’s ability to protect both the hypervisor and the "hypervisor management network" which I assume is actually referring to the the virtual interface of the management functions like VMware’s service console? Are we talking about protecting the service console or the network functions provided by the vKernel?

   Berman: We’ve built a monitor function that uses VMware APIs to watch for changes to/management of the virtual machines. We also have signature templates and customizable policies for network connections to the service console and the host.
    

2. Hoff: The press release makes it sound like protecting the hypervisor is accomplished via an IPS function that isolates VM’s from one another like Reflex and Blue Lane?

   Berman: With all due respect to our colleagues in this space, intrusion detection and protection is one element.  Catbird combines several technologies to extend separation of duties, dual control and strict change control to the virtual infrastructure. Deploying a signature for VMSA-2008-0001 is nice, but detection or prevention of a rogue virtual center administrator from pulling off a Societe Generale hack is priceless.
    

3. Hoff: What exactly does Catbird do (in partnering with IPS companies like SourceFire) that folks like Reflex and BlueLane? don’t already do.

   Berman: Rather than talk about the differences, let’s talk about the most important similarity.  I think I speak for all of us when I say that it’s like we are in a time warp to 1996 and I am explaining why you need a firewall for your DMZ.  Customers have little appreciation for the magnitude of the threats facing their virtual infrastructure.  Once we get past that, then we can talk about why Catbird is the best. (hint: we’re smarter, faster and stronger)
    

4. Hoff: How do you monitor the Hypervisor?

   Berman: We deploy a virtual machine that hooks into the vSwitch environment and that also monitors the ESX hypervisor via the VI API.
    

5. Hoff: You say in the press release that "hypervisor exploits have grown 35% in the last several years."  Which hypervisor exploits, exactly? You mean exploits against the big, fat, Linux-based service console from VMware? That’s not the hypervisor!

   Berman: I believe that the real threat to the virtual infrastructure comes from the collapse of separation of duties and the breakdown in implicit and explicit security controls within the virtual data center.  That being said, the hypervisor management application is probably the most significant area of the attack surface. If I can own the management GUI I own the hypervisor. If I can pull a stack smash against the ESX web server I own the hypervisor. If some poor shlemozzle configures Samba and NFS for the storage network then they become part of the attack surface too. You can blame us for some hyperbole, but the stat came from the CVE database. Gartner/451/Edison report that virtual infrastructure (VI) is less secure than physical and we have private data that shows people are deploying VI with no network security at all – this is just wrong.  I also think that writing about, or writing off the only risk as being some sort of red pill/blue pill hack is also wrong.

Thanks again to Michael for responding.  Look for a follow-on Take5 shortly to dig a little deeper.

/Hoff

One of the very profound impacts virtualization will have on security is the resultant collateral damage caused by what I call the "battle for the datacenter OS" framed by vendors who would ordinarily not be thought of as "OS vendors." 

I call the main players in this space the "Three Kings:" Cisco, VMware and EMC.
Microsoft is in there also, but that’s a topic for another post as I bifurcate operating system vendors in the classical sense from datacenter infrastructure platforms.  Google deserves a nod, too.

The "Datacenter OS" I am speaking of is the abstracted amalgam of virtualization and converged networking/storage that delivers the connected and pooled resource equivalent of the utility power grid.   Nick Carr reflects in his book "The Big Switch":

A hundred years ago, companies stopped producing their own power with steam engines and generators and plugged into the newly built electric grid.”

The "datacenter" and its underlying "operating system," in whatever abstracted form they will manifest themselves, will become this service layer delivery "grid" to which all things will connect; services will be merely combinations of resources and capabilities which are provisioned dynamically.

We see this starting to take form with the innovation driven by virtualization, the driving forces of convergence, the re-emergence of grid computing, the architectures afforded by mash-ups and the movements and investments of the Three Kings in all of these areas.

It’s pretty clear that that these three vendors are actively responsible for shaping the future of computing as we know it.  However, It’s not at all clear to me how much of the strategic overlap between them is accidental or planned, but they’re all approaching the definition of how our virtualized computing experience will unfold in very similar ways, albeit from slightly different perspectives.

One of the really interesting examples of this is how virtualization and convergence are colliding to produce the new model of the datacenter which blurs the lines between computing, networking and storage.

Specifically, the industry — as driven by customers — is trending toward the following:

• Upgrading from servers to blades
• Moving from hosts and switches to clusters and fabrics
• Evolving from hardware/software affinity to grid/utility computing
• Transitioning from infrastructure to service layers in “the cloud”

The topic of this post is really about the second bullet, moving from the notion of the classical hosts/servers plugging into separate network and storage switches to instead clusters of resources connecting to fabric(s) such that what we end up with are pools of resources to be provisioned, allocated and dispatched where, when and how needed. 

This is where I/O virtualization enters the picture.  I/O virtualization at the macro level of the datacenter describes the technology which enables the transition from the discrete and directly-connected model of storage versus networking to a converged virtualized model wherein network and storage resources are aggregated into a single connection to the "fabric."

Instead of having separate Ethernet, fiber channel and Infiniband connections, you’d have a single pipe connected to a "virtual connectivity switch" that provides on-demand, dynamic and virtualized allocation of resources to anything connected to the fabric.  The notion of physical affinity from the server/host’s perspective goes away.

Andy Dornan from Information Week just did a nice write-up titled "Cisco pitches virtual switches for next-gen data centers." 

It’s obviously focused on Cisco’s Nexus 7000 Series switch, but also gives some coverage of Brocade’s DCX Backbone, Xsigo’s Director and 3Leaf’s v-8000 products.

Check out what Andy had to say about Cisco’s strategy:

Cisco’s vision is one in which big companies off-load an increasing
number of server tasks to network switches, with servers ultimately
becoming little more than virtual machines inside a switch.*

The Nexus
doesn’t deliver that, but it makes a start, aiming to virtualize the
network interface cards, host bus adapters, and cables that connect
servers to networks and remote storage. At present, those require
dedicated local area networks and storage area networks, with each
using a separate network interface card and host bus adapter for every
virtual server. The Nexus aims to consolidate them all into one (or
two, for redundancy), with virtual servers connecting through virtual
NICs.

This stuff isn’t vaporware anymore.  These products are real…from numerous entities.  These companies — and especially Cisco — are on a mission to re-write the datacenter blueprint and security along with it.  VMware’s leading the virtualization charge and Cisco’s investing for the long run.  When you look at their investment in VMware, the I/O virtualization play and what they’re doing with vFrame, it’s impressive — and scary at the same time.

Them’s a lot of eggs in one basket, and it’s perfectly clear that there is a huge sucking sound coming from the traditional security realm as we look out over the horizon.  How do you apply a static security sensibility grounded in the approaches of 20 years ago to an amorphous, fluid, distributed and entirely dynamic pooled set of resources and information?

Cisco has thrown their hat in the ring to address the convergence of role-based admission and access control with the announcement of TrustSec which will be available in the Nexus as it is in the higher-end Catalyst switches.  Other vendors such as HP, Extreme and now Juniper as well as up-starts like Nevis and Consentry have their perspectives.  What each of these infrastructure networking vendors have in store for how their solutions will play in the world of virtualized and distributed computing is still to unfold.

How might this emerging phase of technology, architecture, provisioning, management, deployment and virtualization of resources impact security especially since we’ve barely even started to embrace the impact server virtualization has?  One word:

Completely.

More on this topic shortly…

/Hoff

*Update: A colleague of mine from Unisys, Michael Salsburg, prompted me via discussion to clarify a point.  I think that  for at least the short term, the "server tasks" that will be offloaded to I/O virtualization solutions such as Cisco’s will be fairly narrow in scope and logically defined.  However, given that NX-OS is Linux based, one might expect to see a Hypervisor-like capability within the switch itself, enabling VM’s and applications to be run directly within it.

Certainly we can expect and intermediary technology derivation which would include Cisco developing their own virtual switch that complements/replaces the vSwitch present in the VMM today; at this point given the heft/performance of the Nexus, one could potentially see it existing "outside" the vHost and using a high-speed 10Gb/s connection, redirect all virtual network functions to the external switch…

In December I made note of an interesting post on the virtualization.info blog titled "Patch Tuesday for VMware."  This issue popped up today in conversation with a customer and I thought to bubble it back up for discussion.

The post focused on some work done by Ronald Oglesby and Dan Pianfetti from GlassHouse
Technologies regarding the volume, frequency and distribution of patches across VMware’s ESX platform. 

When you combine Ronald and Dan’s data with Kris Lamb’s from ISS that I wrote about a few months ago, it’s quite interesting.

The assertion that Ronald/Dan are making in their post is that platforms like VMware’s ESX have to date required just as much care and feeding from a patching/vulnerability management perspective as a common operating system such as a Windows Server:

So why make this chart and look at the time between patches? Let’s take a hypothetical
server built on July 2nd of 2007, 5 months ago almost exactly. Since
being built on that day and put into production that server would have
been put into maintenance mode and patched/updated eight times. That’s
right eight (8) times in 5 months. How did this happen? Let’s look at
the following timeline:

Maybe it’s time to slow down and look at this as a QA issue? Maybe it’s
time to stop thinking about these platforms as rock solid, few moving
parts systems? Maybe it’s better for us not to draw attention to it,
and instead let it play out and the markets decide whether all this
patching is a good thing or not. Obviously patching is a necessary
evil, and maybe because we are so used to it in the Windows world, we
have ignored this so far. But a patch every 18.75 days for our
"hypothetical" server is a bit much, don’t you think?

I think this may come as a shock to some who have long held the belief that bare-metal, Type 1 virtualization platforms require little or no patching and that because of this, the "security" and availability of virtualized hosts was greater than that of their non-virtualized counterparts.

The reality of the situation and the effort and potential downtime (despite tools that help) have led to unexpected service level deviance, hidden costs and latent insecurity in deployed virtualized environments.  I think Ronald/Dan said it best:

If a client is buying into the idea of server virtualization as a piece
of infrastructure (like a SAN or a switch) only to see the types of
patching we see in Windows, they are going to get smacked in the face
with the reality that these are SERVERS. The reality that the vendors
are sticking so much into the OS that patches are going to happen just
as often as with Windows Servers? Or, if the client believes the
stability/rock solidness and skips a majority of general patches, they wind up with goofy time issues or other problems with iSCSI, until they catch up.

As a counterpoint to this argument I had hoped to convince Kris Lamb to extend his patch analysis of VMware’s releases and see if he could tell how many patched vulnerabilities existed in the service console (the big ol’ fat Linux OS globbed onto the side of ESX) versus the actual VMM implementation itself.  For some reason, he’s busy with his day job. 😉 This is really an important data point.  I guess I’ll have to do that myself ;(

The reason why this is important is exactly the reason that you’re seeing VMware and other industry virtualization players moving to embedded hypervisors; skinnying down the VMM’s to yield less code, less attack surface and hopefully less vulnerabilities.  So, to be fair, the evolution of the virtualization platforms is really on-par with what one ought to expect with a technology that’s still fairly nascent.

In fact, that’s exactly Nand Mulchandani, VMware’s Sr. Director of Security Product Management & Marketing in response to Ronald/Dan’s post:

As
the article points out, "patching is a necessary evil" – and that the
existence of ESX patches should not come as a shock to anyone. So let’s
talk about the sinister plan behind the increase in ESX patches.
Fortunately, the answer is in the article itself. Our patches contain a
lot of different things, from hardware compatibility updates, feature
enhancements, security fixes, etc.

…

We
also want customers to view ESX as an appliance – or more accurately,
as a product that has appliance-like characteristics.

…

Speaking
of appliances, another thing to consider is that we are now offering
ESX in a number of different form-factors, including the brand new ESX Server 3i.
3i will have a significantly different patch characteristics – it does
not have a Console OS and has a different patching mechanism than ESX
that will be very attractive to customers.

I see this as a reasonable and rational response to the issue, but it does point out that whether you use VMware or any other vendor’s virtualization platform, you should make sure to recognize that patching and vulnerability management of the underlying virtualization platforms is another — if not really critical — issue that will require operational attention and potential cost allocation.

/Hoff

P.S. Mike D. does a great job of stacking up other vendors against Microsoft in this vein such as Microsoft, Virtual Iron, and SWSoft.

A couple of weeks ago I penned a blog entry titled "The Battle for the HyperVisor Heats Up"
in which I highlighted an announcement from Phoenix Technologies
detailing their entry into the virtualization space with their
BIOS-enabled VMM/Hypervisor offering called HyperCore.

This drew immediate parallels (no pun intended) to VMware and Xen’s plans to embed virtualization capabilities into hardware.

The marketing continues this week with interesting announcements from Microsoft, Oracle and VMware:

1. VMware offers VMware Server 2 as a free virtualization product to do battle against…
2. Oracle offering "Oracle VM" for free (with paid support if you
   like) which claims to be 3 times as efficient than VMWare — based on
   Xen.
3. Microsoft officially re-badged its server virtualization technology as Hyper-V (nee Veridian)
   detailing both a stand-alone Hyper-V Server as well technology integrated into W2K8 Server.

It seems that everyone and their mother is introducing a virtualization platform and the underpinning of commonality between basic functionality demonstrates how the underlying virtualization enabler — the VMM/Hypervisor — is becoming a commodity.

We are sure to see fatter, thinner, faster, "more secure" or more open Hypervisors, but this will be an area with less and less differentiation.  Table stakes.  Everything’s becoming virtualized, so a VMM/Hypervisor will be the underlying "OS" enabling that transformation.

To illustrate the commoditization trend as well as a rather fractured landscape of strategies, one need only look at the diversity in existing and emerging VMM/Hypervisor solutions.   Virtualization strategies are beginning to revolve around a set of distinct approaches where virtualization is:

1. Provided for and/or enhanced in hardware (Intel, AMD, Phoenix)
2. A function of the operating system (Linux, Unix, Microsoft)
3. Delivered by means of an enabling software layer (nee
   platform) that is deployed across your entire infrastructure (VMware, Oracle)
4. Integrated into the larger Data Center "Fabric" or Data Center OS (Cisco)
5. Transformed into a Grid/Utility Computing model for service delivery

The challenge for a customer is making the decision on whom to invest it now.  Given the fact that there is not a widely-adopted common format for VM standardization, the choice today of a virtualization vendor (or vendors) could profoundly affect one’s business in the future since we’re talking about a fundamental shift in how your "centers of data" manifest.

What is so very interesting is that if we accept virtualization as a feature defined as an abstracted platform isolating software from hardware then the next major shift is the extensibility, manageability and flexibility of the solution offering as well as how partnerships knit out between the "platform" providers and the purveyors of toolsets.

It’s clear that VMware’s lead in the virtualization market is right inline with how I described the need for differentiation and extensibility both internally and via partnerships. 

VMotion is a classic example; it’s clearly an internally-generated killer app. that the other players do not currently have and really speaks to being able to integrate virtualization as a "feature" into the combined fabric of the data center.  Binding networking, storage, computing together is critical.  VMware has a slew of partnerships (and potential acquisitions) that enable even greater utility from their products.

Cisco has already invested in VMware and a recent demo I got of Cisco’s VFrame solution shows they are serious about being able to design, provision, deploy, secure and manage virtualized infrastructure up and down the stack, including servers, networking, storage, business process and logic.

In the next 12 months or so, you’ll be able to buy a Dell or HP server using Intel or AMD virtualization-enabled chipsets pre-loaded with multiple VMM/Hypervisors in either flash or BIOS.  How you manage, integrate and secure it with the rest of your infrastructure — well, that’s the fun part, isn’t it?

I’ll bet we’ll see more and more "free" commoditized virtualization platforms with the wallet ding coming from the support and licenses to enable third party feature integration and toolsets.

/Hoff

Version 1.0 of the Center for Internet Security (CIS) benchmark for securing VMware ESX server is available.  This is specific to version 3.x of ESX and covers the basic best practices of preparing an ESX server for deployment.

We’ve still got a ton of stuff that didn’t make the deadline cut-off for the first version of the document in follow-on iterations, but it’s a good start.  Please sign up if you can contribute to making this document even better.

You can find it here.

/Hoff

This is truly one of those times that we’re just going to have to hold our breath and wait and see…

Prior to VMworld, I blogged about the expected announcement by Cisco and VMware that the latter would be opening the HyperVisor to third party vendors to develop their virtual switches for ESX.

This is extremely important in the long term because security vendors today who claim to have security solutions for virtualized environments are basically doing nothing more than making virtual appliance versions of their software that run as yet another VM on a host along side critical applications.

These virtual appliances/applications are the same you might find running on their stand-alone physical appliance counterparts, and they have no access to the HyperVisor (or the vSwitch) natively.  Most of them therefore rely upon enabling promiscuous mode on vSwitches to gain visibility to inter-VM traffic which uncorks a nasty security genie of its own.  Furthermore, they impose load and latencies on the VM’s as they compete for resources with the very assets they seek to protect.

The only exception to this that I know of currently is Blue Lane who actually implements their VirtualShield product as a HyperVisor Plug-in which gives them tremendous advantages over products like Reflex and Catbird (which I will speak about all of these further in a follow-on post.)  Ed: I have been advised that this statement needs revision based upon recent developments — I will, as I mention, profile a comparison of Blue Lane, Catbird and Reflex in a follow-on post.  Sorry for the confusion.

At any rate, the specific vSwitch announcement described above was not forthcoming at the show, but a more important rumbling became obvious on the show floor after speaking with several vendors such as Cisco, Blue Lane, Catbird and Reflex; VMware was quietly beginning to provide third parties access to the HyperVisor by exposing API’s per this ZDNet article titled "VMware shares secrets in security drive":

Virtualization vendor VMware has quietly begun sharing some of
its software secrets with the IT security industry under an unannounced
plan to create better ways of securing virtual machines. 

VMware
has traditionally restricted access to its hypervisor code and, while
the vendor has made no official announcement about the API sharing
program tentatively called "Vsafe," VMware founder and chief scientist
Mendel Rosenblum said that the company has started sharing some APIs
(application program interfaces) with security vendors.

I know I should be happy about this, and I am, but now that we’re getting closer to the potential for better VM security, the implementation deserves some scrutiny.  We don’t have that yet because most of the vSafe detail is still hush-hush.

This is a double-edged sword.  While it represents a fantastic opportunity to expose functionality and provide visibility into the very guts of the VMM to allow third party software to interact with and control the HyperVisor and dependent VM/GuestOS’s, opening the Kimono represents a potentially huge new attack surface for potential malicious use.

"We would like at a high level for (VMware’s platform) to be a better
place to run," he said. "To try and realize that vision, we have been
partnering with experts in security, like the McAfees and Symantecs,
and asking them about the security issues in a virtual world."

I’m not quite sure I follow that logic.  McAfee and Symantec are just as clueless as the bulk of the security world when it comes to security issues in a virtual world.  Their answer is usually "do what you normally do and please make sure to buy a license for our software on each VM!" 

The long-term for McAfee and Symantec can’t be to continue to deploy bloatware on every VM.  Users won’t put up with the performance hit or the hit in their wallet.  They will have to re-architect to take advantage of the VMM API’s just like everyone else, but they have a more desperate timefame:

Mukil Kesavan, a VMware intern studying at the University of Rochester,
demonstrated his research into the creation of a host-based antivirus
scanning solution for virtualized servers at the conference. Such a
solution would enable people to pay for a single antivirus solution
across a box running multiple virtual servers, rather than having to
buy an antivirus solution for each virtual machine.

Licensing is going to be very critical to companies like these two very shortly as it’s a "virtual certainty" that the cost savings enjoyed by consolidating physical servers will place pressure on reducing the software licensing that goes along with it — and that includes security.

Rosenblum says that some of the traditional tools used to protect a hardware server work just as well in a virtualized environment, while others "break altogether."

"We’re trying to fix the things that break, to bring ourselves up to
the level of security where physical machines are," he said. "But we
are also looking to create new types of protection."

Rosenblum said the APIs released as part of the initiative
offer security vendors a way to check the memory of a processor, "so
they can look for viruses or signatures or other bad things."

Others allow a security vendor to check the calls an
application within a virtual machine is making, or at the packets the
machine is sending and receiving, he said.

I think Rosenblum’s statement is interesting in a couple of ways:

1. His goal, as quoted,  is to fix the things that virtualization breaks and bring security up to the level of physical servers.  Unlike every other statement from VMware spokesholes, this statement therefore suggests that virtualized environments are less secure than physical ones.  Huh.
2. I think this area of focus — when combined with the evolution of the determina acquisition — will yield excellent security gains.  Extending the monitoring and visibility into the isolated memory spaces of the virtual processors in a VM means that we may be able to counter attacks without having to depend upon solely on the virtual switches; it gives you application-level visibility without the need for another agent.

The Determina acquisition is really a red herring for VMware.  Determina’s "memory firewall" seeks to protect a system "…from buffer overflow
attacks, while still allowing the system to run at high speeds. It also
developed "hot-patching" technology–which allows servers to be patched
on the fly, while they are still running."  I’ve said before that this acquisition was an excellent move.  Let’s hope the integration goes well.

If you imagine this chunk built into the VMM, the combination of exposed VMM API’s with a lightweight VMM running in hardware (flash) embedded natively into a server less the bloated service console, it really is starting tohead down an interesting path.  This is what ESX Server 3i is designed to provide:

ESX Server 3i has considerable advantages over its predecessors
from a security standpoint. In this latest release, which will be
available in November, VMware has decoupled the hypervisor from the
service console it once shipped with. This console was based on a
version of the Red Hat Linux operating system.

As such, ESX 3i is a mere 32MB in size, rather than 2GB.

Some 50 percent of the vulnerabilities that VMware was patching in
prior versions of its software were attributable to the Red Hat piece,
not the hypervisor.

"Our hope is that those vulnerabilities will all be gone in 3i," Rosenblum said.

Given Kris Lamb’s vulnerability distribution data from last week, I can imagine that everyone hopes that these vulnerabilities will all be gone, too.  I wonder if Kris can go back and isolate how many of the vulns listed as "First Party" were attributable to the service console (the underlying RH Linux OS) accompanying the HyperVisor.  This would be good to know.  Kris? 😉

At any rate, life’s about trade-offs and security’s no different.  I think that as we see the API’s open up, so will more activity designed to start tearing at the fleshy underbelly of the VMM’s.  I wonder if we’ll see attacks specific to flash hardware when 3i comes out?

/Hoff

(P.S. Not to leave XenSource or Veridian out of the mix…I’m sure that their parent companies (Citrix & microsoft) who have quite a few combined security M&A transactions behind them are not dragging their feet on security portfolio integration, either.)

I’d like to frame this little ditty with a quote that Marcus Ranum gave in a face-off between he and Bruce Schneier in this month’s Information Security Magazine wherein he states:

"Will the future be more secure? It’ll be just as insecure as it
possibly can, while still continuing to function. Just like it is today."

Keep that in mind as you read this post on virtualization security, won’t you?

Over the last few months we’ve had a serious malfunction in the supply chain management of Common Sense.  It’s simply missing from the manifest in many cases.

Such is the case wherein numerous claims of undelivered security common sense are being filed as instead of shipping clue in boxes filled with virtualization goodness, all we get are those styrofoam marketing peanuts suggesting that we’re either "more" or "less" secure instead.  More or less compared to what, exactly?

It’s unfortunate that it’s still not clear enough at this point that we’re at a crossroads with virtualization.

I believe it’s fair to suggest that the majority of us know that the technology represents fantastic opportunities, but vendors and customers alike continue to cover their ears, eyes and mouths and ignore certain realities inherent in the adoption of any new application or technology when it comes to assessing the risk associated with deploying this technology.

Further, generalizations regarding virtualization as being "more" or "less" secure than non-virtualized platforms represents an exercise in tail-chasing that seems more and more to be specious claims delivered in many cases without substantiated backing…

Here’s a perfect example of this sort of thing from a CMP ChannelWeb story titled "Plotting Security Strategy in a Virtual World":

"In many ways, securing virtual servers is little different from securing physical servers, said Patrick Lin, senior director of product management at VMware."

We’ve talked about this before.  This is true, except (unfortunately) for the fact that we’ve lost a tremendous amount of visibility from the network security practitioner’s perspective as now the "computer is the network" and many of the toolsets and technologies are not adapted to a accomodate a virtualized instantiation of controls and detection mechanisms such as Firewalls, IDS/IPS and other typical gateway security functions.

"At the end of the day, they are just Windows machines," Lin said. "When you turn a physical server into a virtual server, it’s no more vulnerable that it was before. There are not new avenues of attack all of a sudden."

That statement is false, misleading and ironic given the four vulnerabilities we just saw reported over the last 3 days that are introduced onto a virtual host thanks to the VMware software which enables the virtualization capabilities.

If you aren’t running VMWare’s software, then you’re not vulnerable to exploit from these vulnerabilities.  This is an avenue of attack.  This represents a serious vulnerability.  This is a new threat/attack surface "all of a sudden."

[Ed: I simply had to add this excerpt from Kris Lamb’s fantastic blog post (IBM/ISS X-Force) that summarized empirically the distribution of VMware-specific vulnerabilities from 1999-2007.]

We pulled all known vulnerabilities across all of VMware’s products
since 1999. I then focused on categorizing by year, by severity, by
impact, by vector and by whether the vulnerability was in VMware’s
proprietary first-party components or in third-party components that
they use in their products.

Once I pulled all the data, sorted and structured it in various ways, it summarized like this:

VMware Vulns by Year	Total Vulns	High Risk Vulns	Remote Vulns	Vulns in First Party Code	Vulns in 3rd Party Code
Vulns in 1999	1	1	0	1	0
Vulns in 2000	1	1	0	1	0
Vulns in 2001	2	0	0	2	0
Vulns in 2002	1	1	1	1	0
Vulns in 2003	9	5	5	5	4
Vulns in 2004	4	2	0	2	2
Vulns in 2005	10	5	5	4	6
Vulns in 2006	38	13	27	10	28
Vulns in 2007	34	18	19	22	12
TOTALS	100	46	57	48	52

So what are some of the interesting trends?

• There have been 100 vulnerabilities disclosed across all of VMware’s virtualization products since 1999.
  
• 57% of the vulnerabilities discovered in VMware products are remotely accessible, while 46% are high risk vulnerabilities.
  
• 72% of all the vulnerabilities in VMware virtualization products have been discovered since 2006.
  
• 48% of the vulnerabilities in VMware virtualization products
  are in first-party code and 62% are in third-party code that their
  non-hosted Linux-based products use.
• Starting in 2007, the number of vulnerabilities discovered
  in first-party VMware components almost doubled that of vulnerabilities
  discovered in third-party VMware components. 2007 is the first time
  where first-party VMware vulnerabilities were greater than third-party
  VMware vulnerabilities.

How do I interpret these trends?

• It is clear that with the increase in popularity, relevance and
  deployment of virtualization starting in 2006, vulnerability discovery
  energies have increasingly focused on finding ways to exploit
  virtualization technologies.
• Combine the vulnerabilities in virtualization software,
  vulnerabilities in operating systems and applications that still exist
  independent of the virtualization software, the new impact of virtual
  rootkits and break-out attacks with the fact that in a virtual
  environment all your exploitation risks are now consolidated into one
  physical target where exploiting one system could potentially allow
  access and control of multiple systems on that server (or the server
  itself). In total, this adds up to a more complex and risky security
  environment.
• Virtualization does not equal security!

I’ve already blog-leeched enough of Kris’ post, so please read his blog entry to see the remainder of his findings, but I think this does a really good job of putting to rest some of the FUD associated with this point.

Let’s continue to deconstruct at Mr. Lin’s commentary:

Even so, server virtualization vendors are taking steps to ensure that their technology is itself up-to-date in terms of security.

OK, I’ll buy that based upon my first hand experience thus far.  However, here’s where the example takes a bit of a turn as it seeks to prove that a certification somehow suggests that a virtualization platform is more secure:

Lin said VMware Server ESX is currently certified at Common Criteria Level 2 (CCL2), a security standard, and is in the process of applying for CCL4 for its Virtual Infrastructure 3 (VI3) product suite.

Just to be clear, Common Criteria Evaluation Assurance Levels don’t guarantee the security of a product; they demonstrate that under controlled evaluation, a product configured in a specific way meets certain assurance requirements that relay a higher level of confidence that a product’s security functions will be performed correctly and effectively. 

CC certification does not mean that a system is not vulnerable to or resilient against many prevalent classes of attack.  Also, in many ways, CC provides the opposite of being "up-to-date" from a security perspective because once a system has been certified, modifying it substantially with security code additions or even patches, renders the certification invalid and requires re-test.

If you’re interested, here’s a story describing some interesting aspects of CC certification and what it may mean for a product’s security and resilience titled "Common Criteria is Bad for You."

I’m working very hard to pull together a document which outlines exposure and risk associated with deploying virtualization with as much contextual and temporal relevance as I can muster.  Numerous other people are, also.  This way we can quantify the issues at hand rather than listening to marketing squawk boxes yelp from their VoxHoles about how secure/insecure virtualization platforms are without examples or solutions…

In the meantime, as a friendly piece of advice, might I suggest that the virtualization vendors such as VMWare kindly limit the outflow of how security information regarding virtualization is communicated? 

Statements like those above made by product managers who are not security domain experts only seeks to erode the trust and momentum you’re trying to gain.

There are certainly areas in which virtualization provides interesting, useful, unique and (in some cases) enhanced security over those in non-virtualized environments.  The converse can also be held as true.

Let’s work on communicating these differences in specifics instead of generalities.

/Hoff

First the Virtualization Security Public Service Announcement:

By now you’ve no doubt heard that Ryan Smith and Neel Mehta from IBM/ISS X-Force have discovered vulnerabilities in VMware’s DHCP implementation that could allow for "…specially crafted packets to gain system-level privileges" and allow an attacker to execute arbitrary code on the system with elevated privileges thereby gaining control of the system.   

Further, Dark Reading details that Rafal Wojtczvk (whose last name’s spelling is a vulnerability in and of itself!) from McAfee discovered the following vulnerability:

A vulnerability
that could allow a guest operating system user with administrative
privileges to cause memory corruption in a host process, and
potentially execute arbitrary code on the host. Another fix addresses a
denial-of-service vulnerability that could allow a guest operating
system to cause a host process to become unresponsive or crash.

…and yet another from the Goodfellas Security Research Team:

An additional update, according to the advisory, addresses a
security vulnerability that could allow a remote hacker to exploit the
library file IntraProcessLogging.dll to overwrite files in a system. It
also fixes a similar bug in the library file vielib.dll.

It is important to note that these vulnerabilities have been mitigated by VMWare at the time of this announcement.  Further information regarding mitigation of all of these vulnerabilities can be found here.

You can find details regarding these vulnerabilities via the National Vulnerability Database here:

CVE-2007-0061 – The DHCP server in EMC VMware Workstation before 5.5.5 Build 56455 and
6.x before 6.0.1 Build 55017, Player before 1.0.5 Build 56455 and
Player 2 before 2.0.1 Build 55017, ACE before 1.0.3 Build 54075 and ACE
2 before 2.0.1 Build 55017, and Server before 1.0.4 Build 56528 allows
remote attackers to execute arbitrary code via a malformed packet that
triggers "corrupt stack memory.

CVE-2007-0062 – Integer overflow in the DHCP server in EMC VMware Workstation before
5.5.5 Build 56455 and 6.x before 6.0.1 Build 55017, Player before 1.0.5
Build 56455 and Player 2 before 2.0.1 Build 55017, ACE before 1.0.3
Build 54075 and ACE 2 before 2.0.1 Build 55017, and Server before 1.0.4
Build 56528 allows remote attackers to execute arbitrary code via a
malformed DHCP packet that triggers a stack-based buffer overflow.

CVE-2007-0063 – Integer underflow in the DHCP server in EMC VMware Workstation before
5.5.5 Build 56455 and 6.x before 6.0.1 Build 55017, Player before 1.0.5
Build 56455 and Player 2 before 2.0.1 Build 55017, ACE before 1.0.3
Build 54075 and ACE 2 before 2.0.1 Build 55017, and Server before 1.0.4
Build 56528 allows remote attackers to execute arbitrary code via a
malformed DHCP packet that triggers a stack-based buffer overflow.

CVE-2007-4496 – Unspecified vulnerability in EMC
VMware Workstation before 5.5.5 Build 56455 and 6.x before 6.0.1 Build
55017, Player before 1.0.5 Build 56455 and Player 2 before 2.0.1 Build
55017, ACE before 1.0.3 Build 54075 and ACE 2 before 2.0.1 Build 55017,
and Server before 1.0.4 Build 56528 allows authenticated users with
administrative privileges on a guest operating system to corrupt memory
and possibly execute arbitrary code on the host operating system via
unspecified vectors.

CVE-2007-4155 – Absolute path traversal vulnerability in a certain ActiveX control in
vielib.dll in EMC VMware 6.0.0 allows remote attackers to execute
arbitrary local programs via a full pathname in the first two arguments
to the (1) CreateProcess or (2) CreateProcessEx method.

I am happy to see that VMware moved on these vulnerabilities (I do not have the timeframe of this disclosure and mitigation available.)  I am convinced that their security team and product managers truly take this sort of thing seriously.

However, this just goes to show you that as the virtualization platforms enter further-highlighted mainstream adoption, exploitable vulnerabilities will continue to follow as those who follow the money begin to pick up the scent. 

This is another phrase that’s going to make a me a victim of my own Captain Obvious Award, but it seems like we’ve been fighting this premise for too long now.  I recognize that this is not the first set of security vulnerabilities we’ve seen from VMware, but I’m going to highlight them for a reason.

It seems that due to a lack of well-articulated vulnerabilities that extended beyond theoretical assertions or POC’s, the sensationalism of research such as Blue Pill has desensitized folks to the emerging realities of virtualization platform attack surfaces.

I’ve blogged about this over the last year and a half, with the latest found here and an interview here.  It’s really just an awareness campaign.  One I’m more than willing to wage given the stakes.  If that makes me the noisy canary in the coal mine, so be it.

These very real examples are why I feel it’s ludicrous to take seriously any comments that suggest by generalization that virtualized environments are "more secure" by design; it’s software, just like anything else, and it’s going to be vulnerable.

I’m not trying to signal that the sky is falling, just the opposite.  I do, however, want to make sure we bring these issues to your attention.

Happy Patching!

/Hoff

Reader Colin was kind enough to forward me a link to a great security and hardening document which begins to address many of the  elements I posted in my recent "Ephiphany…" blog entry regarding virtualization and hardening documentation.

This document was produced by the folks at XtraVirt  who describe themselves as "…a company of innovative experts dedicated to VMware virtualisation, storage, operating systems and deployment methods."  These guys maintain an impressive cache of tools, whitepapers and commercial products focused on virtualization, many of which are available for download.

I’m rather annoyed and embarrassed that it took me this long to discover this site and its resources!

As a wonderful study in serendipity, I’ve recently signed up to contribute to the follow-on to the CIS Virtualization Benchmark that specifically addresses VMware’s ESX environment.   This draft is under construction now, and it represents a good first pass, but continues to need (IMHO) some additional focus on the network/vSwitch elements.

I respectfully suggest that many of the contents of the XtraVirt document, need to make their way into the CIS draft.

One of the other really interesting approaches this document takes is to classify each of the potential hardening elements by risk expressed as a function of threat, likelihood, potential impact and countermeasure as measured against impact to C, I, and A.

Secondly, there is a much-needed section on the VirtualSwitch and network constituent functions.

Here’s the snapshot of the XtraVirt effort:

One of the more difficult challenges found when introducing
virtualisation technologies to a new environment, whether it be your
own or as a consultant to a client, can be gaining the understanding
and support of the IT Security team, especially so if they haven’t been
exposed to virtualisation technologies in the past. 

As a
Solutions Architect having faced this in this situation on several
occasions and being tied up in weeks of claim and counter-claim about
how secure VMware VI3 was, I tried several approaches; one was to
simply email the published VMware security documents to them, and two
was sit down and explain why and how VI3 was inherently secure.

Both
of these approaches could take weeks and at times frustration on both
sides could lead to unnecessary discussions.  Although the VMware
documents are excellent and pitched at the right level, I found that
security team engagement could be limited and it wasn’t always enough
to simply provide these on their own as the basis for a solution.

So the idea was sown to create the ‘VMware® VI3 Security Risk Assessment Template’ that could be repeatedly used as the basis for any VI3 design submission. There’s
nothing particularly clever about it, the information is already out
there, I just felt it needed to be presented in a customised way for IT
Security review and approval.

This MS Word document template is designed to:

· Provide detail of around security measures designed into each major component of VI3

· Provide a ‘best practice’ security framework for VI3 designs that can be repeated again and again

· Detail
real world scenario’s that IT Security personnel can relate to their
environment, including built-in countermeasures and additional
configuration options.

· Significantly reduce the time and stress involved with gaining design approvals.

The idea is to take your own VI3 design and apply it to each of the major VI3 components in this template:

· ESX Server – Service Console

· ESX Server – Kernel

· ESX Server – Virtual Networking Layer

· Virtual Machines

· Virtual Storage

· VirtualCenter

This
means that in most cases it’s just a case of filling in the gaps, and
putting a stake in the ground as to which additional configuration
options you wish to implement.  In all cases you end up with a
document that should relate to your design and the IT Security teams
have a specific proposal which details all the things they want to see
and understand.

The first time I used it (on a
particularly tough Security Advisor who had never seen VMware products
I might add) I had nothing but great feedback which allowed my low
level design to proceed with confidence and saved weeks of explanation
and negotiation.

I’ve reached out to the guys at XtraVirt to both thank them and to gain some additional insight into their work.

I think this is a great effort.

Oh, did you want the link? 😉

/Hoff