Home > Security Conferences, Virtualization, VMware > Epiphany: For Network/InfoSec Folks, the Virtualization Security Awareness Problem All Starts With the vSwitch…

Epiphany: For Network/InfoSec Folks, the Virtualization Security Awareness Problem All Starts With the vSwitch…

September 13th, 2007 Leave a comment Go to comments

It’s all about awareness, and as you’ll come to read, there’s just not enough of it when we talk about the security implications of virtualizing our infrastructure.  We can fix this, however, without resorting to FUD and without vendors knee-jerking us into spin battles. 

I’m attending VMworld 2007 in San Francisco, looking specifically for security-focused information in the general sessions and hands-on labs.  I attended the following sessions and a hands-on lab yesterday:

  • Security Hardening and Monitoring of VMware Infrastructure 3 (Hands-on Lab)
  • Security Architecture Design and Hardening VMware Infrastructure 3 (General Session)
  • Using the Secure Technical Implementation Guide (STIG) with VMware Infrastructure 3 (General Session)

I had high hopes that the content and focus of these sessions would live up to the hype surrounding the statements by VMware reps at the show.   As Dennis Fisher from  SearchSecurity wrote, there are some  powerful statements coming from the VMware camp on the security of virtualized environments and how they are safer than non-virtualized deployments.  These are a bold, and in my opinion, dangerously generalized statements to make when backed up with examples which beg for context:

To help assuage customers’ fears, VMWare executives and security
engineers are going on the offensive and touting the company’s ESX
Server as a more secure alternative to traditional computing setups.

Despite the complexity of virtualized environments, they are inherently
more secure than normal one-to-one hardware and operating system
because of the hypervisor’s ability to enforce isolation
among the virtual machines, Mukundi Gunti, a security engineer at
VMWare said in a session on security and virtualization Tuesday.


"It’s a much more complex architecture with a lot of moving parts.
There are a lot of misconceptions about security and virtualization,"
said Jim Weingarten, senior technical alliances manager at VMWare, who
presented with Gunti. "Virtual machines are safer."

So I undertook my mission of better understanding how these statements could be substantiated empirically and attended the sessions/labs with as much of an open mind as I could given the fact that I’m a crusty old security dweeb.

Security Hardening and Monitoring of VMware Infrastructure 3 (Hands-on Lab)

The security hardening and monitoring hands-on lab was very basic and focused on general unix hardening of the underlying RH OS as well as SWATCH log monitoring.  The labs represented the absolute minima that one would expect to see performed when placing any Unix/Linux based host into a network environment.  Very little was specific to virtualization.  This  session presented absolutely nothing new or informative.

Security Architecture Design and Hardening VMware Infrastructure 3 (General Session)
The security architecture design and hardening session was at the very end of the day and was packed with attendees.  Unfortunately, it was very much a re-cap of the hands-on lab but did touch on a couple of network-centric design elements (isolating the VMotion/VIC Management ports onto separate NICs/VLANs, etc) as well as some very interesting information regarding the security of the virtual switch (vSwitch.)  More on this below, because it’s very interesting.

The Epiphany
This is when it occurred to me, that given the general roles and constituent responsibilities of the attendees, most of whom are not dedicated network or security folks, the disconnect between the "Leviathan Force" (the network and network security admins) and the "Sysadmins" (the server/VMM administrators) was little more than the mashup of a classic turf battle and differing security mindsets combined with a lack of network and information security-focused educational campaigning on the part of VMware.

After the talk, I got to spend about 30 minutes chatting with VMware’s Kirk Larsen (Engineering Product Security Officer) and Banjot Chanana (Product Manager for Platform Security) about the lack of network-centric security information in the presentations and how we could fix that.

What I suggested was that since now we see the collapse and convergence of the network and the compute stacks into the virtualizaton platforms, the operational impact of what that means to the SysAdmins and Network/Information Security folks is huge. 

The former now own the keys to the castle whilst the latter now "enjoy" the loss of visibility and operational control.  Because the network and InfoSec folks aren’t competently trained in the operation of the VM platforms and the SysAdmins aren’t competently trained in securing (holistically — from the host through to the network) them, there’s a natural tendency for conflict.

So here’s what VMware needs to do immediately:

  1. Add a series of whitepapers and sessions that speak directly to assuage the fear of the virtualization unknowns targeting the network and InfoSec security staffers. 
  2. Provide more detail and solicit feedback relating to the technical roadmaps that will get the network and InfoSec staffer’s visibility and control back by including them in the process, not isolating them from it.
  3. Assign a VMware community ombudsman to provide outreach and make his/her responsibility to make folks in our industry aware — and not by soundbites that sponsors contention — that there are really some excellent security enhancements that virtualization (and specifically VMware) bring to the table.
  4. Make more security acquisitions and form more partnerships.  Determina was good, but as much as we need "prevention" we need "detection" — we’ve lost visibility, so don’t ignore the basics.
  5. Stop fighting "FUD" with "IAFNAB" (It’s a feature, not a bug) responses
  6. Give the network and InfoSec folks solid information and guidance against which we can build budgets to secure the virtualization infrastructure before it’s deployed, not scrap for it after it’s already in production and hackbait.
  7. Address the possibility of virtualization security horizon events like Blue Pill and Skoudis’ VM Jail escapes head-on and work with us to mitigate them.
  8. Don’t make the mistakes Cisco does and just show pretty security architecture and strategy slides featuring roadmaps that are 80% vapor and call it a success.
  9. Leverage the talent pool in the security/network space to help build great and secure products; don’t think that the only folks you have to target are the cost-conscious CIO and the SysAdmins.
  10. Rebuild and earn our trust that the virtualization gravy train isn’t an end run around the last 10 years we’ve spent trying to secure our data and assets.  Get us involved.

I will tell you that both Kirk and Banjot recognize the need for these activities and are very passionate about solving them.  I look forward to their efforts. 

In the meantime, a lot of gaps can be closed in a very short period by disseminating some basic network and security-focused information.  For example, from the security architecture session, did you know that the vSwitch (fabric) is positioned as being more secure than a standard L2/L3 switch because it is not susceptible to the following attacks:

  • Double Encapsulation
  • Spanning Tree Floods
  • Random Frames
  • MAC Address Flooding
  • 802.1q and ISL Tagging
  • Multicast Brute Forcing

Basic stuff, but very interesting.  Did you know that the vSwitch doesn’t rely on ARP caches or CAM tables for switching/forwarding decisions?  Did you know that ESX features a built-in firewall (that stomps on IPtables?)  Did you know that the vSwitch has features built-in that limit MAC address flux and provides features such as traffic shaping and promiscuous mode for sniffing?

Many in the network and InfoSec career paths do not, and they should.

I’m going to keep in touch with Kirk and Banjot and help to ensure that this outreach is given the attention it deserves.  It will be good for VMware and very good for our community.  Virtualization is here to stay and we can’t afford to maintain this stare-down much longer.

I’m off to the show this morning to investigate some of the vendor’s like Catbird and Reflex and what they are doing with their efforts.



  1. NRC
    September 14th, 2007 at 04:53 | #1

    Great if not a little depressing write-up. Like many I am feeling this pain, I have my sysadmins tell me how its more secure (obviously what VMware are telling them) and on the other hand the network folks poo-pooing the vswitch.
    The unfortunate trend of features before security is also true of the corporate network, demands for speed and capacity without consideration for how we secure it. Trying to add additional layers to bolster the apparent lack of security is usually costly and frowned upon by those with the cash. Given the new generation of bladeshelves offering 10g uplink capability the whole cost efficiancy of the VMware offering starts to look a little tired by the time we start to add IDS/IPS etc to cope with those speeds.
    I still have to say all this is great but the VMDisk is still a flat file, it still sits on a SAN share with many other VMware images to allow tools like VMotion to run. Whilst you could argue the difficulty of gaining SAN access, via a VM or its host, the issue still exists, gain SAN access and its open day. This could be easily fixed if VMware had some way to ensure the disk image onthe SAN was what it expected, maybe a simple hash written to the local host HDD that is verifiied on boot time – whatever!
    Thanks for the info

  2. colin
    September 16th, 2007 at 16:43 | #2

    I am relatively ignorant as to VM security requirements, but have had quite a bit of experience in the real world of securing ICT assets.
    I fail to see entirely where virtualisation can provide inherent security as against stand alone installations. Surely the same requirements for protecting the host would be the same in a virtualised environment? If your guest OS is insecure, it can be compromised, and where virtualised systems are in a security enclave of the same classification, they are as prone to attack as any real world system?
    I am of the thought (and reading other scribes) the flag in the virtual world is the hypervisor or the vSwitch. Obviously from reading above, some thought has gone into protecting assets attached to the vSwitch, but how long will it be before this is circumvented, or a default configuration can be exploited?

  3. September 17th, 2007 at 18:57 | #3

    I could not agree more. Virtualization's dynamic asset relationships, reduced visibility and additional technology layers, all contribute to an increase in security configuration management and compliance assessment complexity, over non-virtualized infrastructures. The impact of this complexity is exacerbated by the relatively immature nature of enterprise experience and expertise in the evolving technologies at each level of virtualization.
    Recognition of these issues has prompted significant efforts to establish forums for emerging authoritative guidance, including the Defense Information Systems Agency's (DISA) Virtual Computing STIG and the Center for Internet Security's Virtualization Benchmarks. Kirk and Banjot, among others at VMware, have been contributors to these efforts. As the IT Audit community turns its attention to virtualization, I hope to see a more uniformly proactive response from VMware.

  4. September 17th, 2007 at 19:58 | #4

    Hi Dennis.
    Missed you @ VMWorld…it was @ your booth that I was able to sign up to contribute to the new ESX Benchmark. I also attended (and met Christine later) the STIG session.
    Glad to see that you're pushing for this, too. With you on the job it's bound to get worked out faster and better.

  5. September 18th, 2007 at 04:12 | #5

    Your blog is very interesting and useful. I'm not so professionalin this topic, but I have a blog as well about information security.
    You can visit it: http://infosecurityawareness.blogspot.com/

  6. September 26th, 2007 at 05:35 | #6

    Chris – nice post it sounds like you are a fan of Tom Barnett.

  7. GP
    October 12th, 2007 at 14:30 | #7

    You use too many big words.

  8. GP
    October 12th, 2007 at 14:35 | #8

    >Did you know that the vSwitch doesn't rely on ARP caches or CAM tables for switching/forwarding decisions?
    Because the vSwitch is in reality one big fat vHub. Turning on promisc and then sniffing the vSwitch proves that.
    >Did you know that ESX features a built-in firewall (that stomps on IPtables?)
    The ESX built-in firewall IS iptables/netfilter.

  9. October 12th, 2007 at 15:15 | #9

    1) Sorry about the big words. I'm a product of a real schooling (New Zealand)
    2) You can have multiple vSwitches on an ESX host. Enabling promiscuous mode on one vSwitch doesn't make it so on every vSwitch. I'm not sure what you're driving at. It's only a "big fat vHub" if you configure it that way…just like if you use SPAN ports on a switch…
    Furthermore, the MAC address forgery and change mitigation functions are features you'll find in a hub for obvious reasons, hubs aren't switches and don't care about associations between MAC addresses, physical ports and forwarding.
    3) What I meant is that if you modified IPtables by hand rather than use the esxcfg-firewall command, your IPtables entries get stomped on.

  1. May 7th, 2009 at 20:10 | #1