Home > Virtualization > Oh Noes: We Can’t Monitor/Protect Against Intra-VM Traffic!

Oh Noes: We Can’t Monitor/Protect Against Intra-VM Traffic!

I got a press release in my inbox this morning that made me cringe.  It came from a vendor who produces a "purpose-built virtual firewall."

The press release details a customer case study that I found typical of how security solutions are being marketed in the virtualization space today, which again is really more about visibility than pure "security" and preys mostly on poor planning and fundamental issues stemming from treating "security" like a band-aid instead of an element of enterprise architecture.

When we start to cross the streams as to the realities of virtualization, the security implications thereof and making promises to solve problems with products which may or may not be deserving of investment given an assessment of risk, especially in today's trying economic climate, it makes me cranky.

I'm just tiring of the mixing of metaphors in the marketing of these "solutions."

I was specifically annoyed by a couple of statements in the press release and since I haven't had my coffee, I thought I'd point out a few to further underscore what I present in my Four Horsemen presentation regarding where we are in the solution continuum today.

To wit:

[Customer] has selected the [Vendor's] virtual firewall to secure its virtual environment and mitigate an attack before it could hit their network. 

Given the fact that to get to a VM you generally have to (1) utilize the physical network and (2) transit the vSwitch in the VMM, the reality is that an attack has already "hit their network" long before it gets to the VM or the virtual firewall, at least given today's available offerings.  There is no magic security fairy dust that will mitigate an attack presciently.

If you put VM's into production that are already infected, you have other problems to solve…

After moving our production applications to a virtualized environment we realized that we lacked security; I had no visibility into what was going on between VMs and a virtual attack could take down our network,” said [Customer.]  “We sought the same level of security for our virtual environment that we had with our physical network.”

This indicates a lack of proper risk management and planning on the part of the [Customer.]  Further, it underscores an example I use in the Four Horsemen which concerns which tools in a multi-server physical deployment did the [Customer] use to monitor/protect in-line traffic between these physical machines? 

The {Customer] must have done this since the press release suggests they demand the "…same level of security for [their] virtual environment that [they] had with [their] physical network."

Did the [Customer] have each physical server on it's own VLAN/subnet, isolated with firewalls?  Did he SPAN every single port to an IDS/IPS?
 If not, what's the difference here?  The Hypervisor?  What protection mechanisms has the fancy virtual firewall put in place to protect it?  None.

[Customer] was increasingly concerned about the risks of virtual networks, which range from security policy violations such as mixing trusted and un-trusted systems to malware exploits that can propagate undetected within a virtual network. 

Based upon the second paragraph above where the [Customer] admitted they put their virtualized environment into production without visibility or security, they clearly weren't that concerned with the risks.

A large amount of data center network traffic was moving between VMs and [Customer] had no visibility or control over the communication on the virtual network.

So if there were no security or visibility tools in place, how was it determined that traffic was moving between VM's?

Does this mean that all the customer's VM's were in a single VLAN and not segmented? If not and vSwitch configurations via port groups and VLANs were configured around VM criticality, role or function, then they certainly had some insight into what was moving between VM's and the "data center," right?

I must be confused. 

[Customer's] traditional network security tools could not monitor, analyze or troubleshoot inter-VM traffic because communications between VMs on the same physical host never touch the traditional network

Assuming that the VM's weren't in a single VLAN/portgroup on a single vSwitch and instead were segmented via VLANs/subnets, then the only way to get traffic from VLAN/IP Subnet A to VLAN/IP Subnet B (and thus VM A to VM B in these VLAN's) is though a layer 3 routing process which generally means traffic exits the virtual network and hits the physical network…where said "traditional security tools" could see it.

Of course, this doesn't help intra-VM traffic on the same portgroup/VLAN/vSwitch, but that's not what they pointed to above, but assuming they don't look at inter-machine traffic in their physical network on the same VLAN, again I ask what's the difference?

VMs were able to communicate with each other without observation or policy-based inspection and filtering, which left them highly vulnerable to malicious exploits. Additionally, worms and viruses could further spread among physical hosts via unintentional VMotion of an infected VM.

Back to my point above about how the [Customer] monitored traffic between physical hosts…if you don't do it in physical environments, why the fret in the virtual?

Oh and "unintentional VMotion!?" ZOMG!  For a VM to be "infected," excluding direct physical access, wouldn't the threat vector be the network in the first place?  

The [Vendor] virtual firewall was specifically created to mitigate the risks of virtual networks, while maintaining the ROI of virtualization.

What "risks of virtual networks" does this product mitigate in the absence of vulnerability or clearly defined threats that aren't faced in the physical realm?  Let me tell you.  It goes back to the very valid claim that you get better visibility given the integration with the virtualization platform configuration managers to call attention to when CHANGE occurs.

This is the real value of products like this from [Vendor.]  I
n the long run, the big boys who make mature firewalls and IPS products will get to harness API's like VMsafe and combined with the compartmentalization and segmentation capabilities of vShield Zones leaves a very short runway for products like this.

I'm not suggesting that products like this from [Vendor] don't offer value and solve an immediate pain point. I'd even consider deploying them to solve very specific problems I might have, but then again, I know what problem I'd be trying to solve. ROI?  Oy.

However, unlike the picture painted of the [Customer] above, I plan a little better and understand the impact virtualization has on my security posture and how that factors into my assessment and management of risk BEFORE I put it into production.  You should too.


{Ed: I use 'intra-' instead of 'inter-' to reflect the "internal" passing of traffic between VM's using the vSwitch. Should traffic exit the vSwitch/host and hit the network as part of interchange between two VM's, I'd count this as 'inter-" VM traffic.}

Categories: Virtualization Tags:
  1. March 10th, 2009 at 07:19 | #1

    Definitely reminds me of this:
    That being said, it would be nice for the virtual switch to provide a monitoring port, so my virtual snortbox could watch all the fun

  2. March 11th, 2009 at 06:04 | #2

    Re: ''For a VM to be "infected," excluding direct physical access, wouldn't the threat vector be the network in the first place?''
    The exception here is when you put a VM on a DMZ VLAN, which I'm still not sure is a great idea, although it is advertised as one of the uses for VMs.

  3. March 12th, 2009 at 19:08 | #3

    The virtual switch can provide a form of a monitoring port with the use of a promiscuous mode port group so your virtual snortbox can watch all the fun!

  4. Justin Henderson
    August 24th, 2014 at 10:19 | #4

    I definitely think you have some strong points that many individuals are overlooking. However, there are some cases were we’ve forced VMs to reside on the same vsphere host to force inter-vm traffic as it allows speeds greater than 10 gbps without the requirement for any special cabling or NIC cards. In these cases without using a distributed switch to drop traffic to a physical switch or putting an IDS on top of the vSphere host all traffic between these VMs goes undetected.

    Granted, at this high speed you are most likely using a BPF filter to exclude the ports sending this high volume traffic you still may want to see things like smb or other common ports between these systems. Regardless, without something in place all traffic becomes hidden and it becomes difficult to see lateral movements specifically between these two systems. Really I’m talking less than 1% of the overall architecture in this design so you could have all physical IDS sensors and a single virtual IDS sensor and have full visibility.

    The only point I’m trying to make is that in some cases there are some blind spots. However, just like mentioned in this article, if you don’t have everything on the same layer 2 network their shouldn’t be as many blind spots as marketed. This article is spot on I just want to mention that dependent on your overall architecture you may need to consider the few blind spots that could be inter-vm traffic.

  1. No trackbacks yet.