News Flash: If You Don’t Follow Suggested Security Hardening Guidelines, Bad Things Can Happen…
The virtualization security (VirtSec) FUD meter is in overdrive this week…
Part I:
So, I was at a conference a couple of weeks ago. I sat in on a lot of talks.
Some of them had demos. What amazed me about these demos is that in
many cases, in order for the attacks to work, it was disclosed that the
attack target was configured by a monkey with all defaults enabled and no
security controls in place. "…of course, if you checked this one box, the
exploit doesn’t work…" *gulp*
Part II:
We’ve observed a lot of interesting PoC attack demonstrations such as those at shows being picked
up by the press and covered in blogs and such. Many of these stories simply ham it up for the
sensational title. Some of the artistic license and innacuracies are just plain recockulous.
That’s right. There’s ridiculous, then there’s recockulous.
Example:
Here’s a by-line from an article which details the PoC attack/code that Jon Oberheide used to show
how, if you don’t follow VMware’s (and the CIS benchmark) recommendations for securing your
VMotion network, you might be susceptible to interception of traffic and bad things since — as
VMware clearly states — VMotion traffic (and machine state) is sent in the clear.
This was demonstrated at BlackHat DC and here’s how the article portrayed it:
Jon Oberheide, a researcher and PhD candidate at the
University of Michigan, is releasing a proof-of-concept tool called
Xensploit that lets an attacker take over the VM’s hypervisor and
applications, and grab sensitive data from the live VMs.
Really? Take over the hypervisor, eh? Hmmmm. That sounds super-serious! Oh, the humanity!
However, here’s how the VMTN blog rationally describes the situation in a measured response that does it better than I could:
Recently a researcher published a proof-of-concept called
Xensploit which allows an attacker to view or manipulate a VM undergoing live
migration (i.e. VMware’s VMotion) from one server to
another. This was shown to work with
both VMware’s and Xen’s version of live migration. Although impressive, this work by no means
represents any new security risk in the datacenter. It should be emphasized this proof-of-concept
does NOT “take over the hypervisor” nor present
unencrypted traffic as a vulnerability needing patching, as some news
reports incorrectly assert. Rather, it a
reminder of how an already-compromised network, if left unchecked, could be
used to stage additional severe attacks in any environment, virtual or
physical. …Encryption of all data-in-transit is certainly one well-understood mitigation
for man-in-the-middle attacks. But the fact
that plenty of data flows unencrypted within the enterprise – indeed perhaps
the majority of data – suggests that there are other adequate mitigations. Unencrypted VMotion traffic is not a flaw,
but allowing VMotion to occur on a compromised network can be. So this is a good time to re-emphasize hardening best practices for VMware
Infrastructure and what benefit they serve in this scenario.
I’m going to give you one guess as to why this traffic is unencrypted…see if you can guess right in the comments.
Now, I will concede that this sort of thing represents a new risk in the datacenter if you happen to not pay attention to what you’re doing, but I think Jon’s PoC is a great example of substantiating why you should follow both common sense, security hardening recommendations and NOT BELIEVE EVERYTHING YOU READ.
If you don’t stand for something, you’ll fall for anything.
/Hoff
Lemme guess, performance?
That would be my guess as well…the overhead from the encrypt/decrypt would come at the cost of the "wow" factor of moving a guest to another host w/ nothing more than a single ping timeout.
The old 'hard crunchy shell, soft insides' comment springs to mind….
No need to encrypt internally, we've gotta firewall…….
Sure, it might be a chewy center, but if someone is in a position to attack your VM infrastructure by MITMing your VMotioning (wahoo, verbizing words!) images, you have some other serious problems. That was my team's first impression on this attack (and we have 95% of our production environment on VM).
The only thread that can truly only leverage this attack without already preying on other poorer security measures would be the insider. Yes, that admin who is managing your storage devices. And let's just say he has far greater access to do things than some exotic VMotion MITM research.
VMotion has two things that make this POC "exploit" extremely exotic.
1) VMotion doesn't just do its thing over a switch or traditional network or over the Internet. It does so in a storage arrangement, for instance inside a SAN or iSCSI setup. Sure there are attacks against those things, but that's not necessarily something VM makers should be concerned about.
2) VMotion requires the source and destination discs be in sync, and it will track and rectify changes. Not sure what would happen if you changed the disk state on one side, but I think it might corrupt the VMotion activity.
An analogy of this might be saying, gosh, data inside my CPU is passed around unencrypted and attackers might be able to see or adjust that! The CPU must be encrypted when it talks to itself, otherwise WoW hackers can get my account! Ok, that's stretching it, but any little bit helps…
I'm going to rant a bit more about this. 🙂
Encrypting that VMotion traffic, I can imagine, would have a pretty significant impact on the process. And it's really just not necessary.
Research is one thing. It can often take the form of exotic attacks that require special circumstances, situations, or equipment. Sometimes they are useful to places like government spies or defense secret protectors. A good example is the FDE-beating DRAM freezing attack. Almost zero risk to most people and almost zero chance anyone will even bother. But it could still be very useful to high level espionage efforts, or something to be aware of for those trying to stop those efforts.
I don't think this VMotion issue fits into even that scenario. I think we'll only ever see/hear about this in a lab setting, and nothing more. Not even in the dark hallways of the NSA…
On the other hand, research like this can lead to other, more important vulnerabilities in VMs or other things, or possibly just open new avenues of thought. So, one never knows, and I think the researcher is justified in publishing it. It's just the media is not very justified in hyping it.
The shared folders issue that accompanied the VMotion issue in most articles is far more important.
can someone refer me to a reliable resource where the perfomance
impact of data encryption like TLS on local networks is discussed? E.g. the difference between TLS session setup overhead compared with costs of the continous encryption of the data stream?
t.i.a
Hey guys! Jon Oberheide here. A buddy pointed out this post so I thought I'd follow up.
Yep, some of the media coverage was somewhat inaccurate. When I spoke to some of the press, I really wanted to emphasize the awareness of the issue but not hype it up or pretend like it's some doomsday scenario by any means.
In particular, with regards to "taking over the hypervisor", we showed how an in-progress migration could be manipulated by a MITM attacker in order to exploit vulnerabilities in the migration routines of the Xen dom0 daemon. So to clarify, the MITM manipulation of VM state/memory applies to both Xen and VMware migrations, but the remote hypervisor exploits only apply to Xen <= 3.1.0.
I'm not sure when the BlackHat materials will be posted but if you take a look at the presentation, the underlying theme is that we've been continually breaking down isolations barriers that provide security in exchange for new functionality. We've gone from physical machines where state is protected by hardware mechanisms such as the MMU, to virtual machines where the machine state is protected by the isolation boundaries of a software layer such as the VMM/hypervisor (which, as we've seen, often fails in its isolation), to exposing the machine state to the local network and any attackers who happen to be snooping. This transition isn't necessarily a bad thing…we just have to be aware of the security concerns it presents.
Re: LonerVamp, the payload of a VMotion migration is not the disk state but the working memory of the VM. Therefore, it doesn't matter if source and destination disk state is if the VM's kernel is owned or gets VMBR'ed during migration.
Re: Hoff, I'm very happy to see VMware approached the issue on their blog. I think their statement of "plenty of data flows unencrypted" is kind of a cop-out though, as there is an important distinction between a compromise of data integrity and a compromise of host/VM/code integrity.
Anywho, my primary goal was to raise awareness for the numerous organizations deploying such technology who may not have followed best practices or were not aware of the associated risks. Hopefully that has happened at least to some extent.
Any questions/comments appreciated! 🙂
@Jon:
Thanks very much for your comments and your work, Jon. I think the awareness is a good thing, but the press just needs to temper their enthusiasm a tad and make sure they accurately convey not just sensationalist titles.
That being said, in review I *do* have to agree with you that while there are "by design" unencrypted flows, there is plenty of technology available to mitigate this issue (like using SSL/TLS accelerators that could offload the encryption at line rate)
Also, I re-read my introduction and I didn't mean to come off like I was suggesting there was no merit to your work because the attack vector was in an unprotected state, but rather that the press didn't really mention it…
Thanks again, Jon.
/Hoff
Suggestions for why it is not encrypted:
1. They re-used the existing transport layer code developed for vmkernel traffic
2. The developer was not familiar with openssl, opentls or any other _secure_ transport mechanism
3. It's cool/useful to see the vm-state in-transit, encryption would make wireshark useless as a debugging tool
4. Encryption and security were not part of the marketing specification, design specification or functional specification
Woah thank you for most likely being the truth about why traffic is unencrypted Michael Berman. It makes a lot more sense to me now!