All Your Virtualized PCI Compliance Are Belong To Us…
Another interesting example I use in my VirtSec presentations when discussing the challenges of what I describe as Phase 2 of virtualization — virtualizing critical applications and things like Internet-facing infrastructure in DMZ’s — is the notion of compliance failures based on existing and upcoming revisions to regulatory requirements.
Specifically, I use PCI/DSS to illustrate that in many cases were one to take a highly-segmented and stratified "defense-in-depth" architecture that is today "PCI compliant" and virtualize it given presently available options, you’d likely find yourself out of compliance given the current state of technology solutions and auditing standards used to assess against.
Then again, you might just pass with flying colors while being totally insecure.
Here’s a fantastic example from Eric Siebert over at the TechTarget Virtualization blog. Check this out, it’s a doozie!
Having just survived another annual PCI compliance audit, I was again surprised that the strict standards for securing servers that must be followed contain nothing specific concerning virtual hosts and networks. Our auditor focused on guest virtual machines (VMs), ensuring they had up-to-date patches, locked-down security settings and current anti-virus definitions. But ironically, the host server that the virtual machines were running on went completely ignored. If the host server was compromised, it wouldn’t matter how secure the VMs were because they could be easily accessed. Host servers should always be securely locked down to protect the VMs which are running on them.
It seems that much of the IT industry has yet to react to the virtualization trend, having been slow in changing procedures to adjust to some of the unconventional concepts that virtualization introduces. When I told our auditor that the servers were virtual, the only thing he wanted to see was some documentation stating that the remote console sessions to the VMs were secure. It’s probably just a matter of time before specific requirements for virtual servers are introduced. In fact, a recent webinar takes up this issue of whether or not virtualized servers can be considered compliant, addressing section 2.2.1 of the PCI DSS which states, “Implement only one primary function per server”; that is to say, web servers, database servers and DNS should be implemented on separate servers. Virtual servers typically have many functions running on a single physical server, which would make them noncompliant.
So let’s assume that what Eric talks about in section 2.2.1 of PCI/DSS holds true, that basically means two things: (1) PCI/DSS intimates that virtualization cannot provide the same level of security as non-virtualized infrastructure and (2) you won’t be able to virtualize infrastructure governed by PCI/DSS if you expect to be compliant.
Now, this goes toward the stuff Mogull and I were talking about in terms of assessing risk and using the notion of "zone defense" for asset segmentation in virtualized infrastructure.
Here’s a snippet from my VirtSec preso on the point:
Further, as I mentioned in my post titled "Risky Business — The Next Audit Cycle: Bellweather Test for Critical Production Virtualized Infrastructure," this next audit cycle is going to be interesting for many companies…
Yippeee!
/Hoff
Extend those thoughts just a bit, on the thread of 'loss of sensitive data', and include not only the hosting server, but also the clones and copies of virtualized servers that tend to be laying around the infrastructure. In some cases, such as DR or business continuity, we want copies of VM's stored off line or of site to achieve desired RPO/RTO goals. In other cases, the copies are left laying around for the convenience of the server manager. Those copies are a potential security problem, just like loose backup tapes. Releasing a copy of sensitive data from a cloned VM that is stored off to the side 'just in case' generates the same headlines as releasing the original, but because the copies are scattered about uncatalogued on various storage devices, the probability of accidental release is higher.
Ding! – I now have to secure every clone of every VM just the same as the original host, and to do that I need security and auditing and change control on the spot where I store the 'spare' VM's that is at least equal to the security requirements of the most sensitive host copy that I store.
So if I want to hack a datacenter and grab interesting data from a server, I have the virtualized server containing the data as a target, the host server that is hosting the live virtual server as a target, the tape backups (or backup-to-disk-pool) as a target, and the big cheap NAS device where the sys admin stores copies of the VM's off to the side 'just in case'. I'd aim for the latter first.
Hmmm.. will the PCI auditor ask me where all my clones are stored?
@michael:
Better yet (and in one of my slides) is the fact that by design and roadmap progression these files are almost all stored on centralized SAN clusters.
I've asked many times how many of the security pros in my audiences are storage security experts — nobody has yet raised their hand.
You get access to the SAN — as you point out — not only do you get the OS and the applications, you get the data that goes with it…all in one nice tidy bundle!
I'll be posting that slide shortly.
/Hoff
OK – I was thinking not of access to the SAN itself, but rather access to the do-it-all 'utility' servers that system admins have in the back corner and use to store their tool kits, scripts and ummm… spare VM clones. You know, the servers that nobody knows the real name of, that don't show up as a part of any application & that the security and audit dudes don't even know exist. Those servers. They happen to be SAN attached, and happen to have a gold mine of interesting bits on them.
But now that you bring up the SAN itself, the lack of overlap between security people, who tend to be mostly network people, and the storage teams, who probably have never seen a hacked server, is a big issue. Unfortunately there is no overlap.
When you post you SAN slide, I'll have more comments. 😉
Christofer,
This raises some points that I have been trying to raise for some time with various organisations that are virtualising at a great rate of knots – how does your virtualisation strategy affect your regulatory and compliance goals?
This is invariably met with show gazing, various mumbling sounds and eventual shuffling away. The amazing thing is that this exact scenario is played out when asking other QSA's about how to assess security posture in a virtualised environment.
The PCI consortium have not specifically addressed virtualisation as it is a concept that the card processors themselves have not embraced in any way. I believe the approach to a stratified architecture is almost completely lost when mixing assets of multiple classifications under a lone hypervisor.
I think some of the compliance goals can be met if systems of the same classification can live on the same hypervisor, but as one of your commentors has pointed out, the back end data storage again has the same issues. Multiple classifications of data residing on the one storage medium, to my mind, would be an immediate non compliance.
I was recently asked about compliance and PCI/ACSI 33 with regards virtualisation. I hate to admit it, but my response was "if you are attempting to meet something as stringent as ACSI 33, just don't virtualise".
-colin.
Could one intepret the PCI rule to say the VM host server serves the one function of hosting VMs? I guess this might contrast with the answer to the question of whether the PCI-related web server can host multiple web sites or a website that does multiple functions? This is an innocent question; I don't know the answer to either of those.
I posted other ancillary thoughts on my own blog. 🙂
Well, the basic rule of PCI is that any part of the network that credit card info goes over or is stored on is in-scope. Network segmentation on to separate, secure VLANs is the best way to reduce scope so therefore it stands to reason that if any VM on a host server contains CC info then said host servers (and all VMs on it) become in-scope.
I think I'm too close to the issue as treating each host server as its own "VLAN" seems like a no-brainer to me. It's clearly not a best-practice to host VMs that run across multiple secure zones yet…it seems to be fairly prevalent.
Is it simply a matter of convenience vs security? i.e. companies try to consolidate as much as possible even if it means crossing zones? Sounds pretty risky with all the unknowns regarding virtualization security.
It's a matter of convenience/cost vs security for us. We are tinkering with our need for a DMZ so that we have less zones to keep VM infrastructure in. That or keep a DMZ and fully separate our zones. The latter is expensive, though.
I think we'll end up going expensive but separate, but we do have to check out our options since it also means more work for us as moving VMs between zones will become less easy.
I Drink Your PCI Virtualization Milkshake!
PCI compliance, virtualization and security all can work together just like a Dairy Queen Blizzard if a company takes the right steps.. Its smooth and tasty (for your business)! I was reading Chris Hoffs and David Taylors re…
Is an ESX Host a server?
It should be considered similar to the chassis holding a bunch of blade servers.
These have management ports on separate networks, with LDAP authentication over security protocols like ssh and ssl.
And why not treat them as a hybrid device with different network switches, storage controllers, etc?
Vmware has recently removed the word "Server" from after the ESX product name…
It's not a server, it's a hypervisor.
It's not a server, it's a switch.
By defining what a server is and is not a PCI Audit should be pretty straight forward. http://www.vmware.com/products/vi/esx/
@Iben:
The answers to your questions/suppositions are quite simple:
"It all depends upon the auditor."
Most of the folks I've spoken to recently are essentially counting upon the ignorance of the auditors and the general confusion regarding terminology and technology to glide by at this point.
Server/blade/hypervisor/switch … it's all fun and games until someone looses a (PC)I… 😉
"As long as I put in place the same host controls I do in a physical environment and not tell the auditor it's virtualized, it's all good and what they don't know, won't hurt me."
Sad but true.
/Hoff
I fully agree with the splitting of virtualization hosts into different segments, having different physical servers offering different zones of access. But in the real world it is not possible to have 1:1 matching for all possible zones (if you do that you almost void the reason for using virtualization).
However saying that virtualization is mutually exclusive for PCI or other compliance ignores the fact that it's been accepted and used for years in the mainframe world, and they use shared storage which is just segmented & controlled properly (yes do not have 1 nfs export accessible to all zones:)
But when someone says it is not possible or 'in the future there are going to be hacks so your doing insecure things and should not do it' has to remember that with any business there are risks (look at stock traders or auto industry workers). Along with those risks are the decisions of how big is the risk, how much to secure it and how much does the business make. If it costs more to secure it than to operate it's not fiscally worth operating, however if the risks are there but the cost of a loss (fees, customers, etc) are minimal to the size of the business many businesses will go ahead with the operational risk.
What I would like to hear instead of all these nay-sayers is a real disussion on how we can get the modern "PC" virtualization to get the tools and security mechanisms in place to get virtualization to an acceptable point as where mainframe lpars are. Lets discuss how we can not only have configuration management, but also integrity monitoring and anomolous activity detection (in both disk, process and network usage) across not only the virtual machines but also the entire virtualizatoin cloud.
This is a battle with those who work very hard to not work at all (thieves/crooks/hackers/script kiddies/etc) with those who are trying to help business move forward.
As for auditors not knowing where to look or what to do, first be honest with your auditor. Are you getting audited to confirm compliance or are you actually concerned with your customer data and business reputation. If your auditor does not have the technical experience to do an audit properly, I would suggest engaging another (there are quite a few:). If you have concerns that they are not being detailed enough, then do what you can do ensure that you have controls to cover all of your security risks and that they are clearly documented and available to your auditor so it can be documented and on-file if there is ever a compromise or any question as to your compliance.
But most of all, remember compliance is not just when the auditor shows up to validate your environment. Compliance is an on-going effort by everyone involved in your business; from HR, Legal, Marketing, Sales to IT & Security. If someone in HR doesn't do a background check, you are not compliant. If someone in marketing takes database data to create mailing lists and it contains card data, you may not be compliant. If you do not monitor your system logs _daily_, you are not compliant (and remember, system logs include all PCI applications, not just the servers syslog!).
Good luck to you all and for those who spend their entire life just complaining about what insecurities something has or may have but offer no solution; I appreciate the research you do and the publishing / speaking you do to get the word out, but what good is the buy who cried wolf who doesn't stand with his family to defend against it.
Regards
David M. Zendzian
DMZ
<blockquote cite="#commentbody-4158">
David M. Zendzian :
the buy who cried wolf
Woops, was suppose to be "boy who cried wolf".
@David M. Zendzian
So I'll address your jab at the end (corrected) wherein you said:
"Good luck to you all and for those who spend their entire life just complaining about what insecurities something has or may have but offer no solution; I appreciate the research you do and the publishing / speaking you do to get the word out, but what good is the boy who cried wolf who doesn’t stand with his family to defend against it."
<< You've either not heard me speak, don't read the slides/conclusions, or choose not to understand that in many cases using the stick method with the people who write the rules is sometimes the most effective way to bring about change.
If you care to characterize me as someone who simply "cries wolf," then you're being disingenuous when you suggest you appreciate my work/writing/publishing. Further, you don't know me at all.
/Hoff
I apologize for classifying you as such and will look out for your talks/writings/etc to get to better understand what you contribute to the world 🙂
That is true, I should not have made an overall generalization about you specifically but you have to admit that most in the sploit/hacker (kiddie & media whore) world would rather get their name out there as hacking a system than actually trying to find real solutions for real people (and not blaiming microsoft/google/everyone else for not fixing their apps).
One of my real issues is at all of these security conferences, blogs, etc the majority (not all but a huge piece of them) are focused on the big stick, the latest 0day, the company that isn't fixing something and there is little discussion and work on ways for the smaller company (someone who can't hire someone like you or me or a team of us full time). I have been pushing talks at various conferences trying to present ways of integrating inexpensive (off the shelf or opensource) tools into managable security infrastructure tools for small or even large companies. So far it hasn't met the "0day" track requirements and is not even getting much discussion but I can't be the complainer myself if I don't get out and try to do what i'm complaining others don't 🙂
@David M. Zendzian
I'm particularly sensitive to the "Chicken Little" syndrome also, but context is important, so let me give a bit of color just to punctuate the discussion.
A great deal of what I do on a daily basis I don't write about here but I think I have a reasonable pulse on what's going on given who I work for and who my customers are…including participating on the PCI SSC virtualization SIG.
In terms of giving back, I lead the CloudAudit working group, am a co-founder of the Cloud Security Alliance & technical advisor, founder of HacKid (kids-focused security conference,) co-organizer of BeanSec (Greater Boston Area Security community group) and spend a tremendous amount of time raising what I feel is a rational set of problems and call for solutions.
I appreciate the fact that you replied and I'm sorry if I sounded defensive, but hopefully you get a picture that is more realistic.
/Hoff
I have gone back & read a lot of your older posts and I think we both agree in many ways. And I wasn't trying to comment about you, your background or your contributions (just because I comment on your post does not mean I'm commenting about you, just your content:)
And you don't need to blow your horn, it is very apparent you are one of those that participates in everything you are able and that you are focusing in the world of virtualization now. Note I did not say cloud because I personally do not believe that many people understand the difference and I'm sure we could have a great discussion over some good sipping whiskey some day and while you know the difference until people are using clouds different from old infrastructure designs it's just a new tool for those piloting the old ships of business (insert image of monty python and the pirates of accounting taking over the business world)
I've been paying attention to what is coming out of the cloud security alliance, but have not been able to contribute much myself. And it's good to know that both you & I are on the SSC virt SIG (doug & I just finished working on the mapping doc sections 10/11).
Please don't consider my posts as if they are addressing you directly in as much as trying to stimulate discussion on the concepts you bring up. Your posts help motivate me and sometimes typed words do not express the appreciation that the conversation brings.
And as for realistic pictures, I've long been a fan of Einstein who said that sometimes you must rip a hole in reality to solve your problems. For the reality of things, I am a huge fan of virtualization and grid computing; but not so much a fan of marketing speak and people running out to get VC $$ to startup businesses if they have a solid business idea, make a business don't get other peoples $ to make a company.
IMOHO "clouds" are not quite what most people believe they are. I keep running into sysad friends who have to keep explaining to their management that they can't "outsource to the cloud" so they can then get rid of their network/security & IT teams since the "cloud just makes it work". If only it was so simple.
The only thing I see that's really happened in the last 3 years is that one of the holy grails of data-center operations (note only 1 of many), automated system provisioning & configuration management has been almost solved by those who now have automated cloud & hybrid solutions. Stepping from that level of automation into 'cloud' applications still takes development, security, it & networking along with the business to establish what exactly are they trying to do and what is really different from what they were doing.
In the end, if people really look at the costs associated with running their business, either in public, private clouds or closed gardens there are obvious and hidden expenses and none of the "new" or "old" methods really are any different from what they had before, and the costs are really not much different if there are controls you need to have on your data and application. The only real savings I've seen is those that have large data-centers are now able to look at consolidating thousands of servers and save power/water/ac/etc, but for someone with only a web shopping cart there isn't too much of an overall difference.
Thanks for the dialog.
David
-dmz