Home > Risk Assessment, Risk Management, Virtualization > Risky Business — The Next Audit Cycle: Bellweather Test for Critical Production Virtualized Infrastructure

Risky Business — The Next Audit Cycle: Bellweather Test for Critical Production Virtualized Infrastructure

I believe it’s fair to suggest that thus far, the adoption of virtualized infrastructure has been driven largely by consolidation and cost reduction.

In most cases the initial targets for consolidation through virtualization have focused on development environments, internally-facing infrastructure and non-critical application stacks and services.

Up until six months ago, my research indicated that most larger companies were not yet at the point where either critical applications/databases or those that were externally-facing were candidates for virtualization. 

As the virtualization platforms mature, the management and mobility functionality provides leveraged impovement over physical non-virtualized counterparts, and the capabilities to provide for resilient services emerge,  there is mounting pressure to expand virtualization efforts to include these remaining services/functions. 

With cost-reduction and availability improvements becoming more visible, companies are starting to tip-toe down the path of evaluating virtualizing everything else including these critical application stacks, databases and externally-facing clusters that have long depended on physical infrastructure enhancements to ensure availability and resiliency.

In these "legacy" environments, the HA capabilities are often provided by software-based clustering capabilities in the operating systems, applications or via the network thanks to load balancers and the like.  Each of these solutions sets are managed by different teams.  There’s a lot of complexity in making it all appear simple, secure and available.

This raises some very interesting questions that focus on assessing
risk in these environments in which duties and responsibilities are
largely segmented and well-defined versus their prospective virtualized counterparts where the opposite is true.

If companies begin to virtualize
and consolidate the applications, storage, servers, networking, security and high-availability
capabilities into the virtualization platforms, where does the buck
stop in terms of troubleshooting or assurance?  How does one assess risk?  How do we demonstrate compliance and
security when "all the eggs are in one basket?"

I don’t think it’s accurate to suggest that the lack of mature security
solutions has stalled the adoption of virtualization across the board,
but I do think that as companies evaluate virtualization candidacy,
security has been a difficult-to-quantify speed bump that has been
danced around. 

We’ve basically been playing a waiting game.  The debate over virtualization and the
inability to gain consensus in the increase/decrease of risk posture has left us at the point where we
have taken the low-hanging fruit that is either non-critical or has
resiliency built in, and simply consolidated it.  But now we’re at a crossroads as virtualization phase 2 has begun.

It’s time to put up or shut down…

Over the last year since my panel on virtualization security at RSA, I’ve been asking the same question in customer engagements and briefings:

How many of you have been audited by either internal or external governance organizations against critical virtualized infrastructure that are in production roles and/or externally facing? 

A year ago, nobody raised their hands.  I wonder what it will look like this year?

If IT and Security professionals can’t agree on the relative "security" or risk increase/decrease that virtualization brings, what position do you think that leaves the auditors in?  They are basically going to measure relative compliance to guidelines prescribed by governance and regulatory requirements.  Taken quite literally, many production environments featuring virtualized production components would not pass an audit.  PCI/DSS comes to mind.

In virtualized environments we’ve lost visiblity, we’ve lost separation of duties, we’ve lost the inherent simplicity that functions spread over physical entities provides.  Existing controls and processes get us only so far and the technology crutches we used to be able to depend on are buckling when we add the V-word to the mix.

We’ve seen technology initiatives such as VMware’s VMsafe that are still 9-12 months out that will help gain back some purchase in some of these areas, but how does one address these issues with auditors today?

I’m looking forward to the answer to this question at RSA this year to evaluate how companies are dealing with GRC (governance, risk and compliance) audits in complex critical production environments.


  1. March 24th, 2008 at 18:56 | #1

    "In virtualized environments we've lost visiblity, we've lost separation of duties, we've lost the inherent simplicity that functions spread over physical entities provides."
    I'd agreed with that. Having physical, logical and administrative separation between at least some of the components of the technology stack (firewall, network, load balancer, server, database, etc.) makes 'separation of duties' fairly straightforward. Our auditors tend to like that.
    If the entire stack were to fall under a single administrative boundary and even worse, under a single GUI interface that hides all the details, you'd have a hard time maintaining that separation. The result would tend toward not only troublesome audits, but also a higher probability of a having a system administrator error caused security incident.
    I always make sure my network people can't deploy servers & my server people can't enable switch ports, add services to the load balancer or add DNS records, for that very reason.

  2. March 24th, 2008 at 19:27 | #2

    Feels like "worlds in collision" to me. Security pros unfamilar with virtualization and ops pros unfamiliar with security.

  3. March 24th, 2008 at 20:39 | #3

    @Michael: So, in the non-virtualized world, I'd suggest you can get away with that forced SoD…in the virtualization realm, good luck. Exactly what you described is in many cases what you get with virtualized infrastructure.
    Now, this will take a couple of different forms over the next couple of years as we see the toolsets mature and some of the functions become absorbed into different spots, but it's going to challenge the we operationalize IT and Security. It already has.

  1. No trackbacks yet.