Rational Survivability

“Cloud Security Will  Supplant Patching…”

What a sexy-sounding claim in this Network World piece which is titled with the opposite suggestion from the title of my blog post.  We will still need patching.  I agree, however, that how it’s delivered needs to change.

Before we get to the issues I have, I do want to point out that the article — despite it’s title —  is focused on the newest release of Qualys’ Laws of Vulnerability 2.0 report (pdf,) which is the latest version of the Half Lives of Vulnerability study that my friend Gerhardt Eschelbeck started some years ago.

In the report, the new author, Qualys’ current CTO Wolfgang Kandek, delivers a really disappointing statistic:

In five years, the average time taken by companies to patch vulnerabilities had decreased by only one day, from 60 days to 59 days, at a time when the number of flaws and the speed at which they are being exploited has accelerated from weeks to, in some cases, days. During the same period, the number of IP scanned on an anonymous basis by the company from its customer base had increased from 3 million to a statistically significant 80 million, with the number of vulnerabilities uncovered rocketing from 3 million to 680 million. Of the latter, 72 million were rated by Qualys as being of ‘critical’ severity.

That lack of progress is sobering, right? So far I’m intrigued, but then that article goes off the reservation by quoting Wolfgang as saying:

Taken together, the statistics suggested that a new solution would be needed in order to make further improvement with the only likely candidate on the horizon being cloud computing. “We believe that cloud security providers can be held to a higher standard in terms of security,” said Kandek. “Cloud vendors can come in and do a much better job.”  Unlike corporate admins for whom patching was a sometimes complex burden, in a cloud environment, patching applications would be more technically predictable – the small risk of ‘breaking’ an application after patching it would be nearly removed, he said.

Qualys has its head up its SaaS.  I mean that in the most polite of ways… 😉

Let me make a couple of important observations on the heels of those I’ve already made and an excellent one Lori MacVittie made today in here post titled “The Real Meaning Of Cloud Security Revealed:”

1. I’d like a better definition of the context of “patching applications.”  I don’t know whether Kandek mean applications in an enterprise or those hosted by a Cloud Provider or both?
2. There’s a difference between providing security services via the Cloud versus securing Cloud and its application/data.  The quotes above mix the issues.  A “Cloud Security” provider like Qualys can absolutely provide excellent solutions to many of the problems we have today associated with point product deployments of security functions across the enterprise. Anti-spam and vulnerability management are excellent examples.  What that does not mean is that the applications that run in an enterprise can be delivered and deployed more “securely” thanks to the efforts of the same providers.
3. To that point, the Cloud is not all SaaS-based.  Not every application is going to be or can be moved to a SaaS.  Patching legacy applications (or hosting them for that matter) can be extremely difficult.  Virtualization certainly comes into play here, but by definition, that’s an IaaS/PaaS opportunity, not a SaaS one.
4. While SaaS providers who do “own the entire stack” are in a better position through consolidated multi-tenancy to transfer the responsibility of patching “their” infrastructure and application(s) on your behalf, it doesn’t really mean they do it any better on an application-by-application basis.  If a SaaS provider only has 1-2 apps to manage (with lots of customers) versus an enterprise with hundreds (and lost of customers,) the “quality” measurements as it relates to management of defect (from any perspective) would likely look better were you the competent SaaS vendor mentioned in this article.  You can see my point here.
5. If you add in PaaS and IaaS as opposed to simply SaaS (as managed by a third party.) then the statement that “…patching applications would be more technically predictable – the small risk of ‘breaking’ an application after patching it would be nearly removed” is false.

It’s really, really important to compare apples to apples here. Qualys is a fantastic company with a visionary leader in Phillipe Courtot.  I was an early adopter of his SaaS service.  I was on his Customer Advisory Board.  However, as I pointed out to him at the Jericho event where I was a panelist, delivering a security function via the Cloud is not the same thing as securing it and SaaS is merely one piece of the puzzle.

I wrote a couple of other blogs about this topic:

• Patching the Cloud
• What People Really Mean When They Say The Cloud Is More Secure
• Cloud Providers Are Better At Securing Your Data Than You Are…
• Reprise: On-Demand SaaS Vendors Able to Secure Assets Better than Customers?

/Hoff

You may have heard of it.

It’s quite possibly the fundamental underpinning of the entire security industry; a veritable life-source for over-worked security folk.  It’s apparently critical to the success of Cloud, as you can see from the picture to the right.

What is this mystical thing?  The Hoffacchino. Or, Hoffaccino, if you prefer.

You may hear it muttered and wonder “Just what the hell is a Hoffac[h]ino, anyway?”

Go to your local Starbucks and order the following:

The Hoffacc[h]ino

Venti Starbucks Doubleshot on ice. 6 shots, 3 Splenda (can sub sugar,) no classic (syrup,) breve (that’s 1/2 and 1/2 for those of you who don’t speak Strabucktalian.)

I cannot take responsibility for substitutions, because the recipe above took dozens of iterations to perfect for balance acidity, sweetness, caffeine, creaminess and mouth feel.

It’s like an americano over ice (without water to dilute) with splenda and 1/2 and 1/2, and it’s shaken which makes a big difference for some reason.

Now you know.

Everyone groans when they hear it.  Then they try it.  Then they’re hooked.

Sorry.

/Hoff

I was just reading a post by Alessandro at virtualization.info in which he was discussing the availability of trial versions of Cisco’s Nexus 1000v virtual switch solution for VMware environments:

Starting May 21, we’ll see if the customers will really consider the Cisco virtual switch a must-have and will gladly pay the premium price to replace the basic VMware virtual switch they used for so many years now.  As usual in virtualization, it really depends on who’s your interlocutor inside the corporate. The guys at the security department may have a slightly different opinion on this product than the virtualization guys.

Clearly the Nexus 1000v is just the first in a series of technology and architectural elements that Cisco is introducing to integrate more tightly into virtualized and Cloud environments.  The realities of adoption of the 1000v come down to who is making the purchasing decisions, how virtualization is being addressed as an enterprise architecture issue,  how the organization is structured and what pain points might be felt from the current limitations associated with VMware’s vSwitch from both a technological and operational perspective.

Oh, it also depends on price, too 😉

Alessandro also alludes to some complaints in pricing strategy regarding how the underlying requirement for the 1000v, the vNetwork Distributed switch, is also a for-pay item.  Without the vNDS, the 1000v no workee:

Some VMware customers are arguing that the current packaging and price may negatively impact the sales of Nexus 1000V, which becomes now much less attractive.

I don’t pretend to understand all the vagaries of the SKU and cost structures of VMware’s new vSphere, but I was intrigued by the following post from the vinternals blog titled VMware slaps enterprise and Cisco in face, opens door for competitors,:

And finally, vNetwork Distributed Switch. This is where the slap in the face for Cisco is, because the word on the street is that no one even cares about this feature. It is merely seen as an enabler for the Cisco Nexus 1000V. But now, I have to not only pay $600 per socket for the distributed switch, but also pay Cisco for the 1000V!?!?! A large slice of Cisco’s potential market just evaporated. Enterprises have already jumped through the necessary security, audit and operational hoops to allow vSwitches and port groups to be used as standard in the production environment. Putting Cisco into the virtual networking stack is nowhere near a necessity. I wonder what Cisco are going to do now, start rubbishing VMware’s native vSwitches? That will go down well. Oh and yeh, looks like you pretty much have only 1 licensing option for Cisco’s Unified Computing System now. Guess that “20% reduction in capital expense” just flew out the window.

Boy, what a downer! Nobody cares about vNDS?  It’s “…merely seen as an enabler for the Cisco Nexus 1000V?” Evaporation of market? I think those statements are a tad melodramatic, short-sighted and miss the point.

The “…necessary security, audit and operational hoops to allow vSwitches and port groups to be used as standard in the production environment” may have been jumped through, but they represent some serious issues at scale and I maintain that these hoops barely satisfy these requirements based on what’s available, not what is needed, especially in the long term.  The issues surrounding compliance, separation of duties, change control/management as well as consistent and stateful policy enforcement are huge problems that are being tolerated today, not solved.

The reality is that vNDS and the 1000v represent serious operational, organizational and technical shifts in the virtualization environment. These are foundational building blocks of a converged datacenter, not point-product cash cows being built to make a quick buck.   The adoption and integration are going to take time, as will vSphere upgrades in general.  Will people pay for them?  If they need more scalable, agile, and secure environments, they will.  Remember the Four Horsemen? vSphere and vNetworking go a long way toward giving enterprises more choice in solving these problems and vNDS/1000v are certainly pieces of this puzzle. The network simply must become more virtualization (and application and information-) aware in order to remain relevant.

However, I don’t disagree in general that  “…putting Cisco into the virtual networking stack is nowhere near a necessity,” for most enterprises, especially if they have very simple requirements for scale, mobility and security.  In environments that are designing their next evolution of datacenter architecture, the integration between Cisco, VMware, and EMC are critical. Virtualization context, security and policy enforcement are pretty important things.  vNetworking/VNDS/1000v/VN-Link are all enablers.

Lastly, there is also no need for Cisco to “…start rubbishing VMware’s native vSwitches” as the differences are pretty clear.  If customers see value in the solution, they will pay for it. I don’t disagree that the “premium” needs to be assessed and the market will dicate what that will be, but this doom and gloom is premature.

Time will tell if these bets pay off.  I am putting money on the fact that they will.

Don’t think that Cisco and VMware aren’t aware of how critical one are to the other and there’s no face slapping going on.

/Hoff

I’ll be at the Virtualization Congress ’09 / Citrix Synergy at the MGM Grand in Las Vegas for a couple of days this week.

I am presenting on Cloud Computing Security on May 6th at 11:30am-12:20pm – Mozart’s The Marriage of Figaro: The Complexity and Insecurity of the Cloud – VC105

This ought to be a funny presentation for about the first 5 minutes…you’ll see why 😉

I’m also on a panel with Dave Shackleford (Configuresoft) & Michael Berman (Catbird) moderated by the mastermind of all things virtualization, Alessandro Perelli,  on May 6th at 5: Securing the Virtual Data Center (on Earth and on Clouds) – VC302

If you’re around, ping me via DM on Twitter (@beaker) or hit me up via email [choff @ packetfilter.com]

Of course, it’s entirely likely you’ll find Crosby and I chatting it up somewhere 😉

See you there!

/Hoff

Overheard in the backroom of an audit meeting:

Brett: No, no, I just want you to know… I just want you to know how sorry we are that things got so fucked up with us and the Cloud thing. We got into this thing with the best intentions and I never…
Jules: [Jules shoots the man on the couch] I’m sorry, did I break your concentration? I didn’t mean to do that. Please, continue, you were saying something about best intentions. What’s the matter? Oh, you were finished! Well, allow me to retort. What do these Clouds look like?
Brett: Cloud, what?
Jules: What country are you from?
Brett: Cloud what? What? Wh – ?
Jules: “Cloud” ain’t no country I’ve ever heard of. They speak English in Cloud?
Brett: Cloud, what?
Jules: English, motherfucker, do you speak it?
Brett: Yes! Yes!
Jules: Then you know what I’m sayin’!
Brett: Yes!
Jules: Describe what the Cloud looks like!
Brett: Cloud what?
Jules: Say ‘Cloud, what’ again. Say ‘Cloud, what’ again, I dare you, I double dare you motherfucker, say Cloud one more Goddamn time!

Don’t be a square, Daddy-o.

Charles Babcock at InformationWeek wrote an article titled “IBM Launches Appliance For Private Cloud Computing” in which he details IBM’s plans to bundle VMware with their WebSphere Application Server on an x86 platform, stir in chargeback/billing capability, call it “Hypervisor Edition” and sell it as an “appliance” that runs in “Private Clouds” for $45,000.

Bundling hardware with a virtualization platform as an appliance isn’t a new concept as everyone including Cisco is doing that.  However, the notion of bundling hardware with a virtualization platform and a virtual appliance and then labeling THAT an appliance “to disperse those applications to the cloud” is an ironic twist of marketing.

Tarting it up and calling it a “Cloud appliance” (the WebSphere CloudBurst Appliance to be specific) that “…plugs into Private Clouds” is humorous:

IBM this week announced its WebSphere CloudBurst Appliance for deploying applications to a private cloud. IBM is the first major vendor to produce a cloud appliance for its customers, a sign of how the concepts of private cloud computing are getting a hearing in the deepest recesses of the enterprise.

Private clouds are scalable compute resources established in the enterprise data center that have been configured by IT to run a virtual machine upon demand. In some cases, business users are empowered to select an application and submit it as a virtualized workload to be run in the cloud.

The WebSphere Appliance stores and secures virtualized images of applications on a piece of IBM xSeries hardware that’s ready to be plugged into a private cloud, Tom Rosamilia, general manager of the applications and integration middleware division, said in an interview. That image will be cast in a VMware ESX Server file format for now; other hypervisor formats are likely to follow, he said. The WebSphere Application Server Hypervisor Edition is also preloaded on the appliance and can run the virtualized image upon demand. The Hypervisor Edition is also new and both it and the appliance will become available by the end of the second quarter.

Hypervisor Edition is a version of the WebSphere Application Server designed to run virtualized applications on IBM’s x86-based server series. The appliance with application server will be priced at $45,000, Rosamilia said.

Having an application ready to run on a hardware appliance represents a number of short cuts for the IT staff, Rosamilia said. Once an application is configured carefully to run with its operating system and middleware, that version of the application is “freeze dried with its best practices into a virtualized image,” or a complete instance of the application with the software on which it depends.

Additional instances of the application can be started up as needed from this freeze-dried image without danger of configuration error, Rosamilia noted. The application is a service, awaiting its call to run in a virtual machine while on the WebSphere appliance. When it is run, the appliance logs the resources use and who used them for chargeback purposes, one of the requirements for successful private cloud operation, according to private cloud proponents.

Rosamilia said enterprises that have applications that are already configured as a service or sets of services will find those applications fitting easily into a cloud infrastructure. An appliance approach makes it simple “to disperse those applications to the cloud” with a lower set of skills than IT currently needs to configure and deploy an application in the data center.

So now, for the first time ever, you can leverage virtualization to run a “freeze-dried” VM application/service on an x86 server appliance in the datacenter Private Cloud! Awesome. You heard it here second.

Is it any wonder people are confused by Private Clouds? Selling software disguised as a virtual machine, coupled to hardware, but abstracted by a hypervisor as a bundled “appliance” ISN’T Cloud Computing. It’s box pushing.

Not that I should be surprised.

<sigh>

/Hoff

I had to chuckle and then sob when I saw this posting from Reuven Cohen on the Cloud Computing Interoperability Forum (CCIF) regarding the ViMTruder “virtual machine trojan:”

Sergio Castro has released a functional, open source Virtual Machine Trojan called ViMTruder.

I’ve held off for a few days before posting this news. I wasn’t sure if helping spread the news would do more harm then good but, several other blogs have picked up the story, so why not.

So what is a Virtual Machine Trojan? According to Castro virtual machine trojans are seemingly benign virtual machine you download from the Internet contains a trojan. The objective of the trojan is to remotely take control
of the machine for nefarious purposes: steal information, send spam, conduct click fraud, stage denial of service attacks within a botnet, etc.

ViMtruder is written in Python and consists of a client which is installed within a virtual machine, and a control server, which sits in a host on the Internet. The virtual machine, running Linux, is configured to automatically run the VMT client in the background upon boot up. The VMT tries periodically to contact the control server through the Internet using port 80 outbound. Once the control server links with the VMT, you can send it Nmap commands to scan the target LAN where the VMT is connected.

The types of attacks a VMT can execute are different than a normal trojan. The VMT does not have access to the host machine; rather, it has access to the local network. Therefore, a VMT can be programmed to do the following:

1. Sniff traffic in the local network
2. Actively scan the local network to detect machines, ports and services
3. Do a vulnerability scan to detect exploitable machines in the local network
4. Execute exploits  in the local network
5. Brute force attacks against services such as ftp and ssh
6. Launch DoS attacks within the local network, or against external hosts
7. And of course, send spam and conduct click fraud

My first thought is imagine something like this embedded into an EC2 AMI and the potential damage it would cause.

Direct Link:
http://code.google.com/p/vimtruder/
—
–
Reuven
CCIF Instigator

You can read my response at the bottom of the thread in the link at the top of the page.  I am awe struck at the moment.

Keep in mind that frothy hyperbole misrepresenting security risks as unique and “damaging”  as illustrated above are being made by people invited to advise the U.S. government on how to secure Cloud Computing.  Joy.

/Hoff

My buddy George Hulme wrote a great piece on the efforts of the Cloud Security Alliance and the first draft of our “Security Guidance for Critical Areas of Focus in Cloud Computing.”

I had one important point of departure from his assessment that I feel needs discussion wherein George said:

While there are a number of minor issues I’d question in this paper, these are all fixable challenges — and will be strengthened in time, I’m certain. It’s that, despite its comprehensiveness, what is not in this paper that disappointed.

There is no overarching vision in this paper. There is no call to action for the IT community: whether it be the builders, providers, or consumers of cloud services. There’s no inspiration to motivate broad community involvement. This is no small oversight.

Selling the importance of doing cloud computing right from the beginning is the most “critical area of focus” of all.

I wanted to clear up my disagreement with George on those few points he dinged us on, as I feel that we covered all of these things at both our kick-off session at RSA and while we certainly could have “sold” the idea more within the first release of the guidance, page 5 (the introduction) stated the following:

We are continuously bombarded with news of information technology’s next big thing, a disruptive trend in computing with far reaching implications.  Many of these trends are no more than a marketer’s dream – hype sells technology and it becomes difficult to separate real change from an incremental upgrade.  Cloud Computing is having its moment in the sun, as the concept of utilizing computing as an on-demand subscription creates operating and economic efficiencies. Some deride the cloud as nothing new and in many respects they are correct.  Henry Ford’s Model T was not a new invention, but the revolution that ensued cannot be denied.  We believe Cloud Computing to be a very important trend that in many ways is beginning to fulfill the early promise of the Internet and will create unanticipated change in business with its ubiquitous adoption.  Phase one of the Internet was connectivity, with Cloud Computing we are leveraging that connectivity to optimize the utility of computing.

While we do see Cloud Computing as being a major change coming to every business, as information security practitioners, we recognize that there are verities which must not change: good governance, managing risks and common sense.  Cloud Computing is an unstoppable force and we encourage security practitioners to lead and help accelerate its secure adoption aided by common sense, rather than standing on the sidelines and letting the business move forward without us.

Some evangelists of cloud computing encourage us to focus on the model as a black box, the seamless presentation of your information on demand.  Pay no attention to how it works: resources are dynamically allocated, loads are balanced in real time and data is archived automatically.   Our message to the security practitioner is that in these early days of cloud computing, you must look under the hood of your cloud providers and you must do so using the broadest precepts of your profession in order to properly assure that the service engagements meet and exceed the security requirements of your organization.

The Cloud Security Alliance is a grassroots effort to facilitate the mission to create and apply best practices to secure cloud computing.  Incorporated as a not-for-profit organization, our efforts will seek to provide a voice for security practitioners.  However, recognizing that a secure cloud is a shared responsibility, we will be inclusive of all organizations and points of view to fulfill this mission.

What follows is our initial report, outlining areas of concern and guidance for organizations adopting cloud computing.  The intention is to provide security practitioners with a comprehensive roadmap for being proactive in developing positive and secure relationships with cloud providers.  Much of this guidance is also quite relevant to the cloud provider to improve the quality and security of their service offerings.   As with any initial foray, there will certainly be guidance that we could improve upon.  We will quite likely modify the number of domains and change the focus of some areas of concern.  We seek your help to improve this guidance to make version 2.0 of this document an even better asset to the security practitioner and cloud provider.

We will be kicking off numerous online activities and in-person regional events to share our findings and connect with experts to increase our knowledge base.  Here is how you can get involved:

• Visit our website to find out how you can help: www.cloudsecurityalliance.org
• Join our LinkedIn group to collaborate with us: www.linkedin.com/groups?gid=1864210

In my opinion, the introduction conveyed our vision, the call to action, and inspired community involvement.  I’m slightly biased, however.

It could certainly be improved, but I felt that while George did a great job with the rest of his article, he missed the point that we did address these important issues.

Our outreach is currently limited by people’s bandwidth, but as things settle down after RSA and InfoSec UK, you can expect to see much better organizational efforts and messaging around what we are doing and how you can get involved.

Did you come away from reading the paper without a sense of vision, call to action, inspiration or how to get involved?   Please do let me know.

/Hoff

See the diagram to the right?  It is my masterful “Hamster Sine Wave Of Pain.”  The HSWOP demonstrates where and how, over time, we manifest our investment in security controls and approaches.

We waffle between securing the host to the user to information to applications and then to the network and back again.  It’s how it’s always been and how it always will be.  It makes for some timing problems, however.

The gap in approach shows up when we overlay disruptive innovation and technology such as virtualization and Cloud Computing on top of this security response curve and we realize we’re out of synch.  When we’re busy being information-centric from a security perspective and a disruptive networking event occurs…oops.

The inspiration for this post came from a complaint on Twitter this morning from my buddy Rich Mogull in which he lamented that too many people are equating “HIPS (host-based intrusion prevention)” with “Cloud Security.”

The reality is that depending upon the *aaS model you’re referring to, HIPS *is* Cloud Security.  Specifically, in IaaS/PaaS environments when you can’t plumb in virtual network appliances (or physical for that matter) then you’re basically left with whatever the provider gives you at the “network” layer (which is usually not much) or you focus on host-based controls. HIPS is as good as any other solution at that point.

In SaaS environments, you’re dependent upon whatever the provider engineers into their network platforms and the applications themselves.

To generalize, when you’re talking about having security as a visible operational capability presented to the user versus being bundled as part of the service, besides application security and the odd ACL, HIPS/HIDS/AV/Hardening Scripts/etc… is Cloud Security for most folks at the moment.

Ultimately, this Cloud Security gap at the IaaS/PaaS level will close over time as it is beginning to do so technologically with virtualization.

You’ll have more options as the mechanisms for integrating network-based security solutions become available.  At issue here is the fact that security capabilities caused by inflexible policies based on IP addresses, are out of step with connectivity advances and how Cloud services are composed, provisioned, orchestrated and managed.  Hence the host/guest-based security focus.  It’s simply the easiest and most prudent thing to do given our options at the moment.

We’ve seen the hints of advancement with what VMware is doing with VMsafe and their API’s.  As the notion of VDCOS evolves,  I maintain we’ll see this sort of capability appear with IaaS/PaaS vendors in the Cloud, too, and it will expand beyond things like firewalls and IPS’s — we’ll see load balancers and other network-based capabilities emerge through creative plumbing.  We’ll see what other virtualization platforms bring to the table in this scope as introspection capabilities mature (if they do at all…)

We ought to see a bunch of innovative solutions that will emerge slowly as the “internal” virtualization and unified computing capabilities make their way “outward” and become the same platforms powering more mainstream Cloud offerings.  This might take a while.  Perhaps a very long while.

Until then, enjoy your agents.

Same as it ever was…same as it ever was.

/Hoff

GigaOm’s Structure ’09 “Putting Cloud Computing to Work” conference sounded really good. I thought I’d submit a response to their CFP with a perspective on Cloud Security that I’m pretty sure would be unique.

I was excited when I saw a response from GigaOm’s Surj Patel titled: GigaOM’s Structure 09: Speaker Application Status

I was slightly less excited when I read the contents of the email which you can see by clicking on the image below to expand it:

I loved this.  “We ask you to consider engaging our audience not by speaking but via sponsorship.”

So while my talk doesn’t satisfy their requirements, cash does.  Yup, that’s adding value alright.  I don’t mind not meeting their speaking requirements, but slapping me in the face with this kiss-off is insulting.

Bite me.

/Hoff