Archive for the ‘Citrix’ Category

Crosby: Xen and the Art of Marketcycle Maintenance

May 12th, 2008 14 comments

It seems I have fallen victim to a series of misunderstandings these days.

First there was Joanna-Gate and now Simon Crosby, Citrix’s CTO, suggests in a blog entry titled "Chris Hoff & The Mother Of All Misunderstandings" that I’m puffing on the wrong end of my cigars for disagreeing with his position.

I’m a little concerned that Simon’s response to me was issued on what is listed as the "beta" version of Citrix’s official blog.  Perhaps the virtualized version hasn’t made it out of QA yet? 😉

Simon’s response was extremely well crafted to avoid responding to most of my actual points, was contextually oblique at points, and was a fantastic marketing piece for Xen Citrix, but I wish he’d paid more attention to the actual points within my post. 

Further his little quips/comments on his hyperlinks "Who is this guy, anyway?  Think before you type dude, we’re not idiots," etc. didn’t go unnoticed – cute but juvenile)

I am, however, honored that Simon would accord me the high-status of being "…normally fairly clued-in:"

reckon that Hoff, who is normally fairly clued-in,  has put the smoking
end of the cigar in his mouth before thinking through this argument.
He’s horribly confused, but as smug as always, so let me clarify what I
said, and what it means.

…but I can assure you that I’ve only ever done that with a cigar once,
and it was for a much better reason than blogging.  If you must know,
it was Kentucky’s finest bourbon.  That is all I’m going to say about

I’m glad he’s "clarifying" what he said, since I will also.  I seem to have that effect on people.  Must be the accent thing…

The reason for my allergic reaction to Simon’s comments stem from my opinion that it is the responsibility of virtualization platform providers to ensure that their "[virtualized] data center operating system platforms of the future" don’t become the next generation of insecure infrastructure.

Simon sums up his opinion:

In summary an assertion that the virtualization platform vendor has
to fix the sad state of the OS/App world by making it secure is
demanding too much.  It would mean that we have to be experts in every
piece of system software including all of the vulnerabilities of all
OSes and their apps.  In my view the reason the state of security is
poor now is because of the monolithic approaches of traditional OS and
app vendors. 

We will focus manically on our layer, make it
secure, tiny and bulletproof to attack in its own right.  And we will
work closely with experts in security of OSes and Apps to give them an
opportunity to implement guest-level security outside the guest,
through privileged interfaces that themselves are secure.

After 15 years of dealing with this crap, I respectfully suggest that it is not too much to ask and it’s about time we stood up and did.  First  you criticize OS/App. vendors and blame them for the state of security because of their "monolithic approach" and then you go on to propose the exact same thing!

Focusing only on your little patch of grass is short-sighted and it won’t work.  Just like it hasn’t worked in the past.  It’s a disaster waiting to happen, and you’re enabling it. 

I shudder at the potential tunnel vision of virtualization platform providers only focusing on the security of the hypervisor without taking the bigger picture into consideration and expect a piecemeal approach to securing the expanse of the virtualized environment to suffice.

It’s clear you’re making arguments about security from an engineering and code-base perspective that is simply disconnected from the realities of what it means to actually deploy these solutions. 

Virtualization is more than just the hypervisor.  You should know that by now, Simon.  The company that acquired your company knows all about that.  The hypervisor will shortly become a commodity, so in the long term the value brought to bear has to be more than just an ultra-thin layer of code:


…and furthermore, we’re going to deploy many of them:


I wish to make it clear that I hold all virtualization platform vendors to the same level of scrutiny and criticism, not just Citrix. 

I happen to like Xen very much.  I like VMware, also.  I think the latter is more realistic and measured when it comes to addressing the need and approach in recognizing that as a major layer in the infrastructure, there’s more required than to just secure the hypervisor and leave the remaining mess to someone else to solve.

I think Simon’s blog title is apropos, but I think the misunderstanding is his.

It’s important to understand that I’m not suggesting that virtualization platform providers should secure the actual guest operating systems
but they should enable an easier and more effective way of doing so when virtualized.

I mean that the virtualization platform providers should ensure the security of the instantiation of those
guests as "hosted" by the virtualization platform.  In some cases this means leveraging technology present in the virtualization platform to do things that non-virtualized instances cannot. That’s more than just securing the hypervisor.

Securing the hypervisor whilst closing your eyes to the likelihood
that the majority of attacks against it and other guests will come from "guests" within the same system is planting your head in the sand.  That means that there will be a need to ensure that certain behaviors specific to the hosted guests are mitigated to ensure that bad things don’t happen — to the guest or the hypervisor.

Transferring the responsibility to secure the environment to third party security ISV’s in order
to secure the VM’s
and preventing them from compromising one another or the hypervisor is
difficult for me to comprehend, especially when they are playing catch up of what virtualization means within the context of security.

Fundamentally, attempting to mate static and topology-dependent policies to incredibly dynamic and transitive technology delivered by virtualization will simply fail.  Third party security ISV’s will simply require a complete re-tool to even get close to delivering this and will need to provide intimate hooks to allow for this policy/guest affinity to occur in the first place.

I consider the virtualization infrastructure layer as that of an operating system and as such, I would expect that the underpinning mechanicals are as sound and secure as possible while also ensuring that anything running on top of it is as secure as possible, also.

Let’s take Microsoft (with or without Hyper-V) as an example:

Microsoft is fundamentally concerned now with making the OS as
resilient and secure as possible whilst preventing the applications and
interaction with elements riding on top of the OS from doing bad things
to the system as a whole; this isn’t just to protect the OS, but the
assets on it. 

This is really what I’m getting at.  Yes, Microsoft is an OS provider.  Shortly, that OS provider will integrate virtualization directly into the operating system.  That means more, not less, direct integration and security embedded as a function of the virtualization platformCitrix, VMware, etc. are all just operating system vendors of a different shape and size.

It’s unclear to me, Simon, whether your arguments are meant to justify a business model, a lack of planning, a crafty plan to perpetuate the security hamster wheel of pain, or all of the above.  It’s clear to me, however, that you’ve not felt the pain of actually having to use the products you suggest should be deployed in order to secure this mess.

I promised myself I wouldn’t turn this into one of those cut/paste blog pong entries, but the following really confused me:

But we are not in the business of specifically securing guests or their
applications, other than through offering a secure virtualization
platform.  Even VMware with VMsafe simply exposes APIs to third party
security vendors, so that customers can choose their preferred security
partner to secure guests.  I think that the VMware Determina
acquisition was very smart, and that hints to me that VMware sees
itself having a greater role in the security of guest OSes, since it
could choose to be in the vulnerability checking business without 3rd
party security vendors, but thus far they are working very openly with
the ecosystem.

So which is it?  You’ve established that Citrix is not in the business of securing guests or applications (you must mean Xen specifically, because somebody at Citrix spent quite a bit of money on this stuff with their other acquisitions) and that you believe it to be a lousy idea, but you think that VMware’s approach through their Determina acquisition as well as the capabilities of VMsafe is "…very smart?"

Simon, you’re the CTO and I’m the security wonk.  If we didn’t disagree, I’d be alarmed.  However, I think you might want to rethink your approach to how you market the security of your platform.

I’ve got a cigar for you anytime you want one.  I’ll let you light it.


Categories: Citrix, Virtualization Tags:

Citrix’s Crosby & The Mother Of All Cop-Outs

May 8th, 2008 6 comments

Bullshit_button In an article over at, Simon Crosby, the CTO of Citrix, suggests that "Virtualization vendors [are] not in the security business." 

Besides summarizing what is plainly an obvious statement of fact regarding the general omission of integrated security (outside of securing the hypervisor) from most virtualization platforms, Crosby’s statement simply underscores the woeful state we’re in:

While virtualization vendors will do their role in protecting the hypervisor, they are not in the business of catching bad guys or discovering vulnerabilities, said Simon Crosby, chief technology officer of Citrix Systems.

Independent security vendors will play a critical role in protecting virtual environments, he said. "The industry has already decided a long time ago that third party vendors are required to secure any platform," Crosby said. In this interview, Crosby agrees that using virtual technology introduces new complexities and security issues.

He said the uncertainties will be addressed once the industry matures.

I’m sure it’s reasonable to suggest that nobody expects virtualization platform providers to "…catch
bad guys," but I do expect that they employ a significant amount of
resources and follow an SDLC to discover vulnerabilities — at least in
their software.

Further, I don’t expect that the hypervisor should be the place in which all security functionality is delivered, but simply transferring the lack of design and architecture forethought from the hypervisor provider to the consumer by expecting someone else to clean up the mess is just, well, typical.

I love the last line.  What a crock of shit.  We’ve seen how well
this approach had worked with operating system vendors in the past, so why
shouldn’t the "next generation" of OS vendors — virtualization
platform providers — follow suit and not provide for a secure operating environment?

Let’s see, Microsoft is investing hugely in security.  Cisco is too.  Why would the other tip of the trident want to?  VMware’s at least taking steps to deliver a secure hypervisor as well as API’s to help secure the  VM’s that run atop of it.   Where’s Citrix in this…I mean besides late and complaining they weren’t first?

So, in trade for the "open framework for security ecosystem partnership" cop-out, we get to wait for the self-perpetuating security industry hamster wheel of pain to come back full circle. 

The fact that the "industry" has "decided" that "third party vendors are required to secure any platform" simply points to the ignorance, arrogance and manifest destiny we endure at the hands of those who are responsible for the computing infrastructure we’re all held hostage with. 

Just so I understand the premise, the security industry (or is it the virtualization industry?) has decided that the security industry instead of the OS/infrastructure (virtualization) vendors are the one’s responsible to secure the infrastructure — and thus our businesses!?  What a shocker.  Way to push for change, Simon.

I can’t even describe how utterly pissed off these statements make me.


Categories: Citrix, Virtualization Tags:

It’s Virtualization March Madness! Up First, Montego Networks

March 27th, 2008 No comments

If you want to read about Montego Networks right off the bat, you can skip the Hoff-Tax and scroll down to the horizontal rule and start reading.  Though I’ll be horribly offended, I’ll understand…

I like being contradictory, even when it appears that I’m contradicting myself.  I like to think of it as giving a balanced perspective on my schizophrenic self…

You will likely recall that my latest post suggested that the real challenge for virtualization at this stage in the game is organizational and operational and not technical. 

Well, within the context of this post, that’s obviously half right, but it’s an incredibly overlooked fact that is causing distress in most organizations, and it’s something that technology — as a symptom of the human condition — cannot remedy.

But back to the Tech.

The reality is that for reasons I’ve spoken of many times, our favorite ISV’s have been a little handicapped by what the virtualization platforms offer up in terms of proper integration against which we can gain purchase from a security perspective.  They have to sell what they’ve got while trying to remain relevant all the while watching the ground drop out beneath them.

These vendors have a choice: employ some fancy marketing messaging to make it appear as though the same products you run on a $50,000+ dedicated security appliance will actually perform just as well in a virtual form.

Further, tell you that you’ll enjoy just as much visibility without disclosing limitations when interfaced to a virtual switch that makes it next to impossible to replicate most complex non-virtualized topologies. 

Or, just wait it out and see what happens hoping to sell more appliances in the meantime.

Some employ all three strategies (with a fourth being a little bit of hope.)

Some of that hoping is over and is on it’s way to being remedied with enablers like VMware’s VMsafe initiative.  It’s a shame that we’ll probably end up with a battle of API’s with ISV’s having to choose which virtualization platform providers’ API to support rather than a standard across multiple platforms.

Simon Crosby from Xen/Citrix made a similar comment in this article:

While I totally agree with his sentiment, I’m not sure Simon would be as vocal or egalitarian had Citrix been first out of the gate with their own VMsafe equivalent.  It’s always sad when one must plead for standardization when you’re not in control of the standards…and by the way, Simon, nobody held a gun to the heads of the 20 companies that rushed for the opportunity to be the first out of the gate with VMsafe as it’s made available.

While that band marches on, some additional measure of aid may come from innovative youngbloods looking to build and sell you the next better mousetrap.

As such, in advance of the RSA Conference in a couple of weeks, the security world’s all aflutter with the sounds of start-ups being born out of stealth as well as new-fangled innovation clawing its way out of up-starts seeking to establish a beachhead in the attack on your budget.

With the normal blitzkrieg of press releases that will undoubtedly make their way to your doorstop, I thought I’d comment on a couple of these companies in advance of the noise.

A lot of what I want to say is sadly under embargo, but I’ll get further in-depth later when I’m told I can take the wraps off.  You should know that almost all of these emerging solutions, as with the one below, operate as virtual appliances inside your hosts and require close and careful configuration of the virtual networking elements therein.

If you go back to the meat of the organization/operational issue I describe above, who do you think has access and control over the virtual switch configurations?  The network team?  The security team?  How about the virtual server admin. team…are you concerned yet?

Here’s my first Virtualized March Madness (VMM, get it!) ISV:

  • Montegomodel
    Montego Networks – John Peterson used to be the CTO at Reflex, so he knows a thing or two about switching, virtualization and security.  I very much like Montego’s approach to solving some of the networking issues associated with vSwitch integration and better yet, they’ve created a very interesting business model that actually is something like VMsafe in reverse. 

    Essentially Montego’s HyperSwitch works in conjunction with the integrated vSwitch in the VMM and uses some reasonably elegant networking functionality to classify traffic and either enforce dispositions natively using their own "firewall" technologies (L2-L4) or — and this is the best part — redirect traffic to other named security software partners to effect disposition. 

    If you look on Montego’s website, you’ll see that they show StillSecure and BlueLane as candidates as what they call HyperVSecurity partners.  They also do some really cool stuff with Netflow.

    Neat model.  When VMsafe is available, Montego should then allow these other third party ISV’s to take advantage of VMsafe (by virtue of the HyperSwitch) without the ISV’s having to actually modify their code to do so – Montego will build that to suit.  There’s a bunch of other stuff that I will write about once the embargo is lifted.

    I’m not sure how much runway and strategic differentiation Montego will have from a purely technical perspective as VMsafe ought to level the playing field for some of the networking functionality with competitors, but the policy partnering is a cool idea. 

    We’ll have to see what the performance implications are given the virtual appliance model Montego (and everyone else) has employed.  There’s lots of software in them thar hills doing the flow/packet processing and enacting dispositions…and remember, that’s all virtualized too.

    In the long term, I expect we’ll see some of this functionality appear natively in other virtualization platforms.

    We’ll see how well that prediction works out over time as well as keep an eye out for that Cisco virtual switch we’ve all been waiting for…*

I’ll be shortly talking about Altor Networks and Blue Lane’s latest goodies.

If you’ve got a mousetrap you’d like to see in lights here, feel free to ping me, tell me why I should care, and we’ll explore your offering.  I guarantee that if it passes the sniff test here it will likely mean someone else will want a whiff.


* Update: Alan over at the Virtual Data Center Blog did a nice write-up on his impressions and asks why this functionality isn’t in the vSwitch natively.  I’d pile onto that query, too.  Also, I sort of burned myself by speaking to Montego because the details of how they do what they do is under embargo based on my conversation for a little while longer, so I can’t respond to Alan…

Take5 (Episode #5) – Five Questions for Allwyn Sequeira, SVP of Product Operations, Blue Lane

August 21st, 2007 18 comments

This fifth episode of Take5 interviews Allwyn Sequeira, SVP of Product Operations for Blue Lane.  

First a little background on the victim:

Allwyn Sequeira is Senior Vice President of Product Operations at Blue
Lane Technologies, responsible for managing the overall product life
cycle, from concept through research, development and test, to delivery
and support. He was previously the Senior Vice President of Technology
and Operations at netVmg, an intelligent route control company acquired
by InterNap in 2003, where he was responsible for the architecture,
development and deployment of the industry-leading flow control
platform. Prior to netVmg, he was founder, Chief Technology Officer and
Executive Vice President of Products and Operations at First Virtual
Corporation (FVC), a multi-service networking company that had a
successful IPO in 1998. Prior to FVC, he was Director of the Network
Management Business Unit at Ungermann-Bass, the first independent local
area network company. Mr. Sequeira has previously served as a Director
on the boards of FVC and netVmg.

Mr. Sequeira started his career as a software developer at HP in the
Information Networks Division, working on the development of TCP/IP
protocols. During the early 1980’s, he worked on the CSNET project, an
early realization of the Internet concept. Mr. Sequeira is a recognized
expert in data networking, with twenty five years of experience in the
industry, and has been a featured speaker at industry leading forums
like Networld+Interop, Next Generation Networks, ISP Con and RSA

Mr. Sequeira holds a Bachelor of Technology degree in Computer
Science from the Indian Institute of Technology, Bombay, and a Master
of Science in Computer Science from the University of Wisconsin,

Allwyn, despite all this good schoolin’ forgot to send me a picture, so he gets what he deserves 😉
(Ed: Yes, those of you quick enough were smart enough to detect that the previous picture was of Brad Pitt and not Allwyn.  I apologize for the unnecessary froth-factor.)


1) Blue Lane has two distinct product lines, VirtualShield and PatchPoint.  The former is a software-based solution which provides protection for VMware Infrastructure 3 virtual servers as an ESX VM plug-in whilst the latter offers a network appliance-based solution for physical servers.  How are these products different than either virtual switch IPS’ like Virtual Iron or in-line network-based IPS’s?

IPS technologies have been charged with the incredible mission of trying to protect everything from anything.  Overall they’ve done well, considering how much the perimeter of the network has changed and how sophisticated hackers have become. Much of their core technology, however, was relevant and useful when hackers could be easily identified by their signatures. As many have proclaimed, those days are coming to an end.

A defense department official recently quipped, "If you offer the same protection for your toothbrushes and your diamonds you are bound to lose fewer toothbrushes and more diamonds."  We think that data center security similarly demands specialized solutions.  The concept of an enterprise network has become so ambiguous when it comes to endpoints and devices and supply chain partners, etc. we think its time to think more realistically in terms of trusted, yet highly available zones within the data center.

It seems clear at this point that different parts of the network need very different security capabilities.  Servers, for example need highly accurate solutions that do not block or impede good traffic and can correct bad traffic, especially when it comes to closing network-facing vulnerability windows.  They need to maintain availability with minimal latency for starters; and that has been a sort of Achilles heel for signature-based approaches.  Of course, signatures also bring considerable management burdens over and beyond their security capabilities.

No one is advocating turning off the IPS, but rather approaching servers with more specialized capabilities.  We started focusing on servers years ago and established very sophisticated application and protocol intelligence, which has allowed us to correct traffic inline without the noise, suspense and delay that general purpose network security appliance users have come to expect.

IPS solutions depend on deep packet inspection typically at the perimeter based on regexp pattern matching for exploits.  Emerging challenges with this approach have made alert and block modes absolutely necessary as most IPS solutions aren’t accurate enough to be trusted in full library block. 

Blue Lane uses a vastly different approach.  We call it deep flow inspection/correction for known server vulnerabilities based on stateful decoding up to layer 7.  We can alert, block and correct, but most of are deployments are in correct mode, with our full capabilities enabled. From an operational standpoint we have substantially different impacts.

A typical IPS may have 10K signatures while experts recommend turning on just a few hundred.  That kind of marketing shell game (find out what really works) means that there will be plenty of false alarms, false positives and negatives and plenty of tuning.  With polymorphic attacks signature libraries can increase exponentially while not delivering meaningful improvements in protection. 

Blue Lane supports about 1000 inline security patches across dozens of very specific server vulnerabilities, applications and operating systems.  We generate very few false alarms and minimal latency.  We don’t require ANY tuning.  Our customers run our solution in automated, correct mode.

The traditional static signature IPS category has evolved into an ASIC war between some very capable players for the reasons we just discussed.Exploding variations of exploits and vectors means that exploit-centric approaches will require more processing power.

Virtualization is pulling the data center into an entirely different direction, driven by commodity processors.  So of course our VirtualShield solution was a much cleaner setup with a hypervisor; we can plug into the hypervisor layer and run on top of existing hardware, again with minimal latency and footprint.

You don’t have to be a Metasploit genius to evade IPS signatures.  Our higher layer 7 stateful decoding is much more resilient. 

2) With zero-days on the rise, pay-for-play vulnerability research and now Zero-Bay (WabiSabiLabi) vulnerability auctions and the like, do you see an uptake in customer demand for vulnerability shielding solutions?

Exploit-signature technologies are meaningless in the face of evanescent, polymorphic threats, resulting in 0-day exploits. Slight modifications to signatures can bypass IPSes, even against known vulnerabilities.  Blue Lane technology provides 0-day protection for any variant of an exploit against known vulnerabilities.  No technology can provide ultimate protection against 0-day exploits based on 0-day vulnerabilities. However, this requires a different class of hacker.

3) As large companies start to put their virtualization strategies in play, how do you see customers addressing securing their virtualized infrastructure?  Do they try to adapt existing layered security methodologies and where do these fall down in a virtualized world?

I’ve explored this topic in depth at the Next Generation Data Center conference last week. Also, your readers might be interested in listening to a recent podcast: The Myths and Realities of Virtualization Security: An Interview. 

To summarize, there are a few things that change with virtualization, that folks need to be aware of.  It represents a new architecture.  The hypervisor layer represents the un-tethering and clustering of VMs, and centralized control.  It introduces a new virtual network layer.  There are entirely new states of servers, not anticipated by traditional static security approaches (like instant create, destroy, clone, suspend, snapshot and revert to snapshot). 

Then you’ll see unprecedented levels of mobility and new virtual appliances and black boxing of complex stacks including embedded databases.  Organizations will have to work out who is responsible for securing this very fluid environment.  We’ll also see unprecedented scalability with Infiniband cores attaching LAN/SAN out to 100’s of ESX hypervisors and thousands of VMs.

Organizations will need the capability to shield these complex, fluid environments; because trying to keep track of individual VMs, states, patch levels, locations will make tuning an IPS for polymorphic attacks look like childs play in comparison.   Effective solutions will need to be highly accurate, low latency solutions deployed in correct mode. Gone will be the days of man-to-man blocking and tuning.  Here to stay are the days of zone defense.

4) VMware just purchased Determina and intends to integrate their memory firewall IPS product as an ESX VM plug-in.  Given your early partnership with VMware, are you surprised by this move?  Doesn’t this directly compete with the VirtualSheild offering?

I wouldn’t read too much into this. Determina hit the wall on sales, primarily because it’s original memory wall technology was too intrusive, and fell short of handling new vulnerabilities/exploits.

This necessitated the LiveShield product, which required ongoing updates, destroying the value proposition of not having to touch servers, once installed. So, this is a technology/people acquisition, not a product line/customer-base acquisition.

VMware was smart to get a very bright set of folks, with deep memory/paging/OS, and a core technology that would do well to be integrated into the hypervisor for the purpose of hypervisor hardening, and interVM isolation. I don’t see VMware entering the security content business soon (A/V, vulnerabilities, etc.). I see Blue Lane’s VirtualShield technology integrated into the virtual networking layer (vSwitch), as a perfect complement to anything that will come out of the Determina acquisition.

5) Citrix just acquired XenSource.  Do you have plans to offer VirtualShield for Xen? 

A smart move on Citrix’s part to get back into the game. Temporary market caps don’t matter. Virtualization matters. If Citrix can make this a two or three horse race, it will keep the VMware, Citrix, Microsoft triumvirate on their toes, delivering better products, and net good for the customer.

Regarding BlueLane, and Citrix/Xensource, we will continue to pay attention to what customers are buying as they virtualize their data centers. For now, this is a one horse show 🙂

Citrix Buying XenSource — It’s About Time(ing)

August 16th, 2007 No comments

This will be short and sweet.  Citrix’s announcement that they will clip a swell $500 Million to acquire XenSource on the tail of VMware’s IPO makes nothing but sense.  The timing is interesting; waiting for VMware’s IPO both validated the move but one has to wonder if it jacked the price any.

I can’t wait to see how this maps out over time across Citrix’s product lines which are still fairly siloed at this point.  Leveraging XenSource’s technology is a force multiplier across many elements of their offerings. It’s clear what the first moves will be, but I’m really interested in the longer term play.

At any rate, this is a fantastic strategic move for Citrix; these guys are poised to continue their march to take on Cisco as they become a robust platform for application and content delivery.*   If you take a look at their M&A activity over the last few years, it’s on a direct collision course with Cisco in many vectors. 

The big difference is, you can bolt their solution on instead of having to bake it in and these guys already have a footprint and expertise in the server and client consolidation markets.

Orthogonally, I wonder what effect this might have on f5?  Any thoughts there?

Then there’s Microsoft.  This may be a huge opportunity for other players such as SWsoft  to reinforce defensive positioning by shoring up relationships that otherwise might have gone XS’s way.

It’s going to get messy boys and girls.

This acquisition certainly has its challenges, but it really positions Citrix with as a complement to their existing product offerings.


*It gets more interesting strategically from a defensive position given Cisco’s recent investment of $150M in VMware prior to their IPO and my commentary on the matter here.

Categories: Citrix, Virtualization Tags: