Virtualization 3.0: Cloud-wide VM migration and memory de-duplication

For those unfamiliar with my background, I authored a full x86 PC emulation project in the 1990's, called bochs. It was used in one form or another, by a number of virtualization players, including an R&D project at Stanford which became VMware. It's been interesting watching x86 virtualization mature, from the early days of it being used as a developer's tool and then on to server consolidation (what I call virtualization 1.0). Consolidation is an interesting proposition of its own, making use of the VM level of abstraction to pack more work onto less physical resources. But it's still a very static technology in that VMs need to be booted and shut-down to be moved to different hardware.

When VM migration came onto the scene, it unlocked a wealth of additional value of virtualization, and a set of higher-level technologies. In VMware terminology, DRS tapped VM migration to do load balancing within a cluster of machines. DPM uses migration to pack VMs into lesser number of machines, de-powering ones not needed, thus doing cluster-wide power management. I call this whole suite of dynamic VM technologies, virtualization 2.0. Unfortunately, these dynamic features have generally been confined to within a physical site location, or at best within a Campus Area Network.

To get to where virtualization needs to go, we need to be able to look at virtualization as a fabric, stretching or overlaying numerous physical sites. And Cloud Computing will absolutely exacerbate this need. Many things that we've contemplated on a small scale (e.g. load balancing, power management, down-time maintenance), need to be brought to a larger context of a virtualization fabric stretching across physical sites. Virtualization needs to stretch to the cloud. To be sure, there are a number of issues to solve to make this happen, including networking and storage continuity. But I'd like to present a part of this next evolutionary step, virtualization 3.0, which is critical to its success yet unanswered elsewhere to my knowledge.

Memory density in servers continues to go up following its own exponential path. And as virtualization is used for increasingly higher-end workloads, the size of per-VM memory will continue to rise. Just imagine if you piled up all the RAM from all of your data centers, in one spot! Yet, to enable a fluid and dynamic virtualization 3.0 fabric, we need to rapidly allow all kinds of intra and inter-site VM migrations to occur, often driven automatically. That requires a whole new approach to how we manage VM memory; huge volumes of it effectively need to be transported rapidly. On the storage front, there are a number of technologies afoot, which are enablers of virtualization 3.0. But, I've been working for some time on concepts for making VM memory a 1st class citizen of the virtualization 3.0 vision.

We know from various benchmarks that there is a significant amount of redundant memory state across VMs. Within a server, VMware will consolidate some redundancy, on the order of up to 30%. Research which uses a sub-page granularity for reducing redundancy brings that up to 65%. Further research, using a differencing engine, ups this to a phenomenal 90% with similar apps and operating systems, and 65% even across VMs running disparate workloads!

Knowing all of this, what could we do if we managed memory with virtualization 3.0 in mind? In short, we can enable faster and more long distance VM migrations, cross-site load balancing and power management, de-duplicate memory thoughout multiple sites, and even WAN accelerate VM memory! Not only can we make the virtualization fabric quicker, more responsive, and encompass a much large geography, but we can actually increase VM density (and/or decrease physical RAM requirements), thus become more capital efficient and reduce overall power consumption. What's fortunate, is that the buzz around "long distance VMotion" (VMotion is VMware-speak) has finally picked up, starting at the latest VMworld. As storage, networking, and other hurdles are overcome, there is a vast amount of optimization opportunities at the VM memory level.

I put together this PowerPoint presentation to highlight these points. Feel free to pass it along. Note that this is Patent Pending technology. Give me a shout if you'd like to talk more.

Disclosure: no positions, author has related Patent Pending technology