As deployments of virtualised servers proliferate in enterprise IT environments, there are growing concerns associated with creating, managing and protecting these newly created virtual machines (VMs).
Because VMs are so prolific, there are problems with backing them up, managing them, migrating data among them and controlling their sprawl. But new products help IT administrators back up and protect VMs, as well as manage and monitor the connections between them and the networked storage resources they use.
Industry pundits estimate that as many as 70% to 80% of VMs rely on storage resources from Fibre Channel (FC) or IP SANs, or NAS. With such a reliance on shared storage, the problems mount for storage administrators charged with not only managing, but backing up and protecting, the environment.
Virtual machine backup blues
Chief among the concerns in backing up VMs is the cost of software licenses for backup software. Most backup vendors require a software license for each VM protected and a separate license for the physical host computer. If server virtualisation is done as part of a consolidation effort, the licenses are likely already available; but if new virtual servers are being created, additional licenses may be needed.
You may also need to rejigger the backup schedule. Because the job created for backing up one physical server may encompass backing up a number of VMs, rescheduling those jobs to occur consecutively is recommended to avoid bottlenecks.
Storage admins must also tailor each backup job to the virtualisation software they're using. Most backup software packages, such as EMC's NetWorker and Symantec's Veritas NetBackup, will back up VMware environments. There are even some that are focused specifically on VMware. These packages commonly use agent technology in which a software agent backs up each virtual and physical host machine.
Because storage admins are most familiar with the use of agent-based backups, analysts suggest adoption of a multilayered approach to data protection, one that combines agent-based backup with consolidated or serverless backup and snapshot or cloning technology.
Another method of VM backup is the use of a proxy server. Sometimes called serverless or consolidated backup technology, the introduction of a proxy server into the environment can reduce overhead and increase performance. Backup processing is offloaded from the host computer being backed up to this consolidated backup server.
The most commonly used software of this type is VMware's Consolidated Backup (VCB). In VCB, a series of drivers and preconfigured scripts execute the backup. A script takes a snapshot of the VM and mounts it to the proxy server. The script then quiesces the file system within the VM, while the backup software agent creates a virtual disk image of the VM. The mount is then destroyed and the virtual disk is removed from snapshot mode. Snapshot and cloning are also included in software from VMware and Citrix Systems's XenServer Enterprise Edition (formerly XenSource XenEnterprise), as well as in many backup software packages.
Many backup vendors, including CommVault, EMC and Symantec, have recently announced the capability of their software to work with VCB. Carmine Iannace, director of IT at global consultancy firm Brattle Group, has taken a combination approach to backing up his 75 virtual machines. "We install CommVault Simpana backup agents directly into the virtual guests and back them up and restore them just as if they were a physical server," says Iannace. For server failover and redundancy, Iannace uses a product from Neverfail.
Iannance says if the email system malfunctions or if the site goes down in one of the offices, the company could resume its email functionality from another remote office, for the entire firm.
Bare-metal restoration of Iannace's VMs is also handled by CommVault Simpana. "From a physical host perspective, the VMware hosts are quite easy to recover," says Iannace. "We can easily restore those and recover the virtual guests. That's not considered a big roadblock for us."
Jim Klein, director of information services and technology at a prominent school district has also taken a combined approach. "We treat the virtual machines just as if they are physical servers with backup agents and software from the open-source software Bacula," he says.
Because images of VMs are standardised and stored on Klein's SAN, for bare-metal recovery, "we just basically recreate machines on the fly and then restore the data to it. It reduces the amount of backup tape that we need and increases our efficiency." Klein also uses a proxy server to offload processing from his host computers.
Migration of virtual machines
VMs can be deployed easily, moved around and then deleted when you're done with them. But it's that simplicity of deployment that can cause problems.
VMware's VMotion enables the movement of VM images from one physical server to another for maintenance, to create new VMs and for business continuity.
"We use VMotion not only for business-continuity purposes, but so we can do maintenance during the day," says Iannace at the Brattle Group. "We can transport the guests from one physical server to another. That has been a big plus for us since we can actually do maintenance during the day by moving the guests running on one physical host to another in real-time."
VMware recently announced that its Storage VMotion, a new feature in VMware Infrastructure 3.5, enables the nondisruptive migration of VM disk files from one data store to another.
With VMotion, you can move the Virtual Machine Disk Formats from one storage device to another, without powering down the server.
One of the unintended consequences of VM mobility is when a VM fails over to another physical server. Backups may fail because the schedule expects to see the original host machine rather than the new one. Management and reporting software must be adjusted to move and reschedule these mobility issues.
STORServer has unveiled STORServer Agent for VMware Consolidated Backup, software that manages backups of ESX Servers to IBM Tivoli Storage Manager servers. And Veeam Software has introduced Veeam Reporter 2.5 for VMware Infrastructure 3. The product now reports on storage capacity and utilisation by VM, space allocated for each VM and disk status.
Rounding up virtual machine images
VM portability and migration also cause server sprawl. It's difficult for a storage admin to know how many VMs exist and where they're located. Because VMs are so easy to deploy, they'll spring up in the unlikeliest of places. VMs may be deployed for testing new apps, for migrating data and applications during maintenance operations, or simply to mirror data to server failover activities. They'll sprawl across the enterprise network, making it difficult to standardise their provisioning, and to maintain and patch them.
"The number of virtual machines we have at any one time varies based on development and other issues," says Iannace. "We might have times when we approach 90 virtual machines, but we are constantly building them and destroying them for either development or test. But the core 75 are what we have in production."
Another feature of VMware Infrastructure 3.5 is the Update Manager, which automates the patching of ESX Server hosts and VMs nondisruptively. Update Manager makes snapshots of VMs prior to patching; if patching isn't successful, it then rolls back to the pre-patching snapshot. The software works with Linux and Windows VMs.
VMware is also getting into the game of managing and abating VM sprawl with the acquisition of Dunes Technologies last fall. Dunes' Virtual Service-Orchestrator 3.1 (VS-O 3.1) is server-based software that allows IT to track the creation of virtual machines and automate processes for managing the VMs.
Bus contention and performance issues
Bus contention and CPU utilisation are paramount concerns in VM environments. Because all VMs in a physical server share the same PCI bus and CPUs, performance bottlenecks can crop up as VMs contend for the shared resources.
To lessen the effects of shared CPU and bus contention, it's best to implement hefty dual- or quad-core servers that are PCI Express-enabled. Adding dual- or quad-ported HBAs and Gigabit Ethernet adapters to the host servers can also help--each VM can then have its own IO channel to shared storage.
Brattle Group's Iannace deployed Dell Power-Edge servers with dual quad-core CPUs. "We've seen some high utilisation of our backup servers and have upgraded them over time," says Iannace. "In fact, our main physical server is a dual-die, quad-core system--for eight cores total--because we noticed a lot of CPU utilisation."
To overcome the performance limitations of virtualised servers, Klein at the school district says, "We generally buy a high-performance machine, such as a blade server, with extremely fast IO for a virtualisation host." Klein has virtualised five HP blade servers with the open-source Xen hypervisor and Citrix's XenServer.
"We only use Fibre Channel for shared storage, high-speed Serial Attached SCSI [SAS] drives and fast, multicore processors," says Klein. "Since you have multiple loads hitting the same IO channel and bus, it's important that each load be able to get on and off the channel as quickly as possible."
Several vendors have introduced tools and management appliances that alleviate the performance tax virtualised servers extract from the network. Onaro's (soon to be acquired by NetApp) VM Insight lets server and storage admins view the dependencies between virtualised servers and their attached storage, and determine how changes to the FC or IP network and its server connections will affect overall performance. Xsigo Systems offers appliances that try to mitigate virtualised server performance bottlenecks. The Xsigo I/O Director creates virtual IO channels between servers and storage to reduce overhead and increase IO throughput.
VMs can be a boon to businesses because they consolidate operations, but they can also be a burden when IT considers the complexities of backing up, managing and tracking them. It will behoove storage administrators to learn the best ways to protect VMs in their environment and, with management and monitoring tools, control their growth.