Spinning Out of Control: Securely Managing Virtual Sprawl

Server virtualization is taking hold. It boasts so many advantages that it is likely to become the standard for data centers around the world. It saves money by maximizing hardware resources. It reduces the number of physical servers, which reduces power consumption. It also revolutionizes server deployment by allowing servers to be copied as easily as files on the file system. Add to this the benefit of using pre-configured virtual appliances, and you should be convinced that virtual servers are good for your business.

As with any new technology, security tends to be an afterthought. Many companies that venture into virtual technologies expect that their existing security controls will apply to the new virtual environment, but virtual servers require new security approaches and controls.

In an IT department with weak or missing security controls, virtual servers will sprawl across physical hardware and quickly become unmanageable and vulnerable to attack. To jump into virtualization, it’s essential to be innovative about security and to reinvent your security controls.

Segmentation
In the physical server environment, there is some level of built-in segmentation of applications and data. They are divided by networks, and administrative privileges may be separated at the server level. Some data centers even segment their physical servers with separate rooms that require specific authorization for physical access.

Virtual servers may be operating on the same servers, logical disks, and even CPUs. In the past, physical segmentation did not require significant planning. It was more of a natural security control.

Before deploying your virtual server environment, think through how you will segment your virtual machines to compensate for the loss of physical segmentation. If you must comply with standards that require classification of critical cyber assets, you will need to spend more time to ensure that you are not commingling critical and non-critical assets on the same physical host.

Patch Management
Many companies already struggle with patch management. Virtualization will amplify the problem by making it easier to deploy servers and more challenging to patch them. If the interdependencies of applications and the lack of testing personnel hinder your patch management today, remember that virtualization will add another layer of dependencies. Now a single patch may affect more servers, more applications, and more data.

To patch virtual servers, make sure that you have policies, standards, and procedures that define acceptable practices for patching virtual servers. This should include the definition of acceptable network segments for applying patches and hardening standards for host operating systems.
Follow up with routine network vulnerability scans to report on the effectiveness of your patch management procedures. Remember that even your vulnerability assessments may need to change to accommodate specialized virtualization host operating systems, such as VMWare ESX. You may also need to redefine the severity of identified vulnerabilities to increase the priority of those found in host operating systems and hypervisors.

Server Life Cycle Management
Traditional server life cycles define the deployment, on-going operation, decommissioning, and clearing of servers. This requires good asset management controls. These controls will need to be modified to reflect the ease of copying and redeploying existing virtual servers.

Like patch management, you will need to take a close look at your policies related to the deployment, decommissioning, and clearing of servers. Establishing standards will help, but you will also need to look at asset management solutions that automate tracking of virtual assets.

Some solutions tag virtual machines and only allow authorized virtual servers to start up on a given physical host. This will help ensure that rogue devices are not started up without going through proper configuration management procedures.

Security Monitoring
If you are currently monitoring network traffic using an intrusion detection system (IDS), you may need to rethink your security monitoring strategy. Depending on the configuration, your virtual machines may be able to communicate with each other without traversing the network, which will make it impossible to monitor using a traditional IDS.

To begin with, make sure that virtual servers are not hidden behind a network address translation (NAT) scheme that prevents your IDS from distinguishing activity between virtual servers. It may require routine auditing of hypervisor configuration to ensure that each VM receives its own IP address. Also look for opportunities to deploy virtual IDS appliances on the physical servers to monitor inter-VM traffic.

Consider using your IDS to track the virtual server life cycle. This will allow you to detect when a new VM is spun up and to ensure that it is authorized on the network.

Conclusion
When jumping into server virtualization, make sure you first consider how you will need to change your practices to prevent server sprawl and security issues. Proceed with caution, and you can benefit from virtual technologies without adversely affecting your business.

Read More...

Virtualization: Red Pill or Blue?

Virtualization technologies have been compared to the movie, The Matrix. In this, Neo and other humans, are captured in a virtual world. Neo is offered a blue pill or a red pill. The blue pill will return him to his normal unreal world in the matrix. The red will set his mind free by exposing the matrix. When it comes to virtualization technologies, the red pill and blue pill have similar meanings.

Red Pill
The red pill is a small piece of code that, when run on a virtual machine, is able to determine if it is running in a virtual system or a real, physical system. It does this by detecting if the operating system is under the control of a hypervisor, the monitoring process that enables virtualization.

Theoretically, programs operating in a virtual instance should not be able to determine anything outside of the virtual instance. This encapsulation is important to ensure that the operating system operates independently.

The red pill is more than a theory. A few lines of C code are all that are needed to create the red pill code. The following sample was developed by Joanna Rutkowska:

int swallow_redpill () {
  unsigned char m[2+4], rpill[] = "\x0f\x01\x0d\x00\x00\x00\x00\xc3";
  *((unsigned*)&rpill[3]) = (unsigned)m;
  ((void(*)())&rpill)();
  return (m[5]>0xd0) ? 1 : 0;
}

More details about how Rutkowska's red pill works can be found at: http://invisiblethings.org/papers/redpill.html

Blue Pill
The blue pill takes the concept of the red pill further to create an exploit using virtualization. The most popular example of this was also developed by Joanna Rutkowska. The blue pill uses virtualization capabilities built into some microprocessors to trap the computer's operating system into a virtual machine. The blue pill acts as the hypervisor and has complete control of the regular operating system, now trapped in the virtual machine.

Rutkowska's blue pill is said to be completely undetectable, although some have disputed this claim. Whether detectable or not, it is a demonstration of one type of exploit made possible by virtualization technologies.

Bottom Line
The bottom line in this discussion of the red and blue pills is that virtualization technologies may become exploitable. This may allow an attacker to gain access to virtual servers or create an undetectable root kit. If you are using virtualization, stay on top of vulnerabilities in your software and patch regularly.

Read More...

Patch Management In a Virtual World

As more and more companies adopt virtualization in their data centers to reduce the number of physical servers and save money, security strategies need to be developed in parallel. While security may push back on this movement and resist its adoption, it will be far more beneficial to develop security strategies to deal effectively with advancing virtualization technologies.

Patch management is one area that is especially important in virtualization. While the ability to save a virtual machine and copy it easily to other physical servers streamlines deployment, it can lead to outdated software that is vulnerable for exploitation. In addition, the ability to suspend a virtual machine in its current state means that a suspended instance may be activated in a vulnerable state if patches have not been applied.

Here are a few patch management considerations when securing virtual hosts:

1. Keep the host operating system patched and hardened
Securing the host operating system is essential in virtualization. If everything works as it should, virtual machines should not be aware of their own virualization or of other virtual instances on the same physical server. This encapsulation is provided by the host operating system. Unpatched vulnerabilities in the host operating system may lead to the compromise of all of the virtual instances. Keep alert for security patches in the host operating system and have a strategy to quickly apply them.

2. Activate virtual machines securely
When deploying a virtual machine, make sure that it is fully patched before connecting to your production network. This may be accomplished by using isolated network segments where virtual instances can be activated and patched before moving them to a production environment. You may also want to consider network access control products that detect the patch levels and antivirus capabilities of new devices and restrict network access until they are patched.

3. Scan for vulnerabilities
Make sure that your regular vulnerability scans are aware of and have network access to virtual machines. Patch management is a preventive control, but many organizations struggle to keep up-to-date, so network scanning is a detective control to assess and respond to unpatched physical and virtual servers.

4. Establish policies, standards, and procedures
Make sure your policies and standards address the patch management of physical and virtual machines, server hardening, and network locations suitable for patching and deployment. If communicated and enforced effectively, these will help ensure consistency and repeatability of patch management.

5. Watch for third-party supporting products
Virtualization vendors typically do not have comprehensive solutions that address the security management aspects of their products. Keep on the look out for new products as this technology matures. A system that includes patch management and security solutions from multiple vendors may provide defense in-depth and help assure the security of your virtual environment.

What are your strategies for secure patch management in your virtual environment? What supporting products are you using? Please share your feedback in the comments section.

Read More...