Spinning Out of Control: Securely Managing Virtual Sprawl

Server virtualization is taking hold. It boasts so many advantages that it is likely to become the standard for data centers around the world. It saves money by maximizing hardware resources. It reduces the number of physical servers, which reduces power consumption. It also revolutionizes server deployment by allowing servers to be copied as easily as files on the file system. Add to this the benefit of using pre-configured virtual appliances, and you should be convinced that virtual servers are good for your business.

As with any new technology, security tends to be an afterthought. Many companies that venture into virtual technologies expect that their existing security controls will apply to the new virtual environment, but virtual servers require new security approaches and controls.

In an IT department with weak or missing security controls, virtual servers will sprawl across physical hardware and quickly become unmanageable and vulnerable to attack. To jump into virtualization, it’s essential to be innovative about security and to reinvent your security controls.

Segmentation
In the physical server environment, there is some level of built-in segmentation of applications and data. They are divided by networks, and administrative privileges may be separated at the server level. Some data centers even segment their physical servers with separate rooms that require specific authorization for physical access.

Virtual servers may be operating on the same servers, logical disks, and even CPUs. In the past, physical segmentation did not require significant planning. It was more of a natural security control.

Before deploying your virtual server environment, think through how you will segment your virtual machines to compensate for the loss of physical segmentation. If you must comply with standards that require classification of critical cyber assets, you will need to spend more time to ensure that you are not commingling critical and non-critical assets on the same physical host.

Patch Management
Many companies already struggle with patch management. Virtualization will amplify the problem by making it easier to deploy servers and more challenging to patch them. If the interdependencies of applications and the lack of testing personnel hinder your patch management today, remember that virtualization will add another layer of dependencies. Now a single patch may affect more servers, more applications, and more data.

To patch virtual servers, make sure that you have policies, standards, and procedures that define acceptable practices for patching virtual servers. This should include the definition of acceptable network segments for applying patches and hardening standards for host operating systems.
Follow up with routine network vulnerability scans to report on the effectiveness of your patch management procedures. Remember that even your vulnerability assessments may need to change to accommodate specialized virtualization host operating systems, such as VMWare ESX. You may also need to redefine the severity of identified vulnerabilities to increase the priority of those found in host operating systems and hypervisors.

Server Life Cycle Management
Traditional server life cycles define the deployment, on-going operation, decommissioning, and clearing of servers. This requires good asset management controls. These controls will need to be modified to reflect the ease of copying and redeploying existing virtual servers.

Like patch management, you will need to take a close look at your policies related to the deployment, decommissioning, and clearing of servers. Establishing standards will help, but you will also need to look at asset management solutions that automate tracking of virtual assets.

Some solutions tag virtual machines and only allow authorized virtual servers to start up on a given physical host. This will help ensure that rogue devices are not started up without going through proper configuration management procedures.

Security Monitoring
If you are currently monitoring network traffic using an intrusion detection system (IDS), you may need to rethink your security monitoring strategy. Depending on the configuration, your virtual machines may be able to communicate with each other without traversing the network, which will make it impossible to monitor using a traditional IDS.

To begin with, make sure that virtual servers are not hidden behind a network address translation (NAT) scheme that prevents your IDS from distinguishing activity between virtual servers. It may require routine auditing of hypervisor configuration to ensure that each VM receives its own IP address. Also look for opportunities to deploy virtual IDS appliances on the physical servers to monitor inter-VM traffic.

Consider using your IDS to track the virtual server life cycle. This will allow you to detect when a new VM is spun up and to ensure that it is authorized on the network.

Conclusion
When jumping into server virtualization, make sure you first consider how you will need to change your practices to prevent server sprawl and security issues. Proceed with caution, and you can benefit from virtual technologies without adversely affecting your business.

Virtualization: Red Pill or Blue?

Virtualization technologies have been compared to the movie, The Matrix. In this, Neo and other humans, are captured in a virtual world. Neo is offered a blue pill or a red pill. The blue pill will return him to his normal unreal world in the matrix. The red will set his mind free by exposing the matrix. When it comes to virtualization technologies, the red pill and blue pill have similar meanings.

Red Pill
The red pill is a small piece of code that, when run on a virtual machine, is able to determine if it is running in a virtual system or a real, physical system. It does this by detecting if the operating system is under the control of a hypervisor, the monitoring process that enables virtualization.

Theoretically, programs operating in a virtual instance should not be able to determine anything outside of the virtual instance. This encapsulation is important to ensure that the operating system operates independently.

The red pill is more than a theory. A few lines of C code are all that are needed to create the red pill code. The following sample was developed by Joanna Rutkowska:

int swallow_redpill () {
  unsigned char m[2+4], rpill[] = "\x0f\x01\x0d\x00\x00\x00\x00\xc3";
  *((unsigned*)&rpill[3]) = (unsigned)m;
  ((void(*)())&rpill)();
  return (m[5]>0xd0) ? 1 : 0;
}

More details about how Rutkowska's red pill works can be found at: http://invisiblethings.org/papers/redpill.html

Blue Pill
The blue pill takes the concept of the red pill further to create an exploit using virtualization. The most popular example of this was also developed by Joanna Rutkowska. The blue pill uses virtualization capabilities built into some microprocessors to trap the computer's operating system into a virtual machine. The blue pill acts as the hypervisor and has complete control of the regular operating system, now trapped in the virtual machine.

Rutkowska's blue pill is said to be completely undetectable, although some have disputed this claim. Whether detectable or not, it is a demonstration of one type of exploit made possible by virtualization technologies.

Bottom Line
The bottom line in this discussion of the red and blue pills is that virtualization technologies may become exploitable. This may allow an attacker to gain access to virtual servers or create an undetectable root kit. If you are using virtualization, stay on top of vulnerabilities in your software and patch regularly.

Patch Management In a Virtual World

As more and more companies adopt virtualization in their data centers to reduce the number of physical servers and save money, security strategies need to be developed in parallel. While security may push back on this movement and resist its adoption, it will be far more beneficial to develop security strategies to deal effectively with advancing virtualization technologies.

Patch management is one area that is especially important in virtualization. While the ability to save a virtual machine and copy it easily to other physical servers streamlines deployment, it can lead to outdated software that is vulnerable for exploitation. In addition, the ability to suspend a virtual machine in its current state means that a suspended instance may be activated in a vulnerable state if patches have not been applied.

Here are a few patch management considerations when securing virtual hosts:

1. Keep the host operating system patched and hardened
Securing the host operating system is essential in virtualization. If everything works as it should, virtual machines should not be aware of their own virualization or of other virtual instances on the same physical server. This encapsulation is provided by the host operating system. Unpatched vulnerabilities in the host operating system may lead to the compromise of all of the virtual instances. Keep alert for security patches in the host operating system and have a strategy to quickly apply them.

2. Activate virtual machines securely
When deploying a virtual machine, make sure that it is fully patched before connecting to your production network. This may be accomplished by using isolated network segments where virtual instances can be activated and patched before moving them to a production environment. You may also want to consider network access control products that detect the patch levels and antivirus capabilities of new devices and restrict network access until they are patched.

3. Scan for vulnerabilities
Make sure that your regular vulnerability scans are aware of and have network access to virtual machines. Patch management is a preventive control, but many organizations struggle to keep up-to-date, so network scanning is a detective control to assess and respond to unpatched physical and virtual servers.

4. Establish policies, standards, and procedures
Make sure your policies and standards address the patch management of physical and virtual machines, server hardening, and network locations suitable for patching and deployment. If communicated and enforced effectively, these will help ensure consistency and repeatability of patch management.

5. Watch for third-party supporting products
Virtualization vendors typically do not have comprehensive solutions that address the security management aspects of their products. Keep on the look out for new products as this technology matures. A system that includes patch management and security solutions from multiple vendors may provide defense in-depth and help assure the security of your virtual environment.

What are your strategies for secure patch management in your virtual environment? What supporting products are you using? Please share your feedback in the comments section.

Six Sigma for Security, Part 5: Staying in Control

After you have moved through the define, measure, analyze, and improve stages of your Six Sigma for Security project, you are ready to move into the control phase. Your security operations should always operate in the control phase. If so, the control phase for Six Sigma is merely a matter of adding your improvements into your existing controls.

Here are some tools that you may find helpful in the control phase:

• Control matrices
• CTQ dashboards
• Control charts
• Performance graphs

The more of your metrics you can automate, the more successful you will be in the control phase. If you have to spend time every week or every month to pull data, manipulate it, and report on it, you will quickly stop measuring. Use Excel and Access extensively to make routine measurement and reporting a snap.

The control phase does not use the detailed analysis tools used earlier in the Six Sigma project. It only reports on the results. If you find that you are no longer meeting your targets in any CTQ goal, go back through the Six Sigma process again to see what caused the change. You may have something influencing the process causing variation, or the quantity of work may require you to re-examine resource allocation.

Have you made it this far through the Six Sigma for Security process? How has it worked for you?

Six Sigma for Security, Part 4: Making Improvements

You have made your security program measurable. You have identified problem areas that keep you from meeting targets in your critical to quality (CTQ) goals. Now it is time to move on to making improvements and measuring their effectiveness.

One of the great things about Six Sigma is that improvements are tied to specific metrics, with a goal to reduce variation and improve your average. To make this work, continue with your CTQ goals, tied to root causes for problems within those goals. Each improvement project should be tied to these root causes. This is the first stage of the project in which you get to look at solutions.

One of the easiest ways to identify the best improvement opportunities is brainstorming. For each root cause problem, work with a team to list as many solutions as possible. They don't have to be practical or affordable. Your goal is to find as many creative solutions as possible. When working with a team, ask the team not to place any value judgment on any of the solutions. Instead, rapidly throw out ideas and keep moving.

Next rank each improvement idea in each root cause area according to the following attributes, such as on a scale from 1 to 4:

• Ease - Rate each improvement on how easy it is to implement. You are looking for quick wins.
• Impact - Rate the likeliness of the solution solving the root cause problem. You may find good solutions, but they may not be for the problems you are trying to address. Make sure you are tying it to the root cause.
• Affordability - Rate how affordable this solution is.

Multiply the above factors together to create a score for each improvement idea. The top scores are your best choices to make measurable improvements. They combine positive impact with ease of implementation and affordability. By tackling improvement ideas starting with the highest score, you can use your limited resources in the most effective way.

After implementing each improvement, measure again and see if your CTQ metrics have improved. If they have improved dramatically, congratulations. You have found the best solution. If they have improved slightly, try moving down your list of improvements to see if the next solution will help.

It is important to note that not every idea on your improvement list should be implemented. Only the top scoring ideas. The other ideas have less impact, take too long, or are too expensive. After trying your top improvement ideas, discard your list and start over to find valuable solutions.

Finding the best, measurable improvement opportunities for security management departments is important. Companies keep security lean. Six Sigma for Security will help you to make the best use of your time and resources.

Six Sigma for Security, Part 3: Analyzing the Data

Making security measurable is only the beginning of Six Sigma for Security. The purpose of security metrics is not the metrics themselves but what you do with them. The next phase is to analyze the data and find opportunities for improvement.

The analysis phase is what makes Six Sigma stand out. In it you will use statistics to find where you can make improvements. There are two improvements in your measurements that are important: 1) average performance; and 2) variation.

Your average performance is an important measurement because it allows you to track and trend how well you are doing at protecting information assets. But performance may vary widely around an average. For example if your average turn around on a user termination is 15 days, it may be that half of your requests are 30 days. Some may even linger for 90 days. This is very wide variation around the 15 day average.

Your goal in the analysis phase is to quantify variation and identify factors that can positively influence your performance. This is accomplished by using a variety of statistical analysis tools.

One of the most popular and easy-to-use tools is Pareto analysis, which allows you to quickly find the biggest problem areas in your metrics. It also results in a distinctive chart that shows you frequency and a cumulative percentage to visually quantify the impact of improvements.

Analysis is one of the most challenging phases of Six Sigma. Until now the process has been very consistent and predictable. In this phase, you may try analyzing data one way, find nothing worthwhile, and try it again in several different ways. It is a trial and error process in which you don't stop until you find something of significance that can lead to improvement.

At the end of the analysis phase you should end up with root causes for problem areas in your security metrics. If you are confident that you have arrived at the root cause, you are ready to move on to the improvement phase.

Learn more about Six Sigma tools on Wikipedia.

Six Sigma for Security, Part 2: Measuring

After you have defined the Critical to Quality (CTQ) goals of your security program, the next step is to determine how to measure them. Creating a measurement plan for security is like creating a scoreboard. It allows you to gauge at any time how successful you are and your level of protection.

Keep in mind that at the outset of this Six Sigma project we determined that we would not just measure the results of security, which should be few and far between. Instead, we are focusing on measuring the level of protection.

In the measurement phase, we take the four CTQ goals and create a measurement plan for each. Here are sample metrics you may decide to capture.

Policy

• Annual review/approval of policy completed
• Percentage of policies with corresponding controls
• Number of policy exceptions documented
  

Process

• Percentage of controls followed as scheduled
• User access management SLAs
• Incident response times

Partnerships

• Number of security awareness sessions completed
• Number of monthly cross-department meetings completed
• Number of customer contacts made

Payoff

• Internal policy violations
• Security incidents resulting in loss
• Virus/malware incidents
• Compliance gaps

These measurements should get you started. To determine your measurements, brainstorm to create a list of everything that would be nice to measure. Then decide which three or four are the essential metrics. If you have more than three or four measurements for each CTQ, you will quickly be overwhelmed with data.

The next step in building your measurement plan is to identify the data source for each of the metrics. You may find that you don't have a good source of data for some. In these cases, consider ways to begin capturing it now. You may simply use a data collection worksheet to keep track. Keep it simple, but automate it whenever possible.

With your CTQ goals defined, your metrics for each determined, and your data sources identified, you are ready to begin collecting data. For some measurements, you will be able to retrieve historical data and begin your analysis immediately. For new data sources, you may need to capture data for a month or two before you have enough information for analysis.

Complete your measurement plan and capture data as soon as possible. This should be the quickest phase of the Six Sigma for Security project. It will establish a baseline that allows you to set your goals for improvement. It will also allow you to move into one of the most interesting phases -- analysis.

What are some of the metrics you are using to keep score for your CTQ goals?

Six Sigma for Security, Part 1: Defining Success

How do you measure the effectiveness of your security program? Unlike business metrics which have tangible goals, such as revenue growth, inventory reduction, and sales force effectiveness, security's goal is to prevent internal and external breaches. Your goal in security should be to have nothing to measure - no virus attacks, no external breaches, no successful social engineering attacks, no insider exploits.

Although measurement of security incidents and related losses has value, it is not sufficient for measurement of security effectiveness or for justifying the cost of security investment. It is important to have a different standard for making security measurable.

Instead of measuring loss, this article proposes measuring prevention. Security management changes over time, but it is generally accepted to consist of a set of best practices for prevention of business loss caused by security breaches. If you do well at following the standards established by ISO27001, ITIL, ISM3, or CoBIT, you will have exercised due care and will have have provided excellent protection for your information.

But how do you know how well you are doing in following these standards? This is where Six Sigma comes in. Six Sigma is an improvement process that is commonly used in manufacturing. In recent years it has been applied to various types of businesses. The goal of Six Sigma is to define how to measure success, measure the current performance, analyze the data to find root causes of problems, implement improvement projects and test their effectiveness, and move newly implemented processes and practices into a controlled phase. To remember this project life cycle, the acronym DMAIC is used. It stands for:

• Define
• Measure
• Analyze
• Improve
• Control

Applying Six Sigma to security management means walking through the DMAIC process. The most critical stage is the Define stage, where the Critical to Quality (CTQ) goals are defined. These goals are carried through the process and are used to measure success. For most uses of Six Sigma, CTQ goals are defined by finding the voice of the customer. In security management, because of the growing body of knowledge and best practices, they are easier to define.

Consider these four CTQ goals for measuring your security program. They are especially geared toward measuring how well you are performing standard best practices in security management.

1. Policy - Measuring the quality and enforceability of your security policies
2. Processes - Measuring how well you follow defined practices according to the standards you adopt
3. Partnerships - Measuring your efforts to build relationships with other departments and customers
4. Payoff - Measuring incidents and resulting business loss

With the CTQ goals of security defined, the next step is to create a plan for measuring them and begin capturing this data.

These CTQ goals for security are generic. Do they work for your security organization? Please share your comments.

Privacy on the Web, Part 4: How to Hide from Web Beacons

Why would I want to hide from web beacons and consolidated web traffic analysis? I don't have anything to hide. We each make decisions about how much privacy and security to give up to gain convenience. The settings in web browsers - to save passwords, accept third party cookies, and keep authenticated sessions persistent over many days and across many sites - make using the web easier. For some people this is an acceptable trade off. For others it is a more serious matter.

You may want to consider some privacy measures if you are:

• A regular user who wants to keep your web browsing habits out of the hands of marketers
• A parent who wants additional protection for your children online by hiding their IP address
• A member of the military or are involved in covert activities
• A citizen in a country that monitors the web and enforces information standards
• A whistle blower who wants to remain anonymous
  
• A journalist, writer, or blogger who publishes sensitive information
• An activist concerned about privacy
  

At the most basic level, hiding from web beacons is as easy as turning off cookies in your web browser. Unfortunately, many web sites won't work if you do not enable cookies. You can limit the exposure of your web browsing by clearing your cookies frequently, such as each time you close your web browser. This segments the trail of information about your web surfing habits and makes your browsing less identifiable across web sites and over a period of time.

This approach only helps to reduce your exposure to web beacons. It does not protect the normal web traffic that is part of the HTTP protocol. HTTP traffic is the network information that passes from your web browser, over the Internet, to a web server, and back again. It is what makes the web work. Each request contains your IP address and the page you are requesting. It also contains the web address of the web site you visited, if you clicked on a link to get there.

If you are really serious about web privacy and feel that you need to hide from web beacons and HTTP traffic analysis, you need a complete solution that bounces your web traffic off of several relays and manages cookies. One solution for this is Tor and Privoxy.

Tor is short for "The Onion Router". It uses relays distributed across the Internet to hide your HTTP traffic from the web sites you visit. When configured correctly, it provides a high degree of privacy on the web. It does not, however, provide protection from web beacons, which run from the content of the web pages, usually using Javascript.

Privoxy provides a flexible solution for handling of cookies and blocking various types of content. When used with Tor, it provides the content-level privacy from web beacons.

Both Tor and Privoxy are freely available, but they may require some time to learn and configure correctly. There are also commercial solutions available that may simplify the setup and configuration.

How much web anonymity is right for you? You need to decide the right balance of convenience and privacy for yourself. When writing this series of posts, I tried some of the measures described here, but found that I'm more in favor of convenience. I have gone back to allowing cookies, web beacons, and HTTP traffic that can be traced to me.

How much privacy is right for you? Please share your thoughts in the comments section.

Privacy on the Web, Part 3: Analytics, Recommendations, Summarizing, and Anonymity

Since web beacons from outsourcing companies may be able to track your every move, you may wonder what they are doing with your information. This post discusses the positive side of collecting and using this information. It also touches on the issue of anonymity and privacy.

Analytics
If you are running a web site today, you are probably using some form of web analytics. From the multi-billion dollar retailer to the blogger who publishes his rants, web analytics are easy to implement and provide a gold mine of information about your visitors. For this web site, I use a web beacon (some Javascript provided by a major analytics company) to collect traffic data to answer questions like:

• What is the most popular content?
• How did users get to the web site?
• What keywords were used in search engines to get to the site?
• What was the most used landing page?
• How many pages per visit did users view?

Analytics, when used locally by a web site publisher, allow the publisher to enhance content and better reach an audience. For the web publisher, using the web beacon approach to gather these metrics is not only the easiest, it is also the most accurate approach. This is because legitimate web indexing services crawl web sites regularly and inflate traffic data. Web beacons that use Javascript do not record this data, since the indexing services do not execute Javascript.

Analytic data is summarized data. Although the raw data contains information about individuals and their behaviors on the web, companies who use the data aggregate it and use it to draw conclusions about all users -- not individuals.

Recommendations
Companies use data collected from web beacons to feed their recommendation engines. This along with other sources of data helps them to present products that you may be interested in. Some recommendation engines will use the data to group users into virtual communities of people with the same interests, which broadens their ability by recommending products that others in your community are buying. Recommendations work with data at the individual level, but for this use, companies don't view the data.

Recommendation data is information about the individual user. An automated process works with the data to identify you personally and serve you recommendations. The user's web behavior is probably never reviewed by people, unless someone is debugging problems with the recommendation engine.

Targeting the Individual
Some web beacons collect information that you have submitted to the web site you are viewing. This may include your email address, user name, or account number. Companies may use this to follow up. After identifying you personally, you may be tracked to see what you purchased. The company may follow up with you individually or use this information for targeted email marketing.

Crossing the Boundaries
None of this may seem out of line to you. Most organizations that use web beacons to collect information about you have no harmful intent but rather aim to make your experience better. The potential issue lies with the collection of this data by large companies that cross company boundaries. Because they are a common collection point, they have the ability to match data from multiple web sites.

One privacy issue is that the privacy policy of the web site you visit may not be honored by the web beacon data collection company. This information may be provided to third parties or used in ways in which you have not agreed.

Another privacy issue is that this creates more repositories of rich user data that may or may not be protected with adequate security controls. It is subject to insider threats and may be used for corporate espionage and unsolicited email.

What do you think about the collection of web behavioral data that crosses web sites? Does it threaten your privacy?