In my last blog entry I described how you can use Microsoft Virtual PC 2007 to help test various software configurations. You can also use Microsoft Virtual Server 2005 for configuration testing. As a very general rule of thumb, Virtual PC 2007 is intended to host desktop OSes like Windows XP and Windows 2000, and Virtual Server 2005 is intended to host server OSes such as Windows Server 2003. However, I have experimented with many different combinations of host and guest OSes on both Virtual PC 2007 and Virtual Server 2005, and haven't run into any problems. In the screenshots below, I have a physical host machine which is running Windows Server 2008, and I install a virtual guest machine which is running Windows Server 2003. Notice that Virtual Server 2005 uses a Web-based interface instead of a Form-based interface like that used by Virtual PC 2007. Also, Virtual Server 2005 has a fully scriptable interface (although I'm not sure exactly what you can automate yet). Microsoft's Hyper-V virtualization product will ship later this year. I see Hyper-V as a successor to Virtual Server 2005.

 

 

 

Configuration testing can be very painful. There are almost an infinite number of combinations of possible hardware (CPU, RAM, disk, etc.), operating system (Windows XP SP2, SuSE Linux, etc.), and relevant software (IE6, IE7, Firefox 2.0, etc.) to deal with. Here at Microsoft we maintain lots of configuration labs which have all sorts of different physical configurations, but that is a very expensive proposition most organizations just can't afford. One way to help ease (but not eliminate) the pain of configuration testing is to use Virtual PC 2007. With VPC 2007 you can set up virtual "guest" machines with just about any configuration you like, running on a real "host" machine. The image below shows I have created a virtual guest machine running the SuSE 9.0 Linux-based OS with the Konqueror Web browser on my real host machine which is running Windows Vista. Setting up virtual machines is quite easy. VPC 2007 is available as a free download. VPC 2007 is intended primarily for use on host machines which run desktop OSes such as Windows Vista and Windows XP, but also runs just fine on server OSes such as Windows Server 2003. You can create virtual host machines with most desktop OSes, even, as my screenshot shows, non-Microsoft OSes. Each virtual machine consists of a large .vhd (virtual hard drive) file plus a small .vmc (virtual machine configuration) file. This means you can create and then save as many virtual machines as necessary. After installing VPC 2007 on your host machine, you can click on the New button on the VPC mini control panel to launch a wizard to walk you through creating a new virtual machine. Then you insert a bootable OS of your choice into your host CD drive, the install will launch, and you install the guest OS as normal. Next you can add and configure software onto the guest. When you close the VPC control, all changes will be saved to your .vhd and .vmc files and can be used later. Using VPC 2007 is most suitable for basic manual functionality testing. Alternatives and complements to using VPC 2007 for configuration testing are creating multi-boot systems and using the more powerful Virtual Server 2005 product.

Several of my blog entries here have described various mathematical techniques that are really useful in software testing. One of the most very useful techniques is the chi-square test for goodness of fit. This test applies to many situations, but is not often used in practice. I suspect this is because most of the testers I know do not understand the chi-square test and therefore do not recognize situations when the test is useful. The chi-square test can be used to determine how well a set of actual results match a corresponding set of expected results. Here's an example. Suppose you have some software system that is supposed to randomly spit out a total of 100 of the 5 letters 'A' through 'E'. Therefore, you'd expect to get 20 of each letter, subject to a certain amount of variation. Suppose you run the system and get this actual data:

 

A = 12, B = 10, C = 20, D = 30, E = 28

 

The chi-square statistic is the sum of the squared differences between each observed and expected pair of numbers divided by the expected number. In this case the chi-square statistic is (12-20)2/20 + (10-20)2/20 (20-20)2/20 + (30-20)2/20 + (28-20)2/20 = 3.2 + 5.0 + 0.0 + 5.0 + 3.2 = 16.4. The number f degrees of freedom for chi-square is just k-1, the number of categories minus one, in this case df = k-1 = 5-1 = 4. Now we can look up the 95% critical value from any stats book and find it is 9.49. Because our calculated chi-square value of 16.4 is greater than the critical value of 9.49, we conclude that the software system is not performing as it should -- there is less than a 5% chance we'd get the observed data if the system is actually spitting out evenly distributed letters. (Excel can do chi-square -- see the image below -- the .000324 is the probability that the observed numbers match the expected numbers). There's a lot more to chi-square but once you realize the situations in which chi-square can be used, you'll be surprised at just how useful chi-square is in software testing.

 

There are a lot of activities that go on in a software testing environment. Consider a situation where you want a group of people to evaluate a set of options and choose one, single best option. For example, suppose you are creating a Web-based tool program for use on your company's intranet. You have designed four quite different user interface prototypes, and you ask a group of people to evaluate each prototype and rank their preferences from best to worst. Now the question is just how can you make sense of this data? There is a large body of knowledge on various techniques to analyze group evaluations of a set of alternatives designed to select the best option. However, I've noticed that these techniques are almost totally unknown to most software testers. There are many fascinating effects in group analyses. Suppose you have four options (A,B,C,D) and 14 evaluators, and some data like this:

 

A > B > C > D according to 8 people
B > C > D > A according to 4 people
D > C > A > B according to 2 person

 

Suppose you decide to give 3 points for a top ranking, 2 points for a second place, 1 point for a third, and 0 points for a fourth place. Then option A has 26 points, option B has 28 points, option C has 20 points, and option D has 10 points. So option B is best. But then suppose someone observes that option D is such a loser it shouldn't have even been there in the first place. You agree and toss out option D and now the data looks like:

 

A > B > C according to 8 people
B > C > A according to 4 people
C > A > B according to 2 person

 

But now if you re-compute you'll see that option A has 18 points, B has 16, and C has 8, and so now option A becomes the winner instead of option B! Anyway, I am writing up a primer on collaboration techniques for the monthly Test Run column in MSDN Magazine. I also intend to submit a paper on this topic to the Pacific Northwest Software Quality Conference, and present it in October.