There are several interesting ways you can integrate Windows PowerShell and Visual Studio. Why would you want to do this? Consider the following scenario: you are using Visual Studio to create a C# project of some sort (a Windows application, a Web application, a Class Library, etc.) You wish to use some very lightweight PowerShell based test automation to supplement your unit tests and your medium weight C# test harnesses. In some situations you might want some sort of PowerShell - Visual Studio integration rather than using each separately. One neat trick is to create a custom Visual Studio Item Type as a PowerShell test script template so that inside Visual Studio you can right click on your Project name and select Add | New Item from the context menu and have your custom PowerShell template available. The process is quite easy in Visual Studio 2005 and 2008. First, create an empty .zip folder named PowerShellScript.zip on your Desktop. Next use notepad to create your custom PowerShell template. For example:

 

# PowerShell test automation template
write-output "Begin tests"

write-output "End test run"

 

Save this file as PowerShellScript.ps1 on your Desktop. Now use notepad to create this XML meta file:

 

<VSTemplate Version="2.0.0" Type="Item"
xmlns="http://schemas.microsoft.com/developer/vstemplate/2005">
  <TemplateData>
    <Name>My PowerShell Template</Name>
    <Description>My Custom PowerShell File</Description>
    <Icon>Book.ico</Icon>
    <ProjectType>CSharp</ProjectType>
    <DefaultName>PowerShellScript.ps1</DefaultName>
  </TemplateData>
  <TemplateContent>
    <ProjectItem>PowerShellScript.ps1</ProjectItem>
  </TemplateContent>
</VSTemplate>

 

Save as PowerShellScript.vstemplate on your Desktop. You should be able to figure out most of this XML file. Notice the Book.ico icon reference. This entry is optional. I hate creating .ico files so I just used one I found. Now copy files PowerShellScript.ps1 and PowerShellScript.vstemplate (and an icon file if you decide to use one) into the PowerShellScript.zip directory on your Desktop. Now copy the entire .zip folder into the language-appropriate Visual Studio user-defined template directory. On my Vista machine this is at C:\Users\myname\Documents\Visual Studio 2008\Templates\ItemTemplates\Visual C#. You can find the exact location on your particular machine from Visual Studio by Tools | Options | Projects and Solutions | General. Now, the next time you create any kind of C# project, you'll have your PowerShell template available as a New Item entry. Very cool. At this point you can write the PowerShell script within the Visual Studio environment where you get the advantages of general file management and integrated .NET help. Now to execute your tests you'd have to launch an instance of PowerShell, navigate to your script and execute it. In my next blog entry I'll describe an interesting alternative to this last step of the process.

 

Every now and then I'll hear people argue either that software testers absolutely must know how to code, or that software testers absolutely do not need to know how to code. I believe the true fact of the matter is that it totally depends upon the situation. At Microsoft for example, virtually all testers must have some knowledge of coding. Microsoft produces all kinds of software including complex systems software. Testers must know how to code so that can understand and effectively test the systems software they're working on as well as understand, modify, and write test automation which is the only practical way to test certain aspects of some types of systems software. On the other extreme you have pure manual testers who are testing a relatively simple (compared to systems software that is) application program of some sort. Do these kind of testers need to know how to code? Well, it depends. However, I can state this with pretty much 100% confidence: the more any tester knows about coding, they better he'll be able to do his job, and the more career opportunities he'll have. But here's an opinion I have that many of my colleagues (whose opinions I respect a lot) do not agree with. I believe that, in general, knowledge of specific programming languages is (in a way that I'll explain in a moment) actually more important for software testers than it is for software developers. My logic is that it is critically important for software developers to understand algorithms and problem solving. Sure, the programming language used is extremely important so that any algorithm can be implemented accurately and efficiently, but the underlying algorithms are in most (but certainly not all) cases more important than the choice of programming language used. Software testing is primarily an analysis activity so in many (but not all) situations software testers need to be able to understand the algorithms used in the software system they're testing, but not necessarily create very complex algorithms themselves. By sort of a weird default this means that knowledge of different programming language features is more important for testers, so that can quickly implement test automation (and test cases) that test the software system they're working on as thoroughly as possible. Now I'm not saying that this algorithms vs. languages importance idea is a Boolean thing; I'm merely suggesting that all software testers can do their jobs better and expand their career opportunities by learning how to program better and by learning multiple programming languages.

SQL Server 2008, due to be released within a few months, has a ton of new features. I've been looking at RC0 (release candidate 0), trying to determine the implications SQL Server 2008 has for testing. The first step in such an analysis is to nderstand exactly what the new features of SQL Server 2008 are. One of the interesting new features is a new HierarchyID data type. You can use HierarchyID to represent hierarchical data, such as the manager and report-to relationships in a company. Of course  you can represent such relationships but HierarchyID provides a de facto standard. I didn't find any good HierarchyID examples (by that I mean examples which expose aspects of HierarchyID that are relevant to testing implications) so I experimented. Consider this example based on one in the documentation:

 

-- assume we have an existing table of employees

create table tblOrgStructure
(
orgnode hierarchyid primary key clustered,
orglevel as orgnode.GetLevel(), -- computed column
emp_id int unique not null,
emp_name varchar(50) not null,
emp_title varchar(50) null
)
go

insert into tblOrgStructure -- root node
values
(
hierarchyid::GetRoot(), -- orgnode
-- orglevel (will be computed)
111, -- emp_id
'Allen Anderson', -- name
'President' -- title
)

insert into tblOrgStructure values -- child of root
(
hierarchyid::GetRoot().GetDescendant(null,null),
222,
'Bob Baker',
'Vice President'
)

-- child of non-root
declare @managersOrgnode hierarchyid
declare @newOrgnode hierarchyid
-- 1. use manager's ID to get manager's orgnode
select @managersOrgnode = orgnode from tblOrgStructure where emp_id = 222
-- 2. use manager's orgnode to get insert position
select @newOrgnode = MAX(orgnode) from tblOrgStructure where orgnode.GetAncestor(1) = @managersOrgnode
-- 3. insert new node
insert into tblOrgStructure values
(
@managersOrgnode.GetDescendant(@newOrgnode,null),
333,
'Chris Collins',
'Director'
)

 

select orgnode.ToString(),
orglevel,
emp_id,
emp_name,
emp_title
from tblOrgStructure

 

As you can see, the trick is to determine the orgnode which is a HierarchyID type. So, what does this mean for testing? First, we'll assume that the SQL Server 2008 folks have tested the heck out of HierarchyID and its associated functions such as GetDescendant(). So, by testing HierarchyID, I really mean testing an application program (or perhaps a library module of some sort) which uses a SQL table which has one or more HierarchyID columns. And this means you'd have use standard hierarchical data testing techniques -- examine empty organization structures, org structures with just one node, and so on. 

Perl used to be one of my favorite languages for performing test automation. Perl is available on most platforms, has been around for a long time, and has extensive libraries that have all kinds of functionality such as HTTP requests. I don't use Perl very much anymore. Perl has fairly tricky syntax which makes development time a bit longer than with more modern languages such as C# and PowerShell. Anyway, I was putting together an introduction to Perl class last week and discovered a couple of things about Perl and Web programming that I didn't know before. Namely, you can use Perl (instead of the usual VBScript) to create ASP pages on the server side, and you can use Perl on the client side too. Sort of anyway. The trick is that you really have to use something called PerlScript, which is a kind of set of extensions to the Perl language. PerlScript comes from ActiveState, the most common version of Perl used on Windows based systems. Here's an example of PerlScript used to create an ASP page:

 

<%@ Language="PerlScript" %>
<html>
<body>
<h3>ASP with PerlScript Example</h3>
<p>
<%
  $t = localtime
  $Response->write("This page created at $t");
%>
</body>
</html>

 

Pretty cool. You can also use PerlScript on the client side (instead of the usual JavaScript) too. For example:

 

<html>
<head>
<script language="PerlScript">
 sub putMessage {
  $window->document->myForm->Text1->{'value'} = "Hi";
 }
</scipt>
</head>
<body>
 <form name="myForm">
 <input type="text" id="Text1" />
 <input type="button" onclick="putMessage();" />
 </form>
</body>
</html>

 

Also pretty cool but not terribly useful because the technique requires PerlScript on the client. Anyway, Perl is a very flexible language and it's amazing what you can do with it.

 

One of my least favorite software test automation tasks is parsing XML files. For example, a common scenario is to have an XML file which contains test case data (such as test case IDs, inputs, and expected values) and I've got to parse the file. For some reason, although I love coding of all kinds, I just don't get much enjoyment out of parsing XML files. Anyway, when programming in a .NET environment I use five main techniques to parse XML files. My most common approach, and the technique which is usually the only technique you'll find if you research parsing XML files, is to use the XmlDocument class of the System.Xml namespace. I read the entire XML file into memory and then use a combination of methods and properties including SelectNodes(), Attributes, SelectSingleNode(), InnerXml, InnnerText, and Item. It's quite ugly, but that's just the way XML is sometimes. A second approach is to use the XmlTextReader class. This technique works well when you have very simple XML files that have a consistent structure with very few levels. A third approach is to use the XmlSeralizer class. This technique is sometimes a god approach if you need to perform processing of your XML data once it's been parsed into memory, because you end up with an array of objects. A fourth XML parsing technique is to read XML data into memory using the XPathDocument class. This approach is quite rare and I use it only when I need to perform a lot of searching through the XML data -- the XPathDocument class is optimized for search with XPath queries. My fifth XML paring approach is to use a DataSet object. Here I read my entire XML file into memory as a DataSet object using the ReadXml() method. Once in the Dataset object, I can use DataSet methods and properties such as GetChildRows() and DataRow to extract individual XML data. There's no real moral to this blog -- parsing XML files simply isn't very much fun. The five techniques I've listed here meet most of my test automation scenarios. Here's an article I wrote some time ago that gives you all the details and a bunch of complete examples: http://www.ddj.com/windows/184416669.

Identifying risks is an absolutely essential activity for all software projects. A risk is an event which is unpredictable and which has negative consequences. You must identify project risks, so that you can analyze the likelihood and impact of the risks (see a nice overview at http://www.rand.org/pubs/working_papers/2004/RAND_WR112.pdf), and then plan to prevent the risks (in part, through testing) and mitigate the effects of risks if they do occur.

 

Technique #1 - At a very high level of granularity, you take each of the four components of any project (time, cost, quality, features) and ask yourself, what could happen to make any of these four components go wrong. For example, "In my project, what events would make me go over budget?" Pros: easy, good way to get started. Cons: too high level to catch all risks.

 

Technique #2 - At a slightly lower level of granularity than Technique #1 above, you can walk through a taxonomy. This is simply a classification of generic software risks. There are many software project risk taxonomies available. One commonly used taxonomy was published by the Software Engineering Institute. It is available in an Appendix in a document at http://www.sei.cmu.edu/pub/documents/93.reports/pdf/tr06.93.pdf. The SEI taxonomy is structured into various categories and sub-categories as a list of 194 questions. Examples include, "[159] Are the development facilities adequate?", and "[3] Are there any TBDs in the specifications?" Pros: relatively easy and mechanical. Cons: can take a long time, too generic to catch risks specific to specialized projects.

 

Technique #3 - A less generic approach of risk identification is to walk through your software project specification documents, and ask what could go wrong for each feature, activity, and person. Pros: quite specific. Cons: only as good as the specification documents.

 

Technique #4 - If you use traditional project management techniques, you will have a Work Breakdown Structure. For each leaf Work Package activity, you ask yourself, what risks are associated with the activity. Pros: Can be applied to many levels of work package granularity, leverages an existing WBS. Cons: requires a WBS, often emphasizes activities and deemphasizes resources.

 

Technique #5 - An unusual approach, but one which I've used quite successfully, is to storyboard. You assume various roles (end user, developer, etc.) and walk through various scenarios, asking yourself what could go wrong at each step. Think about going on a plane trip: drive to airport (bad traffic risk), park at airport (no parking available risk), etc. Pros: effective in practice, very specific. Cons: more art than science.

 

(Semi) Technique #6 - A theoretical but promising approach is explained in a research paper at http://iag.pg.gda.pl/iag/download/Miler-Gorski_Risk_Identification_Patterns.pdf. The authors present a possible structured way to identify risk by first identifying six project components: activities, artifacts, roles, practices, features and capabilities. Next you run through a list of risk patterns. For example, one specific example of a pattern is: "If Requirements <artifact> loses Clarity <feature> then Development <activity> takes more time than expected." Pros: unknown. Cons: unknown.

 

I think most software developers, testers, and managers should have a basic understanding of estimating the probability that a project will finish on time (or finish behind schedule). The technique is fairly simple. First you break your project down into manageable sized chunks. At a coarse level of granularity these chunks can be milestones (typically measured in weeks or months), or at a fine level of granularity these chunks can be work packages (typically lasting from 4 to 40 hours) that are derived from a Work Breakdown Structure. Next for each chuck you estimate how long it will take, using an optimistic guess, a pessimistic guess, and a most likely guess. Of course this is the hard part, and you have to rely on historical data from similar projects, expert judgment, or some other method. Now for each chunk you compute the duration mean and variance. How you do this depends on which probability distribution you use, but the beta distribution (along with the triangular) is the most common. The mean for a beta distribution is the quantity of (optimistic, plus 4 times most likely, plus pessimistic), all divided by 6. The variance for beta is simply the square of (the quantity of pessimistic minus optimistic divided by 6). Now you compute the sum of the means and the sum of the variances. With these you can compute a Z score as (X - M) / (sqrt(sum of variances)) and use the Normal distribution to compute your probabilities. This sounds a lot worse than it is. Here's a highly simplified example. You have three chunks, A, B, C. The means are 4.0, 5.0, and 8.0 (arbitrary units) respectively. The variances are 4.0, 9.0, and 36.0 repectively. The sum of the means is 17.0. The sqrt of the sum of the variances is sqrt(4 + 9 + 36) = sqrt(49) = 7.0. You want to know the probability that your project will take between 17.0 days (the mean) and 27.5 days. Z = (27.5 - 17.0) / 7.0 = 10.5 / 7.0 = 1.50. Looking up this  value in a Standard Normal Distribution  table you get probability = 0.4332 or 43%. As with any quantitative technique, a.) your result is just a crude estimate, b.) because in most cases even a crude estimate is better than no estimate, c.) your final estimate is only as good as your input data, and d.) the most important value from such an analysis come from setting the problem p, not the final answer.

One of my recent articles in Microsoft's MSDN Magazine describes how to write powerful test automation for SQL stored procedures using LINQ technology ( see http://msdn.microsoft.com/en-us/magazine/cc500645.aspx ). In that article I write, "Determining expected state is the most time-consuming task when testing stored procedures. You must manually determine the expected state of your test bed database after a stored procedure has been called." This weekend an MSDN reader sent me a question about this statement and one which I receive often. The reader's question essentially asked, "Creating test case data is very painful. Is there any way to automate the creation of test case data?" I wrote back to the reader that he had essentially asked one of the big questions in software testing, and that the short answer is that there is currently no good way to automate the generation of test case data. Basically, any test case generation system needs to know what valid inputs are to a particular state, and what the corresponding outputs and expected states are. One approach is to use Model Driven/Based Development where you use a modeling language such as UML to define a software system, and then use tools to automatically generate source code and test case data. The problem with MDD/MBD is that it only works for relatively simple application programs (as opposed to system software) and costs significantly more than traditional techniques. Another approach to automatic test case generation is to formalize the specifications of your software system -- defining inputs and outputs in a very rigorous way. Microsoft Research has been working on an extension to the C# language called Spec# which takes this approach. It is not currently usable in a production environment but I believe Spec# is a good approach in general and is the way in which automatic test case generation will eventually become a practical reality. A third approach to automatic test case generation is Model Driven Testing -- you create a simplified model of the actual system being developed and then test the model instead of the actual system. This is truly a wacky idea that just doesn't work in practice. So, for the next few years at least, test case generation is a time-consuming activity where a tester's intuition and experience play a big role.

I recently spoke at the Microsoft Management Summit. MMS is a huge event and this year had over 4,000 attendees. All four of my talks were introductory level talks about PowerShell, Microsoft’s new shell and scripting environment. Over the course of delivering training to many hundreds of people, I’ve discovered that beginners are often confused about the difference between the select-object and the where-object cmdlets. In short, if you are dealing with a collection of objects, and you think of that collection as a table with rows and columns, then select-object is used to select columns, and where-object is used to select rows. There are several exceptions that I’ll describe in a moment, but this simplification is a useful mnemonic. Consider these PowerShell commands:

 

(1)> $p = get-process
(2)> $p | more
(3)> $p | get-member | more
(4)> $p[0].Name
(5)> $p[0].ProcessName
(6)> $p[0].get_ProcessName()
(7)> $q = $p | select-object name,id,handles
(8)> $q
(9)> $q | format-list
(10)> $r = $q | where-object { $_.handles –gt 500 }
(11)> $r | format-list

 

The (1) command fetches process information into a $p array of objects. The (2) command displays the $p array as a table. Each row represents one process and each column is a property. The (3) command shows that objects can have properties, property aliases, and underlying methods. The (4),(5), and (6) commands shows how to get at the ProcessName “column” of the $p “table”. The (7) command uses select-object to conceptually select just the Name, ID, and Handles columns of the $p “table” and store into a $q object. The (8) command displays the result but illustrates that results can scroll off to the right of a PowerShell shell. The (9) command shows how to print collections of objects nicely. The (10) command uses the where-object cmdlet to select just those rows where the Handles column is greater than 500. Notice that where-object is an implied loop and so the curly brace notation is required. The $_ means “the current object in the loop.”

 

Now there are a few details. Confusingly, the select-object cmdlet supports –first n and –last n parameters; this means you can select rows using select-object. Also some cmdlets support parameters such as –filter, and –include which can be used to select rows. Furthermore, instead of using the implicit loop of the where-object cmdlet, you can use the explicit foreach-object cmdlet. Well, all these (and other) exceptions aside, if you remember the little rule, “select-object selects columns and where-object selects rows”, you’ll be fine in most situations.

 

Microsoft has two free software virtualization products: Virtual PC 2007 and Virtual Server 2005. You can use either product to help perform configuration testing. The idea is to create multiple virtual machines, each with a different operating system, on a single physical machine, and then test your system under development on the different virtual machines. Both Virtual PC and Virtual Server are built upon the same underlying, core code. An advantage of Virtual PC is that it has a very clean and easy user interface. An advantage of Virtual Server is that it is fully automatable. However, I've been doing a deep dive into Virtual Server over the past week and have discovered it is surprisingly poorly documented from the perspective of automating software configuration testing. The problem is not that there isn't enough documentation on Virtual Server -- there is a ton of it -- but rather a.) there are no complete end-to-end examples available, and b.) much of the existing information on the Internet is misleading or just plain incorrect. Anyway, I am putting together a complete (and hopefully accurate) article for MSDN Magazine which will help testers use Virtual Server for test automation. Here's a brief example of what Virtual Server automation looks like using VBScript as the automation language, leaving out a lot of details:

 

' file: demo.vbs
Set objVS = CreateObject("VirtualServer.Application")
Set fso = CreateObject("Scripting.FileSystemObject")
' use fso here to delete virtual machine, if it exists
Set objVM = objVS.CreateVirtualMachine("VirtualMachineName","D:\Someplace")
objVM.Memory = 256
' add and attach virtual hard drive here
' add and attach virtual CD drive here
' attach virtual network to default adapter
objVM.Startup()
' copy test automation to guest machine here
' run  test automation
objVM.TurnOff()

 

I expect to have an article for MSDN Magazine done in about three weeks, and an associated technical training class for engineers workng at Microsoft done about a week later. 

I recently spoke at the Microsoft Management Summit. My talks were introductory PowerShell talks. Yesterday one of the conference attendees asked me if it is possible to call a SQL stored procedure using PowerShell. I answered I wasn’t sure. So, I sat down and determined that you can in fact call a SQL stored procedure using PowerShell. The basic idea is to use “LINQ to SQL” (formerly called DLINQ). Suppose you have an existing database with a stored procedure. The steps are to first use the sqlmetal.exe tool to generate a C# wrapper file that contains all the code you need to interact with the database. Next you compile the C# code into a DLL library. Then you can use PowerShell to instantiate an object which is a proxy for the database. And then you can use that proxy object to call the stored procedure. Let me illustrate.

 

Suppose you have a SQL database named dbMovies which has a stored procedure usp_GetMovieDataByPrice. First launch a Visual Studio command shell and issue the command:

 

>sqlmetal.exe /server:(local) /database:dbMovies /sprocs /code:mapping.cs

 

This creates a C# file named mapping.cs. Next issue the command:

 

>csc.exe mapping.cs /target:library

 

This compiles the C# proxy code into a DLL library named mapping.dll which PowerShell can access. Now launch PowerShell and issue these commands:

 

> [Reflection.Assembly]::LoadFile('C:\mapping.dll')
> $cs = "server=(local);database=dbMovies;Trusted_Connection=true"
> $o = new-object dbMovies($cs)
> $o | get-member
> $ans = $o.Usp_GetMovieDataByPrice(11.11)
> $ans

 

And presto! You have called a SQL stored procedure using PowerShell. Very neat and easy. Once you can call a stored procedure, you can write a test harness which feeds input to the stored procedure and checks for an expected result to determine a pass/fail result.

 

I am a very big fan of Windows PowerShell, Microsoft's new command shell and scripting language. I have found PowerShell useful in software testing and many other areas too. I am delivering a PowerShell talk and facilitating some hands-on PowerShell Lab sessions next week at the Microsoft Management Summit (see http://www.mms-2008.com/ ). MMS is Microsoft's premier IT event and it's very interesting. Anyway, as always, the hardest part about learning or teaching any new programming language or technology is getting over the initial hurdles and quirks. The screenshot below demonstrates some of the PowerShell command line usage quirks. I wrote a custom startup script so that each command would be numbered. My first command:

 

PS(1) C:\> write-host -object "hi" -backgroundcolor "red"
hi

 

demonstrates a relatively verbose usage example. The write-host cmdlet can accept an -object parameter, which is what to print, and an optional -backgroundcolor parameter, which is, of course, the background color to use. My second command:

 

PS(2) C:\> write-host "hi"
hi

 

shows that some (but not all) parameters can be passed by position and therefore the parameter name is optional, as in the case of the -object parameter. Optional parameters are general the first or second parameter(s) and are usually omitted simply to save typing. My third command:

 

PS(3) C:\> write-host -obj "hi"
hi

 

shows that PowerShell is very intelligent about parsing your command line input, and can accept partial parameter nsames as long as the meaning is unambiguous. My fourth command:

 

PS(4) C:\> write-host 'hi'
hi

 

illustrates that PowerShell strings can use either double-quotes or single-quotes. The difference (not illustrated in this example) is that strings inside single-quotes are literals; but certain characters inside double-quotes will be evaluated (such as `n for a newline). My fifth command is:

 

PS(5) C:\> write-host hi
hi

 

and shows you that PowerShell will do its best to infer a type. Here PowerShell correctly believes that "hi" is intended to be a string. In general I prefer to be explicit about putting quotes on strings but that's just a matter of personal preference. My sixth command:

 

PS(6) C:\> "hi"
hi

 

is quite strange and shows you that PowerShell really tries to infer what you mean. Here, the default behavior for a string is to display it. Quite unusual! My seventh command is:

 

PS(7) C:\> hi
The term 'hi' is not recognized as a cmdlet, function, operable program, or script file.
At line:1 char:2
+ hi <<<<

 

and shows that PowerShell can't read your mind. My next two commands are:

 

PS(8) C:\> write-host "hi" -nonewline
hiPS(9) C:\> write-host "bye" -fore "yellow"
bye

 

Some PowerrShell cmdlet parameters (in fact many) do not accept any values. These are called switch parameters in PowerShell terminology. Here I tell write-host to print "hi" but then do not print the default newline. My last command again illustrates that PowerShell only needs enough of a parameter name (-fore instead of the full -foregroundcolor) to unambiguously determine what to do.

These few examples should help get you over the initial quirks of using PowerShell on the command line if you're new to PowerShell. Be sure to check out the MMS 2008 Web site. This year's event is sold out but keep an eye out for next year's event.

 

Windows PowerShell is Microsoft's new command shell and scripting language. You can think of PowerShell as a replacement for the old cmd.exe shell and .bat files. I have found that PowerShell is extremely useful in a software testing environment. I will be presenting a PowerShell talk and three hands-on labs at the upcoming 2008 Microsoft Management Summit (https://www.mms-2008.com/public/home.aspx) from April 28 - May 2 in Las Vegas. I've spoken at MMS before and it is a huge and intersting conference and has all kinds of useful information related to Microsoft based IT topics. This year is already sold out but you mighgt want to consider attending next year. Anyway, here's a brief snippet from part of my PowerShell talk where I describe how PowerShell makes it much easier to inteact with WMI.

 

Windows PowerShell provides you with a great interface to WMI functionality. WMI (Windows Management Instrumentation) is the Microsoft implementation of the CIM (Common Information Model) standard. IT administrators can use WMI to retrieve and manipulate a wide range of information about the hardware and software resources on a computer network. Working with WMI using Windows PowerShell is generally quite a bit easier than using than older scripting technologies. In this Lab Exercise you will learn how to access and control Windows services using the get-wmiobject cmdlet. The screenshot below illustrates the key ideas in this exercise.

 

1. Enter the commands:

 

(default directory)> set-location \
PS C:\> get-wmiobject -list | more

 

The key to working with WMI in Windows PowerShell is the get-wmiobject cmdlet. If you supply a -list switch, the get-wmiobject cmdlet will list all the WMI classes available to you -- typically over 1,000 of them. Many Windows-related WMI class names begin with "Win32_" so if you pipe the output of "get-wmiobject -list" to "select-string Win32_" you can view just those classes. Notice that in this special case, you pipe to the select-string cmdlet rather than the more usual select-object cmdlet.

 

2. Enter the commands:

 

PS C:\> $s = get-wmiobject -class win32_service
PS C:\> $s.count

 

The WMI class which is most often used to control Windows services is the Win32_Service class. The command above fetches information about all the services on your host machine and stores them into a collection named $s. As a general rule of thumb, when you issue a command using the get-wmiobject cmdlet, you will get either a single object (for example, with the Win32_SystemTimeZone class), or a collection of objects (as with the Win32_Service class). You can use the Count property to determine how many objects you have. Note that Windows PowerShell has a built-in get-service cmdlet you can use to control services on a host machine. However, the get-wmiobject cmdlet can access remote machines as well as local machines. For example, the command "get-wmiobject -class win32_service -computername Hercules" will fetch information about all services running on a machine named Hercules.

 

3. Enter the commands:

 

PS C:\> $a = get-wmiobject -class win32_service -filter "name='alerter'"
PS C:\> $a

 

In most situations you want to get a reference to one particular WMI object -- in this case we want a reference to the Alerter service. The command above shows a Windows PowerShell idiom to do this. The -filter switch parameter of the get-wmiobject cmdlet can select just a particular object from a WMI class with multiple objects.

(etc.)

At the upcoming Better Software Conference & Expo (http://www.sqe.com/BetterSoftwareConf/) I will be delivering a 4-hour tutorial titled "Quantitative Techniques in Software Management". One of the topics in the tutorial shows attendees exactly how to perform risk analysis. One of the most important activities in software testing is Risk Management. Risk Management has three phases: Risk Identification, Risk Analysis, and Risk Control. Risk Identification is where you list things that can go wrong with your software testing effort. Risk Analysis is where you determine how likely each risk is, estimate the impact of each risk, and compute a priority for each risk. Risk Control is where you plan what to do for each risk. Here's a simplified example from the tutorial. Suppose you identify three risks: Risk A, Risk B, Risk C. You decide to categorize risk likelihood on a three-point scale (low, medium, high). You decide to categorize risk impact on a two-point scale (low, high). Suppose you determine that Risk A has high likelihood but low impact. Risk B has low likelihood but high impact. Risk C has medium likelihood and high impact. What are the normalized priorities for each risk? Using Rank Sum Weights, Risk A has normalized priority = 0.5000 * 0.3333 = 0.1667. Risk B has priority 0.1667 * 0.6667 = 0.1111. Risk C has priority 0.3333 * 0.6667 = 0.2222. Therefore Risk C has the highest priority, followed by Risk A, then Risk B. The Better Software Conference & Expo runs from June 9 - 12, 2008, in Las Vegas. There are many excellent speakers and topics this year; check it out.

A very simple and useful quantitative technique in software testing is using the Triangular Distribution for estimation in situations with very limited data (as few as two data points). Here's an example from a half-day tutorial "Quantitative Techniques for Software Management" I am presenting at the Better Software Conference and Expo (http://www.sqe.com/bettersoftwareconf/) from June 9-12, 2008 in Las Vegas, Nevada.

Suppose you are managing an internal software project and it is ready to ship. You wish to estimate how many undiscovered bugs your system will ship with. Your system has 100 main modules. You thoroughly examine just 4 of the modules (because the process is very time-consuming and costly) and discover one module has 6 bugs, two modules have 18 bugs, and one module has 24 bugs.

 

a.  Use the Triangular distribution to determine a point estimate of the number of bugs your system will ship with.

 

a = smallest value = 6
b = largest value = 24
c = most common value = 18

mean = (a+b+c)/3 = 48/3 = 16.0 bugs

b.  Compute the Triangular variance and recast your point estimate to a 70% confidence interval (z = 1.04) estimate.

 

variance = (a2 + b2 + c2 - ab - ac - bc) / 18
         = (36 + 576 + 324 - 144 - 108 - 432) / 18
         = 14
std dev = sqrt of variance = 3.7 bugs

 

Therefore, we estimate that the system will ship with 16.0 plus or minus 3.9 bugs. The Better Software Conference has a lot of interesting talks and tutorials related to software project management. Check it out at http://www.sqe.com/bettersoftwareconf/.

 

In my last blog entry I described how you can use Microsoft Virtual PC 2007 to help test various software configurations. You can also use Microsoft Virtual Server 2005 for configuration testing. As a very general rule of thumb, Virtual PC 2007 is intended to host desktop OSes like Windows XP and Windows 2000, and Virtual Server 2005 is intended to host server OSes such as Windows Server 2003. However, I have experimented with many different combinations of host and guest OSes on both Virtual PC 2007 and Virtual Server 2005, and haven't run into any problems. In the screenshots below, I have a physical host machine which is running Windows Server 2008, and I install a virtual guest machine which is running Windows Server 2003. Notice that Virtual Server 2005 uses a Web-based interface instead of a Form-based interface like that used by Virtual PC 2007. Also, Virtual Server 2005 has a fully scriptable interface (although I'm not sure exactly what you can automate yet). Microsoft's Hyper-V virtualization product will ship later this year. I see Hyper-V as a successor to Virtual Server 2005.

 

 

 

Configuration testing can be very painful. There are almost an infinite number of combinations of possible hardware (CPU, RAM, disk, etc.), operating system (Windows XP SP2, SuSE Linux, etc.), and relevant software (IE6, IE7, Firefox 2.0, etc.) to deal with. Here at Microsoft we maintain lots of configuration labs which have all sorts of different physical configurations, but that is a very expensive proposition most organizations just can't afford. One way to help ease (but not eliminate) the pain of configuration testing is to use Virtual PC 2007. With VPC 2007 you can set up virtual "guest" machines with just about any configuration you like, running on a real "host" machine. The image below shows I have created a virtual guest machine running the SuSE 9.0 Linux-based OS with the Konqueror Web browser on my real host machine which is running Windows Vista. Setting up virtual machines is quite easy. VPC 2007 is available as a free download. VPC 2007 is intended primarily for use on host machines which run desktop OSes such as Windows Vista and Windows XP, but also runs just fine on server OSes such as Windows Server 2003. You can create virtual host machines with most desktop OSes, even, as my screenshot shows, non-Microsoft OSes. Each virtual machine consists of a large .vhd (virtual hard drive) file plus a small .vmc (virtual machine configuration) file. This means you can create and then save as many virtual machines as necessary. After installing VPC 2007 on your host machine, you can click on the New button on the VPC mini control panel to launch a wizard to walk you through creating a new virtual machine. Then you insert a bootable OS of your choice into your host CD drive, the install will launch, and you install the guest OS as normal. Next you can add and configure software onto the guest. When you close the VPC control, all changes will be saved to your .vhd and .vmc files and can be used later. Using VPC 2007 is most suitable for basic manual functionality testing. Alternatives and complements to using VPC 2007 for configuration testing are creating multi-boot systems and using the more powerful Virtual Server 2005 product.

Several of my blog entries here have described various mathematical techniques that are really useful in software testing. One of the most very useful techniques is the chi-square test for goodness of fit. This test applies to many situations, but is not often used in practice. I suspect this is because most of the testers I know do not understand the chi-square test and therefore do not recognize situations when the test is useful. The chi-square test can be used to determine how well a set of actual results match a corresponding set of expected results. Here's an example. Suppose you have some software system that is supposed to randomly spit out a total of 100 of the 5 letters 'A' through 'E'. Therefore, you'd expect to get 20 of each letter, subject to a certain amount of variation. Suppose you run the system and get this actual data:

 

A = 12, B = 10, C = 20, D = 30, E = 28

 

The chi-square statistic is the sum of the squared differences between each observed and expected pair of numbers divided by the expected number. In this case the chi-square statistic is (12-20)2/20 + (10-20)2/20 (20-20)2/20 + (30-20)2/20 + (28-20)2/20 = 3.2 + 5.0 + 0.0 + 5.0 + 3.2 = 16.4. The number f degrees of freedom for chi-square is just k-1, the number of categories minus one, in this case df = k-1 = 5-1 = 4. Now we can look up the 95% critical value from any stats book and find it is 9.49. Because our calculated chi-square value of 16.4 is greater than the critical value of 9.49, we conclude that the software system is not performing as it should -- there is less than a 5% chance we'd get the observed data if the system is actually spitting out evenly distributed letters. (Excel can do chi-square -- see the image below -- the .000324 is the probability that the observed numbers match the expected numbers). There's a lot more to chi-square but once you realize the situations in which chi-square can be used, you'll be surprised at just how useful chi-square is in software testing.

 

There are a lot of activities that go on in a software testing environment. Consider a situation where you want a group of people to evaluate a set of options and choose one, single best option. For example, suppose you are creating a Web-based tool program for use on your company's intranet. You have designed four quite different user interface prototypes, and you ask a group of people to evaluate each prototype and rank their preferences from best to worst. Now the question is just how can you make sense of this data? There is a large body of knowledge on various techniques to analyze group evaluations of a set of alternatives designed to select the best option. However, I've noticed that these techniques are almost totally unknown to most software testers. There are many fascinating effects in group analyses. Suppose you have four options (A,B,C,D) and 14 evaluators, and some data like this:

 

A > B > C > D according to 8 people
B > C > D > A according to 4 people
D > C > A > B according to 2 person

 

Suppose you decide to give 3 points for a top ranking, 2 points for a second place, 1 point for a third, and 0 points for a fourth place. Then option A has 26 points, option B has 28 points, option C has 20 points, and option D has 10 points. So option B is best. But then suppose someone observes that option D is such a loser it shouldn't have even been there in the first place. You agree and toss out option D and now the data looks like:

 

A > B > C according to 8 people
B > C > A according to 4 people
C > A > B according to 2 person

 

But now if you re-compute you'll see that option A has 18 points, B has 16, and C has 8, and so now option A becomes the winner instead of option B! Anyway, I am writing up a primer on collaboration techniques for the monthly Test Run column in MSDN Magazine. I also intend to submit a paper on this topic to the Pacific Northwest Software Quality Conference, and present it in October.