One of my recent articles in Microsoft's MSDN Magazine describes how to write powerful test automation for SQL stored procedures using LINQ technology ( see http://msdn.microsoft.com/en-us/magazine/cc500645.aspx ). In that article I write, "Determining expected state is the most time-consuming task when testing stored procedures. You must manually determine the expected state of your test bed database after a stored procedure has been called." This weekend an MSDN reader sent me a question about this statement and one which I receive often. The reader's question essentially asked, "Creating test case data is very painful. Is there any way to automate the creation of test case data?" I wrote back to the reader that he had essentially asked one of the big questions in software testing, and that the short answer is that there is currently no good way to automate the generation of test case data. Basically, any test case generation system needs to know what valid inputs are to a particular state, and what the corresponding outputs and expected states are. One approach is to use Model Driven/Based Development where you use a modeling language such as UML to define a software system, and then use tools to automatically generate source code and test case data. The problem with MDD/MBD is that it only works for relatively simple application programs (as opposed to system software) and costs significantly more than traditional techniques. Another approach to automatic test case generation is to formalize the specifications of your software system -- defining inputs and outputs in a very rigorous way. Microsoft Research has been working on an extension to the C# language called Spec# which takes this approach. It is not currently usable in a production environment but I believe Spec# is a good approach in general and is the way in which automatic test case generation will eventually become a practical reality. A third approach to automatic test case generation is Model Driven Testing -- you create a simplified model of the actual system being developed and then test the model instead of the actual system. This is truly a wacky idea that just doesn't work in practice. So, for the next few years at least, test case generation is a time-consuming activity where a tester's intuition and experience play a big role.

I recently spoke at the Microsoft Management Summit. MMS is a huge event and this year had over 4,000 attendees. All four of my talks were introductory level talks about PowerShell, Microsoft’s new shell and scripting environment. Over the course of delivering training to many hundreds of people, I’ve discovered that beginners are often confused about the difference between the select-object and the where-object cmdlets. In short, if you are dealing with a collection of objects, and you think of that collection as a table with rows and columns, then select-object is used to select columns, and where-object is used to select rows. There are several exceptions that I’ll describe in a moment, but this simplification is a useful mnemonic. Consider these PowerShell commands:

 

(1)> $p = get-process
(2)> $p | more
(3)> $p | get-member | more
(4)> $p[0].Name
(5)> $p[0].ProcessName
(6)> $p[0].get_ProcessName()
(7)> $q = $p | select-object name,id,handles
(8)> $q
(9)> $q | format-list
(10)> $r = $q | where-object { $_.handles –gt 500 }
(11)> $r | format-list

 

The (1) command fetches process information into a $p array of objects. The (2) command displays the $p array as a table. Each row represents one process and each column is a property. The (3) command shows that objects can have properties, property aliases, and underlying methods. The (4),(5), and (6) commands shows how to get at the ProcessName “column” of the $p “table”. The (7) command uses select-object to conceptually select just the Name, ID, and Handles columns of the $p “table” and store into a $q object. The (8) command displays the result but illustrates that results can scroll off to the right of a PowerShell shell. The (9) command shows how to print collections of objects nicely. The (10) command uses the where-object cmdlet to select just those rows where the Handles column is greater than 500. Notice that where-object is an implied loop and so the curly brace notation is required. The $_ means “the current object in the loop.”

 

Now there are a few details. Confusingly, the select-object cmdlet supports –first n and –last n parameters; this means you can select rows using select-object. Also some cmdlets support parameters such as –filter, and –include which can be used to select rows. Furthermore, instead of using the implicit loop of the where-object cmdlet, you can use the explicit foreach-object cmdlet. Well, all these (and other) exceptions aside, if you remember the little rule, “select-object selects columns and where-object selects rows”, you’ll be fine in most situations.

 

Microsoft has two free software virtualization products: Virtual PC 2007 and Virtual Server 2005. You can use either product to help perform configuration testing. The idea is to create multiple virtual machines, each with a different operating system, on a single physical machine, and then test your system under development on the different virtual machines. Both Virtual PC and Virtual Server are built upon the same underlying, core code. An advantage of Virtual PC is that it has a very clean and easy user interface. An advantage of Virtual Server is that it is fully automatable. However, I've been doing a deep dive into Virtual Server over the past week and have discovered it is surprisingly poorly documented from the perspective of automating software configuration testing. The problem is not that there isn't enough documentation on Virtual Server -- there is a ton of it -- but rather a.) there are no complete end-to-end examples available, and b.) much of the existing information on the Internet is misleading or just plain incorrect. Anyway, I am putting together a complete (and hopefully accurate) article for MSDN Magazine which will help testers use Virtual Server for test automation. Here's a brief example of what Virtual Server automation looks like using VBScript as the automation language, leaving out a lot of details:

 

' file: demo.vbs
Set objVS = CreateObject("VirtualServer.Application")
Set fso = CreateObject("Scripting.FileSystemObject")
' use fso here to delete virtual machine, if it exists
Set objVM = objVS.CreateVirtualMachine("VirtualMachineName","D:\Someplace")
objVM.Memory = 256
' add and attach virtual hard drive here
' add and attach virtual CD drive here
' attach virtual network to default adapter
objVM.Startup()
' copy test automation to guest machine here
' run  test automation
objVM.TurnOff()

 

I expect to have an article for MSDN Magazine done in about three weeks, and an associated technical training class for engineers workng at Microsoft done about a week later. 

I recently spoke at the Microsoft Management Summit. My talks were introductory PowerShell talks. Yesterday one of the conference attendees asked me if it is possible to call a SQL stored procedure using PowerShell. I answered I wasn’t sure. So, I sat down and determined that you can in fact call a SQL stored procedure using PowerShell. The basic idea is to use “LINQ to SQL” (formerly called DLINQ). Suppose you have an existing database with a stored procedure. The steps are to first use the sqlmetal.exe tool to generate a C# wrapper file that contains all the code you need to interact with the database. Next you compile the C# code into a DLL library. Then you can use PowerShell to instantiate an object which is a proxy for the database. And then you can use that proxy object to call the stored procedure. Let me illustrate.

 

Suppose you have a SQL database named dbMovies which has a stored procedure usp_GetMovieDataByPrice. First launch a Visual Studio command shell and issue the command:

 

>sqlmetal.exe /server:(local) /database:dbMovies /sprocs /code:mapping.cs

 

This creates a C# file named mapping.cs. Next issue the command:

 

>csc.exe mapping.cs /target:library

 

This compiles the C# proxy code into a DLL library named mapping.dll which PowerShell can access. Now launch PowerShell and issue these commands:

 

> [Reflection.Assembly]::LoadFile('C:\mapping.dll')
> $cs = "server=(local);database=dbMovies;Trusted_Connection=true"
> $o = new-object dbMovies($cs)
> $o | get-member
> $ans = $o.Usp_GetMovieDataByPrice(11.11)
> $ans

 

And presto! You have called a SQL stored procedure using PowerShell. Very neat and easy. Once you can call a stored procedure, you can write a test harness which feeds input to the stored procedure and checks for an expected result to determine a pass/fail result.