In a recent post I described a very easy way to programmatically test an ASP.NET Web service using JavaScript and the XMLHTTP COM object. Windows PowerShell, the new command shell and scripting language, has generated a lot of interest in the testing community here at Microsoft, and so, several people asked me about testing a Web service using PowerShell. Well, because PowerShell can call COM objects, one easy way of testing a Web service with PowerShell is almost identical to testing with JavaScript. In both cases the idea is to use an XMLHTTP object to send a request to the Web service under test, fetch the XML response, and examine the response for an expected value. Suppose you create an ASP.NET Web Service which contains a dummy Web Method named TriMax(x,y,z) which simply returns the largest of integers x, y, and z. Listed below is the basic skeleton of a PowerShell test automation script which sends 4,8,6 to TriMax() and looks for a result of 8. The XMLHTTP object is normally used to send asynchronous requests to an AJAX Web application, but by setting the second parameter to "false" you can send a synchronous request. You can determine the HTTP POST data using Visual Studio by instructing the ASP.NET Web Service to run -- it gives you the POST template that you can copy-paste into your PowerShell script.

 

# testWebService.ps1

 

write-host "`nBegin Web service test"
$req = new-object -com "Microsoft.XMLHTTP"
$req.open("POST", "http://localhost/MyService/Service.asmx", $false)
$req.setRequestHeader("Content-Type", "text/xml; charset=utf-8")
$req.setRequestHeader("SOAPAction", "http://tempuri.org/TriMax")

 

$requestString  = '<?xml version="1.0" encoding="utf-8"?>'
$requestString += '<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">'
$requestString += '  <soap:Body>'
$requestString += '    <TriMax xmlns="http://tempuri.org/">'
$requestString += '      <x>4</x>'
$requestString += '      <y>8</y>'
$requestString += '      <z>6</z>'
$requestString += '    </TriMax>'
$requestString += '  </soap:Body>'
$requestString += '</soap:Envelope>'
 
$req.send($requestString)
$response = $req.responseText
write-host "`nResponse is $response`n"

 

if ($response.indexOf("<TriMaxResult>8</TriMaxResult>") -ge 0) {
  write-host "Pass"
}
else {
  write-host "**FAIL**"
}

write-host "`nEnd test"

 

I was teaching a technical training class on the Microsoft Redmond, WA campus last week when one of the students asked if it was possible to test an ASP.NET Web Service using JavaScript. Yes, it is possible, and quite easy. Suppose you create an ASP.NET Web Service which contains a dummy Web Method named TriMax(x,y,z) which simply returns the largest of integers x, y, and z. Listed below is the basis of JavaScript test automation. The key idea is to use an XMLHTTP object to send a request to the Web Method and grab the resulting XML response. The XMLHTTP object is normally used to send asynchronous requests to an AJAX Web application, but by setting the second parameter to "false" you can send a synchronous request. You can determine the HTTP POST data using Visual Studio by instructing the ASP.NET Web Service to run -- it gives you the POST template that you can copy-paste. Very clean and easy. 

 

// testWebService.js
WScript.Echo("\n Starting test");
var req = new ActiveXObject("Microsoft.XMLHTTP");
req.open("POST", "http://localhost/MyService/Service.asmx", false);

req.setRequestHeader("Content-Type", "text/xml; charset=utf-8");
req.setRequestHeader("SOAPAction", "http://tempuri.org/TriMax");

 

strReq = '<?xml version="1.0" encoding="utf-8"?>'
  + '<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">'
  + '  <soap:Body>'
  + '    <TriMax xmlns="http://tempuri.org/">'
  + '      <x>3</x>'
  + '      <y>7</y>'
  + '      <z>5</z>'
  + '    </TriMax>'
  + '  </soap:Body>'
  + '</soap:Envelope>';

 

req.send(strReq);
var response = req.responseText;
WScript.Echo(response);

 

if (response.indexOf("<TriMaxResult>7</TriMaxResult>") >= 0)
  WScript.Echo("Pass");
else
  WSCript.Echo("**FAIL**");

 

WScript.Echo("\n End test"); 

Creating classic COM objects with C++ is very difficult, and to a large extent .NET has eliminated the need to create classic COM DLLs for applications in a Windows environment. But classic COM objects are still needed for many types of system programming tasks, and legacy classic COM objects will be around for decades. Let me describe a few ways to call classic COM objects using some scripting languages. Suppose you have a classic COM object realized as Utilities.dll and which contains a class named CMyFunctions and associated interface named IMyFunctions which in turn contains a method named TriMax(x,y,z) that returns the largest of integers x, y, and z. A JavaScript call example is:

 

var obj = new ActiveXObject("Utilities.MyFunctions")
var result = obj.TriMax(3,7,5)

 

You can access the COM object using ActiveX technology, passing in the object's ProgID, which is stored in the System Registry and which you can think of as an alias for the COM object. A VBScript call example is:

 

Dim myObj
Set myObj = CreateObject("Utilities.MyFunctions")
result = myObj.TriMax(3,7,5)

 

This is pretty much the same. A PowerShell call example is:

 

$obj = new-object -comObject "Utilities.MyFunctions"
$result = $obj.TriMax(3,7,5)

 

Here the new-object cmdlet marshalls up the COM object. Accessing a COM object from Perl looks like this:

 

use Win32::OLE;
$obj = new Win32::OLE("Utilities.MyFunctions");
$result = $obj->TriMax(3,7,5);

 

You have to use the Win32::OLE language extension. As you can see, the ideas behind calling classic COM objects are essentially the same for most scripting languages. The choice of "best" scripting language for this task depends mostly on non-technical factors such as how familiar you are with a particular language and how much a particular language is used within your development environment. 

During the Q & A of a technical talk I delivered recently, Howard Dierking, the Editor-in-Chief of MSDN Magazine, (where I write a column on software testing called "Test Run") made a good observation. Namely, one of the disadvantages of custom, lightweight software test automation compared to test frameworks is manageability -- because software test automation is now so much easier to write than it was just a few years ago (in particular automation written using PowerShell), you can easily become overwhelmed with test case data files, test harnesses, and test results files. Howard suggested the idea of using Team Foundation Server (TFS) to manage PowerShell-based lightweight custom test automation. So, I've been looking at this idea and there's some good news and some bad news. You can think of Team Foundation Server as a large database that can store source code, perform version control, manage the build process, store bugs and other so-called work items, and store test cases and test results. Various flavors of Visual Studio essentially act as the client software for Team Foundation Server. The Visual Studio Team Test (VSTT) edition has built-in support for unit testing, some types of Web testing, a "generic test", and other test types. You can run tests on a host machine and then use VSTT to "publish" test results to the TFS databases. What I was hoping for was that there would be a simple, standard XML schema format for a VSTT results file. If so, then all you'd have to do to use TFS as a test results data store for custom PowerShell-based test automation would be to emit test results in the proper format and then (somehow) publish to TFS. It turns out that VSTT test results are stored as ".trx" ("Test Results XML") files which are just XML -- however there is no published schema for these files -- they are generated on the fly by VSTT in a fairly complex way. One way around this situation is to use the VSTT "Generic" test type which is a wrapper around a custom test harness. If you code your test harness to return a errorlevel value of 0 for pass or 1 for fail, you can execute the test from VSTT, which will produce a simple .trx file, which can be published to TFS. You can provide enhanced test results by coding your test automation to store results as XML following a simple SummaryResult.xsd schema (installed with VSTT), and then run your custom automation from a VSTT environment. VSTT will auto-magically grab your results.xml file, build up a results.trx file from the XML file, which you can then publish to TFS. There are some other approaches too. Running custom lightweight test automation from within VSTT has its pros and cons. By the way, because the backend TFS data warehouse is so complex, you really don't want to write directly to it. Also, VSTT does support a fairly well-documented custom test type that allows you to create, just once, a test type that is essentially a custom PowerShell script. I have to look at this idea further. Finally, Visual Studio 2008 is likely to have a new, significantly streamlined way of publishing test results. (Thanks to Dominic Hopton of the VS team for some good information during a phone call the other day).
 

I recently returned from a road trip through the Midwest where I spoke to the development teams at companies like Payless Shoes and H&R Block. I was there to represent my parent company Volt (staffing) as well as to evangelize Microsoft technologies and MSDN Magazine (where I'm a Contributing Editor). I learned a ton, including: Web applications and Web services are still hot topics, the job market for Web developers (especially ASP.NET developers) is unbelievably hot, and nobody does Web application UI test automation. Let me address this last point by describing four general approaches to writing Web app UI test automation, and why these approaches are fatally flawed in most work environments, and then describe what I think is a part of the solution to the problem. One approach to Web app UI test automation is to buy, learn, and use a commercial test framework. But commercial frameworks are expensive, have a significant learning curve, lead you towards testing based on the capabilities of the framework rather than the needs of your test effort, and add an external dependency to your effort. A second approach is to use an Open Source framework such as NUnit.ASP or WATIR. The downsides are similar to commercial frameworks except that you don't pay anything but in return you get questionable quality and increased instability in the tool code base. A third approach is to write custom Web app UI test automation using a language such as Java or C#. The downside here is that, in general, the time and effort required to write your automation is too much relative to the payoff you get in bugs found / improved product quality. Put more simply, you usually just don't have time to write test automation. A fourth approach is to write test automation using a scripting language such as Perl or JavaScript. The downside here is that scripting languages are generally a bit too lightweight to use for UI test automation and so the time you gain by using a scripting language is lost due to the increased complexity of your code. Anyway, I've been looking at Web application UI test automation using Windows PowerShell, Microsoft's new scripting language and shell environment and I think this approach is a big advance. PowerShell can call the .NET Framework directly, has built-in cmdlets which shorten the size of your scripts (and therefore development time), has an interactive mode which greatly speeds up script writing and gives you a virtual help system, and has a modern set of command structures and error-handling capabilities. I've written up an article describing the details for MSDN Magazine and hope to see it in print soon. I demonstrated Web application UI test automation with PowerShell to several companies and groups in the Midwest and it generated a lot of interest.