Over the past few days I've been looking at testing WCF services using a network socket based approach. I got a proof-of-concept demo program up and running. As usual, there were several glitches along the way. The key to resolving most of these glitches was to examine network traffic between a WCF service and a client, using a traffic examination tool. I used Fiddler, which has come in handy many times. See screenshot below. In a larger perspective, software testing is often all about understanding some software system. Once a system is fully understood, testing the system manually or programmatically usually comes pretty easily (relatively anyway). I am writing up an article on the technique of testing WCF services using sockets for MSDN Magazine and I'm happy with the way it is going.

 

Suppose you are testing a WCF (Windows Communication Foundation) service. In some situations you may want to view the low level traffic between a WCF client and the service. The easiest way to do this in general is to use a network analyzer such as netmon and capture the traffic as the client sends a request and receives a response. However in some test automation situations you may want to programmatically capture WCF traffic. There are several good blog posts on this topic but I couldn't find a complete end-to-end example so I experimented and came up with an example of how to do this.

 

First, let's assume we have a WCF service name MathService running in either a Console host, or in a Windows Service host, or in an IIS host:

 

 

To programmatically capture WCF traffic you can first create a Console Application client like so:

 

 

Next you can add two classes to the client like so:

 

And then you can wire up the two classes in the client like so:

 

 

When you execute the Console Application client program, WCF traffic will be echoed to the shell as demonstrated in the partial screenshot below.

 

I have been looking at Silverlight lately and have been investigating how to create UI test automation for Silverlight applications. As far as I can tell so far, it looks like there are two main approaches. First, you can use the Microsoft UI Automation (MUIA) library to create C#, shell-based test automation. A second approach is to use JavaScript-to-Silverlight interoperability and create browser-based test automation. I haven't tried ether approach but I'm fairly confident I could work out the details. A third possibility is to somehow use Silverlight technology to create UI test automation for a Silverlight application. I'm not sure how this would work, or if the technique is even possible, but the idea sounds interesting.

The .NET Framework version 4.0 supports a new programming language feature called variant methods. Let me cut to the chase and state I think variant methods are an example of solutions in search of a problem, but they're fascinating anyway. So what are contravariant and covariant methods, and how do you test them? In order to test variant methods you need to understand what they are. In short, MSDN Library documentation states that contravariance is the ability to use a less derived type, and covariance is the ability to use a more derived type than that originally specified. My initial reaction was pretty much WFH (what the heck). I couldn't entirely follow the MSDN examples so I played around with some code myself. After some experimentation, I believe that the scenario where contravariance and covariance are most likely to be used is when you 1.) have a generic interface, and also 2.) have a base class which implements the generic interface, and also 3.) a second class which is derived from / inherits from the base class. For example, suppose you have some generic interface IConsoleDisplayable<T>:

 

public interface IConsoleDisplayable<T>
{
  . . .
}

and a class Person which implements IConsoleDisplayable<>, and also a class Employee which is derived from class Person. Without contravariance and covariance the following code does not work:

 

IConsoleDisplayable<Person> p = new Person("Smith", "Chris"); // OK
. . .
IConsoleDisplayable<Employee> e = p; // error

 

and requires an explicit cast like this:

 

IConsoleDisplayable<Employee> e = (IConsoleDisplayable<Employee>)p; // OK

 

But if you declare the generic interface as contravariant by adding the "in" keyword like this:

 

public interface IConsoleDisplayable<in T>
{
  . . .
}

 

then the code above (without an explicit cast) compiles. Well this is just plain psycho on a variety of levels and seems like the C# language team had way too much time on their hands. Just because you can create some new language feature doesn't necessarily mean you should. (Of course, there's no reason why application or system developers have to use variant methods). In my opinion, the advantage gained by eliminating an explicit cast is not worth the additional complexity of a contravariant interface. But it's an interesting language concept for sure.

 

The new F# programming language allows you to write curried functions. I am not a fan of using curried functions, at least from my point of view as a software tester. By the way, the term "curried" comes from Haskell Curry, a mathematician. Consider this normal F# code:

 

printfn "\nBegin curried function demo\n"
let s1 = "Hello"
let s2 = "world"

 

let prependString p s =
  let result = p + s
  result

 

let r1 = prependString "//" s1
let r2 = prependString "//" s2
printfn "r1 and r2 are %s %s \n" r1 r2

 

The result would be, as you'd expect "r1 and r2 are //Hello //World". Now here's a way to use curried functions:

 

let curriedPrependSlashes =
  prependString "//"

 

let r3 = curriedPrependSlashes s1
let r4 = curriedPrependSlashes s2
printfn "r3 and r4 are %s %s \n" r3 r4

 

The result would be identical, "r3 and r4 are //Hello //World". From a usage point of view curried functions allow you to reduce the number of arguments passed to a function when the function is called. I don't like this from a testing point of view because currying usually involves an extra call to a non-curried helper function, which makes testing a bit more opaque so to speak. In other words, the call:

 

let r1 = prependString "//" s1

 

is more clear to me as a tester than the call:

 

let r3 = curriedPrependSlashes s1

 

But, this is really more opinion than technical and some people whose opinions I respect think curried functions are great things.

 

Fault injection is the process of manipulating a software system under test so that an error is deliberately generated. For example, suppose some SUT has a method Compute() which may throw an exception under certain conditions. In one form of fault injection you modify the source code of the SUT and recompile so that any time Compute() is called by some other method, say Main(), an exception is thrown. The ideas behind fault injection are to test how well the SUT deals with exceptions, and increase code coverage. Fault injection is rather difficult in many situations because you have to either modify the SUT source code and recompile, modify the SUT binaries, or use some mechanism which modifies the behavior of the SUT at runtime. TestApi is a relatively new and growing collection of C# code which can be used to perform software testing tasks on .NET applications, libraries, and services. TestApi is hosted by the CodePlex Microsoft open source project. Here's a snippet of what fault injection might look like:

 

FaultRule r1 = new FaultRule("MySUT.Compute(int, int)");
r1.Condition =  BuiltInConditions.TriggerIfCalledBy("Program.Main(string[])");
r1.Fault =  BuiltInFaults. ThrowExceptionFault(new Exception("Fatal"));
FaultSession session = new FaultSession(new FaultRule[] { r1 });
session.Launch(D:\\MySUT.exe");

 

There are many other ways to perform fault injection testing but the TestApi approach is worth checking out. I am working with some of the developers of TestApi to put together a research paper for presentation at the 7th International Conference on Information Technology New Generations. See http://www.vteOnline.com/ITNG2010/ for details.

I am the chair of the software testing track of the 2010 7th International Information Technology New Generations conference (see http://www.vteOnline.com/ITNG2010/ ). I am working with the authors of a paper that describes a neat software test automation API set called TestApi. In that paper the authors and I have proposed a taxonomy of different approaches to test case automation. In our model we suggest six levels of abstraction that vary across five key characteristics. The six basic structures of test we propose are: custom code, tool, library, API set, framework, and integrated system. The five key distinguishing characteristics are the extent to which the automation code is reusable, how specific or general the type of target automation tasks are, the extent to which the code is documented, the type of licensing associated with the structure which mediates the extent to which the automation code can be shared, and the extent to which the automation code is designed to allow integration with other code such as logging code, management and monitoring code, and other test automation. As with any taxonomy or set of definitions, no real knowledge is generated; the purpose is to establish a baseline and common set of terminology for more effective communication.

One of the fundamental concepts of software testing is that in most situations you cannot exhaustively test all possible inputs to your system under test because there are simply too many inputs. For example, if you are testing some poker game application and some method accepts a representation of 7 cards, if duplicate cardss are allowed, there are 52 to the 7th power inputs = 1,028,071,702,528 test case inputs. Therefore, a significant part of software testing involves various techniques designed to select a subset of useful test case inputs from the set of all possible test case inputs. Equivalence class partitioning divides all test case inputs into subsets that are equivalent in some way. The idea is that you need only test representative inputs from each equivalence class. Determining equivalence classes is much easier said than done however. A close cousin technique is boundary value analysis. Here you use input values at, above, and below the values which define equivalence classes. Research has shown that these boundary values are relatively more likely to cause errors than non-boundary values. Pairwise testing is a technique which uses all possible pairs of input values from different input parameters. Another technique to reduce the number of test case inputs is to simply send random input to the SUT. Although not particularly effective, random input testing is easy to implement and when it does reveal a bug, the bug is usually a serious crashing or hanging bug. A cousin of random input testing is a technique I call called partial antirandom testing. Here you create a set of random test case inputs which are maximally different f6rom each other. The idea is that similar inputs will reveal similar information about the SUT, so maximally different test case inputs will reveal more information than simple random input testing. The current issue of MSDN Magazine has an article I wrote about the technique: http://msdn.microsoft.com/en-us/magazine/ee309511.aspx .

 

 

I am currently working on an article for Microsoft's MSDN Magazine that describes how to write lightweight Web application UI test automation using the jQuery library. One of the technical reviewers of my article asked me about the pros and cons of using the jQuery approach compared with using the Selenium test framework and the Watir test framework. A second reviewer asked about the difference between using jQuery and using plain vanilla JavaScript. Additionally, a third reviewer pointed out that there is a new .NET based test framework named ASP.NET QA.

 

OK, let's start with jQuery vs. plain JavaScript for Web app automation. The two approaches are very similar. Using the jQuery library has the advantages of a consistent selector and manipulator syntax to get references to controls, such as submit buttons, and syntax to set and examine control values. Additionally, jQuery is mostly browser agnostic so you don't have to worry about whether code such as document.all["TextBox1"] will work with Firefox or not. One disadvantage of jQuery vs. plain JavaScript is that jQuery represents an external dependency to some extent. Now for jQuery test automation vs. using Selenium. Selenium is an excellent tool, but in my opinion a non-technical disadvantage of using Selenium is that time spent learning the Selenium framework does not help you directly with improving your developer skills. When using jQuery to write test automation you are learning skills which can be used not only for testing but also for Web development. One advantage of Selenium is that it is a big framework and has a lot of functionality at one higher level of abstraction than jQuery, so writing test automation with Selenium can be quicker than with jQuery (after you get over the initial Selenium learning curve). I am personally not a fan of Watir mostly because I just don't like coding in Ruby nearly as much as I like coding with JavaScript. So for me jQuery has the advantage of using JavaScript which directly maps to improved development skills. An advantage of Watir compared to jQuery is that the Watir framework is quite well known so there is good documentation and support. Although I'm not positive, I believe that Watir works only with Internet Explorer, which if true, is a disadvantage compared to using jQuery. Now there is also Watin, which does not have the Ruby language disadvantage, but using C# or some other .NET compliant language does not directly map to improved Web development skills either. One of the reviewers of my jQuery article pointed out that there is a new "ASP.NET QA" framework available on CodePlex, Microsoft's open source site. I haven't had a chance to examine ASP.NET QA closely but it looks pretty nice.

In some software test automation scenarios you may want the test harness to programmatically send test results via e-mail in addition to, or instead of, saving results to a file or database. In a .NET environment, this is pretty easy to do. The first step is to configure the test host machine, which is running the test automation, as an SMTP server so the host can send mail. On Windows XP, if you have configured IIS, in most cases SMTP will also be enabled, but not configured. In Computer Management, under IIS, open the SMTP Virtual Server Properties dialog. On the General tab, enter the IP address of the host machine. Port 25 will be the default. Then on the Access tab, Connection section, add the IP address of the test host machine. Now your test host can send mail. You can use the older System.Web.Mail namespace, or the newer System.Net.Mail namespace. C# code to send mail this looks like:

 

using System.Net.Mail;

try
{
  SmtpClient c = new SmtpClient("10.20.30.40", 25); // host IP, port
  MailAddress mFrom = new MailAddress("results@testHost");
  MailAddress mTo = new MailAddress("somebody@somewhere.com");
  MailMessage m = new MailMessage(mFrom, mTo);
  m.Subject = "Test Results at" + System.DateTime.Now;
  string mBody = "Number Pass = 120\nNumber Fail = 0";
  m.Body = mBody;
  c.Send(m);
}

 

If your test host machine is not running under a .NET environment, you can use the old CDONTS or the newer CDOSYS ActiveX libraries.

The new F# programming language is interesting. One possible reason for learning a new programming language is an effect that is closely related to something called the Sapir-Whorf hypothesis (SWH). In over-simplified terms, the SWH says that speakers of different languages (English, Japanese, etc.) think differently because of differences in languages. So an extension of this idea is that programmers and testers who use C# may program and test differently than those who use LISP, and so the more programming languages you know, the more programming patterns you have available. That aside, in a recent Introduction to F# Programming class I taught, most students (and I) said they were in the class out of technical curiosity and for intellectual entertainment. One of the examples I put together for the class has a nice set of thematic F# syntax and paradigms. See the image below for a sample run.

 

 1 #light
 2 open System
 3 printfn "\nBegin F# deck of cards demo"
 4 let deck =
 5   [| for suit in ["Clubs"; "Diamonds"; "Hearts"; "Spades" ] do
 6     for rank in [1..13] do
 7     yield (rank,suit) |]
 8
 9 printfn "\nThe unshuffled deck is %A \n" deck   
10 let ro = new Random(0) 
11 let shuffle (a : (int*string) array) =
12   let len = a.Length
13   for i in 0..len-1 do
14     let r = ro.Next(i, len-1)
15     let tmp = a.[r]
16     a.[r] <- a.[i]
17     a.[i] <- tmp
18 
19 deck |> shuffle
20
21 let printDeck (d : (int*string) array) =
22   let n = d.Length
23   for i in 0..n-1 do
24     if i > 10 && i < 49 then printf "."
25     elif i = 49 then printf "\n"
26     else
27       printf "card[%d] = " i
28       if i < 10 then printf " "
29       let (rank,suit) = d.[i]
30       if rank < 10 then printf " "
31       printfn "%d of %s" rank suit
32   
33 printfn "\nThe shuffled deck is: "
34 deck |> printDeck
35
36 printfn "\nEnd deck of cards demo"
37 System.Console.ReadLine() |> ignore

 

Lines 4-7 construct a mutable array of size 52, where each cell holds a tuple composed of an int (a card value from 1-13) and a string ("Clubs", etc.) Line 9 uses the %A format specifier to instruct the F# compiler to use its own judgment on how to print the array of card tuples. Lines 11-17 define a function which uses the Fisher-Yates shuffle algorithm to shuffle an array in place. Line 19 pipes the deck to the shuffle function. Lines 21-31 define a custom print function. Line 37 points out that F# is a .NET language and can access the .NET Framework. One thing I found interesting is that, more than any language I can think of, F# has many different ways to perform programming tasks. For example, Lines 29-31:

 

let (rank,suit) = d.[i]
if rank < 10 then printf " "
printfn "%d of %s" rank suit

 

can also be coded as:

 

match d.[i] with
| (rank,suit) ->
  if rank < 10 then printf " "
  printfn "%A of %A" rank suit

 

which is more F#-ish. Anyway, F# is really interesting in many ways.

 

Based on my personal experience, I have observed a large disconnect between the fields of software testing research (the kind typically done at universities) and software testing practice (as performed by companies which actually produce software). The two groups involved -- university researchers and software test engineers -- seem particularly unaware of each other's work. Well this isn't very surprising I suppose, but I'm thinking that there should be some easy way for software testing researchers and practitioners to be learn about each other's worlds. I speak at quite a few academic conferences and many of the research papers on software testing presented there are ridiculously out of touch with present or future reality, but on the other hand there are some papers that have really interesting and potentially really useful ideas. Now in theory at least, every academic conference looks for papers from practitioners -- the "call for papers" almost always says something like, "The aim of the XYZ Conference is to bring together researchers and practitioners in an effort to highlight the state-of-the-art and to present and discuss ideas and experiences to explore new directions for blah, blah, blah." However, papers from practitioners are almost never accepted at academic conferences. One reason is that writing a good academic paper is really quite difficult. Academic researchers can spend years to produce a single paper, and you have to know the "secret ingredients" of an IEEE research paper content and format. Anyway, I'm rambling today; let me cut to the chase. I am the chair of the software testing track of the ITNG 2010 Conference and I'd love to see papers from practitioners, and I do not discriminate against authors who don't have a Ph.D. degree. If you have an idea for a paper about software testing I can help you get the idea into a formal IEEE format. See http://www.vteonline.com/ITNG2010/ for a description of the conference, and see http://www.vteOnline.com/ITNG2010/Guidelines/guidelines.html for some tips about writing a research paper.

I am a Contributing Editor for MSDN Magazine and write a monthly column about software testing. As part of my background research I talk to as many software test managers as I can. One common theme in discussions with test managers is a pitfall I'll call "the test architecture error". In simple terms, it is easy for software testers to lose sight of their primary goal: analyze a software system in order to find bugs so those bugs can be fixed and improve the quality, reliability, and performance of the target system. According to many of the managers I've talked to, some software test engineers spend far too much time designing their test effort and not nearly enough time actually testing the target software system. I know I've been guilty of this in the past. Sometimes you just have to sit down and simply test the heck out of a system and not worry about architecting a test framework of some sort. A closely related scenario occurs with test automation. It is easy to get seduced into spending weeks creating some test automation only to have your automation be rendered obsolete before it can be used. One reason this can happen is related to how the job performance of software test engineers is sometimes evaluated by inexperienced managers. By creating test automation, a tester has something tangible and often impressive to demonstrate to show progress, even if the automation is not all that useful. It is much harder to demonstrate progress with a list of test cases that may in fact be far more useful than slick test automation. My point is that test automation and test architecture are useful in many situations but may not always the best way to test a system, especially an application which will ultimately be used by actual human beings. Basically, I think that if you focus on doing those test activities that best improve the quality of your target system based on all the complicated factors of your particular development environment -- maybe manual testing, designing frameworks, writing automation, or whatever -- and carefully explain to your management exactly why you are doing what you are doing, the value of your work will be appreciated.

Cryptology is the study of cryptography (schemes for encrypting data) and cryptanalysis (schemes for breaking codes). The field is very complex and testing cryptological software requires a lot of advanced knowledge. One of the things I've was looking at recently is the topic of nonlinear Boolean functions. A Boolean function accepts any number of variables, each of which can have value of only 0 or 1, and returns a single 0 or 1 value. For example f = x1 + x2x3 means "x1 or (x2 and x3). Boolean functions can be represented in a surprising number of ways, including a truth table, an equation in Algebraic normal form, and many others. Boolean functions are a cryptographic primitive -- a fundamental construct -- used for several purposes. It turns out that Boolean functions used in cryptology should have certain properties to make the systems in which they are used less vulnerable to cryptanalysis. One such property is nonlinearity, which is the Hamming distance between a function of n variables, and all possible linear functions of n variables. Whew! Like I mentioned, cryptology is very complex.

Over the past few days I have been looking at browser based test harnesses with JavaScript to test Web applications. Specifically, I have created proof-of-concept Internet Explorer harnesses to perform HTTP request-response testing, and to perform UI testing. In both situations however I used hard-coded test case input. Over the weekend I took a look at reading external test case data, and writing test case results, from a browser-based harness. Basically there are two approaches available with Internet Explorer. First, you can read or write information on the test host machine. Second, you can read or write information on the Web application server machine. To read or write test data on the test host machine, you can use code along the lines of:

 

var result = document.getElementById('TextBox4');
var fso = new ActiveXObject('Scripting.FileSystemObject');
var infile = fso.OpenTextFile("C:\\testCases.txt", 1);
// etc.

 

If this code is located and executed on the test host, it will look for file testcases.txt on the test host.

To read or write test data on the Web server machine you can create a short ASP script which contains similar code, and then invoke the script from the test host along the lines of:

 

<form method='post' action='readData.asp' name='theForm'>
 <input type='submit' />
</form>

 

combined with:

 

var f = document.getElementById('theForm');
f.submit();

 

If this code is located and executed on the Web server, it will look for file testcases.txt on the server. There are many alternatives to browser based test harnesses, and I still don't have a solid grasp of the pros and cons involved, but the approach seems like it may be useful in certain testing scenarios.

The jQuery library is an Open Source collection of JavaScript functions that Web application developers can use to easily create neat client-side effects such as fade-in and menus. But the jQuery library can be used write lightweight test automation too. I have been looking at using jQuery to perform lightweight HTTP request-response testing. See the screenshot below for a proof-of-concept. The screenshot shows that I have a test harness in the form of an HTML page. That page references the jQuery library and contains a script that fires off an HTTP request, fetches the HTTP response, and examines the response for an expected value to produce a test case pass/fail result. This technique might be useful in situations where you want to use an HTML-based test harness rather than a shell-based or Windows Form-based test harness. Additionally, investigating test automation that uses new technologies such as jQuery helps testers stay in sync with developers who are using the new technologies.

 

I am researching the use of jQuery for Web application test automation. One of the things I need to do is access HTML elements in one frame from another frame. A search of the Internet yielded several people asking how to do this but no answers. Anyway, after about two hours of work I came up with a solution to the problem. The screenshot below shows a demo in action. The main page has two frames. When a user clicks on the Set Text button in the page in left frame, control is transferred to a jQuery function which places text into the text box control in the right frame. The key lines of code are:

 

$(document).ready(function(){
 $("#Button1").click(function(){
  $(parent.rightFrame.document).contents().find('#TextBox1').val('Hello from left frame!');
 });
});

 

Anyway, jQuery is very cool and I should have a Web application test automation article written for MSDN Magazine done in a few weeks.

 

 

You have probably heard the terms eigenvalue and eigenvector before but probably aren't sure exactly what these terms mean. They are staples of discrete mathematics courses required for computer science degrees at many universities. For years I've wondered if eigenvalues and eigenvectors have any application to software testing. I don't have an answer. Let me explain what eigenvalues and eigenvectors are, and why I suspect they might be important to software testing. Suppose you are dealing with points x = (x,y) in the two-dimensional plane. Then a 2x2 square matrix A implicitly defines a transformation of any point. Bear with me here. If there exists a vector x and a value λ (Greek letter lambda) that have the relationship Ax = λx then λ  is called an eigenvalue and x is called an eigenvector. For example, if A = [ (6,3), (-2,-1) ], and λ = 0, and x = [1,-2] the λ and x form an eigenvalue-eigenvector pair. OK, so what? One of the surprising things about eigenvalue analysis is the amazing number of real-world problems they apply to in very unexpected ways. For example, using eigenvalue analysis it was shown that the optimal shape of a building support column is curvy, somewhat like the shape of a decorative stair post. Anyway, software testing is all about state changes which can be represented mathematically as vectors and matrices. There may be some interesting and useful ways that eigenvalues and eigenvectors can be applied to software testing.

The open source jQuery library is a collection of JavaScript functions that allow Web developers to create client-side effects such as fade-in and fade-out more quickly than by using plain JavaScript code. I have been looking at using jQuery to write simple, lightweight Web application UI test automation. The main advantage of using jQuery for test automation rather than using raw JavaScript is that the functions in jQuery work with multiple browsers (IE, Safari, Firefox, etc.) The screenshot below shows an example of testing a simple Web application using jQuery. The frame on the left hosts a jQuery test harness script, and the frame on the right hosts the Web application under test. I intend to write an article for MSDN Magazine which describes all the details. 

 

 

One of the core principles of software testing is that in order to determine a pass/fail result for a test case, after executing the test case you must compare the actual result or state of the system under test with an expected result/state. For example, if you are testing the + functionality of a calculator, and the test case inputs are 3.0 and 4.0 (and the expected result is 7.0) then you exercise the SUT to get an actual result and compare that actual result with the expected result. In statistics, the most common way to compare a set of observed values with a set of expected values is to use the well-known chi-square test. However, what is not so well known is that the chi-square test is actually a discrete approximation to the log likelihood test. The chi-square test was developed in the days before calculators when computing logarithms was difficult. Anyway, the point is, in software testing, if you want to compare how close a set of actual values is to a set of expected values, you should probably use the log likelihood g-test rather than the chi-square test. The g statistic is given by 2 * (sum-over-i(Oi * ln(Oi / Ei)) where Oi is an observed value and Ei is the corresponding expected value. For example, suppose you have some system which should emit the three values (4.0, 4.0, 4.0). These are the expected values. Now if the actual results are (3.0, 4.0, 6.0) then the g-statistic is 2 * [(3.0 * ln(3.0/4.0)) + (4.0 * ln(4.0/4.0)) + (6.0 * ln(6.0/4.0))] = 3.139. The closer the g-static is to 0, the closer the actual results are to the expected results; you can look up specific probabilities if necessary.