One of the challenges facing the field of software testing is the perception that test engineers have weak technical skills relative to developers and that the field has too many self-proclaimed experts who are anything but. One example is the upcoming Pacific Northwest Software Quality Conference, in Portland, OR, in October. In past years I have been quite impressed with the technical quality of the speakers at the PNSQC. I'll probably go to PNSQC but find it hard to recommend the event to others this year because the list of invited speakers appears to have been significantly dumbed-down. One quote from the PNSQC invited speakers descriptions goes, "One final secret every project manager must know—there is no 'one right way' to manage a project. Everything depends on your context." Well, there's a real intellectual gem. Another quotes goes, " Some managers who’ve never studied testing, never done testing, probably have never even *seen* testing up close, nevertheless insist that it be rigorously planned in advance and fully documented." Okay, show me some data to support this ridiculous statement. Which managers? Where do they work? What survey or data supports this claim? On the other hand, I just returned from the Software Engineering and Data Engineering (SEDE) conference which had terrific speakers who actually presented useful, new information. One such topic is the relationship between software testing and entropy. In simple terms, entropy is a measure of disorder in a system. The general equation for entropy is:

 

where b is an arbitrary base (often 2 or 10) and p(x) is the probability that some random variable X takes on value x. The math looks scary but the concepts use only freshman level mathematics and are actually very, very simple. The talk at SEDE went on to describe many highly practical ways entropy can be used in software testing scenarios to improve software quality. The perception of software testing conferences will improve when conference organizers reduce the number of fluffy talks such as "Make Your Project Successful" and increase the number of talks that provide interesting and useful new testing information.

 

The use of Web-based applications such as ASP.NET applications, Web Services, AJAX-based applications, and so on, is increasing steadily. A useful technique for testing many such applications is to send a low-level request to the SUT using TCP/IP and sockets. The technique is general-purpose because is works at a very low level of abstraction. For example, suppose you want to test some Web Service which contains a Web Method named GetTitles(). First you could set up the request in SOAP format along the lines of:

 

string input = "some input";
string soapMessage = "<?xml version=\"1.0\" encoding=\"utf-8\"?>";
soapMessage += "<soap:Envelope>";
soapMessage += "<soap:Body>";
soapMessage += "<GetTitles>";
soapMessage += "<filter>" + input + "</filter>";
soapMessage += "</GetTitles>";
soapMessage += "</soap:Body>";
soapMessage += "</soap:Envelope>";

 

Next you could set up the request destination:

 

string host = "localhost";
string webService = "/TestAuto/Ch8/TheWebService/BookSearch.asmx";
string webMethod = "GetTitles";
IPHostEntry iphe = Dns.Resolve(host);
IPAddress[] addList = iphe.AddressList; // addList[0] == 127.0.0.1
EndPoint ep = new IPEndPoint(addList[0], 80); // ep = 127.0.0.1:80

 

Then you could set up a TCP/IP socket:

 

Socket socket = new Socket(AddressFamily.InterNetwork,
 SocketType.Stream, ProtocolType.Tcp);
socket.Connect(ep);

 

And then construct the HTTP wrapper somewhat like:

 

string header = "POST " + webService + " HTTP/1.1\r\n";
header += "Host: " + host + "\r\n";
header += "Content-Type: text/xml; charset=utf-8\r\n";
header += "Content-Length: " + soapMessage.Length.ToString() + "\r\n";
header += "Connection: close\r\n";
header += "SOAPAction: \"http://tempuri.org/" + webMethod + "\"\r\n\r\n";
string sendAsString = header + soapMessage;
byte[] sendAsBytes = Encoding.ASCII.GetBytes(sendAsString);

 

And next you'd fire off the request and fetch the response:

 

int numBytesSent = socket.Send(sendAsBytes, sendAsBytes.Length,
                               SocketFlags.None);
Console.WriteLine("Sending = " + numBytesSent + " bytes\n");

byte[] receiveBufferAsBytes = new byte[512];
string receiveAsString = "";
string entireReceive = "";
int numBytesReceived = 0;

while ((numBytesReceived = socket.Receive(receiveBufferAsBytes,
 512, SocketFlags.None)) > 0 )
{
  receiveAsString = Encoding.ASCII.GetString(receiveBufferAsBytes,
   0, numBytesReceived);
  entireReceive += receiveAsString;
}

 

Your test could conclude by converting the byte[] response into a string result and analyzing the result for an expected value of some sort. Using a low-level TCP/IP and socket aproach such as I've described is often useful for security-related testing. You can be sure that someone will probe your Web-based application using this technique and so you had better test using the technique.

 

I ran across a very cool metric recently. The metric is called category utility and it can be used to measure how well a database of categorical data is clustered into a set of groups. Cluster analysis of numeric data is one of the most widely studied areas in all of data mining. With numeric data it is not too hard to measure how far one row of data, such as (1.0, 2.0, 3.0), is from another row of data, such as (5.0, -1.5, 6.0). Using a difference metric you can then create clusters of data where the data within a cluster is similar, and the data in different clusters is dissimilar. However, it's not so easy to measure how far categorical data, such as (red, small, hot) is from other categorical data, such as (blue, large, cold). Category utility is a neat idea which measures such differences based on probabilities. The equation is:

 

Each possible clustering of data will have a different categorical utility value, with larger CU values indicating better clustering. I'm looking at a research paper for the International Symposium on Visual Computing and I coded up an implementation of a category utility function in C# and the following screenshot shows it in action:

 

A few blog posts ago I described generating all possible permutations. I mentioned that the terms permutations and combinations are used incorrectly on many Web sites. A mathematical permutation of order (size) n is a rearrangement of the numbers 0 through n-1. For example, if n=3, then all possible permutations are {0,1,2}, {0,2,1}, {1,0,2}, {1,2,0}, {2,0,1}, {2,1,0}. Here I've listed the permutations in lexicographical order. A mathematical combination of order (n,k) is a subset of size k of the numbers 0 through n-1 where order does not matter. For example, all combinations of order (5,3) in lexicographical order are {0,1,2}, {0,1,3}, {0,1,4}, {0,2,3}, {0,2,4}, {0,3,4}, {1,2,3}, {1,2,4}, {1,3,4}, {2,3,4}. Here is a class, written in C# (with error-checking removed) that will generate all possible combinations. The class can be called like this:

 

static void Main(string[] args)
{
  Console.Write("\nEnter combination n-value: ");
  int n = int.Parse(Console.ReadLine());
  Console.Write("\nEnter combination k-value: ");
  int k = int.Parse(Console.ReadLine());

  Console.WriteLine("\nAll combinations: \n");
  Combination c = new Combination(n, k);
  while (c != null) {
    Console.WriteLine(c.ToString());
    c = c.Successor();
  }

  Console.WriteLine("\nDone");
}

 

 

 

Notice the approach is to define a Successor() method which returns the next combination object, or null if we are at the last combination. Here's the class:

 

public class Combination
{
  private readonly int n;
  private readonly int k;
  private readonly int[] data;

  public Combination(int n, int k)
  {
    this.n = n;
    this.k = k;
    this.data = new int[k];
    for (int i = 0; i < k; ++i) {
      data[i] = i;
    }
  } // ctor()

  public override string ToString()
  {
    string s = "{ ";
    for (int i = 0; i < k; ++i)
      s += data[i] + " ";
    s += "}";
    return s;
  }

  public Combination Successor()
  {
    if (data[0] == n - k) // last combination element
      return null;

    Combination ans = new Combination(n, k);

    int i;
    for (i = 0; i < k; ++i)
      ans.data[i] = this.data[i];

    for (i = k - 1; i > 0 && ans.data[i] == n - k + i; --i)
      ;

    ++ans.data[i];

    for (int j = i; j < k - 1; ++j)
      ans.data[j + 1] = ans.data[j] + 1;

    return ans;
  } // Successor()

} // class Combination

 

Starting last week, I have been teaching a class to prepare software test engineers for a Certificate in Software Testing with Microsoft Technologies program. The certificate program covers 42 specific topics that were listed by test managers as those topics that testers should have at least a basic understanding of. One of those 42 topics is character encoding. Yesterday morning I decided to make a tiny demo program to use to illustrate the topic. I recalled that one of my colleagues, William Rollison, is an expert at character encoding. I visited his blog and found a tool he wrote, and I decided to (approximately) reverse engineer that tool so that I could have a simple application with source code to distribute to students in the software testing certificate program. The screenshot below shows the result.

 

 

Interestingly, the character encoding logic was simple (in part because I've investigated the topic before) but I spent an hour wrestling with the ListView data control to display results. The demo is written with C#. Here's the code:

 

private void button1_Click(object sender, EventArgs e)
{
  listView1.Items.Clear();
  int charCount = 0;
  string s = textBox1.Text;
  char[] chars = s.ToCharArray();
     
  foreach (char c in chars)
  {
    char[] arrayWithOneChar = new char[] { c };

    byte[] bytes = null;
    if (radioButton1.Checked) // ASCII
      bytes = Encoding.ASCII.GetBytes(arrayWithOneChar);
    else if (radioButton2.Checked)
      bytes = Encoding.Unicode.GetBytes(arrayWithOneChar); // Little Endian
    else if (radioButton3.Checked)
      bytes = Encoding.BigEndianUnicode.GetBytes(arrayWithOneChar);
    else if (radioButton4.Checked)
      bytes = Encoding.UTF7.GetBytes(arrayWithOneChar);
    else if (radioButton5.Checked)
      bytes = Encoding.UTF8.GetBytes(arrayWithOneChar);
    else if (radioButton6.Checked)
      bytes = Encoding.UTF32.GetBytes(arrayWithOneChar);
 
    string bytesForACharInHex = "";
    string bytesForACharInDecimal = "";
    foreach (byte b in bytes)
    {
      bytesForACharInHex += b.ToString("X2") + " ";
      bytesForACharInDecimal += b.ToString() + " ";
    }
    listView1.Items.Add(new ListViewItem(new string[] { charCount.ToString(), c.ToString(), bytesForACharInHex, bytesForACharInDecimal }));
    ++charCount;
  } // next char
    
} // button1_Click()

 

Two of the most common and fundamental programming tasks are generating all possible permutations, and generating all possible combinations. Last week I spent some time looking at what I'd find on the Internet about these two problems. I was surprised by the large number of Web pages and blogs which have really, really bad information. First, the terms permutations and combinations, are used incorrectly on many Web sites. A mathematical permutation of order (size) n is a rearrangement of the numbers 0 through n-1. For example, if n=3, then all possible permutations are {0,1,2}, {0,2,1}, {1,0,2}, {1,2,0}, {2,0,1}, {2,1,0}. Here I've listed the permutations in lexicographical order. A mathematical combination of order (n,k) is a subset of size k of the numbers 0 through n-1 where order does not matter. For example, all combinations of order (5,3) in lexicographical order are {0,1,2}, {0,1,3}, {0,1,4}, {0,2,3}, {0,2,4}, {0,3,4}, {1,2,3}, {1,2,4}, {1,3,4}, {2,3,4}. Anyway, listed below is a tiny class, written in C#, with error-checking removed, that will generate all possible permutations. The class can be called like this:

 

static void Main(string[] args)
{
  Console.Write("\nEnter permutation order (n): ");
  int n = int.Parse(Console.ReadLine());

  Console.WriteLine("\nAll permutations: \n");
  Permutation p = new Permutation(n);
  while (p != null) {
    Console.WriteLine(p.ToString());
    p = p.Successor();
  }

  Console.WriteLine("\nDone");
}

 

 

 

Notice the approach is to define a Successor() method which returns the next permutation object, or null if we are at the last permutation. Here's the class:

 

public class Permutation
{
  public readonly int n;
  public readonly int[] data;

 

  public Permutation(int n) {
    this.n = n;
    this.data = new int[n];
    for (int i = 0; i < n; ++i)
      data[i] = i;
  }

 

  public override string ToString()
  {
    string s = "( ";
    for (int i = 0; i < n; ++i)
      s += data[i] + " ";
    s += ")";
    return s;
  }

 

  public Permutation Successor()
  {
    Permutation result = new Permutation(n);
    int left, right;

    for (int i = 0; i < n; ++i)
      result.data[i] = this.data[i];
     
    left = result.n- 2;  // Step #1 - Find left value
    while ((result.data[left] > result.data[left + 1]) && (left >= 1))
      --left;
     
    if ((left == 0) && (this.data[left] > this.data[left + 1]))
      return null;

    right = result.n - 1;  // Step #2 - find right value
    while (result.data[left] > result.data[right])
      --right;
     
    int temp = result.data[left];  // Step #3 - left and right
    result.data[left] = result.data[right];
    result.data[right] = temp;
     
    int x = left + 1;              // Step #4 - order the tail
    int y = result.n - 1;

    while (x < y) {
      temp = result.data[x];
      result.data[x++] = result.data[y];
      result.data[y--] = temp;
    }

    return result;
  }

} // Permutation

 

I ran into an interesting technical riddle yesterday. I was writing some demonstration UI test automation for a class which is part of a new Certificate in Software Testing with Microsoft Technologies program. My application under test is a .NET Windows application that plays a hypothetical game of two-card poker. I was using the Microsoft UI Automation library to implement my UI test automation. Everything went according to plan until I decided I'd like to examine the contents of a ListBox control as part of the process of determining the final state of the app under test (in order to determine a pass/fail result). It turns out that there are no existing published examples of how to do this. I spent a couple of hours but finally came up with the correct approach. See the screenshot below.

 

 

In words, first get a reference to the ListBox control itself. Next get a reference to that control's ListItem collection. Next use zero-based array indexing to get a reference to a particular item in the collection. And then use the Name property of that item to get the text. The trick is the last step. I had been trying to use the MUIA TextPattern and ValuePattern patterns. It turns out that the Name property of a ListItem item reflects the text contents. So, the code to fetch the contents of a .NET ListBox control using the MUIA library resembles the following:

AutomationElement aeListBox =
  aeForm.FindFirst(TreeScope.Children,
  new PropertyCondition(AutomationElement.ControlTypeProperty,
  ControlType.List));

 

AutomationElementCollection aeListBoxItems =
  aeListBox.FindAll(TreeScope.Descendants,
  new PropertyCondition(AutomationElement.ControlTypeProperty,
  ControlType.ListItem));

 

AutomationElement aeFirstItem =
  aeListBoxItems[0];

 

string s =
  (string)aeFirstItem.GetCurrentPropertyValue(
    AutomationElement.NameProperty);

 

Anyway, like always, the answer seems obvious once you know it. I believe that the MUIA library will become the de facto standard technology for UI test automation of Microsoft programs, once all the little recipes like the one I've presented here become well known.

 

 

I've been working informally with several groups at Microsoft to create a certificate in software testing program. The idea is to provide all testers with a basic knowledge of software testing principles through a certificate program. A certificate program is much more narrowly defined than a certification program such as the MSCSE (Microsoft Certified Systems Engineer) certification. A certificate normally means a person has obtained a very specific goal. Anyway, as part of my research, I reviewed the four primary software testing certifications that exist. They are the IIST CSTP (International Institute for Software Testing, Certified Software Test Professional), the QAI CSTE (Quality Assurance Institute, Certified Software Tester), the ASTQB CTFL (International/American Software Testing Qualifications Board, Certified Tester Foundation Level), and the ASQ CSQE (American Society for Quality, Certified Software Quality Engineer). I've put together a matrix that summarizes these four certifications. The bottom line is that they all target a different audience, and have different goals, with the one exception that they're all designed to generate revenue for their sponsoring companies and organizations. The image below is a snapshot of my matrix. You may need to click on the image to view it.

 

 

All these certifications have strengths and a whole lot of weaknesses. That's why we are putting together a lean, highly focused, relatively inexpensive, Certificate in Software Testing with Microsoft Technologies program.

 

Recently I've been looking at the problem of rule set extraction. Actually, to be more accurate, I've been looking at a particular class of problems and I'm not exactly sure what these problems are called. Suppose you have a set of categorical data like this:

 

(Red, Small, Hot) -> c0
(Red, Small, Cold) -> c0
(Blue, Medium, Hot) -> c1
(Green, Large, Cold) -> c1
(Yellow, Large, Warm) -> c2
(Blue, Small, Hot) -> c2

 

The first tuple, or itemset, means that there is something which has an attribute of color = red, size = small, temperature = hot, and is assigned to category, or cluster c0. The problem I'm looking at is how to programmatically extract a set of rules from this data. For example, a human might conculde that:

 

if (color = Red) then cluster = c0
else if (size = Medium) then cluster = c1
else if (color = Gren) then cluster = c1
else cluster = c2

 

I've been looking for existing work in this area, but haven't found anything that matches this particular problem. The closet area I've found was pointed out to me by a colleague in Microsoft Research. That area is generally called Association Rules. However, Association Rules are slighty different because they look at all the associations within tuple attrbute values rather than the associations between the first n-1 attribute values, and some cluster or category. Anyway, it's a very interesting problem.

I've been working on a set of related programming projects over the past couple of weeks. Classification, cluster analysis, and rule set extraction are closely related topics. Suppose you have a set of data points (also called vectors or tuples) of some sort. These data points could be numeric abstractions such as geometric points, like (0, 3, -1), or the data points might be rows of a SQL database like (Smith, Stan, $21.33, Developer). Now suppose you have a set of known categories, such as c0 = "likely to vote Democratic", c1 = "likely to vote Republican", and so on. Programmatic classification is the process of assigning each data point to a particular category. Programmatic clustering is similar to classification except that you don't have known categories; instead the data points are grouped together into clusters of similar data points. Both classification and clustering can be supervised or unsupervised. With a supervised approach, a set of preliminary training data points are manually classified or clustered, and then this information is used to classify or cluster additional new data points. There is a huge body of research on classification and cluster analysis. However, the majority of this research deals with purely numerical data such as (3.0, 5.0, 2.0). There is much less research on categorical data such as (red, small, hot). The main reason for this is that most classification and clustering algorithms rely on some form of a difference function. It's not too hard to compute a number which represents the difference between (2.0, 3.0, 4.0) and (1.0, 3.5, 2.7), but it's a harder problem to determine the difference between (red, small, hot) and (blue, large, cold). Anyway, I've found what I believe to be some very cool new ways to perform classification and clustering of categorical data. The topic of rule set extraction enters the mix then: after clustering your data, how can you extract a set of if..then rules that correspond to the clustering result? Again, I'm working on some ideas that really fascinate me.

I was working on some data mining related code last week and ran into an old problem I've bumped up against many times in the past, namely, how to generate all combinations of a set of strings. Mathematical combinatorics was my primary area of study during my undergraduate days and I have my own library of code that I've created over the years. For example see http://msdn.microsoft.com/en-us/magazine/cc163957.aspx. However I decided to search the Internet to see what I'd turn up. I was somewhat surprised at a.) the limited amount of information available for such a common problem, and b.) the amount of information available which is just incorrect. One thing I immediately noticed is that many Internet posts confuse combinations and permutations. A permutation of a set of items is a rearrangement. For example, if a set contains {"Adam", "Bill", "Carl"} then one permutation is {"Bill", "Adam", "Carl}. There are a total of n! different permutations of a set of size n. Combinations on the other hand are subsets of size k of the original set where order does not matter. For example, with the above set, if k = 2, then all three possible combinations are {"Adam", "Bill"} {"Adam", "Carl"}, and {"Bill", "Carl"}. Anyway, generating all possible combinations would yield these 7 combinations:

 

(k = 1): {"Adam"}, {"Bill"}, {"Carl"}
(k = 2): {"Adam", "Bill"} {"Adam", "Carl"}, and {"Bill", "Carl"}
(k = 3): {"Adam", "Bill", "Carl"}

 

This problem turns out to be somewhat surprisingly tricky. I'll give some code in a future blog post. Combinatorics, which includes combinations and permutations, is one of the most fundamentally important areas of software testing. 

Suppose you want to test a Web application through its user interface. Over the past couple of years I've written several articles for Microsoft's MSDN Magazine that demonstrate different ways to do this, but in all cases I assumed you are working in a 100% Microsoft environment and using Internet Explorer. Two of the techniques available to you are to write a C++ or C# language program which calls directly into the IE COM-based API set, or to use JavaScript to call into the IE DOM. The JavaScript approach is often a good choice and works well for Internet Explorer but doesn't always work with other browsers because of differences between browser DOMs. For example, with IE v4.0, to get a reference to a button with ID = "Button1" you can write var btn = document.all["Button1"] but with IE v5.0 and above and Firefox you could write var btn = document.getElementById("Button1"). Although you can write your test automation code to sense which browser is being used and then branch to correct JavaScript, an alternative is to use the nice jQuery library. The jQuery library is essentially a set of JavaScript wrapper functions which are for the most part browser-independent. 

I tend to group conferences that are in some way related to software testing into three categories. One category is hard-core academic/scholarly conferences. These are usually attended by college professors and researchers. The WorldCOMP International Conference on Software Engineering Research and Practice (see http://www.world-academy-of-science.org/worldcomp09/ws/conferences/serp09 ) is an example. The goal of these conferences is primarily to provide a channel for researchers to publish scholarly papers.

 

A second category of conferences is commercial conferences. These are generally sponsored by a large company such as Microsoft and attended by people who work in the IT industry, often in mid-management roles. Software testing usually has a very small role in these conferences. The Microsoft Management Summit (see http://www.mms-2009.com/ ) is one example. The goal of these conferences is to market and advertise a company's products and services.

 

A third category of conferences is hard to put a label on but I'll call them industry-organization conferences. These conferences are sponsored by various organizations. The Better Software Conference and Expo (see http://www.sqe.com/BetterSoftwareConf/ ) is a good example. Attendees are often practitioners and individual contributors. The goal of these conferences is, typically, to make money for the sponsoring organization.

 

I enjoy attending and speaking at conferences. I usually go to conferences in Las Vegas because travel there is relatively cheap and easy. The main benefit from conferences in my opinion is that you get away from the day-to-day work grind and come back with fresh ideas.

 

It's hard to recommend specific software testing conferences because they are all quite different and really target completely different types of people. The best way to evaluate a conference is to get an opinion from someone you know who has been to the conference in the past. That said however, the Pacific Northwest Software Quality Conference (see http://www.pnsqc.org/ ) is often a good one (meaning good value) for testing practitioners.

The ADO.NET Entity Framework (AEF) v1.0 is part of Visual Studio 2008 SP1. The AEF examines a SQL database and creates wrapping C# code that can be used in an application to perform insert, update, create, and delete operations on the database. Yesterday I ran into an interesting error and an interesting solution. I created a dbCars DB with two tables, tblMakes ("Audi", "BMW", etc.) and tblModels ("A4", "328i", etc.) The tblModels table has a foreign key make_id into tblMakes, with an FK constraint named fk_tblModels_tblMakes. In AEF v1.0 if you want to use any stored procedures with a table, you must implement all three of insert, update, and delete procs. Anyway, after doing this and compiling a C# console application I got: Error "2048: The EntitySet 'tblModels' includes function mappings for AssociationSet 'fk_tblModels_tblMakes', but none exists in element 'DeleteFunction' for type 'dbCarsModel.tblModels'. AssociationSets must be consistently mapped for all operations." I beat my head on this for a while but a couple of guys on the AEF team, Daniel and Michael, explained. Basically, the FK constraint referred to keys in both tblMakes and tblModels but my usp_DeleteModel only uses the model key as an input parameter. One solution is to recast the usp_DeleteModel to accept a dummy input parameter of the make_id. That worked but just didn't feel quite right. A second solution is to open the Model1.edmx file with an XML editor and place this mapping code into the SSDL section:

 

 

Then, after going back to the visual designer view of the .edmx file I remapped the DeleteFunction entry to the newly defined Custom_DeleteModel (which automatically appeared in the GUI dropdown control). The XML mapping code above has the dummy make_id parameter which keeps the compiler happy, and calls the usp_DeleteModel stored procedure using just the natural model_id parameter. Cool. Here are the particulars:

 

 

Like many v1.0 products the AEF has its glitches but I bet that this one at least will be fixed with v2.0 which I think will ship with .NET Framework 4.0 and Visual Studio 2010.

I ran across an interesting testing situation the other day. I was looking at a library module for 7-card poker which contains a method int Hand.Compare(Hand h1, Hand h2) to determine which of two 7-card poker hands is better according to the rules of poker. This is a surprisingly tricky method to code up and I use it for many different types of software testing investigations. Anyway, consider these two poker hands:

 

h1 = Ac Ad Ah 9s 9c 9d 2c
h2 = Kc Kd Ks Jc Jd Js Qh

 

The first hand is a full house Aces over Nines (usually pronounced Aces full of Nines). The second hand is a full house Kings over Jacks. A call to Hand.Compare(h1, h2) should return -1 because h1 is better than h2. OK, but if you execute this test case should you also execute Hand.Compare(h2, h1) with an expected result of +1? The answer is yes, absolutely. Well this is basically testing 101 information but it got me to thinking about situations where you absolutely need to create symmetric test case input. I didn't come up with any remarkable epiphanies but it did remind me that there are a few testing principles, like you should almost always create test cases with symmetric input, that are easy to overlook.

 

I've always been fascinated by algorithms which are inspired by natural processes. In fact my Master's thesis investigated optimization using genetic algorithms which mimic evolutionary processes by simulating genetic crossover and mutation. During the past few months I've been looking at optimization algorithms which are based on the behavior of honey bee colonies. These types of algorithms have been studied since at least 1997 and perhaps earlier, but have received a lot of attention in the past three years. Different researchers use different approaches and end up with slightly different meta-heuristics. Some of the names I've come across in my research include Bee System, BeeHive, Virtual Bee Algorithm, Bee Swarm Optimization, Bee Colony Optimization, Artificial Bee Colony, and Bees Algorithm. The basic idea is that foraging bees have a potential solution to an optimization problem in their memory. This solution corresponds to the location of a food source. Each food source has an associated quality measure of how well it solves the optimization problem. Foraging bees return to the hive and perform a waggle dance which describes a potential solution and it quality to currently inactive foragers. These inactive bees then go to the food source and examine nearby sources/solutions. There are also scout foragers who randomly search the problem domain space. Anyway, I cooked up a program which uses a simulated bee colony algorithm to solve various problems in software testing and the results were very promising. I've written up a paper and intend to submit it to a software conference at some point.

F# is Microsoft's new programming language. F# is scheduled to ship with the next version of Visual Studio but a Community Technical Preview version is available. F# is a functional language which means among other things that most language constructs are related to functions and return values. I've had quite a bit of experience with Prolog and LISP, two old functional languages and I never really liked them very much. But I've been looking at F# and like what I've seen so far. For test automation, one traditional, non-functional approach is to create test case data and then iterate through the data. Here is a snippet of what this might look like with F#:

 

let testCases =
  [ "001,3,5,8"
    "002,0,0,0"
    "003,2,4,6"
  ]

let dummy =
  Seq.iter(fun(testCase) ->
  let delimits = [|',';'~'|];
  let tokens = testCase.Split(delimits)
  let caseID = tokens.[0]
  let input1 = tokens.[1]
  let input2 = tokens.[2]
  let expected = tokens.[3]
  // send input1 & input2 to SUT
  // check actual return with expected
  if actual == expected then
    printfn "Pass"
  else printfn "Fail"
  true
  )

 

The first part of the snippet sets up an F# List object of test case input. The second part of the snippet uses Seq.iter (sequence iterator) to process each item in the List by applying an anonymous function (indicated by the "fun" keyword) which always returns true. Like anything else, it takes a while to get up to speed with F# but so far it's been an interesting investigation.

 

 

 

I was teaching an intermediate level JavaScript class last week and I used an example that many experienced JavaScript coders don't know about, and that is how to create a JavaScript class with private data members. Consider this:

 

function Book(t,p)
{
  // public
  this.title = t;
  this.pages = p;
  this.toString = function()
    { return this.title + " # " + this.pages; };
}

 

function main()
{
  var b = new Book("Test Automation",123);
  WScript.Echo(b.toString());
  b.pages = 321;
  WScript.Echo(b.toString());
}

 

main();

 

The output is 'Test Automation # 123' followed by 'Test Automation # 321' as you would expect. Here I have a normal Book class in JavaScript where data members title and pages are public and so can be accessed and manipulated by the calling code in the main() function. Here's how to make the data private:

 

function Book(t,p)
{
  // private
  var title = t;
  var pages = p;
 
  // public
  this.getTitle = function() { return title; };
  this.setTitle = function(t) { title = t; };

  this.getPages = function() { return pages; };
  this.setPages = function(p) { pages = p; };

  this.toString = function() { return title + " # " + pages; };
}

 

By using the "var" keyword instead of the "this" keyword, title and pages can no longer be accessed or manipulated directly so I code public functions to perform those operations. If now:

 

var b = new Book();
b.title = "Software Testing"; // ignored
b.pages = 111; // ignored
WScript.Echo(b.toString());

 

the output is 'undefined # undefined'. However if:

 

var b = new Book();
b.setTitle("Software Development");
b.setPages(222);
WScript.Echo(b.toString());

 

the output is Software Development # 222' as you'd expect.

I wrote a paper this week for the 18th Annual Software Engineering and Data Engineering Conference. The title is, "An Empirical Study of the Effectiveness of Partial Antirandom Testing." Most testers are familiar with random testing where you generate random data and send it to the system under test. In most situations, because here is no explicit expected value or state, the goal of random testing is to try and produce and hang crash, or failure in the SUT. The field of hardware testing came up with the concept of antirandom testing. Antirandom testing is a variant of random testing where a set of test case inputs is generated in such a way that each new test input added to the test set is the test input which is the most different from the test inputs currently in the test set. The concept is best explained by example. Suppose that a system has a parameter which accepts 3-bit values and the test effort wishes to generate a test set which contains four antirandom inputs. The complete input domain is:

 

0: 000
1: 001
2: 010
3: 011
4: 100
5: 101
6: 110
7: 111

 

The initial antirandom test set is empty and value {0,0,0} can be arbitrarily selected from the input domain and added to the test set. The next input to be added to the antirandom test set is the value from the domain space which is most different from the current inputs in the test set. In this case input {1,1,1} is the most different from {0,0,0} using virtually any rational definition of different so the antirandom test set now contains {0,0,0} and {1,1,1}. Similarly, assume that value {0,0,1} is selected from the input domain using the algorithm described below and added to the antirandom test set so the set contains {0,0,0}, {1,1,1}, and {0,0,1}. Any of the remaining five values in the input domain are candidates for the fourth and final member of the test set, however exactly which value is chosen depends upon the difference function used. One obvious candidate function is the Hamming distance. Another possibility is the Cartesian distance; suppose this is used as the difference function. The sum of the Cartesian differences between each candidate and the three current members of the antirandom test set are:

 

010: sqrt(1) + sqrt(2) + sqrt(2) = 3.82
011: sqrt(2) + sqrt(1) + sqrt(1) = 3.41
100: sqrt(1) + sqrt(2) + sqrt(2) = 3.82
101: sqrt(2) + sqrt(1) + sqrt(1) = 3.41
110: sqrt(2) + sqrt(1) + sqrt(3) = 4.14

 

Because {1,1,0} is the most different from the existing members of the antirandom test set, it is added to the test set. The fundamental basis of antirandom testing is the premise that similar inputs will likely yield similar information about the system under test, and therefore random inputs which are as different as possible will yield more information than arbitrary random inputs. The example above points out that pure antirandom test input generation requires a complete enumeration of the entire input domain. This may be possible for certain types of hardware testing with a relatively small input byte size or for software systems with a small input domain. However, pure antirandom testing is rarely possible for complex software systems. Anyway, I came up with an approach I call partial antirandom testing which can be used to test software systems.

 

I have just finished writing a research paper for the SERP (Software Engineering Research and Practice) sub-conference of the 2009 World Congress in Computer Science, Computer Engineering, and Applied Computing. The paper presents the results of an investigation into the effectiveness of pairwise testing. In the paper I introduce the idea of what I call partition-based pairwise testing. The idea is best explained by example. Suppose you have a method under test named Evaluate(Card c1, Card c2, Card c3, Card c4, Card c5) which accepts representations of five objects which represent playing cards from a normal deck of 52 cards. Pure pairwise testing would treat all 52 cards as possible parameter values for each of the five system parameters. This would lead to an unmanageable number of test case inputs. The idea of partition-based pairwise testing is to divide the parameter values into some form of equivalence classes and then use some boundary values from each class. Suppose we divide the 52 card deck into low (2 through 6), medium (7, 8, 9), and high (10, Jack, Queen, King, Ace) classes, and ignore suit. If we use just the edge values from each class and arbitrarily select clubs as the suit, we get this set of input values:

 

c1: 2c, 6c, 7c, 9c, Tc, Ac
c2: 2c, 6c, 7c, 9c, Tc, Ac
c3: 2c, 6c, 7c, 9c, Tc, Ac
c4: 2c, 6c, 7c, 9c, Tc, Ac
c5: 2c, 6c, 7c, 9c, Tc, Ac

 

Now if we use a pairwise test set generation tool such as PICT, we generate 50 test case inputs:

 
    c1  c2  c3  c4  c5
----------------------
01: 2c  7c  6c  2c  2c
02: Tc  2c  9c  2c  6c
03: Ac  Ac  Tc  Tc  9c
 (etc.)
50: 7c  9c  6c  2c  6c

 

This test set is much more manageable. Note that because there are many ways to partition an arbitrary set of values, you'd have to create several equivalence classes (for example, partitioning by suits) to get effective test coverage.