James McCaffrey

One of the challenges facing the field of software testing is the perception that test engineers have weak technical skills relative to developers and that the field has too many self-proclaimed experts who are anything but. One example is the upcoming Pacific Northwest Software Quality Conference, in Portland, OR, in October. In past years I have been quite impressed with the technical quality of the speakers at the PNSQC. I'll probably go to PNSQC but find it hard to recommend the event to others this year because the list of invited speakers appears to have been significantly dumbed-down. One quote from the PNSQC invited speakers descriptions goes, "One final secret every project manager must know—there is no 'one right way' to manage a project. Everything depends on your context." Well, there's a real intellectual gem. Another quotes goes, " Some managers who’ve never studied testing, never done testing, probably have never even *seen* testing up close, nevertheless insist that it be rigorously planned in advance and fully documented." Okay, show me some data to support this ridiculous statement. Which managers? Where do they work? What survey or data supports this claim? On the other hand, I just returned from the Software Engineering and Data Engineering (SEDE) conference which had terrific speakers who actually presented useful, new information. One such topic is the relationship between software testing and entropy. In simple terms, entropy is a measure of disorder in a system. The general equation for entropy is:

 

where b is an arbitrary base (often 2 or 10) and p(x) is the probability that some random variable X takes on value x. The math looks scary but the concepts use only freshman level mathematics and are actually very, very simple. The talk at SEDE went on to describe many highly practical ways entropy can be used in software testing scenarios to improve software quality. The perception of software testing conferences will improve when conference organizers reduce the number of fluffy talks such as "Make Your Project Successful" and increase the number of talks that provide interesting and useful new testing information.

 

The use of Web-based applications such as ASP.NET applications, Web Services, AJAX-based applications, and so on, is increasing steadily. A useful technique for testing many such applications is to send a low-level request to the SUT using TCP/IP and sockets. The technique is general-purpose because is works at a very low level of abstraction. For example, suppose you want to test some Web Service which contains a Web Method named GetTitles(). First you could set up the request in SOAP format along the lines of:

 

string input = "some input";
string soapMessage = "<?xml version=\"1.0\" encoding=\"utf-8\"?>";
soapMessage += "<soap:Envelope>";
soapMessage += "<soap:Body>";
soapMessage += "<GetTitles>";
soapMessage += "<filter>" + input + "</filter>";
soapMessage += "</GetTitles>";
soapMessage += "</soap:Body>";
soapMessage += "</soap:Envelope>";

 

Next you could set up the request destination:

 

string host = "localhost";
string webService = "/TestAuto/Ch8/TheWebService/BookSearch.asmx";
string webMethod = "GetTitles";
IPHostEntry iphe = Dns.Resolve(host);
IPAddress[] addList = iphe.AddressList; // addList[0] == 127.0.0.1
EndPoint ep = new IPEndPoint(addList[0], 80); // ep = 127.0.0.1:80

 

Then you could set up a TCP/IP socket:

 

Socket socket = new Socket(AddressFamily.InterNetwork,
 SocketType.Stream, ProtocolType.Tcp);
socket.Connect(ep);

 

And then construct the HTTP wrapper somewhat like:

 

string header = "POST " + webService + " HTTP/1.1\r\n";
header += "Host: " + host + "\r\n";
header += "Content-Type: text/xml; charset=utf-8\r\n";
header += "Content-Length: " + soapMessage.Length.ToString() + "\r\n";
header += "Connection: close\r\n";
header += "SOAPAction: \"http://tempuri.org/" + webMethod + "\"\r\n\r\n";
string sendAsString = header + soapMessage;
byte[] sendAsBytes = Encoding.ASCII.GetBytes(sendAsString);

 

And next you'd fire off the request and fetch the response:

 

int numBytesSent = socket.Send(sendAsBytes, sendAsBytes.Length,
                               SocketFlags.None);
Console.WriteLine("Sending = " + numBytesSent + " bytes\n");

byte[] receiveBufferAsBytes = new byte[512];
string receiveAsString = "";
string entireReceive = "";
int numBytesReceived = 0;

while ((numBytesReceived = socket.Receive(receiveBufferAsBytes,
 512, SocketFlags.None)) > 0 )
{
  receiveAsString = Encoding.ASCII.GetString(receiveBufferAsBytes,
   0, numBytesReceived);
  entireReceive += receiveAsString;
}

 

Your test could conclude by converting the byte[] response into a string result and analyzing the result for an expected value of some sort. Using a low-level TCP/IP and socket aproach such as I've described is often useful for security-related testing. You can be sure that someone will probe your Web-based application using this technique and so you had better test using the technique.

 

I ran across a very cool metric recently. The metric is called category utility and it can be used to measure how well a database of categorical data is clustered into a set of groups. Cluster analysis of numeric data is one of the most widely studied areas in all of data mining. With numeric data it is not too hard to measure how far one row of data, such as (1.0, 2.0, 3.0), is from another row of data, such as (5.0, -1.5, 6.0). Using a difference metric you can then create clusters of data where the data within a cluster is similar, and the data in different clusters is dissimilar. However, it's not so easy to measure how far categorical data, such as (red, small, hot) is from other categorical data, such as (blue, large, cold). Category utility is a neat idea which measures such differences based on probabilities. The equation is:

 

Each possible clustering of data will have a different categorical utility value, with larger CU values indicating better clustering. I'm looking at a research paper for the International Symposium on Visual Computing and I coded up an implementation of a category utility function in C# and the following screenshot shows it in action:

 

A few blog posts ago I described generating all possible permutations. I mentioned that the terms permutations and combinations are used incorrectly on many Web sites. A mathematical permutation of order (size) n is a rearrangement of the numbers 0 through n-1. For example, if n=3, then all possible permutations are {0,1,2}, {0,2,1}, {1,0,2}, {1,2,0}, {2,0,1}, {2,1,0}. Here I've listed the permutations in lexicographical order. A mathematical combination of order (n,k) is a subset of size k of the numbers 0 through n-1 where order does not matter. For example, all combinations of order (5,3) in lexicographical order are {0,1,2}, {0,1,3}, {0,1,4}, {0,2,3}, {0,2,4}, {0,3,4}, {1,2,3}, {1,2,4}, {1,3,4}, {2,3,4}. Here is a class, written in C# (with error-checking removed) that will generate all possible combinations. The class can be called like this:

 

static void Main(string[] args)
{
  Console.Write("\nEnter combination n-value: ");
  int n = int.Parse(Console.ReadLine());
  Console.Write("\nEnter combination k-value: ");
  int k = int.Parse(Console.ReadLine());

  Console.WriteLine("\nAll combinations: \n");
  Combination c = new Combination(n, k);
  while (c != null) {
    Console.WriteLine(c.ToString());
    c = c.Successor();
  }

  Console.WriteLine("\nDone");
}

 

 

 

Notice the approach is to define a Successor() method which returns the next combination object, or null if we are at the last combination. Here's the class:

 

public class Combination
{
  private readonly int n;
  private readonly int k;
  private readonly int[] data;

  public Combination(int n, int k)
  {
    this.n = n;
    this.k = k;
    this.data = new int[k];
    for (int i = 0; i < k; ++i) {
      data[i] = i;
    }
  } // ctor()

  public override string ToString()
  {
    string s = "{ ";
    for (int i = 0; i < k; ++i)
      s += data[i] + " ";
    s += "}";
    return s;
  }

  public Combination Successor()
  {
    if (data[0] == n - k) // last combination element
      return null;

    Combination ans = new Combination(n, k);

    int i;
    for (i = 0; i < k; ++i)
      ans.data[i] = this.data[i];

    for (i = k - 1; i > 0 && ans.data[i] == n - k + i; --i)
      ;

    ++ans.data[i];

    for (int j = i; j < k - 1; ++j)
      ans.data[j + 1] = ans.data[j] + 1;

    return ans;
  } // Successor()

} // class Combination

 

Starting last week, I have been teaching a class to prepare software test engineers for a Certificate in Software Testing with Microsoft Technologies program. The certificate program covers 42 specific topics that were listed by test managers as those topics that testers should have at least a basic understanding of. One of those 42 topics is character encoding. Yesterday morning I decided to make a tiny demo program to use to illustrate the topic. I recalled that one of my colleagues, William Rollison, is an expert at character encoding. I visited his blog and found a tool he wrote, and I decided to (approximately) reverse engineer that tool so that I could have a simple application with source code to distribute to students in the software testing certificate program. The screenshot below shows the result.

 

 

Interestingly, the character encoding logic was simple (in part because I've investigated the topic before) but I spent an hour wrestling with the ListView data control to display results. The demo is written with C#. Here's the code:

 

private void button1_Click(object sender, EventArgs e)
{
  listView1.Items.Clear();
  int charCount = 0;
  string s = textBox1.Text;
  char[] chars = s.ToCharArray();
     
  foreach (char c in chars)
  {
    char[] arrayWithOneChar = new char[] { c };

    byte[] bytes = null;
    if (radioButton1.Checked) // ASCII
      bytes = Encoding.ASCII.GetBytes(arrayWithOneChar);
    else if (radioButton2.Checked)
      bytes = Encoding.Unicode.GetBytes(arrayWithOneChar); // Little Endian
    else if (radioButton3.Checked)
      bytes = Encoding.BigEndianUnicode.GetBytes(arrayWithOneChar);
    else if (radioButton4.Checked)
      bytes = Encoding.UTF7.GetBytes(arrayWithOneChar);
    else if (radioButton5.Checked)
      bytes = Encoding.UTF8.GetBytes(arrayWithOneChar);
    else if (radioButton6.Checked)
      bytes = Encoding.UTF32.GetBytes(arrayWithOneChar);
 
    string bytesForACharInHex = "";
    string bytesForACharInDecimal = "";
    foreach (byte b in bytes)
    {
      bytesForACharInHex += b.ToString("X2") + " ";
      bytesForACharInDecimal += b.ToString() + " ";
    }
    listView1.Items.Add(new ListViewItem(new string[] { charCount.ToString(), c.ToString(), bytesForACharInHex, bytesForACharInDecimal }));
    ++charCount;
  } // next char
    
} // button1_Click()