I will argue that there is a need for a new, Microsoft-sponsored software testing certification. The Wikipedia entry on software testing certification hits the nail on the head with regards to the weaknesses of all existing software testing certifications: "No certification currently offered actually requires the applicant to demonstrate the ability to test software. No certification is based on a widely accepted body of knowledge." Any certification can have one or more, possibly conflicting, goals. For example, the goal of a software testing certification may be to provide prospective employers with a measure of confidence that an engineer can actually perform software testing Or a goal may be to promote a specific set of technologies and/or products, such as Microsoft .NET and Visual Studio, or perhaps Test Driven Design. Or a goal may be to establish a community of software test engineers who share a common set of terminologies and practices with a view towards enhancing the rigor and effectiveness of the field of software testing in the very long run. Or a perfectly reasonable (but self-serving) goal for an organization is to create a software certification which is designed primarily to generate money for the granting organization by means of charging for required training courses, training materials, and exam fees.

 

There are a handful of companies who grant product-related certifications for their products. Examples include Rational/IMB, Mercury/HP, and Segue. Obviously these companies' goal is to drive sales of their products. There are currently three main (and lots of smaller) non-product-related software certification granting organizations: the International Software Testing Quality Board (ISTQB), the International Institute for Software Testing (IIST), and "Software Certifications", formerly named the Quality Assurance Institute. The ISTQB (see http://www.istqb.org) has a pseudo-academic feel to it and is attempting to create a universally accepted body of knowledge. The ISTQB offers the "ISTQB Certified Tester" certification. This approach is laudable but fatally flawed. The ISTQB is almost entirely terminology based, which leads to a.) there is effectively a limited, discrete number of exam questions, b.) therefore the exam questions are readily available on the Internet, and c.) there is no direct correlation between holding this certification and the ability to actually perform software testing. One actual and typical ISTQB certification exam question is:

 

A deviation from the specified or expected behavior that is visible to end-users is called a(n):

a.) error
b.) fault
c.) failure
d.) defect

While vocabulary is important for clear communication, I argue that questions like this do not correlate with a person's ability to perform good software testing. The IIST (see http://www.testinginstitute.com) has a much more commercial feel than the ISTQB and clearly aims to make profit -- they require you to take 10 days of their training which could cost upwards of $5000. The IIST offers the Certified Software Test Professional (CSTP) and Certified Test Manager (CTM) certifications. The problems with the IIST certifications are the same as the problems with the ISTQB certification: they are primarily vocabulary-based, the exam questions are easily available and memorized, and there's no direct correlation between passing the certifications and testing ability.

 

"Software Certifications" (see http://softwarecertifications.org) is almost purely commercial and has a outdated feel. They offer a Certified Software Quality Analyst (CSQA) and a Certified Software Test Engineer (CSTE) certification. As with the ISTQB, the IIST, Software Certifications exams are primarily vocabulary based (including essay and short answer questions) and suffer the same flaws.

 

Anyway, my position is that Microsoft should sponsor a lightweight software testing certification or specialty. The primary goal of such a certification would be to provide at least a small measure of confidence that someone who holds the certification can actually perform certain types of software testing. A secondary goal would be to evangelize and promote Microsoft technologies, .NET in particular. From an economic standpoint, a certification that I am promoting would not necessarily make money for Microsoft directly, but it should be self-supporting at the very least, through the revenue generated by sales of curriculum and content materials to training partners and by certification exam revenue.

 

In part because the lines between software development and software testing are blurring, I suggest that the Microsoft software testing certification exam be roughly 80%-90% based on coding and design techniques and only 10%-20% based on vocabulary, and even those few vocabulary-related questions be carefully crafted so that they expose testing principles. For example, not, "Which of the following is the definition of 'white box testing'?" (followed by possible definitions) but rather, "In a white box testing scenario which of the following statements is most often true?" (followed by something like, "Creating boundary condition test cases is easier than in a black box testing scenario." Besides correlating with actual software testing ability better than a vocabulary-based exam, a huge advantage of a coding/design based software testing certification exam is that you can create essentially an infinite number of exam questions which probe into a particular testing concept. The idea is to present a testing situation -- system under test description, code, goal of the test, and so on, and then present code or design based alternatives as answers. Simply by changing the testing situation, you  create a new question which probes into the same knowledge concept, and stay ahead of the relentless posting of certification exam questions on the Internet. Based on surveys asking Test Managers what testing techniques they expect software testers to know, I have put together a preliminary course outline of 10 topics that can be covered in 3-4 hours each (perfect for a 4-day commercial course or a 1-semester college class), and which can be nicely tested in a 40-question multiple choice exam which takes 90 minutes, ideal for a Prometric (Microsoft's certification exam delivery partner) certification scenario or college final exam.

 

Basic knowledge of combinatorics is essential for all software test engineers. A mathematical combination is a subset of a collection of items where order doesn't matter. A mathematical permutation is a rearrangement of a collection of items. It is very useful to be able to programmatically manipulate combinations and permutations. Today I was pretty much blocked waiting for some hardware to arrive from Dell, and waiting for some phone messages to be returned, so I decided  to poke around combinatorics with PowerShell. The screenshot beloiw shows that I succeeded in getting a crude set of combination functions working. Because PowerShell is a scripting language it doesn't support classes or structures, so I had to use global variables for n (the total number of items in the collection), k (the subset size), and values (an array to hold a combination element). Here is the main() function I used:

 

# demo.ps1
# string combinations with PowerShell

function main
{
  write-host "`nbegin combinations with PowerShell demo`n"

  $data = ("ant","bat","cow","dog","elk","fox","gnu")
  write-host "Initial data is: $data `n"

  makeCombination 7 3 $data

  for ($i = 0; $i -le 34; $i++) {
    if ($i -lt 10 ) { write-host " " -nonewline }
    write-host "[$i] " -nonewline
    displayCombination
    incrementCombination
  }
   
  write-host "`nend combinations with PowerShell demo`n"
}

 

And here is the rather messy makeCombination() method:

 

function makeCombination($n,$k,[string[]]$d)
{
  $script:nVal = 0
  $script:kVal = 0
  $script:values = @()

  $script:all = @() # save all original strings
  $script:id  = @{} # hash table of original string ids

  if ($d.length -ne $n) {
    throw "Incorrect data array size: array size must equal n"
  }
  else {
    $script:nVal = $n
    $script:kVal = $k

    $script:values = new-object string[] $k
    for ($i = 0; $i -lt $k; $i++) {
      $script:values[$i] = $d[$i]
    }

    # save all strings
    $script:all = new-object string[] $n
    for ($i = 0; $i -lt $d.length; $i++) {
      $script:all[$i] = $d[$i]
    }

    # the 'id' of each string
    for ($i = 0; $i -lt $d.length; $i++) {
      $script:id[$d[$i]] += $i
    }
  } # else
} # makeCombination()

 

Anyway, it was a good exercise that taught me a few things about PowerShell syntax, and supported my notion that PowerShell makes an excellent base-language for lightweight software test automation.

 

Bummer. I've been experimenting with test automation using Windows PowerShell. Everything was wonderful until last week; I performed proof-of-concept demonstrations for almost all of the key test automation tasks including API testing, low-level (via Win32) Windows application UI testing, HTTP request-response testing, low-level (via COM and the IE DOM) Web application UI testing, Web Services testing, and so on. I found that PowerShell is very well-suited for all the test automation tasks I investigated. Then I looked at what I presumed would be by far the easiest type of automation with PowerShell, Windows application UI test automation via the Micosoft UI Automation (MUIA) library/framework. Why did I think this would be trivial? Because PowerShell is essentially a text-based .NET application and therefore all I would be doing is calling code in a .NET Framework library (the MUIA lib that is) from a .NET application (i.e., PowerShell). What could possibly go wrong? Well, as it turns out, the MUIA library employs COM interop to call into Win32 functions such as FindWindow() and SendMessage(). Whenever you use COM interop you must use either a single-thread approach, which ultimately calls the CoInitialize() function, or a multithreaded approach which ultimately calls the CoInitializeEx() function. The MUIA library apparently uses a simple STA (single-threaded apartment) model but unfortunately PowerShell makes all calls using a MTA (multi-threaded apartment model). The effect is that calling certain MUIA library methods from PowerShell just doesn't work. I speculate that PowerShell uses multiple threads; one of these threads calls into the MUIA library; the MUIA library uses a single thread which requires no synchronization plumbing to call into an underlying COM object which actually provides the automation functionality; the return bubbles back up to PowerShell but either returns on a different PowerShell thread or simply gets lost somehow because of the differences between the MUIA library STA and the PowerShell MTA. Bummer. 

I've been examining various types of software test automation using Windows PowerShell (the new command shell and scripting language). My long-term goal is to determine if PowerShell makes sense as a base for a Microsoft software testing certification specialty. Last week I experimented with the Microsoft UI Automation (MUIA) library. The MUIA library is a set of DLLs that ship with the 3.0 version of the .NET Framework and essentially provide you with a uniform way to manipulate and examine UI elements on all types of Windows application programs, including Win32 apps, Form-based apps, and WPF/XAML based apps. The bottom line is that because PowerShell supports direct calls into the .NET Framework, you can call the MUIA library from PowerShell. Conceptually the technique is easy but I ran into quite a few technical problems that caused me some grief. I'll update this topic as I learn more.