The vast majority of software testing is functional verification testing. In this scenario, you have a test case which consists of one or more inputs to the system under test, and an expected value of some sort. The result of the test case is pass or fail, nothing else. However, in some testing scenarios, you must assign metrics. A good example is performance testing, where you measure the time required for some component of your system under test. But what if you need to assign a metric to some component which is inherently subjective? For example, suppose you are writing a search engine and you want to measure how good your system's results are relative to previous builds of your system. A very elegant math construct called a rank order centroid (ROC) can do this for you. Suppose your current build is #00012 and you want to measure its quality against previous builds #00011, #00010, and #00009. You simply rank the four builds from best to worst -- say, #00012 is best, #00009 is 2nd best, #00011 is 3rd, and #00010 is 4th. You can convert the ranks into ratings like this:

 

#00012: (1 + 1/2 + 1/3 + 1/4) / 4 = 0.5208

#00009: (0 + 1/2 + 1/3 + 1/4) / 4 = 0.2708

#00011: (0 +  0  + 1/3 + 1/4) / 4 = 0.1458

#00010: (0 +  0  +  0  + 1/4) / 4 = 0.0625

 

The pattern should be clear to you. Notice that the weights sum to 1.0 (subject to rounding error). So, rank order cenroids are a neat way to turn subjective test results in the form of rabnks (1st, 2nd, etc.) into ratings (0.5208, 0.2708, etc.) which can be analyzed mathematically and tracked over time. Check out my MSDN Magazine article "Competitive Analysis Using MAGIQ" at http://msdn.microsoft.com/msdnmag/issues/06/10/TestRun/default.aspx for more information.

IronPython is an implementation of the Python scripting language which runs in a .NET environment. IronPython supports interactive console sessions as well as dynamic compilation. I think IronPython has great promise as a general purpose test automation language. If you are writing lightweight test automation in a 100% .NET environment, then you are probably better off using C# -- although Python does have some very neat features (especially the List and Dictionary objects). But if you are working in a heterogeneous environment (meaning both .NET and non-.NET, and non-Windows environments), then IronPython is a very attractive choice. Because IronPython is .NET compliant you have the enormous power of the .NET Framework available to you. Because IronPython is "true" Python, you can write non-.Windows-based automation too. In other words, in a heterogeneous environment, IronPython meets all your test automation needs so you don't need, say, C# for your Windows automation and Perl for your non-Windows automation. See http://www.codeplex.com/Wiki/View.aspx?ProjectName=IronPython for more information about IronPython.

When using SQL Server Mixed Mode authentication for test automation, you must deal with SQL Users and SQL Logins. These two objects are quite different, but often confused. The essential difference is that SQL Logins are server-scope objects which are used to control authentication and access to the server, while SQL Users are database scope objects which are used to control authorization and permissions on a particular database. Much of the confusion results from the fact that there is a mapping between Logins and Users, and that the names of related Users and Logins are usually the same. Luckily the SQL Server 2000 users-logins model has been improved greatly in SQL Server 2005. Here's a script template which creates a database for test results storage:

 

use master

go

 

if exists (select * from sys.sysdatabases where name='dbTestResults')

 drop database dbTestResultss -- note: drops any associated users too

go

 

if exists (select * from sys.syslogins where name='harness')

 drop login harness

go

 

create database dbTestResults

go

 

use dbTestResults

go

 

create table tblResults

(

caseid  char(3)  primary key,

result  char(4)  not null, -- 'pass' or 'fail'

-- other fields here

)

go

 

create login harness with password='secret'

go -- create a server-scope login object for the harness to use

 

create user harness for login harness

go -- create a database-scope user with same name

 

exec sp_addrolemember 'db_datareader', 'harness'

-- allows user 'harness' (associated with login 'harness')

-- to read (using SELECT) any table in dbTestResults

exec sp_addrolemember 'db_datawriter', 'harness'

-- allow INSERT and UPDATE

go

 

-- end of script

 

The harness code should be fairly self-explanatory. You can use the "create login" command to create a means for a user (actual person or virtual user) to connect to the SQL Server machine. Then you use the new-to-SQL-2005 "create user" command to create a user associated with the login. Finally we can grant permissions to the newly created user, by invoking the sp_addrolemember system stored procedure.

In many software test automation scenarios, you will store test case data and test results in a SQL database. It is not uncommon for beginners to be confused about several issues related to SQL Server authentication and authorization. When you install the software for a SQL Server machines you can specify either Windows Authentication mode or Mixed mode. In Windows Authentication mode, SQL Server uses your Windows user credentials to establish who you are (where "you" can be mapped to an actual person or just a virtual user account), which then establishes your authorization permissions. In theory this is efficient, secure, and tidy. However, in practice, because the Windows security model is so complex, when writing test automation connection issues often arise. Furthermore, if you are operating in a non-homogeneous environment (with non-Microsoft servers and technologies), in many cases pure Windows Authentication mode just doesn't work. So, most of my test automation colleagues prefer setting up SQL Server with Mixed Mode Authentication. For example, suppose your test harness needs to connect to a database dbTestCases. You've set up Mixed Mode Authentication and created an account named 'harness' which has password 'secret'. To connect using Windows Authentication, your code will resemble:

 

string sConn = "Server=(local);Database=dbTestCases;Trusted_Conection=yes";

 

And to connect using SQL (Mixed) Authentication, your code would resemble:

 

string sConn = "Server=(local);Database=dbTestCases;UID=harness;PWD=secret";

 

You can see there's not much difference in the code but in my experience, the Mixed Mode authentication version is less troublesome.