| James's profileJames McCaffreyBlogLists |  Tools  Help |
|
January 31 Configuration TestingConfiguration testing is the process of testing a system under development on machines which have various combinations of hardware and software. In many situations the number of possible configurations is far too large to test. For example, suppose you are a member of a test team which working on some desktop user application. The number of combinations of operating system versions, memory sizes, hard drive types, CPUs alone could be enormous. If you target only 10 different operating system versions, 8 different memory sizes, 6 different hard drives, and 7 different CPUs, there are already 10 * 8 * 6 * 7 = 3,360 different hardware configurations. Then if you add various software components such as Web browser version, anti-virus software, earlier versions of the system under development, the number of possible configurations quickly becomes unmanageable. Note that not all software development efforts necessarily face an unmanageable number of configurations. For example, software developed for specific, non-PC hardware devices may have only a handful of hardware and software configurations. Because the number of possible configurations to test is typically too large to effectively test, it is crucial that the planning effort for a software testing effort clearly identify which platforms will be supported. If, as is often the case, the number of combinations of hardware and software is simply too great to test thoroughly, then the test planning effort must carefully prioritize testing different configurations based on a number of factors such as the size of the user base, and the risk associated with an undiscovered bug in a particular configuration. Hardware configuration testing is often performed in test labs which simply consist of a large number of machines with different types of hardware. When a testable build of the system under development is released, the build is installed onto each machine in the lab, and a suite of tests is run on each machine. Because there are usually a large number of machines in a test lab, in most situations a significant amount of automation is performed to install the system under test and execute test suites. Setting up a test lab can be expensive so in many situations hardware configuration testing is well suited to outsourcing to organizations which specialize in this type of testing. Software configuration testing can be very time consuming because of the time required to install and uninstall software including operating system software. For this reason, software configuration testing is also often heavily automated. One configuration testing approach which is not generally possible for hardware testing, but is possible for software testing, is to test on virtual machines. A virtual machine consists of a single file, often called a VHD, or virtual hard drive, which when installed on a host machine can simulate a particular real software configuration. Multiple virtual machines, each with a different software configuration, can be installed and run on a single physical host machine, and tested simultaneously. A software configuration test effort can maintain a library of virtual machines in the form of a set of .vhd files which can be used as needed for testing. There are both commercial virtual machine systems as well as systems which are freely available. On Microsoft platforms, free virtual systems include Virtual PC and the very closely related Virtual Server (the differences between the two are very small and mostly technical in nature), and the 64-bit machine based Hyper-V system. There are several free or open source virtual machine software systems for Unix platforms. One such system is VirtualBox.
January 25 Static Code AnalysisStatic code analysis is the process of examining the source code or object code of the system under development, as opposed to examining the behavior of the SUD while the system is executing. There are many types of static code analysis. One of the oldest and most widely known types of static analysis is called cyclomatic complexity. Cyclomatic complexity is a metric (value) which is the number of independent paths through a software module. Cyclomatic complexity is based on graph theory where a program's control flow is modeled as a graph. Consider this artificially simple example:
int max(int a, int b)
{ int answer; if (a > b) answer =a; else answer = b; return answer; } The cyclomatic complexity of this module is 3. There are two paths through the function, and by definition a return statement adds another path, for a total of three paths. In practice, cyclomatic complexity is usually determined by using a software tool rather than computed by hand. The basic principle of cyclomatic complexity is that more complex modules are more likely to contain logic errors than simple modules. For cyclomatic complexity, a general rule of thumb is that a module that has complexity higher than 10 should be examined to see if the module can be refactored into smaller, less complex modules. In some software development scenarios, a maximum complexity value is established. After each build, the modules of the SUD are examined using static code analysis techniques. If any module exceeds some maximum complexity threshold value, the build fails the complexity test.
Cyclomatic complexity was devised in the 1970s before the widespread use of object oriented programming languages. OOP static analysis examines the structure of classes, methods, properties, and other features that are associated with OOP design. Examples of characteristics which could indicate an overly-complex design and therefore susceptibility to bugs include the number of data fields and methods in a class, the number of input parameters in a method, the number of static variables, the number of method calls made by a particular method, and so on. In some development scenarios, an OOP based static code analysis is performed on the system under development and any classes which have a high design complexity value are flagged for review and closer examination.
In a Microsoft .NET programming environment, FxCop is a standard static code analysis tool which primarily examines the SUD for usage patterns rather than complexity characteristics. It examines a system's .NET intermediate language code derived from the system's source code for a wide range of potentially error-prone patterns. Examples of usage warnings include, "Do not call overridable methods in constructors", "Collection properties should be read only", "Do not raise reserved exception types", and so on. Many other programming environments have similar tools. For example, in a Java programming environment the open source FindBugs tool examines bytecode for similar potential usage trouble spots.
Another type of static code analysis is dependency analysis. Dependency analysis examines the source code of a system to determine how many methods are called by method in the system, and similarly, how many different methods call a particular method. The idea is that a method which has many calls to other methods may be overly complex and is more likely to contain bugs.
January 17 Kernel DebuggingThe distinction between user mode software and kernel mode software is relevant to testing. In most cases when application software contains a crashing bug, the application itself may crash but the operating system will usually keep running. However when kernel mode software, often device drivers, crashes the entire operating system may be brought down. In a Windows environment this is sometimes called the blue screen of death (BSOD) because of the color of the error screen displayed after a system crash. Ordinary debugging is the process of determining the cause of a software application failure. Ordinary debugging can be accomplished in several ways because after a crash the host machine is still operating and the crashed program can be examined. For example in a Windows environment, testers can use the cdb.exe (command line debugger) or use the integrated debugging capabilities of the Visual Studio integrated development environment. Kernel debugging is the process of determining the cause of an operating system crash. Kernel debugging requires special tools because after a system crash there is no way to directly examine the host machine.
In a Windows environment, a common kernel debugging tool is the kdb.exe (kernel debugger) command line program. Unix variants usually have similar programs, for example, kdbg for use with the Linux kernel. The kdb.exe tool is somewhat tricky to use and so in Windows environment most kernel debugging is performed using the windbg.exe tool which provides a GUI interface around both cdb.exe (for application debugging) and kdb.exe (for kernel debugging). There are two ways to capture an image of a crashed test host machine for analysis by the windbg.exe / kdb.exe tool. First, a tester can configure the test host to save an image of itself after a system crash, typically as a file named MEMORY.DMP directly on the host. Then after a crash, the test host machine is restarted and the memory image can be analyzed using the kdb.exe tool. The second alternative is to connect the test host machine to a second machine, using an RS-232 interface with a null modem cable. Then when the test host crashed, its memory image is sent to the second machine where it can be analyzed.
In a Windows environment, to perform kernel debugging a tester must have a .pdb file (program database) which is specific to the operating system running on the test host machine. The .pdb file can be downloaded from a publicly available database of such files maintained by Microsoft, or in the situation where the test host machine is connected to a second analysis machine, the analysis machine can access the library of .pdb files over the Internet via a Web server maintained by Microsoft. In some situations software testers are responsible for generating and capturing system crash dumps and then turning these dumps over to developers for analysis. In other situations, testers are responsible for both collecting and analyzing crash dump data. January 09 Module Testing and Buddy TestingI am working on a project that lists and briefly desribes 40 fundamental topics every software tester at Microsoft should know. One of them is module testing. Module testing is arguably the most fundamental type of software testing. Most non-trivial software systems are composed of basic building blocks such as DLLs (dynamic link libraries) and .NET assemblies in a Windows environment, SOs (shared object files) in a Unix-like environment, and Java packages. The idea is simple: if the basic building blocks which make up a software system are not correct, then the software system as a whole cannot possibly be correct. In a Windows development environment the most common scenario is that developers create a library composed of one of more classes which in turn contain one or more methods which must be tested. When developers use native Win32 code (typically with the C++) language the result is a usually file with a .dll extension. Native code often, but not always, uses COM technology so that the module library can be called from multiple languages such as C++, C#, and JavaScript. When developers use managed .NET code (typically with the C# or VB.NET language) the result is also usually a file with a .dll extension although the internal structure is entirely different from a native code DLL file. Managed code libraries do not use COM technology -- in fact one of the motivations for creating the .NET environment was to reduce the need for programming with the very difficult to use COM. Here's a simplified example. Suppose a developer is part of a team working on a software system which performs some mathematical calculations including the harmonic mean of two numbers. (The arithmetic mean is a normal average; the harmonic mean is an average used when the data points are rates such as speed measured in miles per hour). The developer uses C++ to create a COM based, dual interface library named MyMathLib.dll which contains a class named MyMathFunctions which in turn contains a method HarmonicMean(). At this point the devloper could check the new C++ source code into a source code repository where it would eventually be compiled, perhaps by a build team, and the HarmonicMean() method would be available for other developers to use in the software system and for testers to test. An alternative approach is for the developer to request a buddy test. Here the developer would not check the library source code into the source code repository. Instead the developer would compile the source and then give the resulting DLL to a test engineer. The test engineer would then test the methods in the DLL. For example, if using JavaScript, the tester would copy the DLL to a test host machine. Then the tester would register the DLL on the test host using the regsvr32.exe tool and create a small JavaScript test harness that tests the method in the DLL along the lines of: WScript.Echo("Begin test\n"); One of the ideas of buddy testing is to have some measure of confidence that new source code is relatively free of bugs before the code is checked into a source code repository.
January 03 The Synchronize and Stabilize Software Development Methodology ParadigmMany product groups at Microsoft use a software development methodology which is called the Synchronize and Stabilize Model. One of the earliest software development methodologies created is called the waterfall model. Here the requirements, design, implementation, and testing phases are done once each, in order. This approach is often used for military and medical systems. Another methodology is the spiral model. Here, instead of one meta requirements-design-implementation-coding phase, the multi-step process is repeated several times, with each cycle typically taking two to four months. This approach is often used for commercial software applications. A popular form of software development methodology which appeared about ten years ago is called agile development. Here the same multi-step, requirements-design-implementation-testing process is repeated over and over, with each cycle taking typically about two to six weeks. The synchronize and stabilize methodology is characterized by several things. The requirements-design-implementation-test cycle is repeated over and over until certain milestones are reached. These milestones have different labels such as MM1 (major milestone 1), Alpha, Beta Release, CTP (community technical preview). Often, the test team is separate from the development team, and builds of the system under development are produced daily. This approach has been very successful but requires a lot of resources and money. If you are a software tester you need to understand the differences in software development methodologies because testing approaches which work well in one type of development methodology will often not work well in a different type of development methodology.
|
|
|
