posted 06-23-2002 08:33 AM         

A test harness is software that performs automated testing of a core component of a program or system.
Therefore, in addition to using an automated testing tool, a test harness could be developed to allow deeper testing of core components. Usually written in a more robust programming language, such as a stand-alone C++ or VB program, a custom-built test harness will typically be faster and more flexible than an automated test tool script, which is constrained by the test tool’s specific environment.

I think an interesting topic for discussion here would be related to what type of “home-grown” test harnesses others have developed. This discussion could include a summary description of the tool’s architecture, which language was used, who developed the tool (the development team or the testing team, etc?), and/or how successful the effort was (management buy-in, schedules, ROI, etc).
If for example you’ve decided against developing a test harness, what were the reasons?

Looking forward to your feedback,

Elfriede

posted 06-23-2002 10:30 AM           

There were a number of advantages to doing so.

1. We were able to put push through 16,000 + test cases in half an hour with it. It is doubtful whether we could have done much more than 600 in that time via GUI automation.

2. It circumvented the grid control recognition problems with the app.

3. It circumvented a lack of testability of the app, because the variable that carried information as to whether a test had passed or failed was not exposed via the GUI. You got a message but could not tell from it's display whether it was a warning or an error.
I was able to get access to this variable via the test harness.

4. It would have obviated the need for expensive test environment replication. The app would have had to be taken off thin client architecture to allow Gui automation to work. That meant new hardware, software, database licenses, comms/networking etc to get this environment up and running. The test harness simply imported the code components from the server and ran off a standalone laptop.

The harness I built generated test cases automatically by parsing business rules and plugging them into a Perl program written by James Back called Allpairs which created a pairwise orthogonal array of the test data. 2 things then happened. Firstly a test oracle of expected results was generated, and then this was passed to the the code components being tested where the actual result was compared to the expected. The harness had access to the source code of the app and automatically encapsulated each business rule into a component prior to passing the orthogonal array data to it.

I have since issued a genericised re-engineered version of the harness to
genericise as a freeware VB ActiveX Control (although I am still writing tutorials for it and adding functionality). The freeware control allows for the creation of a data dictionary , where not only SUT parameters are defined, but also aliases (for expressions) and file lookups. The test cases are generated from a combination of parsing the business rules and an optional (sometimes) specification of data in the data dictionary. The harness also automatically takes care of boundary value analysis for numeric (to the correct number of decimal places) and date fields.

I will soon write a UNIX/LINUX version, going to try writing it in AWK and BASH (to see how far I get). Ultimately though I suspect the best architecture for the tool would be Prolog for it's AI capabilities, and/or Python for it's object orientation, platform independence and GUI capabilities.

posted 06-25-2002 07:07 AM         

quote:

---

Originally posted by Elfriede Dustin:
I think an interesting topic for discussion here would be related to what type of “home-grown” test harnesses others have developed. This discussion could include a summary description of the tool’s architecture, which language was used, who developed the tool (the development team or the testing team, etc?), and/or how successful the effort was (management buy-in, schedules, ROI, etc).
If for example you’ve decided against developing a test harness, what were the reasons?

---

The test harness we built was constructed within our tool's environment, so this may not be the type of feedback you are looking for. We purchased SilkTest and began automation one test case at a time using a two person test automation group within the test group. Progress was quite slow since our applications have a great deal of functionality and are fairly complicated to learn/test. After a few months our manager decided that he wanted the whole test group involved in the test automation effort. Since none of the other members of our group had any programming experience (and really didn't have the time or inclination to learn the test automation language) I decided to build an interface to our tool - a test harness. This interface was built using SilkTest itself. I used the paper written by Carl Nagle (http://members.aol.com/sascanagl/Default.htm) as the basis for my harness. This paper describes the architecture in detail. The harness allows the testers to input commands via an Access database or an Excel table. The commands and window/object identifiers are documented in a powerpoint file. What we ended up with is an easy to use interface that allows testers to quickly construct and execute automated tests via test tables. Minor Interface changes are easy to handle, and new applications also proved easy to add since all the back-end functionality was already in place.

The success of the effort has varied among different teams within our group. One group has utilized the harness for many of their more tedious testing tasks and it has helped them tremendously. Another group has used it minimally and has only seen minor benefits from the tool. The main problem that I have seen is that management pushed the use of the tool but did not schedule any time or put any deadlines in place. As with most groups, we are constantly busy and very few people had the time/inclination to automate tests. Since there weren't any official milestones only a few people made the extra effort to work on the automation. It has definitely paid off for those folks, so it was worth their effort.

posted 06-25-2002 07:18 AM         

Currently, I've created what I guess you could call a test harness, using a Perl script to launch one of our applications, monitor its progress, record processing time, and capture the test results to a log file.

------------------
Charles Reace

posted 06-26-2002 05:59 AM         

!! The constant
#DEFINE TESTING True

!! create the harness
IF TESTING is True
oHarness = CREATE INSTANCE("StoplightHarness")
ENDIF

!! The actual class
DEFINE CLASS StopLight
LightColor = "RED"

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
METHOD ChangeColors

IF LightColor = "RED"
LightColor = "GREEN"

ELSEIF LightColor = "YELLOW"
LightColor = "RED"

ELSEIF LightColor = "GREEN"
LightColor = "YELLOW"

END IF
END METHOD
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

END DEFINE

#IF TESTING is True
!! This stuff doesn't get included in the exe if
!! TESTING is False

!! The test harness
DEFINE CLASS StopLightHarness

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! This method runs automatically
!! when the class is instantiated
METHOD Initialize
!! instantiate the object being tested.
oStopLight = CREATE INSTANCE("Stoplight")
!! now call each test.
StopLightHarness.TestRed()
StopLightHarness.TestYellow()
StopLightHarness.TestGreen()

END METHOD

METHOD TestRed()
SET oStopLight.Color = "RED"
DO oStopLight.ChangeColor()
CONFIRM oStopLight.Color = "GREEN"
IF NOT, Sound the alarm for the developer.

END METHOD

METHOD TestYellow()
SET oStopLight.Color = "YELLOW"
DO oStopLight.ChangeColor()
CONFIRM oStopLight.Color = "RED"
IF NOT, Sound the alarm for the developer.

END METHOD

METHOD TestGreen()
SET oStopLight.Color = "GREEN"
DO oStopLight.ChangeColor()
CONFIRM oStopLight.Color = "YELLOW"
IF NOT, Sound the alarm for the developer.

END METHOD

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

END DEFINE

As changes are made to the actual class, I jump down and make changes
to the harness as well. The nice thing about having them in the same
file, is that it makes it easy to keep them synchronized in the source
control database. My test harnesses are usually larger than the
classes they test. I also retroactively add tests to the harnesses
when QA finds bugs that the harness missed.

posted 06-27-2002 01:22 PM         

Our new approach is to develop a harness for each major group of objects. The goal of each harness is to cover all methods and properties of the objects, and also to cover all scenarios presented in the object's use case. This way we can directly test the business layer at the base of our suite of products. The bug ticket created usually lists the root cause now (instead of some GUI symptom which could have shown up in numerous tickets), and the time to investigate the ticket for developers is less because of the same reasons (root cause presented).

Myself and another QA resource developed these harnesses in C#. There were a total of about 9 harnesses, and the project took 2 resources roughly 3 months. We are currently automating these harnesses.

One thing to note about our approach: our harnesses were designed to be somewhat robust. I.e., they could be used by any tester to enter various scenarios. It is not just a linear execution of script which touches most scenarios (i.e., it is a little GUI application itself). This allows access for both manual testing and automation. The important aspect is the direct testing of the business layer (both the intitial funcional testing which catches bugs as far upstream in the process as usually possible , and the continued regression possibilities) which QA has full access to now. We know when method X of object Y fails and we can point to a harness scenario to prove it.

Summary:
We started with about 15 use cases (one for each major object area). We created 9 GUI based test harnesses in C#. We ended up with just over 200 test cases based on these harnesses and uncovered roughly 100 bugs in the initial functional testing. We are currently automating some of these test cases. One advantage here is that we (QA) have full access to this code so any automation hurdles are usually easier to deal with because we can control the harness GUI completely.

Anthony Allison

posted 07-05-2002 12:00 PM         

A test harness adapter is a module that “adapts” each system under test to be compatible with the test harness, which executes pre-defined test cases against systems, through the adapters, and stores the results in a standard format so that results can be automatically compared from one run to the next.

For each system to be tested, a specific adapter was developed that is capable of interacting with the system, directly against its DLLs or COM objects, for example, and executing the test cases against it.

To test the two systems with a test harness, it required two different test adapters, and two separate invocations of the test harness, one for each system. The first invocation produced a test result, which would be saved then compared against the test result for the second invocation.

The test harness adapter works by taking a set of test cases and executing them in sequence directly against the application logic of each system under test, bypassing the user interface. This allowed for maximum throughput of the test cases. Results from each test case were stored in one or more results files, in a format, such as XML, that was the same regardless of the systems under test. Result files were retained for later comparison to the results files generated in subsequent test runs.
To compare the results of the tests, a custom-built result comparison tool that knows how to read and evaluate the result files, and output any errors or differences found, was developed. It is also possible to format the results so they can be compared with a standard “file diff” tool.

Test harness test cases may be quite complex, especially if the component tested by the harness is of a mathematical or scientific nature, as it was the case for our testing efforts. Since there are sometimes millions of possible combinations of the various parameters involved in calculations, there are also potentially millions of possible test cases. Given time and budget constraints, it was not economically feasible to express/test all possible test cases, however, many tens-of- thousands of test cases were developed and executed using the test harness. In order to narrow down the set of test cases fed to the test harness we used various approaches, among them was the approach described here http://www.stickyminds.com/sitewide.asp?ObjectId=3130&ObjectType=ART&Function=edetail

Note: This test harness description is copyright “Effective Software Testing: 50 specific ways to improve your software testing,” Addison Wesley (Winter 2002)

Elfriede