posted 01-25-2002 08:56 AM         

We currently have WinRunner, Astra, and XRunner sitting on the shelf ready for action, and we are trying to develop this architecture so we can have a fully robust automation framework. However, the lack of information is slowing us down. Please let me know if you have any input on this.
Thanks!

posted 01-25-2002 12:14 PM         

The user group is also talking about creating a WinRunner version.

posted 01-28-2002 11:35 AM         

1) Automation efforts can be highly personal.
- While generic ideas may exist in the various automation tasks, the actual implemenation from one project to another may be so varied that the framework cannot be used as is across various projects. (insert companies for projects to get a larger sense of this problem).

2) Designing data driven tests (and their frameowrks) entails actually knowing quite a bit about the product under test.

3) Framworks are really the engine of the test automation effort. Plugging X into the system will get you Y, but how does your automation get that result? You have to build it and verify it. These tools out of the box might get you some of the results you need, but I doubt that it will give you all of the results you need (unless you application under test happens to be quite simple).

---

In sum, I would reccomend that you take the information from these articles are a starting point in your automation projects. Figure out what it is exactly that you want the system to do. From here, you should be able to create or use other peoples test cases to find out how you can create data-driven tests, where adding new test cases is simply a case of adding data into the data files. Then the hard part really begins. YOu'll have to design the framework. YOu'll have to implement that framework, and you'll have to maintain it as well.

I always treat automation projects as a software project, where the goal here is to test specific portions of the application under test. I doubt that you can use any single tool to test all aspects of your (or any) application, but with the combination of tools (both COTS and homegrown) you'll be well on your way to repeatable, reliable automated tests. It won't happen overnight, but its a start.

Good luck!

posted 01-29-2002 10:21 AM         

I have to agree that you can't put out code out in the open. It's also a 'company property'.

The key thing that everyone seems to agree is separate of data and test code. I still see people who hard code logics with data, then cut&paste, and cut&paste to thousands of nightmare lines!!!

The important thing is to truly identify what you are looking for, and given that I am handing my project over to someone, I can spend a little more time to give you an example in code.

I worked in one company, where all the data was stored in Oracle, and basic calculations were done in Stored Procs. Because C++ code was not there, we started with Stored Proc validation tool in Perl. So, here's a simple logic.

TestCase Code:
MyTableSum(Test1-2-3, MasterCard, CreditLine, 0)
MyTableSum(Test1-2-3, MasterCard, UserName, 1)

Test Main function:
Sub MyTableSum(sCaseId, sTableName, sColumnName, bExpectResult){
if (SPAddSumValuesTotalofTable(Table_x, Column_y)){
bTestSuccess = 0
}
compareAndPrintResult(bExpectResult, bTestSuccess)
}

Sub SPAddSumValuesTotalofTable{
-Execute function here in stored proc call using database interface-
-Call database using the table name and column values, and match against the returned value above and return boolen value.
}

sub compareAndPrintResult{
if (bExpectResult == bTestSuccess){
Print sCaseId . " matched."
else
Print sCaseId . " mis-matched!"
}
}

So, the main script will call a function called MyTableSum, which will execute the test case, called SPAddSumValuesTotalofTable, which will return a boolean of success or failure. (Actual test code). The beauty of writing like this is, you can change the report formatting to suite your needs. (Including html). The other part is test execution code is separate, and you will isolate out the code change to just the functional part.

So, we have the Stored Procedure, but within the code, validation logic is calculating the database. So, why not enhance it? In my last job, this test code handled TCP/IP message (wrote a C++ code that hooked into the perl), and java code as well. Validation logic stayed the same, and we simply added extra logic in the main functionality to determine which code path to use.

So, the important thing here is 'modularity'. The benefits are as follows:
1. Consistancy. Using one report function, you can generate a consistant report format. You can now convert data to html, or csv as you please.
2. Migration strategy. Let's say you decide to migrate to another test language, since the main test scripts contain the data, yo haven't lost all of your invested time and resource. while your engineers write respecitive modules in another language.
3. Maintainance is cheap, since you can slide modules in and out, while main test scripts and other parts are untouched.

so, let's say development says their messagin is ready, you simply enhance your test code as follows with couple of optional arguments:

Test Main function:
Sub MyTableSum(sCaseId, sTableName, sColumnName, bExpectResult, testMode){
if (SPAddSumValuesTotalofTable(Table_x, Column_y, testMode)){
bTestSuccess = 0
}
compareAndPrintResult(bExpectResult, bTestSuccess)
}

Sub SPAddSumValuesTotalofTable{
case (testMode)
if TCP-IP Messaging, {Execute the messaging and obtain data}
if JavaServlet, {Execute Java interface, and get value}
else StoreProc, {Execute function here in stored proc call using database interface}
end case}
//Validation call is identical, so do the same for everything!
-Call database using the table name and column values, and match against the returned value above and return boolen value.
}

Test scripts are in tact, and you just slid in additional functionalities! You can cut and paste all the test cases, and simply add test switches to do end, and OULA!!!! You have regression covered all the Java and TCP-IP messaging in one shot! (Now, you know it isn't that easy, since things will break all over...) The best thing is when development is stumbled at how many test cases you've executed shortly after their stuff is done, and you've also validate their infrastructure so you can pinpoint the issues.

I hope this gives you some aspect.

posted 03-14-2002 10:43 AM         

What is the output of the Test Automation Framework? Does this framework that I have worked on create scripts readable by the automation tool? So in the case if I was using WinRunner as my capture/playback, would the output of the framework give me TSL functions?

Thanks

posted 03-15-2002 12:20 AM         

Your output should be a Pass or Fail, and maybe any other information you want. Your input would be the data. The TSL functions should be part of the framework.

Cheers,

Ross.

------------------
Ross Ryan
rossryan@ireland.com

posted 03-15-2002 06:24 AM         

1) From the Software Test Automation book, Chapter 22 "Testing with Action Words" by Hans Buwalda

2) Hans Buwalda new book "Integrated Test Design and Automation: using the TestFrame Method"

ping

posted 03-15-2002 01:41 PM         

quote:

---

Originally posted by pcl9097:
What I am confused about is this:

What is the output of the Test Automation Framework? Does this framework that I have worked on create scripts readable by the automation tool? So in the case if I was using WinRunner as my capture/playback, would the output of the framework give me TSL functions?

Thanks

---

I think you are really missing the big picture about a test framework. You are definitely not alone - I read many of the articles and books mentioned and I didn't get it either. After a not-so-successful attempt at implementing a data-driven scheme, I went back and studied about test harness's again - and I think I understand it now.

A test automation framework is an environment that allows you to create test scripts rapidly for new or existing applications. The framework is created as an independent entity - it does not depend on your application(s) directly. It also does not work with record/playback scripts (at least I don't see how it would). When you record a script, you have hard-coded your window declarations and your input data. The idea of the test harness is to get away from that.

What you build when you create a harness is essentially a layer of abstraction over top of your test tool. In my case we built a two-tiered harness. One tier processed inputs from either an Excel table or an Access database and formatted the commands for the backend. The backend was basically designed to interact with GUI components at the component level. There is a separate module for dealing with each component - PushButtons, TextFields, ComboBoxes, etc. It doesn't care what GUI or page the component is on, it only cares about dealing with the specified component. The front end passes it a component and a command, and possibly input text if needed. Any application we care to use is 'learned' within the tool. This application map is used to generate an API.

The input tables are generated based on an API that the test harness creaters create. The API identifies every GUI component by name as outlined in the GUI map within the test tool. A second document outlines all the commands that are available for a given component. Every command must be coded into the harness for the component that it applies to. Using these two documents (power point slides in my case), any tester can generate an automated test case fairly easily. You only have to know how to kick off a test case in the automated tool - you don't have to know anything about coding in the tool's language.

The commands that you implement within the harness will determine what output you receive. In my case, I have two basic types of commands - verification commands and non-verification commands. Verification commands result in information being entered into an Output log file. This log file is used to determine if a verification step either passed or failed. Verification steps can also perform screen snapshots or generate their own output files. I try to make sure that enough information is provided so the tester can examine the logs and be comforable that the verification really passed, or to provide enough information so that the tester can tell what went wrong.
Non-verification commands are used to drive the system to the desired state prior to a verification command. These commands do not enter any information into the Output log. I have a debug log that contains all this information. Any time a command is entered an entry is placed in the debug log. This provides a great deal of information about what is going on and allows any problems to be 'debugged'. All log entries are time-stamped so they can be cross-references.

I'm sure there are other ways to implement a harness, but this is one of the most popular (from what I have read). It allows one or a few automation engineers to develop a utility that allows a whole group of testers to create automated tests.

posted 03-15-2002 07:48 PM         

------------------
www.turbotester.com

posted 03-18-2002 08:20 AM         

automation3_18.pdf

As you can probably tell, I'm awfully confused. I have drawn up a flow diagram of what I think this is all about (see attachment). Let me know if this makes any sense.

What I am having a hard time understanding (right now) is #3 of the diagram. What is actually driving the Automation Tool? If what you say is true, that a test automation framework is an environment that allows you to create test scripts rapidly, then are these test scripts readable by the Automation Tool; in TurboTester's case, the rapidly generated scripts are TSLs?

I appreciated all of your help. Thanks.

posted 03-18-2002 09:43 AM         

quote:

---

Originally posted by pcl9097:
What I am having a hard time understanding (right now) is #3 of the diagram. What is actually driving the Automation Tool?

---

posted 03-18-2002 10:32 AM         

Your Navigation Script will process the input commands and format them into the right format to call the ‘backend’. In your diagram the backend is the ‘Automation Tool’ block. This will contain your functions for dealing with each window component type. There will be a module for TextFields. This module will receive all commands that deal with text fields. It will contain all the functions (set text, get text, verify text, etc.) related to the text fields. These modules are generic – they should work with any application. I have found that I can keep mine about 90—95% generic, but that certain commands require application knowledge to work.

In addition to the predefined components (TextField, ComboBox, PushButton, etc.) I found a need to implement a set of custom commands that were application-specific. I keep these in different files and label the file after the application.

Here is a long example that will hopefully explain the interaction between my script processing engine and my backend command processor. I’ll use the terminology and diagram from Nagle’s Figure 4 in this example.

Say we want to test that you can successfully enter a user Id in the UserId field. The test steps are as follows:

1. Invoke the login window.
2. Enter the UserID.
3. Verify the UserID has been set.

We use a three-tiered table structure as outlined in Nagle’s document. The Cycle table and Suite tables call lower level tables. For simplicity, I’ll only deal with the lowest level table right now. My lowest level test table (the Step Table) will contain the following commands:

MainWin .... Component . Action . InputText
LoginWindow ........... Invoke
LoginWindow . UserId .. SetText .... UserName
LoginWindow . UserId .. VerifyText . UserName

I have a SilkTest script called CycleDriver which processes the highest level tables – the cycle tables. A SilkTest script call SuiteDriver processes the intermediate tables and a StepDriver script processes the lowest level tables.

StepDriver reads in the MainWin and the Component and formats that into the identifier for the specified component. For the first command, there is no component. StepDriver determines what type of window LoginWindow is – in this case it is a JavaMainWin. StepDriver calls JavaMainWin and passes it the window name, the action to perform, and any additional parameters needed. JavaMainWin is a module that processes any commands related to JavaMainWin’s. It contains the instructions for all these commands. The JavaMainWin module sees that this particular command is an ‘Invoke’ command. It will call the invoke method for this window. StepDriver then processes the next command by figuring out the component name – in this case LoginWindow.UserId (this is the user ID text field). It determines that this is a TextField and routes the command to the TextField processing module. The TextField module receives the command to ‘SetText’ for LoginWindow.UserId and has the associated input text ‘UserName’. It goes into the SetText routine which contains the instructions to do the actual settext command. The StepDriver module then routes the next command in the same way – it passes the component name (LoginWindow.UserId), the command (VerifyText), and the input text (UserName) to the TextField module. The TextField module routes the command to the VerifyText routine. This routine will do whatever you code it to do – in this case it will ‘get’ the text from the UserId field and compare it to the input text value specified in the command. If they match it will print a Success message to the output log. If they don’t it will print a Failure message.

------------------
Atlee

posted 03-18-2002 02:26 PM         

So if we had to test this on an App running on Unix, I would have to also create a "Navigation Script/Core Data Driven Engine" for XRunner?

Thanks.