posted 12-28-2002 02:57 PM         

Automated Testing - FAQs

This forum is for Automated Testing related general discussions, i.e.
how to get started using automated testing tools; automated testing
tool evaluations; which tool to use; what to automate; what not to
automate; when to automate; managing automated testing expectations;
automated testing strategies; when to use a vendor provided tool and
when to develop a tool in-house; and other experienced-based practices
and key concepts.

Before posting any questions, please review the following frequently
asked questions (FAQs). If you don't find your question here or
the answer is not to your satisfaction, please feel free to post a new
question.

Content of the Automated Testing FAQs, to date:

FAQ1: I am new to automated testing, where do I start?
FAQ2: Where can I find articles about Automated Testing?
FAQ3: Where can I find books about Automated Testing?
FAQ4: I have a question related to a specific tool, e.g. Winrunner, Robot, QARun, etc
FAQ5: Which is the best testing tool on the market?
FAQ6: Where can I find testing tool comparisons and evaluations?
FAQ7: What is a Test Harness? (also referred to as Test Oracles)
FAQ8: How can I manage false expectations that come with Automation?
FAQ9: What is the Return on Investment (ROI) for Automated Testing?
FAQ10: What is a Keyword or data driven framework?
FAQ11: Why automate?
FAQ12: When to automate?
FAQ13: What to automate?
FAQ14: How to get tool evaluation copies?
FAQ15: Testing tool suggestions for various development languages, i.e. Java, C++, DHTML?
FAQ16: Testing tool suggestions for SAP, PeopleSoft, Siebel, etc?
FAQ17: Testing tools for different Operating Systems and other environments
FAQ18: Where can I get tool training on-line?
FAQ19: Where can I get free testing tools?
FAQ20: Is 100 percent automation possible?

==================================================================
FAQ1: I am new to automated testing, where do I start?

If you are new to test automation you might want to start by reading the automation
articles or books mentioned in FAQ2 and FAQ3. For starters, you might want to read
the following posts:

Automation will it fail me?
I'm new and I need your opinions about tools
I am new, please help
Steps for automated testing
Automators vs Manual Tester Ratios
Programming languages and testing

FAQ2: Where can I find articles about Automated Testing?

There are numerous test automation articles available. You might want to use a search
engine such as Google or start by reviewing the following posts:

Automated testing vs Human testing
Implementation of Automated Tools
Test Automation articles

FAQ3: Where can I find books about Automated Testing?

QALinks Books
The Best Books
Automated Testing reference books
Books about Automated Testing tools

FAQ4: I have a question related to a specific tool, e.g. Winrunner, Robot,
QARun, etc

Specific questions related to specific tools are to be posted under the relevant tool forum.
Scroll down on the main qaforums page and you will find forums for most all the major tools on the market.
Please post specific tool questions in the relevant tool forum, as this is a general automated testing discussion forum.

FAQ5: Which is the best testing tool on the market?

This question is asked often and the responses will vary, depending on the user's experience and other criteria.
See for example this post: Automated Testing Tools

Mercury has the largest market share. However, it is important to evaluate the tool based on your system-engineering environment, available budget, schedules, skills available, and other
testing needs. Other criteria is listed in the Test Tool Comparison/Evaluation in FAQ6, next.

Sometimes you might have to develop a tool in-house, because none of the vendor provided tools are compatible with your system-engineering environment or for other reasons a test harness will have to be developed. See the related discussion below on Test Harnesses.

FAQ6: Where can I find testing tool comparisons and evaluations?

see

Comparisons for the simple minded
How long to evaluate GUI-automation test tools
Document comparing automated test tools
Mercury or Rational?
Tool comparison
Which of the automated test tool has the best service
Automation Tool Criteria
Test Tool evaluation
Tool Evaluation

FAQ7: What is a Test Harness? (also referred to as Test Oracles)

Automated testing does not only involve using capture/playback tools or a modification of the tool's script generation/language - whether data driven or keyword driven - instead, it often requires the development of a test harness. For additional info, check this post: Test Harnesses

Also, see
Test Harness
Can Perl script be used for testing automation?
Integration Test with Test Drivers
Components of an ideal test harness
A non-GUI linux test harness
Test oracles

FAQ8: How can I manage false expectations that come with Automation?

Along with the idea of automated testing come high expectations. Although automated testing can produce a return on investment if implemented effectively, an immediate pay-back is not always achieved.
It is important to manage the high expectations that come with automated testing. See the following post for related info:
Misconceptions of Mgmt

FAQ9: What is the Return on Investment (ROI) for Automated Testing?

Automated testing can provide several benefits when it is implemented correctly. The test engineer must evaluate whether the potential benefits fit the required improvement criteria and whether the pursuit of automated testing on the project is still a logical fit, given the organizational environment, i.e. tool budgets, schedules, skills available, etc. Also review
the following posts:

Justify the use or purchase of the testing tools
Automated Testing Vs Manual - Time Saving?

FAQ10: What is a Keyword or data driven framework?

see
"Test-Plan" driven Method (Keith Zambelich)
Test engine
Key-Word Driven Methodology
Expected results for Data Driven tests
How Many Of You Are Using Table/Keyword-Driven Methodology?
Table/Keyword Driven (3rd Generation) Automation Questions

FAQ11: Why automate?

see
Why automate?
Is automated testing the answer?

FAQ12: When to automate?

see
When should I start using an Automated tool for Regression testing
Automated or Not automated that is the question
Manual Testing Vs. Automated Testing

FAQ13: What to automate?

see
How much automated testing?
Criteria for Automated/Manual Testing

FAQ14: How to get tool evaluation copies?

see
Evaluation copy...........
More info to follow

FAQ15: Testing tool suggestions for various development languages, i.e. Java, C++, DHTML?

see
What is the best automation tool for Lotus Notes/Domino, VB and Java
Best ActiveX Control recognition in Automated Tool!!
Testing VB.Net Applications?
Unit testing tools for VB.Net
Best tool for Java
Automated Testing - FoxPro
Automation Tools to run against
Apps with Stingray Controls

FAQ16: Testing tool suggestions for SAP, PeopleSoft, Siebel, etc?
Mercury is the leader in this market..
placeholder

FAQ17: Testing tools for different Operating Systems and other environments

There are specific forums for Macintosh and Unix - scroll down on the main page.
There is also a separate forum for PDAs and embedded systems - scroll down on the main page.

Also review the following
Test automation tools for PDAs
Best tool to test mainframe
Test tool best for AS400
Linux Automation Tool

FAQ18: Where can I get tool training on-line?

see

Qatraining.net provides on-line training
Also see this related discussion
Tutorials on WinRunner

FAQ19: Where can I get free testing tools?

see

Free GUI test tools
Open Source Test Suites

FAQ20: Is 100 percent automation possible?

see

Is 100 percent automation possible?

More info to follow