|
Author
|
Topic: Test Tool evaluation
|
FL_AUTOTESTER
Member
Posts: 7
Registered: Nov 2002
|
posted 11-07-2002 02:08 PM
    
Hello All, Is there a set of criteria to be considered when evaluating a Test tool. I want to compare WinRunner against SilkTest and select one to suit our Automation needs. Any docs or sources of info would be helpful. Thanks.
|
jpensyl
Moderator
Posts: 894
Registered: Dec 2000
|
posted 11-07-2002 02:17 PM
Simply search this site for:
"tool evaluation""tools comparison" You will get plenty of reference material.
------------------
JP
|
BillThomson
Member
Posts: 26
Registered: Oct 2002
|
posted 11-08-2002 10:43 AM
IMHO, the 4 most important criteria are:
1) Reliability - if the SUT (System Under Test) behavior is consistent, the test tools behavior should be equally consistent. This can be evaluated by running a set of correctly implemented automated test cases against a known, stable SUT hundreds of times. The outcome should be the same every time. You have to be able to trust the outcome of the automated test cases are related to SUT behavior rather than tool behavior otherwise testers will mistrust and eventually despise the test tool.
2) Ease of failure analysis - How easy is it to isolate the failure cause of a test case. Intuitive and complete logs, "step by step" debugging capabilities etc are key here. You don't want the tool to return some obscure failure code (error 22) that you need to go look up in a 1000 page appendix of a manual.
I would recommend injecting a variety of intentional failures in test scripts during the test drive and observing the behavior of the tool.
3) Ability to synchronize state test test script with the SUT. For example, if you need to wait for the SUT to complete a transaction before proceeding with the next one, the test scripting capability of the test tool needs to provide this.
4) Test case language flexibility. The test case language needs to allow communication with other test drivers. For example, a test scenario might require inputing some data via a GUI and then verifying a text file was produced. The needs for this will obviously depend on the type of SUT.Notice how I did not put "ease of coding test cases" on the list. It was not an oversight. I think this is far less important that the criteria above. Test case creation is the least expensive part of test case automation in the long run. ------------------
Bill Thomson
Test Architect (Demolition Man)
Testability = 1/complexity
|
Mimak
Member
Posts: 7
Registered: Nov 2002
|
posted 11-13-2002 02:38 AM
Hello all
The best way I recommanded some one who want to make tools evaluation is by reading the two books:
- Software Test Automation by Mark Fewster/Dorothy Graham chapter 10 and Automated software testing by Elfriede Dustin, Jeff Rashka and John Paul from page 67 to 101. You can reach me by email at mkbern@yahoo.com------------------
|
RagerXS
Member
Posts: 81
Registered: Jul 2001
|
posted 11-14-2002 08:02 PM
Bill made an excellent post without recommending a tool. Smart, because ultimately it is up to the individual... But if you want a head start you can read this thread: http://www.qaforums.com/Forum34/HTML/001517.html After reading that thread, get evaluation copies of each tool you are considering and take a look at them yourself. ~ Fred ------------------
|
gerardy01
Member
Posts: 38
Registered: Oct 2002
|
posted 11-18-2002 03:37 PM
One point I would have in the evaluation is the service and support available from the vendors. This would cover what on-site support they are willing to provide for the initial setup and installation period, what the support arrangements are (is it 24 hr or business hours only - not good if you need support at 10 pm!), do they have a support website and how good is it, etc etc. Check through the various tool specific parts of the forums to get a feel for each vendor support and service.This is important as the tool may be good, but if there's no support to help you with your problems, it can become expensive shelfware very quickly. ------------------
|
BillThomson
Member
Posts: 26
Registered: Oct 2002
|
posted 11-22-2002 05:44 AM
I agree with the good support requirement and should have added that to my list. We have a tool in house right now that is 5 figure shelfware  ------------------
Bill Thomson
Test Architect (Demolition Man)
Testability = 1/complexity
|
Gregor.Schmid
New Member
Posts: 5
Registered: Nov 2002
|
posted 11-25-2002 01:14 AM
I'd like to add another point to Bill's list (with which I fully agree otherwise): Adaptability.If the tool you use is not able to adapt to small changes in the SUT, you may have to spend more time maintaining your scripts than you're saving by automation. Best regards,
Greg ------------------
|
RagerXS
Member
Posts: 81
Registered: Jul 2001
|
posted 12-11-2002 08:37 AM
quote:
Originally posted by Gregor.Schmid:
I'd like to add another point to Bill's list (with which I fully agree otherwise): Adaptability.If the tool you use is not able to adapt to small changes in the SUT, you may have to spend more time maintaining your scripts than you're saving by automation. Best regards,
Greg
I'm biased to SilkTest, having chosen it over WinRunner even though my company's SQA dept has WinRunner and the Mercury rep was willing to deal...So I have to point out the specific adaptability of Object Oriented programming. With WinRunner, if the AUT changes you need to run in update mode for each page/screen/window that is affected by the change. If there are many similar pages that all got the same fundamental change, that is a lot of maintenance. But with a tool like SilkTest where you would have made these similar pages/screens/windows into a class (and then each individual page/screen/window an instance of the class), you need only make a change in one location -- at the class level. ~ Fred ------------------
|
coling
Guru
Posts: 260
Registered: Aug 2000
|
posted 12-11-2002 09:12 AM
Hi,Have you checked out http://www.qadownloads.com/White_Papers/ ------------------
Hope this helps. Regards,
Colin.
|
andrew_msn
New Member
Posts: 2
Registered: Feb 2003
|
posted 02-10-2003 02:40 PM
I find that with web development cycles cycling sometimes as fast as every 60 days, that a deeply penetrating product like SilkTest (which I wholeheartedly agree is the hands-down most powerful tool) may represent a bit of overkill. I have noticed time and time again QA department's scripting development not being able to keep up with web product changes. I am beginning to change my prospective and consider a new view... that of quick and dirty automation. Something I wouldn't have considered even a a year ago when my experience was primarily with shrink-wrapped software with a 12-18 month SDLC. Now I am beginning to see the advantages in being able to whip out a couple 100 simple functional scripts in a few weeks and if necessary throwing them away the next cycle. The converse of which would be to spend durations not supported by the current web SDLC... Developing data-driven, deep testing automated suites, only to loose time maintaining them from build to build... even if the scripts were highly modal/portable. Interesting and opposite to what we as techno-johns & janes might have naturally gravitated to. ------------------
Andrew Marin
|
