|
Author
|
Topic: How long to evaluate GUI-automation test tools
|
Jim O'Reilly
New Member
Posts: 4
Registered: Nov 2001
|
posted 01-08-2002 04:05 PM
    
Hi, folks. I need some advice. I'm in the process of evaluating tools for automated testing of GUI applications (Visual C++ with MFC - and later with StingRay Studio - on NT4). The tools that seem to have promise (web searches and browsing this forum :-), etc.) are Rational Robot, WinRunner, SilkTest, and Visual Test.I'll be using these tools primarily to test the GUI itself (execute GUI actions and compare the resultant GUI windows/screens against a reference) - a test harness approach will be used to provide fuller coverage of tool internals. I plan to test-drive each of these tools - anyone have any idea how much time I should allocate for evaluating each of the tools? Any advice welcome, Regards, Jim O'Reilly
jim_oreilly@agilent.com ------------------
|
turbotester
Guru
Posts: 375
Registered: Dec 2001
|
posted 01-08-2002 06:43 PM
Jim- Time should be your last consideration - your first consideration should be: How skilled are the people who are going to be using the tools? Do they have scripting knowledge regardless of language? Do they have SQL knowledge? Do they know how to write good manual tests that can then be easily interpreted for automation testing? Which tools give you the easiest time in learning the language? And finally- once the idea of record and playback has finally left your head as the approach you may think you are gonna use, which tool gives you the easiest tools to provide ease of maintenance (meaning which ones make it easy for you to NOT have to rewrite every time something changes in the appication.) GUI Map recording is the key - gure out which tools are the easiest to use, most powerful n terms of recognizing the objects of your AUT, and which allows for the easiest method of maintenance? Time at the tool in the beginning should give you an idea of which one is best by which one you come away thinking "Yeah, I could learn this etty quick" Not - "Wow look at how many bells and whistles this thing has!" Hope that helps some. Feel free to contact me at turbotesting@hotmail.com - especially if you are doing testing of a client/server tool. I may have something for you that may make this decision easier!the ------------------
Brian K. Qualters TurboTesting Concepts
|
Ross
Advanced
Posts: 113
Registered: Jul 2001
|
posted 01-09-2002 01:45 AM
Hi Jim,You should check out TestPartner by Compuware as well. It is pretty much a brand new tool in it's first version (even though its labelled V5) that will replace QARun. I've been playing around with it, and while I'm not vouching for it over other tools, you should definately consider it for an evaluation.
I've used WinRunner and Silk before and strongly agree with previous comments by turbotester about skill of your testers. Silk would probably be harder to learn to use than WinRunner or TestPartner but its language is more powerful.
------------------
Ross Ryan
rossryan@ireland.com
|
akaStacy
Member
Posts: 35
Registered: Sep 2000
|
posted 01-09-2002 05:45 AM
I'd suggest you make up a basic set of test cases (very simple ones) that prove whatever you need to prove by evaluation - say if you have a custom controll, you'll need a test that uses that controll to see if the tool would recognize it. I'd try to keep it down to 4 - 6 test cases, but it will depend on your requrements. Then, I'd allocate a day per tool for installation and general playing around and another half a day per tool per test case. I guess that might sound too optimistic, but you have to have some time frames, at least management usually wants it  . Also, if you find out that it takes much longer than you thought, at least you'll be prepared for some training in your near future.Hope this helps. ------------------
|
JeffNyman
unregistered
|
posted 01-09-2002 06:03 AM
I would agree with Stacy - plan out your activities, particularly in the actual environment that the tool will need to work in and this is pariticularly the case if you use custom controls of any sort as you indicated that you did.Also look for things like the robustness of the scripting language of the tool and consider that you might want to be build test engines or perform data-driven testing at some point. (As you said, you are thinking of test harnesses.) So, how easy is that to do in each tool? Plan some time to just play around with the language and learn its nuances, particularly some of the not immediately apparent things. For example, SilkTest has the GenerateDecl() method that I have put to good use but this is not something that is immediately obvious in the tool. Think of different things, even outside the scope of your immediate concerns, that you might like the tool to do and see if you can do those things. That can all be extra selling points later on. Also, on a wider level, hopefully you have a test methodology that is generally in place because a testing tool should complement an existing methodology - not serve in place of one and certainly not serve to create one in the first place. So make sure that the tool can match your testing methodology to different degrees. I tend to give, at minimum, a one week evaluation and, generally at maximum, a two week evaluations to tools. (Having, said that, I have also worked with almost all of them repeatedly so you start to be able to shave off that time after awhile.) But the two weeks is to determine how robust the tool is, overall, and the different things it can do. It also pays to do side-by-side comparisons in some cases: how do the tools execute, how easy are they to program, how effective is error-handling for unattended execution, etc. It also depends if you want to evalute tools concurrently. Usually I do this because I can then show the company how one tool does things as opposed to how another does it. I also leave time to write up a justification and cost-benefit analysis for the tool. You have to factor that in. Above all remember: this is the tool and scripting language you are going to have to live with. So it pays to take the time to make sure you make the right choice. ------------------
|
testgeek
Advanced Guru
Posts: 829
Registered: Feb 2001
|
posted 01-09-2002 07:11 AM
The book "Quality Web Systems" contains a Tool Evaluation appendix, comparing these
major tools: Mercury, Rational, Empirix, Compuware, Segue. You can download the matrix at www.qualitywebsys.com It gives a general comparison overview, but it does not substitute an in-house evaluation of the tool (see Dustin's Automated Software Testing book for more info on that.)
|
jgottlieb
Moderator
Posts: 1442
Registered: Apr 2001
|
posted 01-09-2002 08:13 AM
Don't forget, you're the customer. You can evaluate as long as *you* feel reasonable. You'll have to get new licenses (they typically run out on a specified date), but that's a minor issue.------------------
"I can see my house from here!"
Jordan Gottlieb - Keeper of the QA Forums FAQ
Qualitech Solutions, Inc.
jgottlieb@qualitechsolutions.com
|
Jim O'Reilly
New Member
Posts: 4
Registered: Nov 2001
|
posted 01-09-2002 10:26 AM
Thanks to all who responded so far (and I'm hoping for more responses!). I had already come across several of the resources mentioned (particularly the tool evaluation matrix which testgeek mentioned). I'm also planning to pick up a copy of Dustin's book "Automated Software Testing".The time issue I was concerned with was not so much how long the vendors' evaluation licenses are good for (those can *always* be extended :-)), but rather what length of time is appropriate for evaluating each tool - installing it, learning to use it, and creating tests with it. I'm making up a schedule for the evaluations, and I have no idea if my guesstimates are grounded in reality. I'm trying to make sure I set realistic expectations for the amount of time such an evaluation would take - I don't want them to think this can all be done in a couple of weeks for all 4 tools. (The marching orders were "figure out a way to do GUI testing" - and I'm not sure that they had any idea of the scope of the task). In particular, turbotester's, akaStacey's, and JeffNyman's comments were useful in helping me to calibrate my approach. I had already begun listing the types of things I need to do with the tool, and will use that to help check out each tool. Their comments tell me that my approach is not off-base. I guess the thing to do is to continue to learn about the tools, refine and expand the list of things I need the tool to do, and break the tasks down into subtasks whose duration can be more easily estimated. One thing I haven't really seen a discussion of is licensing schemes, and it's impact on cost to roll out the tools. (I already have one copy of WinRunner - purchased on the advice of a contractor who's no longer working with us, due to budget cuts :-( - I'm in the semiconductor test equipment business - times are tough all over!). Since I need to install the package on multiple systems (for developers creating the tests and for running the tests on multiple machine configurations), WinRunner's node-locked licensing is a disadvantage. If WinRunner had a floating license scheme, I probably wouldn't even be considering other tools. But at ~ $5K/seat, if we needed 5-10 seats, it starts to add up, especially in comparison to Robot and SilkTest, which are about the same cost/seat on a floating license scheme. In general, node-locked licenses really bug me - I come from a background in unix-based IC EDA tools and unix-based tester SW, where the licenses are invariably NOT node-locked. Any feedback from the forum on this issue would be appreciated. Jim O'Reilly
Agilent Technologies
jim_oreilly@agilent.com ------------------
|
turbotester
Guru
Posts: 375
Registered: Dec 2001
|
posted 01-10-2002 03:48 PM
Jim-
I have beem working with MI prodcuts for the last five years and although I definitely usually do not get involved in the licensing issues (that's why we have salespeople) I could swear that MI has concurrent user license agreements for their test suite. I.E. - if you bought five licenses - no more than 5 people can be using the tool at the same time - but if you need to put the tool on 10 machines because 10 people have been trained to use it - that's fine. I may be wrong but I know that's what they do with TestDirector. I worked at a place that had 10 licenses but had WinRunner on probably 25 machines - 10 lab machines - and 12 "tester" machines and the leads' machine and the supervisors' machine and my machine. Never had a problem because we never really all used it at the same time------------------
Brian K. Qualters TurboTesting Concepts
|
QAGirl
Moderator
Posts: 2424
Registered: Aug 2001
|
posted 01-10-2002 05:28 PM
We asked this same question of our sales rep recently, and the basic response was that 'theoretically', WinRunner or QTP should be uninstalled on a given machine prior to being installed on another - but there is no technical mechanism to prevent having it on more machines than you have licenses. You're considered within the license agreement (to my understanding) if you are only using the number of licenses that you have at a given point in time.------------------
"They were painters and they were painting themselves a lovely world.."
|
Terry Horwath
Moderator
Posts: 211
Registered: Jul 1999
|
posted 01-10-2002 09:30 PM
You may find some of my evaluation suggestions useful in the following whitepaper I wrote a few years ago:
http://www.lakefolsom.com/whitepapers/TestToolEvalCriteria.pdf -Hope this helps, Terry Horwath ------------------
|
Jim O'Reilly
New Member
Posts: 4
Registered: Nov 2001
|
posted 01-11-2002 09:28 AM
Terry Horwath wrote:
"You may find some of my evaluation suggestions useful in the following whitepaper I wrote a few years ago:"Terry, thanks. I had already found a copy of your paper (probably as a result of searching this forum, but I couldn't swear to it). Regarding the licensing issue, thanks to turbotester and QAGirl for the info about MI (WinRunner) licenses. I'll check with our MI sales person and see what they say. In any case, as was stated, there's no technical barrier (just a legal/moral barrier :-) ) to loading WinRunner on multiple machines. The install process even gives you the option of loading it on a server! Regards
------------------
Jim O'Reilly
Agilent Technologies
jim_oreilly@agilent.com
|
