posted 06-05-2001 08:45 AM
Hi all!I'm evaluating tools for a new company. I decided to go with a standard school grading system so the stakeholders can easily understand my report. I've written the portions that matter most to my projects, and the projects we know about in the future. Some of them are blue sky type criteria that I don't think have been implemented in a tool currently available. I thought this would be a good point of discussion in regards to what you'd add for your company's projects. Some point of knowledge for evaluating testing tools would be extremely helpful to new automation groups. We should begin with a description of the environment we will utilize the tool to test to give a little background on our thoughts.
Anyway, my projects are all web delivered apps with 95% of the Java functionality running on the backend. It's supposed to be displayed utilizing IE 4.01+, NS 4.7+, and Opera 5.0. We're going for W3C HTML 4.01 Transitional compliancy.
Automation Tool Criteria
Ease of Use
A. Tool has an intuitive interface.
B. Tool has an average interface, keep the manual handy.
C. Read manual before installation, after that you’re okay.
D. Tool was designed by a very sadistic people
F. . . . and so was the manual.
Support of Browsers
A. Tool fully supports IE, Netscape, and Opera.
B. Tool fully supports IE and Netscape.
C. Tool supports only old releases of IE and Netscape.
D. Tool supports only IE or Netscape.
F. Tool doesn’t seem aware of what a browser is.
HTML Support
A. Tool has settings to allow tag validation on a custom level, from fully W3C HTML 2.0 to HTML 4.01 Transitional compliancy to anything goes.
B. Tool has a fixed method for dealing with tag validation and both tells you what it is and does it accurately.
C. Tool does no tag validation, but works with page elements correctly.
D. Tool does tag validation, but doesn’t tell you what document type compliancy it’s going after.
F. Tool does tag validation, doesn’t tell you what compliancy it’s looking for (or has a vendor defined non-standard compliancy) and is inconsistent in its reports.
Java Support
A. Tool not only is not confused by Java, it points out coding mistakes.
B. Tool is not confused by Java.
C. Tool is sometimes confused by Java.
D. Tool is not aware of what Java is.
F. Tool reports any Java/Javascript as an error.
Mock Test One: Navigation
(This test checks the tools ability to handle different navigation methods. Straight hypertext linking, drop down nav, dynamic link nav (those that change depending on the hour of the day, for example), java toolbar nav, back & forward & refresh buttons on the browser, images as links, multiple page navs, across multiple domains and page based redirection)
A. Tool is never confused. The tool recognizes each type of navigation and categorizes it appropriately.
B. Tool sometimes misinterprets a navigational item but always recovers and performs the expected action.
C. Tool lacks support for one of the not-commonly used navigation methods.
D. Tool lacks support for any navigation method that doesn’t involved an A HREF tag
F. Tool was designed by a monk in the mountains of Tibet who had no clue what the Internet is.
Mock Test Two: Forms
(This test covers the tools ability to deal with different form elements and implementations of those elements. This includes drop down menus, list boxes, radio buttons, checkboxes, textboxes, password boxes, text areas, file browse text boxes, custom buttons, submit buttons and reset buttons.)
A. Tool deals with all forms of forms without a problem. Tool correctly categorizes each.
B. Tool sometimes deals with form items incorrectly, such as labeling a file browse textbox as a normal textbox, but the mislabeling doesn’t hurt the tools functionality.
C. Tool can’t deal with one of the form items, but there is a workaround.
D. Tool can’t deal with most form items, workarounds are tedious and time consuming.
F. Tool is unaware forms exist. Tool is actively hateful towards any user that attempts to deal with them.
Conditional Support
A. Tool can perform if…then…else’s, do while’s, for each’s, with no problem.
B. Tool has a more limited set of conditional support, but what it does have works beautifully.
C. Tool has one conditional that usually works.
D. Tool has no straight forward conditional support, but you can see a way to hack around it.
F. Tool doesn’t appear to have a language at all you can access.
Dynamic Data Sets
A. Tool supports multiple data sets and can basically turn anything hardcoded into a variable to be looked up in an easy manner.
B. Tool supports a single data set but can turn anything hardcoded into a variable.
C. Tool supports data sets but has limited support for variables.
D. Tool supports data sets somewhat and variable insertion is somewhat difficult.
F. Everything is hardcoded. Tool will make anyone from a development background leave the country and live in an Eskimo village.
Error Recovery
A. Tool can deal with any network, page, or script anomaly with the grace of a dancer.
B. Tool can deal with most errors without aborting.
C. Error recovery must be dealt with heavily by the automation person, but it can be done.
D. The tool aborts the script almost any chance it gets.
F. The tool has the grace of a dancer that has been drinking tequila heavily for several hours.
posted 06-05-2001 08:45 AM
