|
Author
|
Topic: Auotmators vs Manual Tester Ratios
|
UniqueDave
New Member
Posts: 2
Registered: Dec 2001
|
posted 12-27-2001 08:30 AM
    
I have looked through the various articles that have been suggested here regarding ROI analysis for Test Automation. This is all good information however, I did not notice anything specific about the ratio of persons who create automation, vs. persons who create/perform manual tests. Would anyone have any information on these ratios, (i.e., on a test team that has X number of QA engineers, how many are devoted to test automation and how many do only manual testing), either through personal experience or published resources.
Thanks...Dave
|
QAGirl
Moderator
Posts: 2424
Registered: Aug 2001
|
posted 12-27-2001 10:18 AM
Hi, Dave... I'd have to say that in my own personal experiences... "it depends". (You'll find we say that a lot here!)Keep in mind that these ratios are going to depend on how robust your automation is, how much your application allows for automation, and at what process of automation you're in. For example, when you first begin automation, and are striving to create an automated suite, you're going to probably need many more bodies devoted to automation than you will solely for maintenance and upgrading that same suite (if you want to get it done in a realistic timeframe). By the same token, if you are only able to automate 50-75% of your application effectively, you can't cut back too much on manual testers, as you'll still have to verify and regress the remaining 25% of the functionality. Does that help at all? If you could give a bit more detail about the context of your question, it might help. ------------------
~* Happy Holidays!! *~
|
JeffNyman
unregistered
|
posted 12-27-2001 10:23 AM
quote:
Originally posted by QAGirl:
Keep in mind that these ratios are going to depend on how robust your automation is, how much your application allows for automation, and at what process of automation you're in.
This is exactly right in my experience as well. For example, in places where a robust test engine was setup that had a massive amount of error-handling and was modularized for easy inclusion of new material, we could have very few automation people in relation to our manual testers. There is also the notion of time. The automated testers will, depending upon their skill level, either spend more time coding good test scripts or more time maintaining bad ones. This depends on the nature of how "throwaway" your scripts are and also if the automation is being used incorrectly. (It is best for regression testing and combination/permutation testing. At least in a functional test mode.) Generally having automation means having a good test methodology. (It should anyway.) And, in that case, I try to have the ratio of testers I need for each aspect (manual and automated) fall out as a metric of the methodology and this depends, as QAGirl noted, on the type of application and how much of it is amenable to automation. This also has to be considered in direct proportion to how robust your automated test scripts are at any given time. ------------------
|
awhiteleather
Advanced
Posts: 174
Registered: Oct 2000
|
posted 12-27-2001 10:27 AM
For our shop we have the whole team involved in our automation effort. Two of us are dedicated to building a test harness and the remaining 10 folks can/will build their own automated procedures using the interface we provide.This has worked out well for us so far, and I believe it will continue to work fine. As QAGirl mentioned there are a lot of variables that change the number of people working on the automation effort so it really does depend on your unique situation. ------------------
Atlee
|
UniqueDave
New Member
Posts: 2
Registered: Dec 2001
|
posted 12-28-2001 08:03 AM
Thanks to everyone who has responded. I do understand that the ratio is variable based on many factors so perhaps I should put the question into context as QAGirl suggested.
I have been directed to do an ROI analysis, (sigh), on our automation effort here and one of the figures that I *MUST* supply is the ratio of automation engineers vs. manual testers that other shops use. Yes, I know it doesn't mean a whole lot out of context but....
In any case, I have only one concrete example from a past position, (as well as some 'slush 'examples from other past positions).
So, if anyone can help out with figures from their shop, (and maybe include a brief blurb on the app they are testing, functionality coverage, etc.), It would most appreciated.
Thanks again and a Happy and safe New Years to all...Dave------------------
|
turbotester
Guru
Posts: 375
Registered: Dec 2001
|
posted 12-31-2001 07:22 PM
Dave-
My personal experience, having done test automation for the last 5 years and being CPS and CPI certified in Mercury Interactive's TestSuite is a bit different than the others responding to your question. First of all, I must say that I am totally in favor of breaking up your team into people who are what I call "functional analysts" and people who are "automators". The reason for this is pretty simple. To do quality automation it is much easier to get a smaller number of people on the same page in terms of automation approach or methodology then trying to get your whole team, where there is usually varying degrees of skill and understanding of the tool, to script in the same manner. Find those people who appear to have the inclination for scripting and make them your automators. The second reason for this apprach is that automators need to devote all their time to automating - the more responsibilities you put on them to do other work - i.e. analyze the AUT and write the manual documentation - the lower the quality seems to get for all the work throughout the process - think about all the documentation that must be read, analyzed and then repackaged into test cases and then adding the responsibility of creating a quality automated test case on top of this - what you usually wind up with is diminshed quality of documentation on top of less than top-notch scripting as the person uses the "I know what I'm doing" excuse throughout the whole process due to the pressure to get it done. Breaking the responsibilities up has in my experience, always created better quality work throughout. This is truly from experience!!! Also - you always want to have seperation of knowledge to enforce natural checks and balances - if you have the same person automating the test cases as the one who wrote - how often is that person going to same "this is wrong" or "I need better documentation"? I'll answer that this way - how many times have your developers said "Yeah - that's my fault I screwed that up?" And, ideally - if the time is there - you want a check for the automators where they go back to the manual testers and show them the automated tests that have been written for them to make sure the automated scripts prove what the functional analysts intended to prove. Checks and balances!!!Finally - having worked at three different shops in the last 3 years (in between teaching gigs with Mercury) I have found that a ratio of 3 to 4 functional analysts to 1 automator woks just fine as long as the one automator is skilled at what he/she does. I recently worked in Seattle for a year and 3 months with a company that started out with two functional analysts and one automator - me. that was great. then the ratio went to 12 to 1 as they were ramping up their testing efforts and I spent alot time trying to show them how to actually write tests to begin with. Then it went down to 6 to 1 with layoffs. I finally got permission to train another automator and that was grat again as progress was being made much faster in terms of automation. Another key here though is the ratio also depends on having well written test case documentation that is clearly understandable to the automator (all tests should be written so Joe Blow off the street could walk in and do it but that so rarely happens!) Junk manual tests beget junk automated tests so worry about that end first! Make your projection dependent on having well qualified and well trained manual testers!!!!! Good luck and let me know if you need help as I am free now. ------------------
Brian K. Qualters TurboTesting Concepts
| |
