|
Author
|
Topic: Manual Testing Vs. Automated Testing
|
Reji Itteera
Member
Posts: 90
Registered: Feb 2002
|
posted 10-16-2002 09:53 PM
    
We have been Testing an application manually and has 2000 testcases. Now the client wants to Automate the same using Rational tool. All testcases are for testing the functionality. How many testcases can be automated in a day by a Testing Engineer who knows the Testing Tool and the scripting language? Will Automated testing take more time than Manual Testing? If YES how many times more? Any matrice? What is the saving that automation brings about for the client in terms of revenue? Any associates who had similar experiences or knows this kind of estimation kindly reply to this thread. And I hope this thread helps many others who are here. Thanks to all of you well in Advance. Cheers ------------------
Reji Itteera
Testing Engineer
|
PritiPramodini
New Member
Posts: 1
Registered: Oct 2002
|
posted 10-17-2002 02:49 AM
Automated Testing is a good solution for Functional Testing(Regression Testing to be more specific) if the test cases are repeatable in nature. Experts say that Automated Testing is an enhancement of Manual Testing. Remember Automated Testing involves a complete cycle of Software Developement. It does not bring immediate schedule or effort reduction. Only if you would have used the right process and techniques, you can rely on Automated Testing. It involves a lot of effort to get familiar with tools, scripting language, develop the scripts and then maintain them in the future.
Now it is upon you to decide why should you use Automated testing, If you have enough time and your software is going through incremental builds and patches, and this has to continue over a long period, then may be you can give a thought over Automating your test cases. Last point, Manual testing will still be needed even if you go for Automated testing.------------------
|
Reji Itteera
Member
Posts: 90
Registered: Feb 2002
|
posted 10-17-2002 03:30 AM
Thanks for your reply. Manual testcases for the product is already written and the product is in a stable state. The requirement of automation is from the client.And regarding automation knowledge and process we are the best in the industry. I actually wanted a rough estimate when compared to Manual testing. Sure we are going to save some time when we do regression testing. I wanted to know the figures if anyone had similar experience.------------------
Reji Itteera
Testing Engineer
|
joholo
Member
Posts: 10
Registered: Aug 2002
|
posted 10-17-2002 05:17 AM
Hi,
I know this has been discussed before on this forum. In the book Automated Software Testing, Elfriede uses information from www.imbus.com, they have done some sort of survey of when test automation pays off. I did a search on 'imbus' and found some nice threads...Hope those threads will provide some answers. They helped me when I did a similiar survey. Greetings
joholo ------------------
|
Lee Weston
New Member
Posts: 1
Registered: Sep 2002
|
posted 10-18-2002 05:28 AM
Hi Reji,We are currently in the process of deciding whether to automate our regression tests so I have been doing some reading on the subject. When trying to estimate the time taken for automation of a single test most articles quote Cem Kaner who estimates that it takes between three and ten times as long to create, verify and minimally document the test. However, Bret Pettichord in his "Success with test automation" article says that as a rule of thumb test automation will take AT LEAST ten times the time it would take to execute the tests manually. From what I've seen of automation I would be inclined to agree with Bret if you want to record a test that will find bugs and be durable over time. You also need to remember that script maintenance is a major issue, and will take a lot of time. Regards,
Lee.
|
BillThomson
Member
Posts: 26
Registered: Oct 2002
|
posted 10-18-2002 09:05 AM
Reji,I have been involved with test case automation since 1985. Here is my $.02 
---------------------------------------
Some people may claim all test cases are less costly automated
VS manual because once created they are free. In reality, test
case creation is the least expensive item in the total lifetime
cost of automated or manual test cases. With this in mind,
here is a fairly simple "mental "equation that will help you
weigh the costs and benefits of automation for a given test case
type:
TTC = TCC + TMC + (TET * (TEC + TTSC))
Where
TTC = Total lifetime cost to create, maintain and
execute and troubleshoot a test case
TCC = Test case creation cost - Higher for well designed
automated test cases.
TMC = Test case maintenance cost
TET = Number of times a test case will ultimately be executed
TEC = Test case execution cost - Very low for automated test cases.
TTSC = Test case troubleshooting cost - Lower in most
cases for manual test cases.
Considering the equation above, test cases that get executed
frequently and for which creation, maintenance and
troubleshooting cost relatively little extra for
automation VS manual are prime candidates for automation.
Another conclusion that can be drawn from studying this
equation is that for automation provides the biggest
payback if the test cases are automated as early as
possible.
------------------
Testability = 1/complexity
|
RSBarber
Moderator
Posts: 852
Registered: Jul 2002
|
posted 10-18-2002 09:41 AM
If I were new to test automation and read that thread I think I'd find a new career choice! My experience is somewhat different.First, another statistic - Rational software says that to "break even" with automated vs. manual the test will need to be executed 5 times. Personally, I think the payback is at 3. Mind you, I don't like doing manual testing. I will also say this. It is very important to suppliment automated testing wtih manual testing. It is also important to recognize those test cases that just don't make sense to automate. Let me give you an example why I don't agree with these 5 and 10 times numbers. Let's say you have an applicaiton that requires users to log in. Let's say there are 5 different roles, thus 5 test cases to test if logging in correctly for those roles works, and at least a few more about handling invalid log-ons etc. In less than half a day, (maybe under an hour depending on the application), I can write a single automated script to test all of those things with an infinate number of users. How long would it have taken me to test that manually 5 times? And document my results as opposed to just printing the results? I have no idea, but I bet it's longer than 4 hours! What I do know is that if I had to execute those same 10 test cases on 5 seperate occasions manually I would want a new job. ------------------
Scott Barber
NOBLE(STAR
Sr. Performance Engineer
sbarber@noblestar.com
http://www.noblestar.com
http://www.perftestplus.com
|
BillThomson
Member
Posts: 26
Registered: Oct 2002
|
posted 10-20-2002 02:37 PM
Scott and Reji,I am also a big fan of automation but also
quick to recognize it's limitations. I don't
think a blanket "breakeven" number can be
applied to manual VS automated test cases
which is why I presented the simple
equation. IMHO the "payback" number will
range from 0.2 to >100 depending on the
application type, automated test tool reliability and test case failure rate.
Some of the IP telephony call processing
test cases we create are easier to code
up in a test script than perform manually.
Other complex OA&M test cases are >10
times more difficult to automate than to
run manually and profoundly more difficult
to troubleshoot in an automated form. So,
we have to evaluate each class of test cases
and use the dreaded equation to help us
decide which ones make sense to automate. Bill Thomson
Test Architect (Demolition Man) ------------------
Testability = 1/complexity
|
