|
Author
|
Topic: Response Times versus other metrics
|
banff5366
Member
Posts: 27
Registered: Mar 2001
|
posted 10-17-2002 12:12 PM
    
I have a question to pose to all of you performance testers. We do quite a bit of business transaction profiling, and user profiling. BUt one of the things we are struggling with is metrics for performance testing. Every plan I put together I stress the importantance of certain metrics. Response Times, CPU, memory allocation, session information, backend process time, etc.
What I want to know is. If I run my tests (load tests that actual model production usage) and my response times are fine, is there a reason to collect all those other metrics? Lets assume my tests exercise the system to x+1 in terms of usage.
We also run reliability tests that test for stability and uptime. Again if response times are ok, server doesn't crash, and there are no hangups, is there a reason to collect those other metrics? Or should those metrics be used in troubleshooting mode when you identify a problem?------------------
|
RSBarber
Moderator
Posts: 852
Registered: Jul 2002
|
posted 10-17-2002 12:20 PM
My opinion (I might just incite a riot by saying this) No. User Experience is what want to be good, if it's good why waste your time?Now I am going to hedge my bet a little bit. I would still monitor all those metrics at least once or twice under peak or above peak loads. Just to make sure. If you are doing capacity planning, you will want those metrics at many different user loads so you can do that "black magic" math that extrapolates when and where it will break. I guess it all depends on the purpose of your testing. I believe that if you are testing from the perspective of the users, when more users than you expect could possibly ever use the system, are using the system and are happy, then you're done. ------------------
Scott Barber
NOBLE(STAR
Sr. Performance Engineer
sbarber@noblestar.com
http://www.noblestar.com
http://www.perftestplus.com
|
bnayak
Member
Posts: 39
Registered: Oct 2002
|
posted 10-17-2002 10:32 PM
I would rather monitor atleast the following statistics for future analysis. CPU Usage
Memory Usage
Network Usage If I keep this statistics for x+1 users load and may be if something wrong happen on x+2 user tomorrow I would have a chance to compare. However you can automate the statistics collections so that its not a big pain anyway. ------------------
Biswajit
Senior Performance Engg
Oracle Corp
http://www.oracle.corp
http://bnayak.tripod.com/perf/
|
coling
Guru
Posts: 256
Registered: Aug 2000
|
posted 10-18-2002 02:34 AM
Hi,Yeah I would mainly agree with Scott. If however you aren't just looking at Load Testing from an End User perspective then you should still monitor your other metrics. For example if the application you are testing is not the only application hosted on the server your using, then monitoring the other metrics may be a pre-requisite of the test! If you are already running 'reliability' tests which haven't shown any huge memory leaks or instability, then there probably isn't any great added value to doing this unless you always want to have the information to hand for any unexpected troubleshooting? ------------------
Hope this helps. Regards,
Colin.
|
