posted 06-18-2002 01:46 PM         

Thanks in advance,

Alex

posted 06-21-2002 07:02 PM         

There are a number of important factors to determine, such as what input data can be parameterized, how to vary user think-time, what numbers of each kind of user profile to use in a given scenario, etc.

Always be reluctant to load test in the production environment, since load testing to find system capacities usually crashes the system.

posted 06-24-2002 08:38 AM         

I agree with webhaskett's comments; you need to define what are desireable/acceptable performance criteria requirements before developing your testing strategy. For example; 90+% of our anticipated user base must be able to perform the 10 most commonly used functions within x seconds using a client machine with a y specification during our peak processing hours of M-F 9am-10pm EST .........

With regard to scalability, most applications will typically benefit from throwing more hardware at the system, but at the same time few systems are so scalable that doubling the hardware will provide twice the capacity. So what we are really being asked to determine is "what is the scalability ratio" that the proposed solution appears to have. i.e. if you double the hardware will you get a 10% improvement in capacity (a ratio of 10/100 = 1:10) or 80% (80/100 = 4:5), obviously the closer to 1:1 the better. So before developing your scalability testing strategy, determine what the scalability requirement/"desirement" is.

Hope this helps
Steve

posted 06-24-2002 11:01 AM         

Hope this helps.

posted 06-27-2002 12:48 PM         

One, proving that the site/application can scale to XXXX users. The other showing that the system has no bottlenecks that can't be solved without adding hardware.

Adding hardware should be the last concern, as most issues found during proper load testing uncover configuration and design problems.

Points to consider:

About 30% of issues are in the infrastructure, networking, router configuration, firmware, etc. Depending on your architecture this number could be larger or lower.

Your methodology needs to include stress tests for the various servers - independent of the application. This ensures that throughput, connect/disconnect rates, garbage collection, etc can be handled by your existing web/app/db servers. Again, these are sources of bottlenecks as well.

Then you get to the application itself. At that point you need to be pragmatic and prioritize the work. You can't test everything, but you need to define the load test models that you want. Document the model
preciself, including number of users, types
of operations, arrival rates, iterations, think times, etc.

There is merit in testing against the production system. As long as you:

[a] plan the event during off-peak hours
[b] take backups
[c] have support staff on hand to assist and recover
[d] approach testing gradually.

The purpose of testing in this manner is not to crash the system, but to gradually increase load and examine various components as they become busier. Often this will expose configuration issues.

The QA lab environment will not help you in
troubleshooting production issues.

If you need help doing this Mercury also offers a service called ActiveTune, which
does this for you.