posted 11-18-2002 11:10 AM         

1. CPU
2. Memory
3. HD
4. Bandwidth

I have QA Load Software to perform Load test,
How many Vusers (25,50 ...250)I need to run?
What

Thanks in advance.

posted 11-18-2002 11:53 AM         

it depends... (ie on factors like, the predictability of 'workload resource usage characteristics', the workload itself and the 'physical' environment you have to test on, or rather 'how close is this to the 'real world')

My suggestion to you would be to read this paper (from Microsoft) entitled "Capacity Model for Internet Transactions", it does cover the 3 tier (web/app/db) that you reference.
http://www.microsoft.com/technet/treeview/default.asp?url=/technet/prodtechnol/comm/comm2002/plan/capmodit.asp

The paper does focus on MS technology but you should get an idea of the overall process needed.

Also, I am interested, is your question an "in theory" question or are you planning to conduct 'capacity planning' tests for which the target is 100,000 + users/customers?

Regards

Ian

posted 11-18-2002 12:29 PM         

The first thing you have to know, as Ian mentioned, is your usage model. Is that 100000 users an hour? a day? a year?

Take a look at User Experience, not Metrics articles 2,3,4 on http://www.perftestplus.com They discuss user modeling for the purposes of Performance testing.

The total number of users is not going to be of much help to you in capacity planning unless you know their patterns.

Of course, you could try applying SPE if you are able to determine things like the expected number of clock cycles on the web server per average request. But even doing that, you will need to be able to determine how many people will be doing what concurrently.

posted 11-18-2002 02:50 PM         

A thanks for responds. They are really help full for me. I am still puzzling with how I can determine my concurrent user for 100000 customers base business. If I don’t have and any information previously and this test conducting with brand new software and my client have 100000 customers? Only thing is I have to use VU.

a) How many simultaneous VU I need for 100,000 customers base business.
b) Ex: If It is correct if I users 50 simultaneous VU for 100000 customers base business.
c) I can run 50 to 250 simultaneous VU periodically and collecting data from each server (Web Server, Application server and DB (Memory, Disk and CPU))
d) How Can I calculate best hardware for those servers?

Can I use same data to calculate hardware need for 1,000,000-costumer bases business?

(My Software has lots of COM+ object and running on Application server. My Web server is database driven and I have 2 DB, one for web server and other one for actual data)

I have client, he have 100000 customers in his business. He wants to use my software on his business. Before he take my software I need to find out what the hardware requiremen for my software for his business.

Thanks
DA

posted 11-18-2002 05:00 PM         

Even when you understand the process, ie read Scott's website, no one can answer, this type of question for you:-

a) How many simultaneous VU I need for 100,000 customers base business.

without knowing the nature of the business, for example (for all we know) your software could be providing a video lending library service to 'one armed chef's' who need special bread making instructions 'as and when new bread recipes come out', or you could be running a 'horse betting syndicate', how would we know the answer from what you have stated ?

Regards

Ian

posted 11-18-2002 05:12 PM         

quote:

---

Originally posted by Ian:
'one armed chef's' who need special bread making instructions 'as and when new bread recipes come out'

---

Hmm, picturing 100,000 of these chefs. LOL
How do we model 1-armed usage? Multiply the user wait times by two?

:-)

posted 11-19-2002 03:22 AM         

You can predict many different performance related metrics using this. Instead of actually testing on different hardware (which may be costly and time consuming) you can predict the performance results on different configurations with this tool.

Once you have collected performance data from your existing configuration, you can do what if analysis on different factors like,
* Changing a Physical Resource for eg. Adding or Upgrading CPU, Disk, Memory, etc. and
* Changing Resource Usage (Work Load) Increase/decrease the # of users, Increase/decrease the transaction volume etc.
This will help you identifying required hardware configuration for different workloads (users/transactions)

Also check the article at the following link which mentions how this tool helped in identifying correct size of processor while reducing overall cost. This is just one example, I can provide you more.
http://www.idg.co.nz/webhome.nsf/UNID/24C7C760BC3967FDCC256C610072598B?Open Document&Highlight=2,BMC

Please feel free to contact me for more info.

------------------
prasad_patwa@yahoo.com

posted 11-19-2002 07:30 AM         

You are absolutely correct. Reading my stuff will not answer his question. All it will do is help him think through the process of what those "one armed chef's" are doing when. And that is just the first step.

A capacity planning exercise/tool is going to be quite useful in this case.

Maybe it is truly unknown how these users will access the system on a daily/hourly basis. But with good capacity planning, at least you could say...

"if this many users access the system on an hourly basis, this configuration/hardware gives acceptable performance. To double that number, this configuration/hardware is required." etc.

posted 11-19-2002 09:43 AM         

quote:

---

Originally posted by DA:
I have QA Load Software to perform Load test,
How many Vusers (25,50 ...250)I need to run?
What

Thanks in advance.

---

Back to your original question. I have a little formula I use to determine the VUs needed for testing an application. I used it to successfully load test an application that was to have 100,000 users over a three hour period, only using a 500 VU license for QALoad. You may agree with this formula or you may think I am crazy. But here it goes....

OK you have 100000 users. What is the average number of pages that each of these users visit during a session? Next, what is the average length of time of these users' sessions? Divide the first number by the second and the result is the pageviews/second rate for a single user. Multiply this number by the maximum number of active sessions that the application will have at any given time. The result will equal the number of VU's you will need to load test this application.

So let's put some numbers with this. An average user visits 9 pages and stays on the site for 15 minutes (or 900 seconds). You have 100000 users and the max that will be on the site at any given time is 20% (or 20000 users).

9/900 equals .01 pageviews/second for 1 user.
.01 * 20000 = 200 VUs needed to perform your load testing.

Now...this formula assumes that you will build your scripts with no THINK/SLEEP time in them. This is 200 VUs going as fast as possible. NOTE: you should use a top end machine or machines to perform the load testing as you want to stress the app and not the load generating machines. At the end of the test, the key metrics you want to pull from the results are the pages/second and the response times for each of those pages. Don't focus on Virtual User level.

I am open to any suggestions and/or disagreements with this formula. Anything to make it (and me) better is always welcome.

posted 11-19-2002 10:11 AM         

Your formula certainly has merit as we look at the amount of pages served, activity started etc.
The only thing it cannot do is generate the overhead associated with having the actual 100,000 users connected. Things like state management, cookie management are completely driven by the ACTUAL amount of connected users.

In our case we found that a 300 user stress test would not bring out the problems related to state management and cookie management but our 1000 user realistic test would.

Looking at load on our servers, the 300 stress test was keeping the machines busier.

posted 11-19-2002 10:54 AM         

Your right regarding reducing 'think time' to increase RPS (and thereby simulate a larger user base), this is a common misconception.

The 'essential' Web Server Load Drivers are:-

1) Connection rate
2) Bandwidth
3) RPS
4) Concurrent connections

These combinations will have a 'resource usage' profile. The 'Request' in RPS, is the most 'problomatic' for any general discussion as the Request could be for a 'static' page or into an 'app' then on to the 'DB', this is why the answer is always 'it depends'.

By reducing 'think time' one can increase Bandwidth and RPS to simulations that approximate a larger 'Actual' user base (using formula like the example given). However, as indicated by Roland, one does not simulate the actual Connection Rate and Concurrent Connections (both of which use valuable resources, ie non-pageable memory for the extra TCP connections).

As has already been stated 'it depends', is this an issue for predicting the capacity requirements for a given SUT ? If the SUT has predictable memory usages (ie static files, no DB, all 'Requests' are simular or fit a known pattern) then the resource requirements for the 'extra' connections can be 'predicted' and the 'reduce think time' approach could be valid.

Regards

Ian

------------------

[This message has been edited by Ian (edited 11-19-2002).]

posted 11-19-2002 11:04 AM         

Yea...it's tough to get that info when you have a small VU license for the tool you are using. Generally I try fighting for a license with greater number of VU's but end up doing what I can with a 500 VU license.

You say that 1000 user test uncovered problems w/ state mgmt and cookie mgmt. What metrics were you reviewing to determine these problems? This may help me out with my current site to be tested. Normally I look at resource utilization for 1 user, then I look at it for 10 users, 100 users, etc... This helps me give a best guesstimate as to when a particular server will be in trouble. I also like to run a pre-load load test to crank up the number of active sessions, and then run a load test against that environment. I then compare the results from this test to a similar test executed against a freshly cycled environment. Of course, I had to raise the session timeout value from a couple of minutes to a couple of hours.

Sorry for the rambling.....

posted 11-19-2002 12:55 PM         

quote:

---

Originally posted by Test24x7:
You say that 1000 user test uncovered problems w/ state mgmt and cookie mgmt. What metrics were you reviewing to determine these problems?

---

No specific metrics, but we were starting to get errors in our script execution and we traced it back to our state management process.