posted 10-21-2002 11:36 AM           

And for you performance gurus out there, does what I知 doing sound like it makes any sense at all? Also: to what extent do people use server logs to gather your performance model information? We have IIS server logs but I知 not sure how helpful that can be since you壇 have to parse out the file. Anyone doing this?

posted 10-21-2002 11:55 AM         

Take a look at the methodology and articles at http://www.perftestplus.com The first 4 articles cover the topics you are asking about fairly well.

As for time of day - I'd recommend a slightly different approach. I would develop a 3 tier distribution model - best, expected and worst case scenarios.

Server logs are great if you can parse them out effectively. They can also be a huge pain and not be worth the effort if you have no way to parse them.

posted 10-21-2002 01:34 PM           

quote:

---

Originally posted by RSBarber:
Server logs are great if you can parse them out effectively. They can also be a huge pain and not be worth the effort if you have no way to parse them.

---

I will check out the articles you mention.

As far as the server logs, we have no way that I know of to parse them right now. There is talk about either (a) buying something that will do it for us or (b) building something in-house. My boss also wants to fit them into the load tool but I do not see any load tools out there (we are currently using LoadRunner) that allows importing and parsing of logs.

Regarding your three-tier distribution of "best, expected and worst case scenarios" I take it you just mean this over the duration of whatever the load test is? In other words, if I have a load test that I am planning to run for one hour I should basically plan a normal (expected) load for that one hour and simulate that then ramp that up to simulate a worst case.

posted 10-21-2002 02:10 PM         

Actually, in that scenario, I meant executing 3 seperate tests. Same total load, but with the worst case distrubution favoring performance intentensive activities, and best case favoring performance non-intensive activities.

You'll want to focus your tests to maximize the max expected user load, yet allowing for some ammount more than that (25%) before hitting the "knee" in performance (were response times get really bad really quickly as opposed to it just getting gradually slower).

posted 10-21-2002 04:52 PM         

See list of Features, http://webtool.rte.microsoft.com/

Regards

Ian

posted 10-22-2002 06:53 AM         

Interesting you mention that book. Reading that book (and several others) is what led me to start writing the articles I previously mentioned. There are a lot of great books out there with terriffic information, but I haven't found one that talks about the actual execution of performance testing/tuning collectively between the release of build 1 and deployment.

posted 10-23-2002 03:30 AM           

quote:

---

Originally posted by RSBarber:
There are a lot of great books out there with terriffic information, but I haven't found one that talks about the actual execution of performance testing/tuning collectively between the release of build 1 and deployment.

---

This has been my biggest problem. Two of the best books I have found come from Segue: their main eConfidence guide and another small booklet called "Load Testing for eConfidence." Granted these are somewhat geared to their own tools but most of it actually seems to be fairly general. (The second booklet has a full case study that has been generalized.) Most of the books I see are high on theory but low on practical details. I understand that these books have to try to account for many and varied systems but in the time I've been doing this I've found many things that are general enough that could apply to most systems, regardless of what tool you're using.

What I've basically done is take a bits and pieces from two books: "Capacity Planning for Web Performance" and "Scaling for E-Business." I've implemented those bits and pieces as was needed and pretty much discarded the rest -- at least for right now. One thing I often see the books not mention is that you have to consider user types relative to data concerns as well. Most books talk about just delineating user types and where they go. But my major issue here is that different user types (based on security restrictions, for example) will perform differently due to the amount of data they can be passed relative to their security permissions.

Also there are practical details of issues like where you have a system that has a performance load when plans for categories are put through a "actualizing process." But once you put those plans through the process, it is a one shot deal: yet the load tools are going to all operate on that same set of plans (even if you parameterize). So you have to work out how to create realistic loads with these tools based on data concerns. Three of the five projects I am working on now all have this concern.

posted 10-23-2002 05:05 AM         

The Web Testing Handbook by Steve Splaine has a really good few chapters about performance testing considerations. Things like user types, data, browsers, connections, etc. It won't tell you HOW to handle these things, but it is the best I have found so far in putting all the considerations into one place.

posted 10-23-2002 06:56 AM           

quote:

---

Originally posted by RSBarber:
I agree wholeheartedly. I'd be interested in the Segue bookletts - how are they available?

---

We got them through Segue. Both are available from the Members-Only area of their site as PDF documents. You also get sent them as actual books if you purchase one of their products. Personally, I think these books should be more widely available than they are because I have found them to be actually practical in terms of guiding my thinking.