posted 04-01-2002 09:13 PM         

You have got it wrong here.

The collective term is environmental testing. Performance, Load and Stress are different types of environmental testing.

performance testing: Testing conducted to evaluate the compliance of a system or component with specified performance requirements. [IEEE]

performance requirement.

(IEEE) A requirement that imposes conditions on a functional requirement; e.g., a requirement that specifies the speed. accuracy, or memory usage with which a given function must be performed.

load testing - Involves stress testing applications under real-world conditions to predict system behavior and performance and to identify and isolate problems. Load testing applications can emulate the workload of hundreds or even thousands of users, so that you can predict how an application will work under different user loads and determine the maximum number of concurrent users accessing the site at the same time.

Also:

stress testing - A structural testing technique that measures whether the application environment is properly configured to handle expected, or potentially unexpected, high transaction volumes. Ideally, stress testing emulates the maximum capacity the application can support before causing a system outage or disruption.

I get frustrated that this is not common knowledge for all testers.

posted 04-02-2002 04:55 AM           

quote:

---

Originally posted by rtehve:
The collective term is environmental testing. Performance, Load and Stress are different types of environmental testing.

---

In some circles it is. In others it is not. However, that is actually a dated reference at this point, except among the IEEE as a general rule. That is why I said it depends on who you talk to or what source you reference. However "environmental testing" takes on a lot of connotations, certainly not just related to software. Environmental testing is often used to apply to the environment something will have to work in. For examplel, I have done environmental testing in relation to hardware components such as for heat, cold, etc. So you are using a limited definition of that term and one that, at least in my experience, is not being used in that same way anymore.

Again, though, it depends on who you are quoting. You quote the IEEE but the IEEE is not the be-all, end-all distinction. In fact, I serve on their standards settings boards in a few categories for the IEEE and I know performance is one of those areas that is often debated as it has been said by some that IEEE has not kept up with the times. However, even with that, I agree with IEEE in many cases. For example:

quote:

---

performance requirement.

(IEEE) A requirement that imposes conditions on a functional requirement; e.g., a requirement that specifies the speed. accuracy, or memory usage with which a given function must be performed.

---

I agree. Memory usage (or lack thereof) can be one component of stress in terms of a stressing of a system. And that certainly speaks to performance.

And basically your definition of load testing is pretty much what I said, when you consider that "real-world conditions" that you mention can be representative and unrepresentative (just as stress can be sustained or maximal). Load testing is not, however, as you say a way to "predict how an application will work under different user loads and determine the maximum number of concurrent users accessing the site at the same time". That is more the job of workload and customer characterization. Those form workload models that you apply to resource models. It is those resource models that derive performance predictive models. Load testing is just one way of validating the model. It is not predicting anything. What it is doing is confirming the predictions of a model. But, again, I realize not everyone uses the terms this way as well.

Also, keep in mind that most current studies on performance and related techniques make a clear distinction between stress testing and volume testing. See the work by Mark Crovella, Azer Bestavros, and Paul Barford.

I highly recommend "Generating Representative Web Workloads for Network and Server Performance Evaluation" as just one example.

quote:

---

I get frustrated that this is not common knowledge for all testers.

---

I hear you. But I think you might be limiting your defintions, at least based on current research. Check out papers like:

• "Workload Characterization: Issues and Methodologies" by Maria Calzarossa, Luisa Massari, and Daniele Tessera
  
• "Web Server Workload Characterization: The Search for Invariants" by Martin F. Arlitt and Carey L. Williamson
  
• "The Operational Analysis of Queuing Network Models" by Peter J. Denning and Jeffrey P. Buzen
  
• "Self-Similarity in World Wide Web Traffic: Evidence and Possible Causes" by Mark E. Crovella
  
• "A Methodology for Workload Characterization of E-Commerce Sites" by Daniel A. Menasce and Virgilio A.F. Almedia
  
• "A General Methodology for Characterizing Access Patterns and Analyzing Web Server Performance" by Arun K. Iyengar, Edward A. MacNair, Mark S. Squillante, and Li Zhang

There are many others and they show a lot of distinctions between performance, load, stress, volume, and transaction testing. None of those papers, nor any of the papers they reference, refer to things as environmental testing (although that was true about ten years ago).

Also, as I said, stress testing can be used in the sense where it is not generated workloads but you are testing a resource violation. For example, when a given resource completely runs out of space/memory/etc then even having one user hit it can be indicative of stress. Then having multiple users hit it is another measure of stress - sort of like boundary conditions that you would do in functional/behavioral testing.

posted 04-02-2002 09:54 AM         

quote:

---

Originally posted by rtehve:
JeffNyman
You have got it wrong here.
................. [IEEE]

---

Once I witnessed a committee developing a standard (including terms definition).
Basically they finished with a monstrous definitions combining phrases suggested by different committee members. So using an IEEE definition is not a proof at all.

posted 04-02-2002 04:47 PM         

Irrespective of how many whitepapers you want to quote still does not change the fact that Performance testing, Load Testing and Stress testing are three different types of tesing with three different objectives. You defintely have the wrong slant on this.
You refer to stress testing and load testing as types of performnace testing...completely wrong ...sorry.

You have fallen in to the common trap that alot of people do with These types of getting these terms confused. The most common one is people get load testing and performance testing confused. Some think they are the same thing.

Also when you refer to resource violations of course this can include such items as disk usage, bandwidth, memory or even CPU time (one which you missed BTW). These are all factors taken into account when doing any of these three types of testing, in particular stress testing.

posted 04-02-2002 05:01 PM         

If you don't like IEEE definitions (or any definitions) could you volunteer how you conduct the following types of testing;

Performance Testing
Load Testing
Stress testing.

After you have described how you conduct these then we can see how it stacks up against the IEEE standards defnitions and see if it is a match.

Regarding Definitons:

"Verification is the act of reviewing, inspecting, testing, checking, auditing, or otherwise establishing and documenting whether or not items, processes, services or documents conform to specified requirements (ANSI/ASQC A3-1978).

Validation is, according to its ANSI/IEEE definition, `the evaluation of software at the end of the software development process to ensure compliance with the user requirements'. Validation is, therefore, `end-to-end' verification."

Yes they are a bit wishy/washy definitons.

Prefer the Military FCA/PCA approach used in the 2067a Software Engineering Standard ( I think this has been superseeded ). IEEE standards are by far the best commercial standards I have used BTW.

posted 04-02-2002 05:24 PM           

quote:

---

Originally posted by rtehve:
Irrespective of how many whitepapers you want to quote still does not change the fact that Performance testing, Load Testing and Stress testing are three different types of tesing with three different objectives.

---

Well, first, I never said load and stress testing were not different. In fact, they are so we are in agreement there. However, I did say they are part of performance testing, at least by some people's definitions - one that I happen to agree with. You do not agree with that and that is fine. There really is no consensus on this. My point with the white papers, however, was that many other researchers on these same topics share my same viewpoint. While that does not make us "correct", it makes it a little harder for you to categorically state I have the "wrong slant". What you mean is that I have the "wrong slant" relative to your opinion with the sources you prefer.

So my point in showing the white papers was to show you that perhaps you might not want to be so categorical in assuming you personally are correct in your assertions to the point that others of us are "wrong" are on that "wrong slant" you mention.

quote:

---

You refer to stress testing and load testing as types of performnace testing...completely wrong ...sorry.

---

Again, this is more your opinion based on the sources you agree with (and, incidentally, posted in a confrontational manner that does nothing to make your point). As I said, many sources of performance testing (as a strict term) define it as "a form of testing that provides a relative measure of the capacity, stability, and scalability of a given system under expected user/visitor conditions prior to deployment which aims to assure that the software performs in accordance with operational specifications as stated in Service Level Agreements or relative to an operationally agreed upon industry standards, or in relation to an established operational profile."

Testing for scalability and stability, as mentioned in that definition, is often done via the use of stress and load testing. The point is that for many researchers (and I am just one of them) performance testing has taken on wide scope such that it is a test approach that has various applicable test types. (This, admittedly, was not the case ten years ago but, as you know, thought patterns on these things sometimes change.) For example, transaction testing is one type of performance measure and, as such, it is a type of testing for performance - or, in other words, it is a form of performance testing. At least by some definitions. In fact, the current talk among some of the IEEE standards boards is that the term "transaction testing" has replaced how the term "performance testing" used to be applied, thus giving "performance testing" a wider scope.

quote:

---

You have fallen in to the common trap that alot of people do with These types of getting these terms confused.

---

Again, this is by your usage of the term which you seem to think is categorically correct. I do not think my terms are categorically correct either but I do show references that suggest that if I am totally wrong, I am in good company as many other researchers must be totally wrong as well - by your definitions at least.

To be honest, in my experience what people often confuse are the differences between performance analysis and performance simulation. (Performance testing encapsulates both of those, again by the sources I agree with and utilize because testing is a broad concept that does not just have to apply to execution.) Also, people confuse the differences between capacity planning and performance testing - two very different things, both of which are often subsumed under the moniker of performance engineering, which the IEEE has been parading about a lot in the magazines Computer and Software. Also people often fail to take into account the distinctions of transaction testing and volume testing. (Although, I would maintain that sustained stress testing is actually taking on the role of what volume testing was considered.)

quote:

---

Also when you refer to resource violations of course this can include such items as disk usage, bandwidth, memory or even CPU time (one which you missed BTW). These are all factors taken into account when doing any of these three types of testing, in particular stress testing.

---

Agreed. My idea was not to be exhaustive in the list but simply to make a point. Stress testing is sometimes considered in a very granular manner as is performance testing as a whole.

As an example consider that at its most basic level testing for performance applies anytime an application has to do any form of processing and this is true whether that processing is an internal formula (such as a Microsoft Excel sheet calculating a formula for a group of cells) or the interaction of a client with a server (such as a Web page making HTTP requests to a Web server or a client/server business application performing database queries against a database server). What we are considering is transactions. A transaction can be thought of, perhaps loosely, as any actions that a user or visitor takes to initiate some form of processing by the application under test. That processing may involve communication with a server (in terms of a Web application) or it may involve communication with a localized client interface (like a kernel, such as when doing a Mathematica calculation, as just one example).

The interesting thing, in all of this, is that you have done nothing to refute my points (which I would welcome). You have simply stated that people are "completely wrong" or have the "wrong slant". You have mentioned that "performance testing is a rate of measure. How quickly does the system respond." Many researchers now equate that definition with transaction testing and then "transactions under load" are then being used as the transferable metric to strict load testing.

Again, however, we are both probably quoting more from sources that we tend to agree with or that we have used in our work environments that we have found amenable. Also remember as well that a large consensus of thought is that the term "performance" as applied to IT systems means "a general term applied to systems that denotes how well the system fulfills the tasks expected of it." Since performance, as the basis term, is a general term, that means performance testing is, likewise, a general term which is why many researchers have taken it to be a test approach rather than a test type as I mentioned earlier.

posted 04-02-2002 06:00 PM         

Not being confrontational. I just regard what you stated not supported by an testing defnition I know.

IEEE clearly defined performance testing, hence you oppose this with a wishy/washy self generated view...It is considered by many...sorry not good enough. It is fine to oppose the IEEE definition, but what you have presented does not in any way define Performnace Testing accurately, but more in a ..."awh this is what alot of people think it is". I would give you credit if you just came out and said you don't like the definition.

Not saying I am right, I just prefer to follow the standard definitions,(obviously you don't like to) as opposed to following the point you are trying to drive at.

posted 04-02-2002 06:59 PM           

quote:

---

Originally posted by rtehve:
I just regard what you stated not supported by an testing defnition I know.

---

Understood. But just because you do not know of it does not mean it does not exist or that it is not valid. In QA we often have to be open-minded to other possible definitions, even if they go against what we currently know.

quote:

---

IEEE clearly defined performance testing, hence you oppose this with a wishy/washy self generated view...It is considered by many...sorry not good enough.

---

Actually, I pointed out just a few of the "many" in terms of the papers I referenced. Is that "wishy-washy"? Is that a "self-generated view"? Again, this sounds awfully confrontational to me. And, again, you have not refuted any of the points. Just called them "wishy/washy" and disregarded them since they do not follow the standard you support, which is exactly what I stated was probably happening. And, once again, IEEE is not the standard setter for everything and their word is not the be-all, end-all. As I mentioned, I work part-time with the standards setting bodies for the IEEE and, as such, I know there is actually much debate in this area within the IEEE because they have been taken to task (within their own ranks) for not keeping up with changing times.

quote:

---

It is fine to oppose the IEEE definition, but what you have presented does not in any way define Performnace Testing accurately, but more in a ..."awh this is what alot of people think it is". I would give you credit if you just came out and said you don't like the definition.

---

Not only did I say I indicate that I do not like the definition, I presented a very common alternative that I do like, I showed how that alternative fits into one view of the terminology, and I backed that up with other references (some that are written by people who have been working in this field for thirty years or more) from which I base my thoughts. I am not sure what else you are looking for in terms of "giving me credit".

quote:

---

Not saying I am right, I just prefer to follow the standard definitions,(obviously you don't like to) as opposed to following the point you are trying to drive at.

---

Saying I "obviously do not like to" shows that you treat IEEE as the exclusive source for defintions (and, further, that you make unwarranted assumptions about myself). How do you know that I am not following another group that also calls itself a "standard"? What about Imbus? Do you follow their standard definitions? What about Rational's standard definitions for performance testing and what it means? Neither of those groups use the IEEE for their exclusive source so I would guess you disagree with them. But then which standard do we all agree with? Also, IEEE is just one standard among many others. Thus making an a priori assumption that it is the only valid one is a curiously sanguine attitude for a QA professional to take.

Personally, Robert, and this is just my opinion: you have to work on your presentation style. While you may have valid points, your manner of presenting them (at least in my opinion) is not the most effective nor is it the most conducive to discussion.

posted 04-02-2002 07:27 PM         

If it is the case that are following another group :.."How do you know that I am not following another group that also calls itself a "standard"?"...then who are they ? You have yet to reference anything other then a bunch of white papers.

Why don't you be effective/conducive and high light who they are and how their definitions are some how better.

IEEE standards/definitions are an excellant source and by far the best commercially orientated set of standards I have seen.

Can you also elaborate on the benefit of defining stress testing and load testing as part of performnace testing and why IEEE has got it so wrong ?

posted 04-02-2002 07:32 PM           

Personally I like SWE-BOK CMU/SEI-99-TR-004: "Performance Testing: This unit is concerned with knowledge about validating the performance requirements of a system. This unit includes knowledge about techniques to instrument performance measures like logging, event counts, event duration, and sampling. It also includes knowledge about methods for tuning a system for optimum saturation, load, and throughput threshold." This definition also says that load, stress, volume, isolation, stability, transaction testing are all parts of performance testing.

posted 04-02-2002 08:09 PM           

quote:

---

Originally posted by rtehve:
If it is the case that are following another group :.."How do you know that I am not following another group that also calls itself a "standard"?"...then who are they ? You have yet to reference anything other then a bunch of white papers.

---

I was more speaking to your overt assumption that I was not following a standard beyond that of the IEEE before you even asked. You just assumed that I must not be since my definitions did not match the IEEE. Actually, I think the "standard" is currently in flux and, to be honest, I do not take any one definition as better or worse than others at least in general. However, I do tend to like those approaches advocated by the SWEBOK (as i-tester has pointed out) and Rational's to some degree that allow for an umbrella notation for "performance testing".

Also, speaking to the other point, have you read the papers? You are quick to dismiss a "bunch of white papers" but many of those are from authors that are part of the IEEE and serve on the boards that are determining some of the focus for performance and reliability concerns as they relate to testing. Also, referencing a "bunch of white papers" shows a lot of the current thinking on a given topic. Many of the IEEE standards were in fact derived from white papers by their members that were then brought up for review in the standards commitees.

quote:

---

Why don't you be effective/conducive and high light who they are and how their definitions are some how better.

---

Well, I like the notion that "performance" is treated generally - meaning as a general term. And I like this because, of course, it is. Performance, as I think we both agree, can be related to stress and can be related to load and it can also be related to transactions. Stress and load are able to give us one measure of the overall performance of a system. So I like the definition of performance (that is not operational, granted) that suggests:

Performance: A general term applied to systems that denotes how well the system fulfills the tasks expected of it.

I like that because of the definition that I already mentioned of performance testing. Here it is again:

Performance Testing: A form of testing that provides a relative measure of the capacity, stability, and scalability of a given system under expected user/visitor conditions prior to deployment which aims to assure that the software performs in accordance with operational specifications as stated in Service Level Agreements or relative to an operationally agreed upon industry standards, or in relation to an estalbished operational profile.

Notice that this says it is a form of testing. Testing, however, is a broad concept and in this case it is suggesting that it is a relative measure of various aspects of what performance means. This definition also speaks to service level agreements, which the IEEE definition does not, and it speaks to operational profiles, which the IEEE definition does not. Those are generally considered to be crucial aspects of a performance test effort within an organization.

Consider this in contrast to Imbus' definition (which I am not really fond of):

Performance Testing: Test of a system, in which the system's parameters are varied and the workload is kept stable, for the purpose of optimizing the system's performance.

The point is that with this type of nomenclature performance testing takes on the scope of being a test approach that utilizes one or more testing types, all of which can be relevant to the operational profile and the service level agreements. In fact, each can be exercised in different ways by the different testing types: stress, load, volume, transaction. (Personally, however, I agree with those that say that volume testing tends to be done, by proxy, when you do stress and load testing.)

Again, you said in your one post: "Performance testing is a rate measure. How quickly does the system respond." That, however, concretizes the term "performance" in a way that limits what the definition of performance is. IEEE also follows the idea that performance, as a general term, is defined as "how well the system fulfills the tasks expected of it." However "how well" the system "fulfills the tasks" that it must do is more than just how quickly it responds. So if the definition of performance testing, as you have stated it, does not follow the general definition of what is meant by "performance" and I personally find that to not be an amenable way of looking at things. How something "performs" is not limited just to how quickly it responds. "Performance" is a broader term than that. And, actually, as i-tester has pointed out, the definition of the IEEE does not say that it is just a rate measure as you are saying.

quote:

---

Can you also elaborate on the benefit of defining stress testing and load testing as part of performnace testing and why IEEE has got it so wrong ?

---

Actually, if you read my words you will notice I did not say the IEEE has "got it so wrong". In fact, that was the whole point. I said "IEEE is not the be-all, end-all distinction" and, as such, I was not speaking to their correctness or incorrectness. In fact, as their definition stands it could support my viewpoint slightly more than yours (at least that you have posted so far) because they do not limit themselves to just a measure of rate. They more point to "performance requirements" as the basis. As you pointed out they talk about "speed, accuracy, or memory usage" in terms of a performance requirement but I am sure we could agree there is more to performance than just that. (As you mention, CPU time is another measure.) As we know, all of those can be tested for under the rubric of stress, load, and transaction testing. (For example, I could test the speed of a given transaction set under stress or under representative load.)

Beyond that, I have somewhat explained above my thoughts on why I agree with those who place stress and load testing as types within an approach of testing for performance.

posted 04-03-2002 04:27 AM         

quote:

---

Originally posted by rtehve:

IEEE clearly defined performance testing, hence you oppose this with a wishy/washy self generated view...

---

Just as a side note to the general discussion (and without comment on whether provided definitions have been 'wishy/washy') - but well within the turn that some of this discussion has taken - to accept any one standard within Quality Assurance as The standard is not the best approach, in my opinion.

There have been many discussions on these forums and in various venues about the fact that most of the existing 'standards' are lacking - many of those who have written them will agree with those statements!

It far better serves the Quality mindset, the companies we work for, and our own career improvement to evaluate all of the various sources and come up with definitions that work within the context of our organizations.

------------------
~ Annemarie Martin ~
annemarie[dot]martin2[at]verizon[dot]net

posted 04-03-2002 05:44 PM           

quote:

---

Originally posted by JeffNyman:
They more point to "performance requirements" as the basis.

---

That's true but with that Proceedings of the IEEE International Test Conference that I mentioned, they've also said their going to broaden out the term of performance requirement. I'm not sure if that goes to the operational profile or not. Like I said, the new IEEE statement just says that performance testing is an "umbrella term" but it doesn't mention as much what they want to include in that except to say that isolation testing in relation to performance is going to be given more visibility.