The online community for software testing & quality assurance professionals
  
Active Topics    Today's Topics
Sponsors:
Lost Password?

Home
BetaSoft
Jobs
Training
News
Links
Downloads

News Group:
software.testing


Testing
Automation
Performance
Engineering
Miscellaneous
Statistics
Poll
  QA Forums
  Performance & Load Testing
  Difference between Load, stress, and Performance (Page 1)

Post New Topic  Post A Reply
profile | register | preferences | faq | search

UBBFriend: Email This Page to Someone!
This topic is 2 pages long:   1  2 
next newest topic | next oldest topic
Author Topic:   Difference between Load, stress, and Performance
anandapramanik
Member

Posts: 7
Registered: Jul 2002

posted 08-02-2002 01:38 AM     Click Here to See the Profile for anandapramanik   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by anandapramanik Visit anandapramanik's Homepage!
Hello!
I'm very confussed can any body tell me the major differences between Load , Stress and Performance Testing.

thanx in advance.
Ananda

------------------

IP Logged

JoeW
Advanced

Posts: 110
Registered: Jun 2001

posted 08-02-2002 02:43 AM     Click Here to See the Profile for JoeW   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by JoeW
This has been discussed many times before, and there are still different opinions. I like the Rational Unified Process Definitions, as follows :

quote:

"Performance testing" is a class of tests implemented and executed to characterize and evaluate the performance related characteristics of the target-of-test such as the timing profiles, execution flow, response times, and operational reliability and limits.

Included within this class are :

quote:

"Load testing" - Verifies the acceptability of the target-of-test's performance behavior under varying operational conditions (such as number of users, number of transactions, etc.) while the configuration remains constant.

"Stress testing" - Verifies the acceptability of the target-of-test's performance behavior when abnormal or extreme conditions are encountered, such as diminished resources or extremely high number of users.


Basically Performance testing is the overall process, Load testing checks that the system will support the expected conditions, Stress testing tries to break it (which is the best bit of course!)

------------------

IP Logged

anamadhey
Member

Posts: 32
Registered: May 2002

posted 08-02-2002 03:09 AM     Click Here to See the Profile for anamadhey   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by anamadhey Visit anamadhey's Homepage!
Dear friend,
Yes it is very confusing. I passed thru the same phase before knowing what it is.

REad below;

Load and stress testing are subsets of performance testing.

Performance testing means how best something performs under a given benchmark. For example How mucn time you take to run 100 meters without carrying any load (no load is the benchmark) ?

Load testing is also performance testing but under various loads. The previous example if extended would be How much time you took to run the same 100mts but carrying a load of 50 kilos, 100 kilos .... ?

Stress testing is performance under stress conditions. Extending the same example as before How much time you took to run 100 meters with load or no load when a strong wind was blowing in opposite direction ?.

Extending performance, load.. testing to s/w or h/w application.

example

PERformance : 1000 txns per minute with 1000 users concurrent

Load : How many txns when 2000, 3000, 4000 concurrent users.

Stress : How many txns when 1000, 2000, 3000 concurrent users... under conditions like server memory very low, data transmission line poor, etc...


I hope it is clear now

------------------

IP Logged

anandapramanik
Member

Posts: 7
Registered: Jul 2002

posted 08-03-2002 01:45 AM     Click Here to See the Profile for anandapramanik   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by anandapramanik Visit anandapramanik's Homepage!
Hello!
thanx 2 u all.
thanx, JoeW ur definitions clear my doubt and thanx a lot anamadhey ur example practically solve my doubt.

Ananda

------------------

IP Logged

scottqae
Member

Posts: 17
Registered: Aug 2002

posted 08-18-2002 04:18 PM     Click Here to See the Profile for scottqae   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by scottqae
I just did a survey of several QA classics on this very distinction. There is nothing close to universal agreement or terminological consistency between various authors. The replies here capture the basic distinction, however, I consider load testing to be the general category, with performance and stress testing as species of that genus. I'll explain why.

All performance and stress testing require workload definition as part of the test, and they both normally require load generating tools (unless using internal feedback loops to generate load). It may be no load, minimal, normal, above normal or extreme. I define load testing thus:

"Any type of testing where realistic (or hyper-realistic) workloads are characterized, simulated and submitted to the system under test."

Extreme loads are used in stress testing. Usually normal loads are used in performance testing. Minimal loads are usually used in benchmark testing. Anyway, load, not performance, is the shared characteristic across all these types of testing.

In stress testing, some people say you should be measuring performance. But no, in stress testing you should be looking to break the application and expose bugs that are likely to appear under stress, such as data corruption, buffer overflows, poor handling of resource depletion, deadlocks, race conditions, etc. The fact that performance metrics such as utilization, throughput and response time can be measured during stress testing is irrelevant to the purpose of stress testing. If you are testing to see that the application does not meet its performance objectives, then you are performance testing, not stress testing. Totally different goals, which make it clear that stress testing is not a species of performance testing.

Another reason for putting performance and stress under load testing is another type of testing known as background testing. In background testing you use workloads (usually normal) to exercise the system under test while you run functional and/or regression tests against the system. The goal of background testing is to test functionality in more realistic conditions, i.e., with a realistic background workload, like it will have in production.

Background, stress, and performance testing are all described wonderfully in Boris Beizer's timeless *Software System Testing and Quality Assurance*.

Another reason for making load testing the more general category is that load testing is the least frequently used or defined term in the QA literature. To be sure, some people use it. Robert Binder's otherwise excellent *Testing Object-Oriented Systems: Models, Patterns, and Tools* includes load testing as a "variation on" performance testing, but he lists stress testing as a separate category (in a section titled "Performance and Stress Testing"). Note, the load and performance section of his book is nothing more than a cursory overview.

And while I don't much favor John Musa's *Software Reliability Engineering*, (ironically Robert Binder does), I would almost agree with him when he defines "load testing" as:


“executing operations simultaneously, at the same rates and with the same other environmental conditions as those that will occur in the field. Thus the same interactions and impact of environmental conditions will occur as can be expected in the field. Acceptance test and performance test are types of load test.”

My only qualm is that his definition only includes normal conditions, and so doesn't take stress testing into account.

Glenford Myers' classic *The Art of Software Testing* doesn't define load testing, but does define performance, stress, and volume testing.

The bible of performance testing and analysis, Raj Jain's, *The Art of Computer Systems Performance Analysis*, doesn't talk about load testing at all as a separate category.

The more recent *Performance Solutions, a Practical Guide to Creating Responsive Scalable Software*, by by Connie U. Smith and Lloyd G. Williams, talks a lot about performance testing and some about stress testing, but never mentions load testing separately.

The RUP definition for performance testing given above is terrible as it conflates many different testing goals requiring different tools and techniques under one rubric, which loses its true meaning by being applied to too many things. They also have lumped in reliability testing, which is normally considered yet another type of testing, though one could argue that MTTF and MTBF and other reliability metrics are species of performance metrics. In my opinion, the RUP definition is closer to a summary of system testing activities than of performance testing, which is a specific type of system test aimed at showing that a system does or does not meet its objectives for throughput, response time, or utilization.

anamadhey's definitions all sound like variations on performance measurement to me, since they all have to do with measuring response time under various workloads (no load, normal load, and above normal load). Another thing to keep in mind is that while measurement is a necessary part of performance testing, in itself it is not software testing.

Scott Stirling
Workscape

[This message has been edited by scottqae (edited 08-18-2002).]

[This message has been edited by scottqae (edited 08-18-2002).]

IP Logged

mwsrosso
Advanced

Posts: 131
Registered: Sep 2001

posted 08-19-2002 03:21 AM     Click Here to See the Profile for mwsrosso   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by mwsrosso
A lot of good information but no-one has mentioned "Failover Testing" which I consider to be part of the Performance Testing process.

Mark.

------------------

IP Logged

RSBarber
Moderator

Posts: 1244
Registered: Jul 2002

posted 08-19-2002 12:02 PM     Click Here to See the Profile for RSBarber   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by RSBarber Visit RSBarber's Homepage!
Lots of good stuff written. In simplest terms.

Performance tests - test performance in any combination of ways you can think of.

Load - tests "stuff" under realistic, accurately modeled loads.

Stress - tests "stuff" under un-realistic, i.e. "stressful" conditions.

------------------
Scott Barber
NOBLE(STAR
Sr. Performance Engineer
sbarber@noblestar.com
http://www.noblestar.com

IP Logged

anandapramanik
Member

Posts: 7
Registered: Jul 2002

posted 08-19-2002 10:21 PM     Click Here to See the Profile for anandapramanik   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by anandapramanik Visit anandapramanik's Homepage!
Hello Mark,
You discussed about "Failover Testing", What's that?

Ananda

------------------

IP Logged

Franckhalegoi
New Member

Posts: 1
Registered: Feb 2002

posted 08-21-2002 12:58 AM     Click Here to See the Profile for Franckhalegoi   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by Franckhalegoi
I m french so excuse me first for my poor english.

To study load stresss or performance , you had to study a time transaction.

In first you will load that transaction a large number of time with an injector as loadrunnel or QA load. If the respond time is correct compaRate to the user requierement. You can say that your load test is correct. But perhaps the user direction will ask you to know exactly on which load rate your application will failed. that is the stress test. But on the stress test many time you will need to load fewer representative transaction to simulate your production.

The performance test is completly different. In fact for load and stress test your interest is based on the transaction response time. You always study the transaction in a outside view.

Suppose that, Your load test is failed. You need to understand why. So you will study exactly what happen inside the transaction.

To do it you can program a conversation control product (Winrunner ,QArun) with timeclock and use some data base monitor.

that is my point of view

------------------

IP Logged

JoeW
Advanced

Posts: 110
Registered: Jun 2001

posted 08-21-2002 06:29 AM     Click Here to See the Profile for JoeW   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by JoeW
Thanks to scottqae for the detailed analysis - yes it probably does make more sense for "load" and "performance" to swap places. Then again, stress testing is not necessarily a break test, it could just be abnormal load, in which case performance measurements are still made.

I tend to call myself a "Performance/Load Tester" and not worry to much. An additional explanation of "I like to break things" takes the stress testing aspect into account.

------------------

IP Logged

venugopalm
New Member

Posts: 2
Registered: Mar 2003

posted 03-31-2003 02:31 AM     Click Here to See the Profile for venugopalm   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by venugopalm
Hello,

RSBarber, Your definitions are very candid and clear. I want to know what does "stuff" represent in your definitions of Load and Stress testing. Is it the Performance or something else. Can you please clarify on that.

------------------
Thanks
M.Venu Gopal

IP Logged

RSBarber
Moderator

Posts: 1244
Registered: Jul 2002

posted 03-31-2003 04:19 AM     Click Here to See the Profile for RSBarber   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by RSBarber Visit RSBarber's Homepage!
Wow - this is 7 month old thread! Let's see what I can do with this (you may have seen a couple of posts down that there is another discussion ongoing to better categorize/define these, but this should be workable.

Performance tests - test performance in any combination of ways you can think of.

Load - tests performance of the applicaiton under realistic, accurately modeled loads.

Stress - tests performance and stability of the applicaiton or application component under un-realistic, i.e. "stressful" conditions with the intent to find both where it ultimately fails and how it fails.

------------------
Scott Barber, Sr. Performance Engineer
sbarber@noblestar.com
http://www.noblestar.com
http://www.perftestplus.com

IP Logged

bach01
Member

Posts: 6
Registered: Feb 2003

posted 04-14-2003 01:35 PM     Click Here to See the Profile for bach01   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by bach01
>>Stress - tests performance and stability of the applicaiton or application component under un-realistic, i.e. "stressful" conditions with the intent to find both where it ultimately fails and how it fails.<<

As an add on to this, on Stress Testing, your intent is not just to find where, when and how it fails, but ultimately it is so that you can grab some measurements so that you and ultimately your production team can predict when your site is approaching failure.
That way you can hopefully install some tripwires, and allow for some proactive measure to be taken. i.e. possible restrict new logins, search and destory possible runaway queries, add more processors' or systems to the mix....
So Stress Testing is not just about breaking the systems (which of course is fun), but in also collecting usable data to predict failure, and allow you to, hopefully, be proactive in alleviating a disaster.

Matthew S. Brown
Sr Test Engineer


------------------

[This message has been edited by bach01 (edited 04-14-2003).]

IP Logged

rosscollard
Member

Posts: 31
Registered: Dec 2002

posted 04-14-2003 09:34 PM     Click Here to See the Profile for rosscollard   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by rosscollard
BASIC DEFINITIONS

This is an excerpt from my forthcoming book on performance and load testing.

While there is no universal consistency in how people use terms like performance test and robustness test, I can say that the definitions provided here are as much in the mainstream as any others.

The Definition of Performance Testing

The purpose of performance testing is to measure a system’s performance under load. As Humpty Dumpty said, a word can mean whatever one chooses it to mean, so it is worth our time to examine what we mean by the words “measure”, “performance” and “load”.

Performance testing is a measurement of performance characteristics, although sometimes the use of the word “testing” confuses people. Some performance professionals feel strongly that it is important to not use the term “performance testing”, but to call it performance measurement instead. They are concerned that this measurement will get confused with feature testing and debugging, which it is not. They point out that measurement is only testing if the collected measurements are checked against pre-established goals for performance, and that measurement is often done without preconceptions of required performance.

These people have a good point: clarity of terminology is important. But since most people use the term “performance testing” we will go with the majority and use it too.

The term performance can mean response time, throughput, availability, error rate, resource utilization, or another system characteristic (or group of them), which we are interested in measuring. “All promise outruns performance.” Ralph Waldo Emerson

Performance testing simulates the typical user experience under normal working conditions. The load is a typical, representative mix of demands on the system. (And, of course, there can be several different representative loads -- the work load at 2 p.m., at 2 a.m., etc.) Another name sometimes used for a performance test is a capacity test, though there is a minor difference in these terms as we will see later.

First, the performance testers need to define what the term performance means in a specific test situation -- that is, what the objectives are and what we need to measure in the test. The answer to this question is that we measure performance usually as a weighted mix of three characteristics of a system: throughput, response time and availability. In real-time systems, for example, the users need a guarantee that a task will always be completed within a fixed time limit. Performing a task correctly but a millisecond too late could literally be fatal.

The term load simply means the mix of demands placed on a system while we measure its performance and robustness characteristics. In practice, most loads vary continually, so later we will address the challenge of determining the most appropriate load(s) for testing. The terms work load and benchmark are sometimes used as synonyms for load. A benchmark usually means a standard load, one used to compare the performance of systems, system versions, or hardware environments, but the benchmark is not necessarily the actual mix of demands at any one user installation. The term work load is a synonym for a load, and you see both of the terms in this book: they are interchangeable.

Definition of Load Testing

In contrast to a performance test, a load test is a measurement of performance under heavy load: the peak or worst-case conditions. Because loads can have various sizes, more precise terms for this type of testing are peak-load testing or worst-case-load testing.

A performance test usually is done with a typical, representative load, but this measurement may not tell us much about the system’s behavior under heavy load. For example, let’s assume that the peak load on a system is only 15% more than the average load. The system performance may degrade gracefully – the system runs 15% slower at peak load. Often, though, the performance under load is non-linear: as the load increases by a moderate amount (in this case, 15%), the response time does not increase by a comparable percentage but instead becomes infinite because the system fails under the increased load.

Definition of Stress Testing

A stress test is one which deliberately stresses a system by pushing it beyond its specified limits. The idea is to impose an unreasonable load on the system, an overload, without providing the resources which the system needs to process that load.

In a stress test, one or more of the system resources, such as the processor, memory, or database I/O access channel, often “maxes out” and reaches saturation. (Practically, saturation can happen at less than 100% of the theoretical usable amount of the resource, for many reasons.)

This means that the testware (the test environment, test tools, etc.) must be sufficiently robust to support the stress test. We do not want the testware to fail before we have been able to adequately stress the system.

Many bugs found in stress testing are feature bugs which we cannot see with normal loads but are triggered under stress. This can lead to confusion about the difference between a feature bug and a stress bug. We will address this issue in the upcoming section entitled: “Testing Performance and Robustness versus Features”.

Some testers prize stress testing because it is so fruitful in finding bugs. Others think it is dangerous because it misdirects projects to fix irrelevant bugs. Stress testing often finds many bugs, and fixing these bugs leads to significant delays in the system delivery, which in turn leads to resistance to fixing the bugs. If we find a bug with a test case or in a test environment which we can’t connect to actual use, people are likely to dismiss it with comments like: "The users couldn’t do that.", “.. wouldn’t do that” or “... shouldn’t do that.”

Stress, Robustness and Reliability

Although stress, robustness and reliability are similar, the differences among them mean that we test them in related but different ways.

We stress a system when we place a load on it which exceeds its planned capacity. This overload may cause the system to fail, and it is the focus of stress testing.

Systems can fail in many ways, not just from overloading. We define the robustness of a system by its ability to recover from problems; its survivability. Robustness testing tries to make a system fail, so we can observe what happens and whether it recovers. Robustness testing includes stress testing but is broader, since there are many ways in which a system can fail as well as from overloading.

Reliability is most commonly defined as the mean time between failure (MTBF) of a system in operation, and as such it is closely related to availability. Reliability testing measures MTBF in test mode and predicts what the system reliability will be in live operation.

Robustness and reliability testing are discussed in the companion volume to this book, entitled “System Robustness Testing”.

Ross

------------------

IP Logged

RSBarber
Moderator

Posts: 1244
Registered: Jul 2002

posted 04-15-2003 11:52 AM     Click Here to See the Profile for RSBarber   Edit/Delete Message Copy This Message   Reply w/Quote Search for more posts by RSBarber Visit RSBarber's Homepage!
Thanks for that Ross, I'm going to make sure this thread is linked in the FAQ.

------------------
Scott Barber, Sr. Performance Engineer
sbarber@noblestar.com
http://www.noblestar.com
http://www.perftestplus.com

IP Logged


This topic is 2 pages long:   1  2 

All times are PT (US)

next newest topic | next oldest topic

Administrative Options: Close Topic | Archive/Move | Delete Topic | Top
Post New Topic  Post A Reply
Hop to:

Contact Us | BetaSoft Inc. | Privacy Statement

Copyright © 1997-2003 BetaSoft Inc.


Ultimate Bulletin Board 5.45c