posted 11-05-2002 12:48 AM         

Both load and performance tests come under black box testing.

Load tests: Study the behaviour of the program at its work.It considers Volume test, stress test and storage tests.

Volume test:Study the maximum number of the tasks the program can deal with.

Stress test: Study the program's response at the peak bursts of activity.

Storage test: Study how the memory was stored either in the resident memory area or on the disk.

Coming to performance test: Performance testing can be done by both blackbox and whitebox testing techniques.

White box performance testing gives finer analysys because you can use hardware base execution monitors so that you can know how much time it is taking to excute particular application.

The main objective of performance testing is performance enhancement. The performance test describe which modules are executing most often or using the most computer time and those modules will be re-examined and retested to run more quickly.

Testing groups do black box performance tests. For this they use benchmark performance tests to compare the latest versions performance with the previous versions performance. Poor performance can reflect in bugs especially when a part of the program which is executing fast is now running slow. Those modules will be re-examined and retest to run quickly.

In usual peformance and load tests are both treating the same.

I think i clarified your doubts.

posted 12-30-2002 07:58 PM         

The answer to your question is "it depends".

What do you want to stress? Do you want to see how many rows you can add to a single table in a minute? How many complex outer-join queries you can run? How fast queries return when you request information from every table simultaneously?

By definition, a stress test is a test that involves an unexpectedly high load. It is my opion that the best stress tests are a high/fast load that represents the actual expected usage. For example, if your usage pattern was expected to be 100 people doing the following tasks over a 15 minute period:
65% - updating address
15% - placing order
10% - checking order status
5% - running a report
5% - administering data

Then a good stress test might be 200 people doing that same percentage of tasks over 15 min - or maybe 5 min - or maybe 1000 people - or maybe 500 address updates in a 5 min period. That kind of thing. Apply stress to areas where performance is suspect at expected usage levels and see what happens.

These are by no means all of the possible examples, but rather a sample thought process.

posted 12-31-2002 06:51 AM         

"Testing conducted to evaluate a system or component at or beyond the limits of its specified requirements with the goal of causing the system to fail."

Source: IEEE Std 610.12-1990.

The human analogy would be the cardiac stress test, steadily increasing physical exertion until the body reaches an oversubscribed state.

posted 12-31-2002 07:21 AM         

I don't know that I agree with the goal of causing the system to fail, but more with a goal of finding the failure points of the system. Subtle difference, but if I apply the most stress I am able to apply and the system survives, I would not consider that a failed stress test.

sematics, semantics. Sometimes I think I should have been an english major.

posted 03-29-2003 10:50 PM         

The basic idea is that you drive the program to failure in order to watch <i>how</i> the program fails. For example, if the test involves excessive input, you don’t just test near the specified limits. You keep increasing the size or rate of input until either the program finally fails or you become convinced that further increases won’t yield a failure.

The fact that the program eventually fails might not be particularly surprising or motivating. The interesting thinking happens when you see the failure and ask what vulnerabilities have been exposed and which of them might be triggered under less extreme circumstances.

posted 03-30-2003 07:29 AM         

Can we thus agree to define a stress test as "A test designed to exercise the system under unexpected (stressful) conditions to determine when and how that system will fail under those conditions."

The only problem is the word "unexpected". If I expect my system to handle 1000 hourly users (for example) and it starts failing at 200, then a 200 user test would qualify as a stress test, even though that does not qualify as an unexpected contition.

I guess the point I am trying to make is that anything can be a stress test - if it stresses the system. You really don't know what that will be until you start executing tests and monitoring the system.

This is one of the big thought differences between functional and performance testing. While you will typically have a set of test cases/requirements/test scenarios for a performance test, most of them are designed not really to be reported in a Pass/Fail sort of way, but rather to gather information to develop new tests to uncover more information. Eventually (if you are lucky) all of the "positive" test cases will pass and you and your team will not be able to come up with any new ways to "break the system" without applying ridiculous conditions.

posted 03-30-2003 10:22 AM         

An "unexpected" input is one not imagined by the system designer. I don't think we want to define this style of testing in terms of the limits on the imagination of the designer.

In approach that I'm describing, you keep pushing the program until it fails -- or until you decide it won't fail and give up.

The question is not necessarily whether it can handle that particular input stream (the idea is to find a stream, no matter how far out of spec, that it cannot handle), but what you can learn from the failure that might help you harden the system under less extreme circumstances.

posted 03-30-2003 10:40 AM         

Unexpected not in terms of what a developer or tester can imagine, unexpected as in not what the system was designed to handle - but I'm not tied to that term.

I am trying to make a distinction that may be unnecessary. I am trying to draw a definable line between stress and load (I'm not happy with that term either, but I'm kinda stuck with it) testing. Load testing SHOULD test the performance under expected usage scenarios. These tests SHOULD NOT (but usually will) cause stress to the system. These "load-tests" cause stress, and therefore should be analyzed as stress tests, but were not designed to be stress tests.

Stress tests are designed/intended to cause stress (failure). Some of (many) do not end up causing any stress or failure, and are thus modified until stress or failure is caused.

As I said, this distinction is completely un-necessary to an experience performance tester - however - in the last years I have spent so much time and energy trying to explain that stress testing isn't load testing and that neither alone is enought, etc (I'm sure you are familiar with the discussion) that I keep trying to find a simple/elegant way for people who haven't yet had the "lightbulb experience" to flip the switch, so to speak, and see the distinction.

I'm not disagreeing with anything you have written, Cem. I am exploring vocabulary and definitions in an attempt to simplify what the majority of the market finds to be a compex topic.

posted 03-30-2003 07:26 PM         

(a) Read Jorgensen's work on hostile data stream testing. Easiest to reach reference on the web, at the moment, is Jorgensen, Alan: “Testing With Hostile Data Streams,” http://streamer.it.fit.edu:8081/ramgen/cs/Spring03/CSE5500-Sp03-05/trainer.smi

(b) Years ago, Tandem used to be fanatically committed to fault-tolerance. In the early 1980's, I met one of their testers, who did things like disconnecting a disk while the database was being updated. The system had to fail-over--failure of one storage device could not be allowed to cause data loss. More extreme testing involved simultaneous crashing of many devices at once ("earthquakes happen!"). At some point, failure is certain. But if the system is well enough instrumented, you can learn what part of the software failed first and how, and what other failures occured that were independent of the first failure. Each of these is a point of vulnerability, which you MIGHT be able to exploit with much less drastic tests, now that you know about them.

posted 03-30-2003 08:10 PM         

I've actually been having this conversation with several people recently (including Ross Collard and Danny Faught). They prefer the terms Performance and Robustness. I have always divided the same topic areas into 3 categories. Speed, Scalibility and Stability. Different types of stress tests are prevelant in both Scalability and Stability testing.

<shrug> I don't know. It's an organizational method that I came up with. It's not perfect. I don't think any really are - to create a rule with no exceptions is to create a rule that is too complex to really be applicable - at least most of the time.

Growing up, I used to like Math and Physics because they obeyed the rules - then I got kinda good at physics and they told me "All that is true... until you approach the speed of light..." All at once I decided that it was pretty much safe to assume that every rule had exceptions, until proven otherwise.

Cool link, by the way! Thanks for sharing it with us.