posted 04-12-2002 09:03 AM
This is the introduction for the methodology I wrote (and adapted from examples in literature) for our internal Performance Testing manual. This methodology is independent of the kind of tool you want to use.
***********************************
3 Performance Testing Methodology
This chapter describes the activities in the general methodology for performance testing. This systematic approach will form part of the Methodology and Testing Standards within WCB. The previous chapter talks about when certain activities need to take place and what the general objective is of these activities. In this chapter, we try to focus on how we plan to do this.
When starting with performance testing, you have to realize that most situations are unique. The metrics, workload, and evaluation techniques used for one project or phase are often not fully applicable for the next project or phase. Nevertheless, there are steps common to all performance testing exercises, following these steps will help define the performance test exercise in an efficient and consistent manner.The methodology consists of the following 10 steps:
1. Set Goals and System boundaries
2. Define Services/Components and possible outcomes
3. Select Metrics
4. List Parameters
5. Select Factors to Study
6. Select Evaluation Technique
7. Select Workload
8. Design Tests
9. Run, Analyze and Interpret data
10. Present Results
3.1 Set Goals and System boundaries
The first step in any performance testing planning is to state the goals and define what constitutes the system by defining the system boundaries. Clarifying the boundaries of your performance test is critical and will give you a clear scope and an ability to communicate this to the stakeholders.
3.1.1 Goals
Setting goals involves deliberate interaction with system users, decision-makers, technical- and production support personnel.
Goals for new systems are typically related to:
* Business Requirements
* System Requirements
* Service Level Agreement Requirements
Goals are also known as Test requirements.
3.1.2 System Boundaries (Scope)
Identifying the system boundaries is essential for the definition of your scope.
The definition of the system environment may vary depending upon the goals of the study. The system definition may consist of the CPU, memory or I/O subsystem, the network, the database, the business areas, or the entire computing system itself. How the environment is defined affects the choice of performance metrics and workloads needed to compare alternative objectives.
* Describe the system
* Describe the interfaces with other systems
* Describe the Business Areas that are part of the System Under Test (SUT)
* Describe the parts of the system that will be targeted for the test.
Also, describe the parts that specifically are not included. An out-of-scope statement will remove any uncertainties about the system to be tested.
3.2 Define Services/Components possible outcomes
At a minimum, your overall description should include the following system components:
* Hardware configuration + Settings
* Operating system + Settings
* Support software
* Major applications
* Business Functions
* Business User Scenarios
3.3 Select Metrics
Once the goals have been defined and there is a clear understanding of the system or systems to be evaluated, we can move on to identifying and gathering relevant performance metrics.
The choice of metrics may be dependent on the tools available for collecting and measuring system data and vice versa.
Desirable properties for metrics:
* Be robust (that is, repeatable, precise, insensitive to minor changes)
* Suggest a norm
* Relate to specific services, processes, databases, or messages
* Suggest an improvement strategy
* Be a natural result of a service, process, database, or message
* Be simple (and easy to explain)
* Be predictable and traceable (This allows predictions to be compared with actual experience.)
3.4 List Parameters
List parameters that are known or suspected to affect performance.
The list can be divided into system parameters and workload parameters.
* System parameters include both hardware and software parameters.
* The Workload parameters are the characteristics of user's requests. They contain parameters based on the business scenarios that the application has to support.
The list of parameters might not be complete after the first pass of the analysis. You will be able to extend the list later once you gained more insight into the application under test.
3.5 Select Factors to Study
The total list of parameters can be divided into two parts: those that will be varied during the evaluation and those that will not. The parameters to be varied are called factors and their values are called levels. In general, the list of factors and their values is larger than what the available resources will allow. It is therefore better to start out with a very limited amount of factors and very few levels.
In selecting factors, it is important to consider economic, political and technological restrictions. This will make it possible to select factors that will mitigate the risk for the organization the most.
3.6 Select Evaluation Technique
Performance management is the process of ensuring that a computer system meets predefined performance objectives consistently and efficiently, both now and in the future. In order to evaluate future system behavior, the analyst must use predictive models.
Generally, there are four modeling techniques commonly used for performance management:
* Linear projection
Involves collecting past data, extending or implying trends using scatter plots and regression lines, and comparing the trend line with the current capacity of the system.
* Analytic modeling
Uses queuing theory formulas and algorithms to predict response times and utilization projections from workload characterization data and essential system relationships.
* Simulation
Is the process of exercising your system as if it is used in the real world.
* Benchmarking
Is process of defining the tests and results of your first (or any test for that matter) test as the standard and comparing all subsequent test results with the first test.
3.7 Select Workload
Workload characterization involves studying the user and machine environment, observing key characteristics, and developing a workload model that can be used repeatedly.
Once a workload model is available, the effect of changes in the workload and system can be easily evaluated by changing the parameters of the model. In addition, workload characterization can help you to determine what's normal, prepare a baseline for historical comparison, comply with management reporting, and identify candidates for optimization.
Depending upon the evaluation technique being used, workloads may be characterized in different forms. Some attributes used for classifying a workload include:
* Resource usage
* Application type
* Geographical orientation
* Visibility
* Organizational unit
* Processing type
* Similar growth
* Business process
3.8 Design/Build Tests
Once you have the workload, a list of factors and their levels selected, you need to decide of sequence of tests that offer maximal information with minimal effort. Tests can be designed that will target multiple factors at once. The designed tests will be build into "runnable" tests by the Performance Test Group. Once the tests are there, different scenarios will be created to support the testing effort.
3.9 Run tests, analyze and Interpret data
Armed with the proper workloads and data and with a thorough understanding of the environment and goals of the evaluation, you are ready to run your tests and collect your information. In case of an already existing system, you compare the newly found data to actual system measurements or to data found in earlier tests.
Once the tests are verified and the results appear to be representative of the real system, you are ready to use this baseline model to compare alternative scenarios. The results of each run of the model will provide the information needed to draw conclusions about the set of proposed alternatives.
Conclusions made about each alternative should take into account the variability and randomness of the results. None of your recommendations should be based on conclusions that are extremely sensitive to small changes in the model parameters.
3.10 Present Results
The final step of all performance testing activities is to communicate the results of your analysis and results to the decision-makers. Since this report is the primary deliverable of the performance testing process, it is particularly important that it is clear and reliable. It should give specific, practical findings and recommendations, with little technical jargon and no vague generalities. The project report should serve as a valuable reference and as a convenient baseline for comparing future performance with current performance.
*****************************************
Hope this helps.
------------------
Roland
posted 04-12-2002 08:00 AM
