Performance Testing – 10,000 foot view

Introduction

This document discusses the topic of “performance testing” and different strategies and levels of detail that can be targeted by performance testing.  It attempts to clarify and distinguish these terms and possible strategies.

Pre

The goal of performance testing is to measure the performance of a system, with the intent of identifying and documenting the performance.  Typically, this information is used to determine if a system is “fast enough”. However, the term “fast enough” is very subjective and therefore, difficult to consistently define.  Nonetheless, it is not necessary to define the term “fast enough” before conducting a performance test.

Ways of Testing

Bronze – use the app and see if it seems fast enough – completely unscientific. It is difficult to determine the actual speed of any part of the system, to a reasonable degree of accuracy.  However, that might be good-enough. Start by finding the pages/features which are obviously not good-enough. That might be plenty to start with. If not, keep going.

Silver – automate several tests to exercise the app (in a loop) and measure the system – seems more scientific because you can generate a large-enough statistical sample.  The test is less dependent on human error.  This is better than bronze, but maybe not ideal, because it does not isolate each possible bottleneck.

Gold – design a test to measure each part (sub-system, components) in a system. Run the tests several times, varying the number of clients.  Calculate the performance and response times of each element of a system to the deepest granularity possible.  – This is the most scientific approach and most likely to yield useful information. It will obviously also be the most expensive and time-consuming.

Results

The next step, beyond testing usually goes one of two ways:

  1. The performance is acceptable.  We are done.
    – or –
  2. The performance is not acceptable.  The system must be optimized further.

… If the system must be optimized, then there is more to this process

Optimization

  1. Analyze the system and determine options that can be applied to optimize the system.  If the “gold” test was performed, this step will be much simpler because you will clearly see where the process is slow (and how much time could be regained) or which parts are already fast (and not likely to yield much ROI for any optimization efforts).
  2. Determine time estimates for each optimization technique and determine which are practical to try
  3. Apply optimization
  4. Retest – perform a release testing cycle, until the system passes tests
  5. Re-measure (restart of this entire process)
  6. Determine if the optimization was sufficient and we can stop or we must continue

After a few optimization attempts, the cost of optimization will go up and the ROI will go down.  At a point, it is best to determine if it is more realistic or economical to consider other options, such-as purchasing better/more hardware, simply adjusting your expectations, or [migrating-to vs starting-over-with] a different technology/architecture.