Different performance tests including Peak performance and stability

Peak performance test:-
This test is done to analyze the application peak performance. To achieve peak performance we have to slowly ramp up the users and see the response times and TPS with increase in users. At a certain point TPS will stay flat and response times increase and in some cases TPS decreases and thus increases response time with increase in users.
The point where TPS starts staying flat is the peak performance point and this is how we do peak performance test.


Break point test :- 
We do break point test in the same line as peak performance but we dont stop the test once we reach peak point, we continue to increase users and see when the application breaks, that means we see the errors per sec graph along with vusers and transaction response time and transaction per sec graphs.
As we increase load on the application beyond peak performance, application cannot handle it anymore and at some point it starts throwing errors and you see a spike in the errors per sec and that's the break point for that application for the given test conditions.

Typical reasons for the application to reach breakpoint:- 
1. Queue length is increasing and the wait time at that queues for new request reach max wait time and thus will throw error
2. No Connections available to take new requests
3. Threads are busy and are waiting for other system resources
4. Concurrency of the objects being instantiated reached its limit
5. DB connections are full
6. JVM heap memory isn't sufficient.


Stability test:-
We have to know the peak performance point to perform stability. Once we are done with peak performance test and know the max number of users app supports we run stability test for an hour or 2 at the peak point or 5-20% more load than peak point. Note that we assume break point doesnt come immediately after peak performance point.
eg - if peak performance is at 90 users and break point is 100 users then we can run stability at 95 or 90

As part of performance testing, other tests we do are Regression, Load, Stress, Soak/Endurance.

Performance environment scaling and results extrapolation

In this article we shall discuss about the points we have to consider when we get a new environment or existing environment for performance testing.

Rule 1 - The configurations of Production and test env should match
eg - if they have 4 web, 2 app and 2 Db servers
test env might be 2 web, 1 app and 1 DB but not 1 web, 2 app, 1 Db etc....

Rule 2 - JVM heap memory should match
Prod 2 JVM with 4GB heap each
test has 1 JVM with 4 GB heap but not 1 JVM with 3GB

Rule 3 - The log settings of test and prod should match

Rule 4 - Check for the connection limits of all servers (limit should be w.r.to the performance tests we perform on that env)

Depending on the above responses and some more info specific to that env, we do the scaling.

Results extrapolation:-
Rule 2 is most important while doing extrapolation as the available memory on that server is not important but the heap memory allocated is important and that is what we compare with production - not the total memory of server.
If you just have 1 JVM and have to extrapolate to prod with 4JVM's is not practical and results might not be accurate.
For accuracy we might need more than 1 JVM so that we can test with 1 JVM and then with 2 JVM's and then map the results to see the pattern and through curve fitting we can get a formula that can be used to extrapolate.
Note:- if you need more accuracy, you also need more samples

Apart from Rule 2, we also consider all the rules mentioned above to extrapolate the results.