Measuring performance honestly
Why a thousand requests per second means nothing, and what to look at instead
A load test that passes does not say the system holds: without a baseline and without the context of production, that number is not comparable to anything. The series lines up what it takes for a measurement to mean something, and which methods answer which question.
A thousand requests per second means nothing
It is a number without a denominator. A thousand requests per second of what, with how much data, with which distribution of endpoints, against which state of the database, and compared with what? Without a baseline, a load test that passes does not say the system holds: it says that on that day, with that scenario, it did not break.
The series lines up what it takes for a measurement to mean something. First the context: the baseline, the realistic scenario, and the criterion decided beforehand for saying whether the result is acceptable. Then the reading instruments, which are the second half of the problem.
Because the average hides by construction the minority having a bad time, and it is exactly that minority which calls support. RED tells you when it is broken, USE tells you why: they are not alternatives, they answer two different questions and you need both.
What you will learn
- ✓ Understand why a throughput number without a baseline is not a result
- ✓ Read percentiles instead of averages, and see the minority having a bad time
- ✓ Know when you need RED and when you need USE, and why they are not alternatives
Articles in the series
- 01A thousand requests per second means nothing 6 min
A load test that passes does not mean the system holds: without a baseline and without project context, a performance number is true and useless.
→ - 02RED tells you when it broke. USE tells you why. 7 min
Averages hide the minority that suffers, by construction. Percentiles, the RED method, the USE method, and the rule for using them in the right order.
→