Alerting before it is too late
From predictive saturation to SLO burn rate, all the way to who receives the notification
An alert that fires when the disk is already full has answered the wrong question. The question is always the same, when should you say something is not working, and the series walks it on three levels. It closes on the stretch almost nobody looks after: what happens between the rule that fires and the person who has to act.
An alert that fires when the disk is full has answered the wrong question
The rule copied from the first tutorial reports that the disk is full now. By the time it fires, usage is at 90%, the logs are already failing and some service is returning ENOSPC. It is not a threshold problem. The right question was a different one: will it fill up within a window where I can still act without waking anyone?
The series walks three levels of the same question. The first looks at physical resources and the trend they are consumed at. The second moves the subject from the resource to user impact: not when the disk saturates, but at what rate the error budget of the service is burning. The third deals with the stretch almost no repo looks after, the one between the rule that fires and the person who has to act.
The common thread is that alerting well is not a property of a single query. It is a property of the whole system: from the metric to the rule, from the rule to the routing, from the routing to the payload, from the payload to the person who at three in the morning has to figure out what to do.
What you will learn
- ✓ Tell saturation as a current state apart from saturation as a trend, and know which one you need
- ✓ Recognize the four traps of linear regression before putting it in a pager
- ✓ Alert on the rate the error budget is being consumed instead of on a fixed threshold
- ✓ Install the three canonical pairs from the SRE Workbook, and know where the numbers come from
- ✓ Route by severity, suppress duplicates and put a runbook in the payload
Articles in the series
- 01USE and the Golden Signals do not mean the same thing 11 min
USE defines saturation as queue present now, the Golden Signals include predictions. Two compatible frameworks, two alerts with opposite profiles.
→ - 02Which alert for which resource 11 min
Five PromQL examples from TLS to the connection pool, the four traps of linear regression, and ten resources with the alert each one needs.
→ - 03The static threshold answers the wrong question 11 min
A sustained 0.5% for an hour burns 30% of the monthly budget without firing anything. Alert on the rate of consumption, not on the threshold.
→ - 04Three window pairs, and why all three are needed 13 min
SRE Workbook table 5-8 row by row: 14.4× on 1h+5m, 6× on 6h+30m, 1× on 3d+6h. Where the numbers come from and why you never install just one.
→ - 05After the alert fires: severity, routing and the contract with whoever receives it 8 min
Severity as a routing contract, inhibit rules and runbook_url in the payload: the three minimum building blocks that make an Alertmanager alert actionable.
→