The 19 Principles of Systems Monitoring
published:
Most monitoring failures aren't failures of the system being watched. They're failures of the watching. A check that quietly stopped running. A status file nobody noticed was three days old. An hourly "I'm up" message whose absence nobody caught until a customer called.
These principles are the bar monitoring has to clear before anyone should trust it. They don't say what tool to use or how to configure it. They say what has to be true when you're done.
Trustworthy results
1. Observe the service directly. A check asks the service itself and judges the answer. It does not rely on what another process said about the service: not a status file, not a flag, not a relayed result. Every hop between the service and the judgment is a place where the truth can fall out.
2. A result is only valid if it's current. A result that can go stale without anyone knowing cannot be trusted. Cached health endpoints are the classic trap: they keep saying "healthy" after the thing behind them has died.
3. Silence is never success. A timeout, an empty response, or an answer the check can't parse is a failure. "I don't know" is never "fine." Monitoring fails closed.
4. A missing heartbeat is itself an alarm. Some systems can only tell you they're up: the backup that emails "complete," the router that phones home hourly. When they stop, nothing arrives, so nothing is the alert. Something automated has to expect the heartbeat, know the interval and grace period, and raise the alarm when it doesn't show. People are not that mechanism.
5. The monitoring is itself monitored. If your monitoring server loses power, it won't tell you. Something independent, on separate hardware, expects a regular signal from your monitoring and alerts when it stops.
Meaningful signals
6. A running process or an ICMP reply is not a working service. Every service needs at least one check that does what a user does, from where a user is. White-box monitoring tells you what's happening inside; only black-box checks tell you whether the thing works.
7. Impact sets urgency. One half of a redundant pair failing at 2 AM can wait for morning if no user noticed. A failed login page cannot.
8. Watch the vital signs. Every user-facing service is observed for latency, traffic, errors and saturation. Together they show whether it's working and whether it's about to stop.
9. Alert on symptoms, not causes. A symptom is something a user can feel: errors, slow pages, failed logins. A cause is an internal number that may or may not matter: CPU, memory, queue depth. Wake a human for symptoms. Record causes for the human you woke.
10. Know what failed. One sick database should look like one sick database, not forty red lights. Parent, child and dependency relationships are defined in the monitoring system and kept accurate, so a failure points at its source.
11. Signals are stable. A check that goes red, green, red, green isn't ten alerts. It's one problem, instability, and it's reported as one.
Respect for people's attention
12. Every alert deserves action. An alert that interrupts a person requires action and comes with guidance. An alert nobody acts on isn't noise; it's an incident.
13. False alarms are measured and bounded. Google SRE's limit is at most two incidents per 12-hour shift. The common industry bar is that at least 80% of pages need real action, with mature teams past 90%. Track it, and never hit the number by loosening checks until real failures slip through. The fix for noise is a better signal, not a higher threshold.
14. As simple as possible. Monitoring nobody understands or uses is removed. Every check and alert earns its place.
Operational discipline
15. Every check has an owner. A named human. Unowned checks don't stay.
16. Planned work is planned for. Suppressing a check for maintenance has a scope and an expiry, and the suppression ends on its own. It never depends on someone remembering.
17. Checks do no harm. They don't change data, add meaningful load, or expose anything sensitive.
18. Checks are proven before they ship. Break the service on purpose and watch the check catch it. Break the check on purpose and watch it refuse to say "healthy." Nothing goes to production until both are done and documented. Until then you have a hypothesis, not monitoring.
19. Every miss is an incident. When monitoring fails to catch a failure, or cries wolf, review it like an outage and fix what let it through.
The question behind all of them
Each principle answers the same question: if this check says "healthy," how sure am I that it's true?
Monitoring you can trust isn't monitoring that's never wrong. It's monitoring built, on purpose, to be unable to lie.
Further reading: Google's Monitoring Distributed Systems and Being On-Call; Microsoft's Health Endpoint Monitoring pattern; Amazon's Implementing health checks.