Technical~/notes/pager-should-be-hard-to-set-off.md
The pager should be hard to set off
on-callmonitoringsre
Just finished wiring up paging for a side project and I have feelings about it
The first false alarm is what teaches people to ignore the real one. So I made the pager hard to trigger on purpose
The rules
Chat alert on the first failed health check. A phone call only after two in a row. A lost packet or a cold start clears up by the next check, and nobody needs to be woken up for that
One page per outage, then quiet until it recovers. A day-long outage should be one call, not a hundred
Only page for things users can actually feel. If an internal admin panel dies at 3am, nobody can fix it then any better than at 9am
Write the page so someone who just woke up can act on it