Responding out of hours.
WHAT IT REQUIRES
Someone reachable Alerts that reach them Access to fix things Documentation of what to do
WHAT TO ALERT FOR
Customers unable to use the product.
WHAT NOT TO ALERT FOR
Anything that can wait.
WHY
Alerts that are not actionable train people to ignore alerts.
WHAT TO PREPARE
Runbooks for known failures Access that works from a phone, where possible A way to reach whoever else may be needed
WHAT A RUNBOOK SHOULD CONTAIN
How to confirm the problem What to check, in order How to resolve or mitigate Who to escalate to
WHY WRITTEN
At three in the morning, memory is unreliable.
WHAT ROTATION REQUIRES IN A SMALL TEAM
Honesty about what is sustainable.
WHAT IS NOT SUSTAINABLE
One person permanently on call.
WHAT TO DO IF THAT IS UNAVOIDABLE
Reduce what alerts Define hours you do not respond Communicate those to customers
WHY COMMUNICATE
Customers accept stated limits and resent unstated ones.
WHAT TO MEASURE
How often alerts fire, and how many were actionable.
WHAT TO FIX
Whatever wakes people repeatedly.
WHAT TO RECOGNISE
That out-of-hours work has a cost, and it should be acknowledged.
