The summary.
MONITOR FROM OUTSIDE
A system cannot reliably report its own unavailability, and an application can report health while serving an error page.
Check that a real page loads and contains expected content.
THREE THINGS THAT FAIL SILENTLY FOR WEEKS
Backups, scheduled tasks, and certificate renewal. Monitor all three explicitly.
AN ALERT MUST MEAN SOMEONE ACTS NOW
Anything else trains people to ignore alerts, including the important ones.
Send them by a route independent of the failing system — an alert about mail failing, delivered by mail, does not arrive.
MONITOR BUSINESS OUTCOMES, NOT ONLY TECHNICAL HEALTH
Every check can pass while no orders have been placed since a deployment three hours ago.
A broken form or a silently disabled feature produces no errors at all. This is the check that protects revenue.
DO NOT USE AVERAGE RESPONSE TIME
It conceals the slow requests, which are the ones people notice. Use percentiles.
SYNCHRONISE CLOCKS AND USE A REQUEST IDENTIFIER
Without both, correlating an incident across systems is impractical.
REVIEW EVERY ALERT THAT FIRED, MONTHLY
If one was not actionable, change it or remove it.
