Diagnosing live systems.
WHAT TO ESTABLISH FIRST
What exactly is failing For whom Since when Whether it is constant or intermittent
WHAT TO LOOK AT
Error logs, filtered to the period Metrics around the time Recent deployments and changes
WHAT THE FIRST QUESTION IS
What changed.
WHAT TO USE
Request identifiers, tracing one failing request through every log entry.
WHY
It is the difference between diagnosis and guessing.
WHAT TO BE CAREFUL WITH
Changing things in production to test a theory Enabling verbose logging and forgetting it Attaching debuggers to live systems
WHAT TO PREFER
Reproducing in a non-production environment.
WHAT TO DO WHEN YOU CANNOT REPRODUCE
Add logging around the suspected area, deploy, and wait.
WHAT INTERMITTENT FAULTS USUALLY INVOLVE
Concurrency Timeouts Caching Data-specific conditions
WHAT TO RECORD
What you found, and what resolved it.
WHAT TO ADD AFTERWARDS
Monitoring that would have detected it sooner.
