Understanding a running system.
WHY IT MATTERS
You cannot operate what you cannot see, and problems in production are diagnosed with what was recorded beforehand.
WHAT THE THREE KINDS ARE
- Logs: what happened
- Metrics: how much and how often
- Traces: the path of a request through the system
WHAT LOGS SHOULD CONTAIN
A timestamp A severity What happened Enough context to identify the case An identifier linking related entries
WHAT THEY SHOULD NOT CONTAIN
Passwords, tokens, or personal data beyond what is necessary.
WHAT STRUCTURED LOGGING PROVIDES
Logs that can be searched and aggregated by field.
WHAT METRICS TO COLLECT
Request rate Error rate Latency, at several points of the distribution Resource use Business outcomes
WHY THE DISTRIBUTION
Average latency conceals that a proportion of users have a poor experience.
WHY BUSINESS OUTCOMES
Every technical check can pass while nothing useful is happening.
WHAT TRACING PROVIDES
Where time went, across services.
WHAT TO DESIGN IN
A correlation identifier, passed through every call.
WHY
Without it, related events across services cannot be connected.
