Being ready when it breaks.
WHAT TO PREPARE IN ADVANCE
Access that works when the cluster does not A documented way to reach nodes directly Runbooks for the common failures Somewhere to communicate, hosted elsewhere
WHY ACCESS SPECIFICALLY
If authentication depends on something inside the cluster, you may be locked out.
WHAT TO TEST
Reaching a node directly, and inspecting containers without the orchestrator.
WHAT THE COMMON PLATFORM FAILURES ARE
Control plane unavailable Nodes not ready Storage unable to attach Certificate expiry A full disk on nodes An add-on failing
WHAT TO KNOW FOR EACH
How to detect it What still works What to do
WHAT KEEPS RUNNING WHEN THE CONTROL PLANE IS DOWN
Existing workloads, usually.
WHY THAT IS REASSURING
An outage of the control plane is not necessarily an outage of the service.
WHAT STOPS WORKING
Deployments, scaling and recovery from failures.
WHAT TO DO FIRST IN ANY INCIDENT
Establish whether users are affected.
WHY
It determines urgency, and whether to act or investigate.
WHAT TO RECORD
Every command run.
WHAT TO PRODUCE AFTERWARDS
An account of the cause, and what prevents recurrence.
