Rebuilding after a serious failure.
WHAT TO DEFINE
How much data you can lose How long you can be down
WHAT THOSE DETERMINE
Backup frequency and the recovery approach.
WHAT THE RECOVERY PATH USUALLY IS
Provision a cluster or host Apply manifests from version control Restore data from backups Point traffic at it
WHY THAT ORDER
Infrastructure, then configuration, then data.
WHAT MAKES IT POSSIBLE
Everything being described in files rather than created by hand.
WHAT BREAKS IT
Objects created interactively Secrets nobody has outside the cluster Storage tied to a region that is unavailable
WHAT TO DO ABOUT SECRETS
Hold them in a manager, or encrypted in version control with the key held separately.
WHAT TO CONSIDER ABOUT REGIONS
Whether backups are in a different one.
WHY
A regional failure otherwise takes both.
WHAT TO PRACTISE
A rebuild, at least once.
WHY
It reveals what is missing while it does not matter.
WHAT TO MEASURE DURING THE PRACTICE
How long it took.
WHAT TO COMPARE THAT AGAINST
What you told the business.
WHAT TO DOCUMENT
Every step, in an order someone else can follow.
WHAT TO KEEP ACCESSIBLE
That document, outside the systems it describes.
