Verifying behaviour under failure.
WHAT TO TEST
Behaviour when a dependency fails Behaviour when a dependency is slow Behaviour when resources are exhausted Recovery after failure ends
WHY SLOWNESS SPECIFICALLY
It is worse than failure, since it consumes resources while producing nothing.
WHAT TO VERIFY
That timeouts exist and are enforced That failure is contained rather than cascading That the system recovers without intervention
WHAT CASCADING FAILURE IS
One component's failure exhausting resources elsewhere until everything fails.
WHAT PREVENTS IT
Timeouts Limits on concurrent calls Circuit breakers stopping calls to a failing dependency
WHAT TO TEST ABOUT CIRCUIT BREAKERS
That they open That they close when the dependency recovers That behaviour while open is acceptable
WHAT CHAOS TESTING IS
Deliberately introducing failures to verify resilience.
WHERE TO START
A test environment, with a single known failure.
WHAT TO PROGRESS TO
Production, carefully, once confidence is established.
WHAT TO ALWAYS HAVE
A way to stop the experiment immediately.
WHAT TO TEST ABOUT DATA
That failure partway does not leave inconsistency.
WHAT TO TEST ABOUT RESTARTS
That the system recovers to a working state automatically.
WHAT TO MEASURE
Time to recover.
