What to do when the other side fails.
WHAT FAILURES OCCUR
Timeouts Connection refused Server errors Rate limiting Malformed responses Correct responses with unexpected content
WHAT TO RETRY
Timeouts, connection failures, server errors and rate limiting.
WHAT NOT TO RETRY
Client errors, which will fail identically.
WHAT BACKOFF MEANS
Waiting longer between each attempt.
WHY
It gives a struggling service room to recover.
WHAT TO ADD TO THE DELAY
A random element.
WHY
Otherwise every client retries simultaneously, repeating the overload.
HOW MANY ATTEMPTS
Few, with a total time limit.
WHAT TO DO WHEN RETRIES ARE EXHAUSTED
Record it, and decide whether to queue or fail.
WHAT A CIRCUIT BREAKER DOES
Stops calling a failing service for a period, failing fast instead.
WHY THAT HELPS
It stops you exhausting your own resources on calls that will fail.
WHAT TO DO WHILE OPEN
Return a sensible degraded response, if one exists.
WHAT TO BE CAREFUL WITH
Retrying operations that are not idempotent.
WHY
You may create duplicates.
WHAT TO USE
An idempotency key, where supported.
WHAT TO DO WHEN THE OUTCOME IS GENUINELY UNKNOWN
Record it as pending and reconcile later.
WHAT TO NEVER DO
Assume failure and repeat the action.
