Break it, and watch it recover.
The endpoint below is real and deployed. Set it to fail, send traffic, and watch exponential backoff with jitter, idempotency keys and a dead-letter queue do their jobs — with the actual HTTP status codes and measured latencies, not a mock.
down, send a burst, and let everything dead-letter. Then switch to healthy and press replay — nothing is lost, and nothing is applied twice.In flight0
Backing off0
Delivered0
Dead letter0
Every attempt, to scale
One row per message. Bars are real measured request durations; the thin connectors are the backoff waits between attempts. The gaps widen because that is the whole point.
Send some traffic to populate this.
Exponential backoff, with jitter
Doubling the wait is the easy half. The shaded band is the part that matters at scale: without randomisation, every client that failed during the same outage retries at the same instant and knocks the service over again the moment it recovers.
Three things a retry loop gets wrong
Retrying immediately makes an outage worse. A service returning 503 is usually telling you it is already overloaded. Three fast retries from every client turns a brownout into an outage. The delay is not politeness — it is the mechanism that lets the downstream recover.
Synchronised retries cause a second outage. If every client backs off by exactly one second, every client that failed during the same window returns at exactly the same instant. The service comes up, takes the full thundering herd, and falls over again. Jitter — randomising each client’s wait within a range — is what turns a spike back into a spread.
A timeout is not a failure. This is the one that quietly corrupts data. When a request times out, you do not know whether the write landed. Retrying may apply it twice. An idempotency key is what makes that safe: the server records the key on first success, and a retry carrying the same key returns the original result without re-applying anything. Try it — every message here carries one, and the replay flag in the trace shows when the server recognised it.
And once the retry budget is spent, the message has to go somewhere inspectable. A dead-letter queue is the difference between “we lost seventeen orders last Tuesday and found out at month-end” and “seventeen orders are queued, here they are, replay when the vendor is back.”
The rules, and where they come from
| Rule | Here | Precedent |
|---|---|---|
| Exponential backoff | 800ms, doubling per attempt | Standard across AWS SDKs, Google APIs, Stripe |
| Equal jitter | Actual wait is random between half and full of the nominal delay | AWS Architecture Blog, “Exponential Backoff and Jitter” |
| Bounded attempts | 4, then dead-letter | Retrying forever is a queue that never drains |
| Idempotency keys | Per-message key; server no-ops a repeat | Stripe’s Idempotency-Key header |
Honour Retry-After | Sent by the endpoint on 503 and respected as a floor | RFC 9110 §10.2.3 |
| Dead-letter with replay | Failed messages held and re-drivable | AWS SQS redrive |
| Timeouts are failures | 2s client timeout, aborts and counts as an attempt | A request with no deadline is a leak |
Source: functions/api/sim/endpoint.ts ·
public/retry/index.html