Retry & dead letter
When one source goes down mid-cycle, does the pipeline degrade rather than fail, recover without intervention, and avoid double-applying anything it already did?
- success
- a failing source costs its own rows, not the whole cycle
- failure
- anything lost silently, or anything applied twice
- constraint
- the entire retry budget must fit inside a 60-second refresh cycle
Two systems, two completely different costs of failure
CryptoFundingWatch refreshes five venues on a fixed cycle. When one returns 503 mid-cycle there are three options, and two of them are wrong. Failing the whole cycle throws away four good venues because of one bad one. Retrying inline until it works blows the cycle budget — and a refresh that takes longer than its own interval is a system that falls progressively further behind and never catches up.
The third option is to serve four fresh rows and one stale-flagged, and let the next cycle try again. Which produces the constraint this bench actually exists to test: the retry budget is bounded by the refresh interval. Back off too politely and you miss the cycle; give up too fast and you drop a venue that was one attempt from responding.
LatticePulse inverts the cost. Polling failures are cheap there for the same reason — the next poll re-fetches, nothing is lost. But an alert is not a poll. It happens once, it is the product, and both failure directions hurt: a dropped alert is the thing the user pays for, and a duplicate alert at 3am is how you lose them. That asymmetry is why dead-lettering and idempotency matter enormously on the alert path and barely at all on the ingest path.
The dangerous case for both is the timeout. A request that never returns tells you nothing about whether it landed. Retrying might duplicate; not retrying might drop. Without an idempotency key there is no correct choice, only two ways to be wrong.
Exponential backoff, with jitter and a budget
Doubling the wait is the easy half. The shaded band is the part that matters at scale — and the dashed line is the part that matters on a clock.
Break it, and watch it recover
The endpoint is real and deployed. Set it to fail, send traffic, and watch backoff, jitter, idempotency keys and the dead-letter queue do their jobs with actual status codes and measured latencies.
down, send a burst, let everything dead-letter. Switch to healthy and replay. Nothing is lost, and the trace shows where the server recognised a key and declined to apply it twice.In flight0
Backing off0
Delivered0
Dead letter0
One row per message. Bars are real measured request durations; thin connectors are the backoff waits. The gaps widen because that is the point.
Send some traffic to populate this.
Building the retry path
Derive the attempt count from the cycle budget
Start from the refresh interval, subtract a safety margin, and work out how many doublings fit. Picking “3 retries” because it sounds reasonable is how a poller ends up taking longer than its own interval.
Set a timeout on every request, without exception
A request with no deadline is a leak — it holds a connection, a worker and a slot in the cycle indefinitely. The timeout counts as a failed attempt, which is exactly what it is.
Jitter the delay, don’t just double it
Equal jitter: wait half the nominal delay plus a random half. Without it, every client that failed during the same outage returns at the same instant and knocks the recovering service over again.
Attach an idempotency key before the first attempt
Not on retry — on the first send. The key has to be stable across all attempts of the same logical operation, which means generating it when the operation is created, not when it is retried.
Honour Retry-After as a floor
When a service tells you when to come back, it knows something you don’t. Take the larger of its number and your own backoff; never the smaller.
Dead-letter with replay, and alert on depth
Once the budget is spent the message goes somewhere inspectable and re-drivable. A queue nobody watches is just a slower way to lose things, so depth is a monitored metric, not a debugging aid.
| Rule | Here | Precedent |
|---|---|---|
| Exponential backoff | 800ms, doubling per attempt | Standard across AWS SDKs, Google APIs, Stripe |
| Equal jitter | random between half and full of nominal | AWS Architecture Blog, “Exponential Backoff and Jitter” |
| Bounded attempts | 4, sized to fit a 60s cycle | Retrying forever is a queue that never drains |
| Idempotency keys | per-message, server no-ops a repeat | Stripe’s Idempotency-Key header |
Honour Retry-After | sent on 503, respected as a floor | RFC 9110 §10.2.3 |
| Dead-letter with replay | held and re-drivable | AWS SQS redrive |
| Timeouts are failures | 2s client timeout, counts as an attempt | A request with no deadline is a leak |
Where this runs
CryptoFundingWatch
Five venues on a fixed cycle. The retry budget is sized to fit inside the interval, and a venue that exhausts it is stale-flagged rather than dropped — the next cycle re-fetches it anyway. Dead-lettering is almost redundant here, which is itself the finding.
Cost of failure: low — data is re-fetchable
LatticePulse
An alert fires once. Dropping it loses the product; sending it twice at 3am loses the user. Both directions are expensive, so the alert path needs the full apparatus — idempotency key, dead-letter queue, monitored depth — that the ingest path barely uses.
Cost of failure: high in both directions
What the bench showed
In a polling system, the next cycle is the real retry. This was the finding that surprised me, and it only became visible by running the same bench against both systems. CryptoFundingWatch re-fetches everything every cycle, so an in-cycle retry only needs to cover a transient blip — four attempts, under six seconds, roughly 9% of the interval. Retrying harder than that buys nothing, because a genuine outage will still be an outage sixty seconds later, and the cycle will handle it. Dead-lettering an ingest fetch is close to pointless; stale-flagging it is the correct answer.
Alerts invert every one of those conclusions. An alert is one-shot, so there is no next cycle to save you, and the full apparatus earns its keep. The general lesson is that “add retry logic” is not a decision — the right retry policy depends entirely on whether the operation is repeatable and what each failure direction costs. The same codebase needs two different answers.
Retrying immediately makes an outage worse, and synchronised retries cause a second one. A 503 usually means the downstream is already overloaded; the delay is the mechanism that lets it recover. And if every client backs off by exactly one second, they all return at the same instant — jitter turns a spike back into a spread. Neither is subtle once stated, and both are routinely omitted.
A timeout is the case that quietly corrupts data. You cannot know whether the write landed, so retrying may duplicate and not retrying may drop. The idempotency key is what converts that into a decidable question — press slow then replay in the bench above and watch the server decline to apply the same key twice.
Bench source: functions/api/sim/endpoint.ts ·
public/retry/index.html
Related: bench 01 — auth & token lifecycle