Retry logic looks harmless in isolation. When a gateway, application service and SDK each apply their own retry policy, a single failed operation can multiply into a dependency outage. That interaction is the starting point for my retry-storm analysis.
The problem is not that retrying is always wrong; it is that each layer can add traffic at the worst possible moment.
Suppose an API gateway, service A and service B each allow three attempts. A single end-user operation can trigger up to 27 downstream calls if every layer independently retries a failing dependency. That's a worst-case arithmetic illustration, not measured traffic from this site.
Timeouts complicate mutating requests further. A client can give up before the server commits; sending a fresh operation creates the possibility of a duplicate side effect.
The controls that bound the failure
- One retry owner per operation path. Document which layer is responsible; don't hide nested retries inside SDK defaults.
- Bounded attempts and deadline. Stop retrying when the operation's useful response window has expired.
- Exponential backoff with jitter. Spread retries rather than synchronizing on fixed intervals.
- Idempotency for mutations. Preserve the operation key across retries; don't assume a timeout means the operation failed.
- Overload feedback. Circuit breaking and rate limiting are more useful than blindly increasing request concurrency when a service is unhealthy.
A retry budget is a control-plane decision about reliability and cost. An idempotency key is an application contract about effects. Neither replaces the other.
One reproducible experiment
A useful reproduction uses a synthetic dependency with a fixed failure window, then compares fixed-delay and jittered retries under a known seed. The result is incomplete if it only records eventual success; peak in-flight requests, work amplification and tail latency determine whether a retry policy protects or overloads a service.
The existing idempotency reference test checks a different boundary: duplicate mutation suppression in one process. It should not be used as evidence for distributed retry-storm behavior.
The next step would be a controlled load test: fix the dependency failure window, choose one retry owner, and measure peak in-flight work, tail latency, and total dependency requests across multiple retry policies. No distributed benchmark is claimed here.
A review of the call chain should document:
- Where the gateway or SDK may retry the same operation.
- Which service owns the retry budget and preserves idempotency keys.
- How downstream overload and timeout ambiguity are surfaced to the caller.
Source
AWS Builders' Library: timeouts, retries and backoff with jitter
Scope: Architecture analysis and a proposed load test; no distributed benchmark was run for this note.