Idempotency when HTTP timeouts lie about server-side effects

A request timeout is an ambiguous observation, not a database rollback. I built a retry model around a client that misses the response after the server commits an order, because the next decision—retry, reconcile or escalate—is where duplicate writes are created.

The only defensible answer from the client is unknown. An HTTP timeout tells you the client did not receive a satisfactory response before its deadline; it does not roll back the server's transaction. If the client blindly retries, it may create the same order twice.

Take a synthetic scenario with these timing assumptions:

• The client timeout is 350 ms.
• The server commits at 600 ms.
• Response delivery adds 100 ms.
• The caller is configured for two additional retries.

The original request is still running when the client deadline expires. Under the lab's simplified assumption that every attempt reaches the server and eventually commits, three submissions can produce three writes. The client may still report a timeout despite all of them completing.

This is not a performance benchmark or a model of every HTTP stack. Real cancellation, connection state, queues, server concurrency, and late response handling will vary.

Instead of asking the server to execute the same logical action again, the client supplies a unique operation identifier. The same identifier is reused across retries of that one action. For a different action, generate a new key.

A server can associate the key with a fingerprint of the intended request, its processing state, and its final response. When the same request is retried, the server can replay the outcome without repeating the side effect. Reusing a key for a different request body should be rejected.

The emerging Idempotency-Key HTTP header is documented by MDN as non-standard/experimental. Do not assume every service supports that exact header or guarantees identical semantics.

POST /orders
Content-Type: application/json
Idempotency-Key: 1f65cbe7-55de-43c7-a1f2-88c35f470432

{"sku":"book-123","quantity":1}

The header by itself provides zero protection if the backend ignores it or stores keys without an atomic operation claim.

For a system with transactional storage, record the action under a uniqueness constraint scoped appropriately to the tenant/account and operation. The store must arbitrate concurrent arrivals.

CREATE TABLE operation_requests (
  tenant_id text NOT NULL,
  operation_key text NOT NULL,
  request_fingerprint text NOT NULL,
  status text NOT NULL,
  response_json jsonb,
  created_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, operation_key)
);

The table is a schema illustration, not a complete transaction. The implementation still must define how the operation and result are recorded atomically, or reconciled safely after a crash. It needs retention limits, request fingerprinting, replay/ownership checks, concurrency control, and error recovery.

Important cases:

1. First request: Claim the operation key, perform the mutation once, and store its terminal result.
2. Duplicate while processing: Do not execute twice. Return an explicit in-progress response or wait under a bounded policy.
3. Duplicate after success: Return the recorded result, subject to the API's published semantics.
4. Same key with different parameters: Reject the mismatch; never treat it as a new valid request.
5. Expired key: Apply a documented retention and replay policy; avoid silent reuse ambiguity.

For high-value operations, the database transaction and externally visible side effects (such as messaging or payments) may also need an outbox or provider-specific idempotency support. A local table alone cannot roll back a call already accepted by an external system.

Retries have side effects on availability even when logically safe. Limit attempts and use backoff with jitter, timeouts at each dependency, and a total deadline. A service in distress can be made worse by synchronized bursts of retries.

Do not retry every error automatically. A deterministic validation error, authorization failure, or conflicting idempotency key should not get treated like a transient 503.

When a server commits despite the client's timeout, the client may not know whether the operation succeeded. A retry without a stable idempotency key can duplicate the write. Even when the server records only one write, the client may need an operation-status endpoint to resolve the ambiguous outcome.

• Generate a unique key for every new logical mutation.
• Reuse that key across all retries of the same operation.
• Scope the key and request fingerprint to the authenticated tenant.
• Make deduplication atomic across workers, not just in-memory.
• Persist an unambiguous terminal outcome or reconciliation state.
• Test concurrent duplicates and crashes between commit and response.
• Do not claim exactly-once delivery simply because a header exists.

I prepared this reference with AI assistance. The timing and transactions are made-up examples, not measurements from a production incident.