Retry with Exponential Backoff

What is retry with exponential backoff?

When a request fails due to a transient error (network blip, momentary overload, rate limit), retrying immediately is often the right move. But naive retry (retry immediately, every 1 second) causes thundering herd: all clients retry simultaneously, flooding an already-stressed service and preventing recovery.

Exponential backoff solves this by increasing the delay between each retry attempt exponentially, giving the downstream service breathing room.

The Thundering Herd Problem

Imagine 1,000 clients all hitting a service at 09:00:00. The service blips at 09:00:01. If all 1,000 retry at 09:00:02, 09:00:03, 09:00:04... they create a synchronized stampede. The service sees 1,000 requests per second indefinitely until it either recovers or crashes permanently.

Exponential Backoff Formula

delay = min(base * 2^attempt, max_delay)

Example with base=1s, max_delay=60s:

• Attempt 1: wait 1s
• Attempt 2: wait 2s
• Attempt 3: wait 4s
• Attempt 4: wait 8s
• Attempt 5: wait 16s
• Attempt 6: wait 32s
• Attempt 7: wait 60s (capped)

Adding Jitter

Even with exponential backoff, if all clients start at the same time, they still retry in lockstep (all wait 1s, then all wait 2s). Jitter adds randomness to desynchronise clients.

import random

def backoff_with_jitter(attempt: int, base: float = 1.0, cap: float = 60.0) -> float:
    delay = min(base * (2 ** attempt), cap)
    return random.uniform(0, delay)  # full jitter

Types of jitter:

• Full jitter: random(0, calculated_delay) - best for avoiding thundering herd
• Equal jitter: calculated_delay/2 + random(0, calculated_delay/2) - ensures minimum wait
• Decorrelated jitter: min(cap, random(base, last_delay * 3)) - AWS recommended

Idempotency Requirement

Retrying is only safe if the operation is idempotent - executing it multiple times has the same effect as executing it once.

• ✅ Idempotent: GET requests, PUT with full resource, DELETE
• ❌ Not idempotent by default: POST (creating a resource), charging a payment

Make POSTs idempotent with an idempotency key: client generates a unique key per logical operation; server deduplicates by storing processed keys.

POST /payments
Idempotency-Key: a3f9d2e1-4c8b-...

Max Retries and Deadlines

Set a maximum retry count (e.g., 5) AND a total deadline (e.g., 30 seconds). Without a deadline, exponential backoff can retry forever. The deadline propagates from the upstream caller - if the user's request times out in 10 seconds, there's no point retrying for 30 seconds.

What Errors to Retry

• ✅ 503 Service Unavailable - service overloaded, retry
• ✅ 429 Too Many Requests - rate limited, use Retry-After header
• ✅ 500 Internal Server Error - maybe transient
• ✅ Network timeouts, connection refused
• ❌ 400 Bad Request - client error, retrying won't help
• ❌ 401 Unauthorized - auth error, retrying won't help
• ❌ 404 Not Found - resource doesn't exist, retrying won't help

Circuit Breaker Integration

Retry and circuit breaker work together:

• Retry handles transient single-request failures
• Circuit breaker stops retrying when a service is persistently down

Without a circuit breaker, exponential backoff just delays the inevitable exhaustion of retry budgets for all clients.

Interview Angle

Explain the thundering herd problem first - it shows you understand why naive retry is harmful. Then: exponential backoff + jitter + idempotency keys + max retries + deadline. Most candidates forget jitter and idempotency.