Circuit Breaker Pattern

What is the circuit breaker pattern?

The circuit breaker pattern prevents cascading failures in distributed systems. When a downstream service is failing, continuing to send it requests wastes resources (threads, connections, timeouts) and amplifies the failure. The circuit breaker detects the failure and short-circuits calls - immediately returning an error or fallback instead of waiting for a timeout.

Analogy: a household circuit breaker trips when current exceeds safe levels, protecting the wiring. Similarly, a software circuit breaker trips when error rates exceed a threshold, protecting the system.

States

Closed (Normal Operation)

• Requests flow through to the downstream service
• Failures are counted within a rolling time window
• If failure rate exceeds threshold (e.g., > 50% in 10 seconds), transition to Open

Open (Failure Mode)

• All requests are immediately rejected without calling the downstream service
• Caller receives an error or fallback response instantly
• After a configured timeout (e.g., 30 seconds), transition to Half-Open

Half-Open (Recovery Probe)

• Allow a limited number of probe requests through
• If probes succeed -> transition to Closed (service has recovered)
• If probes fail -> transition back to Open (service still failing)

[Closed] ─── failure threshold exceeded ───► [Open]
   ▲                                             │
   │ probe succeeds                    timeout   │
   └──────────── [Half-Open] ◄──────────────────┘

Failure Threshold Configuration

• Error rate threshold: 50% failures in a rolling window triggers open
• Minimum request volume: Don't open circuit on 1 failure out of 2 requests - require at least 20 requests
• Slow call threshold: Treat requests taking > 2s as failures (timeouts count)
• Recovery timeout: How long to stay Open before probing

Fallback Responses

When the circuit is Open, you need something to return. Options:

• Cached data: Return last known good response
• Default/degraded response: Return a safe default (empty list instead of personalised recommendations)
• Error to client: Fail fast with a meaningful error (better than waiting for timeout)
• Queue for later: Enqueue the request and process when service recovers

Half-Open Concurrency Control

In Half-Open state, only allow N concurrent probe requests (e.g., 5). If all 5 succeed, close the circuit. Using semaphores prevents flooding a recovering service.

Implementations

• Java: Netflix Hystrix (deprecated), Resilience4j, Spring Cloud Circuit Breaker
• Go: gobreaker library
• Python: pybreaker, circuitbreaker
• Service mesh: Istio/Linkerd handle circuit breaking at the infrastructure level without code changes

Circuit Breaker vs Retry

Retry with backoff and circuit breaker are complementary, not alternatives:

• Retry: try again when failure might be transient (network blip)
• Circuit breaker: stop trying when failure is sustained (service is down)

Stack them: retry 3 times with exponential backoff; if downstream has failed > 50% of calls over 30 seconds, open the circuit and stop retrying entirely.

Interview Angle

Mention circuit breakers when designing any system with external service dependencies. The interviewer wants to hear: state machine (closed/open/half-open), failure threshold, fallback strategy, and the insight that it prevents cascading failures not just individual request failures.