Timeouts, Retries, Exponential Backoff and Circuit Breaker

So that one slow or down service does not bring the whole system down, we add timeouts, smart retries and a circuit breaker.

In microservices, when one service becomes slow, its callers' threads run out waiting; this causes cascading failure.
Temporary glitches in the network (packet drop, pod restart) are common; one retry makes the request succeed.
If you retry without backoff, you put more load on an already struggling service (a retry storm) and it goes completely down.
A circuit breaker fails fast: instead of calling a down service again and again, it returns a fallback response immediately.
User experience is better: instead of hanging for 30 seconds, the user gets a fallback or a clear error in 1 second.

Timeout: set a max wait time on every remote call, for example connect timeout 1s, read timeout 2s. Never leave the default infinite timeout.
Retry: retry only on transient errors (timeout, 503, connection reset), not on client errors like 400/401. Max 2-3 attempts.
Exponential Backoff: double the wait between retries, for example 100ms, 200ms, 400ms, so the server gets time to recover.
Jitter: add a random delay to the backoff so thousands of clients do not retry together (this avoids a thundering herd).
Circuit Breaker states: CLOSED (normal), OPEN (failure rate threshold crossed, for example 50%, so calls are rejected immediately), HALF-OPEN (send a few test calls; if they pass, go to CLOSED).
Fallback: when the circuit is open, return cached data, a default value or a graceful message, for example top sellers instead of recommendations.
Bulkhead: a separate thread pool or connection limit per dependency, so one slow service does not eat all the threads.
Retry budget / deadline propagation: pass the total request deadline downstream and limit retries to 10-20% of overall traffic.

Timeout, retry, backoff and circuit breaker: the Netflix home page

1.A The user opens the home page; the API Gateway (timeout 1s, retry 2x + jitter, bulkhead pools) calls profile, ratings, artwork and recommendations
2.The Recommendations DB is slow: response goes from 200ms to 8 sec; without a timeout threads would wait 8 sec, the caller's threads run out, cascading failure
3.B Retry + exponential backoff + jitter: Try 1 (timeout 1s) -> wait 100ms + jitter -> Try 2 fails -> wait 200ms + jitter -> Try 3 fails
4.C Circuit breaker: CLOSED (normal, all calls go through) -> failure rate 50%+ -> OPEN (calls rejected immediately for 30 sec, fallback) -> after 30 sec HALF-OPEN (a few test calls)
5.In HALF-OPEN, 5 test calls pass -> CLOSED, a test fails -> back to OPEN
6.Fallback: cached Trending Now / Top 10; the page loads in ~1 sec

Retry only on transient errors (timeout, 503), not on 400/401. Bulkhead: a separate thread pool per dependency. Tools: Resilience4j, Polly, Istio

Tool	Type	How it is used
Resilience4j	Java library	Applied in Spring Boot apps through CircuitBreaker, Retry, TimeLimiter, Bulkhead annotations. The modern replacement for Hystrix.
Netflix Hystrix	Java library (maintenance mode)	Netflix popularised the circuit breaker pattern. It is now in maintenance mode; new projects use Resilience4j.
Polly	.NET library	Retry, circuit breaker, timeout and fallback policies are defined for .NET apps, together with HttpClientFactory.
Istio / Envoy	Service mesh	Timeout, retries and outlier detection (circuit breaking) at the sidecar proxy level through YAML config, with no app code change.
AWS SDK	Cloud SDK	The AWS SDK has built-in exponential backoff with jitter; it retries automatically on throttling errors. (AWS App Mesh has been retired; look at Istio or Linkerd for a mesh.)
Linkerd	Service mesh	A lightweight mesh that gives retries, timeouts and retry budgets automatically.

Point	Resilience4j	Polly	Istio / Envoy
Where it runs	Inside the app (Java JVM)	Inside the app (.NET)	Sidecar proxy, outside the app
Language	Java / Kotlin only	C# / .NET only	Language independent
Fallback logic	Yes, you can call a custom method	Yes, rich fallback policies	No, it only returns an error/503
Config	Code + application.yml	Policy builder in code	Kubernetes YAML (VirtualService, DestinationRule)
Circuit breaker	Based on failure rate + slow call rate	Based on consecutive / advanced failure	Outlier detection: ejects bad pods
Overhead	Very low, in-process	Very low, in-process	Extra hop + sidecar CPU/memory
Best for	Spring Boot microservices	ASP.NET Core services	Polyglot Kubernetes platform

Resilience4j: when you have Java/Spring services and need business-specific fallback.
Polly: when you have a .NET stack and need retry + breaker with HttpClient.
Istio/Envoy: when services in many languages run on Kubernetes and you need a central, uniform policy.
Best practice: basic timeout/retry from the mesh and fallback logic from an app library; do not double the retries across both layers.

Real world: the Netflix home page when the recommendation service is down

To build the Netflix home page, several microservices are called: user profile, recommendations, ratings, artwork.
Scenario: the recommendation service's DB became slow and the response went from 200ms to 8 seconds.
The API gateway has a 1 second timeout, so the request does not hang for 8 seconds.
It retries 2 times with backoff 100ms, 200ms + jitter, and still fails.
When the failure rate crosses 50%, the circuit breaker goes OPEN; for the next 30 seconds the recommendation service is not called at all.
Fallback: instead of the personalized list the user sees cached Trending Now / Top 10, and the page still loads.
After 30 seconds it goes HALF-OPEN; if 5 test calls pass, the circuit goes CLOSED and the personalized rows come back.
Netflix open-sourced this very idea as the Hystrix library.

Adding retries on every layer (client 3 x gateway 3 x service 3 = 27 calls): this creates a retry storm.
Retrying a non-idempotent call (payment, order create) without an idempotency key: a duplicate order gets created.
Layer your timeouts: the caller above must have a timeout larger than the total time of the service below it (including retries), otherwise the caller gives up first and the downstream work is wasted.
Interview tip: be sure to explain the 3 states of a circuit breaker and why jitter is needed.