Backpressure, Load Shedding & Graceful Degradation

When traffic exceeds capacity, instead of the whole system crashing, reject some requests or switch off features so the core keeps running.

Every system has a max capacity; beyond it latency rises, queues fill up and finally it crashes.
On an IPL final or a sale launch, traffic can be 10-50x normal, and keeping that much capacity every day is expensive.
Showing some users "Please try again" quickly is better than the site being down for everyone.
By switching off non-critical features (recommendations, reviews) you can protect the critical flows (login, checkout, video play).
Backpressure slows the producer so a fast producer does not overwhelm a slow consumer.

Backpressure: the consumer signals its capacity; bounded queues, TCP flow control, Reactive Streams request(n), the Kafka consumer pull model.
Load shedding: on overload, reject extra requests immediately with 503 / 429, e.g. CPU > 85% or queue length > limit.
Priority shedding: drop low-priority traffic first (bots, prefetch, analytics) and protect checkout and payment until the end.
Concurrency limits / adaptive limits: a service takes at most N in-flight requests at a time, and N adjusts automatically based on latency.
Graceful degradation: reduced experience when a dependency fails or on overload, e.g. a static cached page, low video quality, hidden recommendations.
Feature kill switch: turn a heavy feature OFF instantly at runtime with a feature flag, without a deploy.
Virtual waiting room: keep users on a queue page and send them to the site at a controlled rate (e.g. 5000 per minute).
Combine with autoscaling: scaling takes minutes, and until then shedding keeps the system alive.

Hotstar IPL final: the spike went above capacity

1.Crores of viewers, a 10-50x spike when Dhoni bats
2.Envoy / Nginx: shed extra requests at max in-flight or CPU > 85% / queue > limit, 429 / 503 + Retry-After (cheap reject)
3.Priority order: Video playback and Login / auth stay ON (CRITICAL); bots, prefetch, analytics are dropped first
4.LaunchDarkly feature flag kill switch: Recommendations and Social features OFF at runtime, without a deploy
5.Adaptive bitrate: video 1080p -> 480p, CDN bandwidth saved, the stream keeps running

K8s HPA autoscaling takes minutes; until then shedding + kill switch keep the system alive. After the spike the flags go back ON.

Three approaches: slow down vs drop vs reduce

1.Backpressure (slow down): Producer -> bounded queue -> Consumer; wait via request(n)
2.Load shedding (drop): 100 requests arrive, 80 served, 20 get 503; reject is cheap and done first
3.Graceful degradation (reduce): full page -> degraded page; HD video -> 480p, recommendations / reviews hidden, cached / static fallback

Do not keep unbounded queues: memory fills up and it crashes. The core flows (login, checkout, play) survive.

Tool	Type	How it is used
Queue-it / Cloudflare Waiting Room / Akamai Visitor Prioritization	Virtual waiting room	Queue traffic at the edge and send users to the origin at a fixed rate.
LaunchDarkly / Unleash / AWS AppConfig	Feature flag / kill switch	Turn features ON/OFF or roll out by percentage at runtime; switch off heavy features on overload.
Envoy / Istio / Nginx	Proxy / service mesh	Load shedding at proxy level with max connections, max pending requests and rate limits.
Netflix concurrency-limits / Resilience4j Bulkhead	Library	Overload control inside a service with adaptive concurrency limits and bulkheads.
Kafka / RabbitMQ / Reactor (Reactive Streams)	Messaging / reactive	Natural backpressure through pull-based consumption, prefetch limits and request(n).
Kubernetes HPA / AWS Auto Scaling	Autoscaling	Add pods/instances as load grows; long-term capacity alongside shedding.

Point	Virtual Waiting Room	Load Shedding (Envoy/Nginx)	Feature Kill Switch
Where it sits	Edge / CDN, outside the site	At the proxy or inside the service	A flag check in app code
User experience	Sees queue position, enters later	Some users get an error / retry	Site works, some features vanish
Reaction speed	Best for pre-planned events	Automatic, milliseconds	Manual or auto, seconds
Fairness	FIFO or randomized, fair	Random, unfair	Same for all users
Protects	The whole site / origin	A specific service	A heavy dependency / DB
Best for	Sale launch, ticket booking	Unexpected spikes	Optional features like recommendations, search, reviews

Waiting room: when you know when the spike will come, e.g. a 12 o clock sale or Tatkal at 10.
Proxy load shedding + concurrency limits: a default safety net in every service, for unexpected spikes.
Kill switch: when you must switch off a heavy feature immediately to save the core flow.
Backpressure: in async pipelines (Kafka, streams) where the producer is faster than the consumer.

Real world: Hotstar IPL final traffic spike

On Hotstar (JioHotstar) concurrent viewers have reached crores during IPL matches, and a sudden spike comes when Dhoni bats.
The team plans capacity and pre-scales in advance because autoscaling is too slow for such a spike (according to public talks).
Such a scenario: if the spike goes above capacity, non-critical features like personalized recommendations or social features are turned OFF via the kill switch.
Core flows like video playback and login get priority, and load shedding applies to the other APIs.
Adaptive bitrate lowers video quality a little to save CDN bandwidth, and the stream does not stop.
When the spike ends, the flags go back ON and users get the full experience again.

Unbounded queues: memory fills up and it crashes, and latency grows so much that the responses become useless.
Doing expensive work even while rejecting (DB call, logging); shedding must be cheap and happen first.
Not sending Retry-After with a 503, so everyone retries immediately and adds more load.
Interview tip: clearly state the difference between backpressure (slow down), load shedding (drop) and degradation (reduce).