Rate Limiting and Throttling
Rate limiting puts a limit on how many requests one user/IP/API key can send in a time window. If it is exceeded, the response is 429 Too Many Requests.
5.1 Why is it used?
- If a single user sends 1000 requests in 1 second the server can go down, which is why rate limiting is applied.
- It stops brute-force logins, OTP spam, scraping and small DDoS attacks.
- Fair usage: one big client should not eat all the resources, so other users also get service.
- Paid API plans: free tier 100 req/day, premium 10,000 req/day. Enforcing the quota.
- Protecting downstream services (DB, payment gateway) from overload. Throttling controls the load.
5.2 How it works
- 1.10:00 AM: lakhs of requests from users + bots at once
- 2.Edge / WAF (Cloudflare / AWS WAF): IP limit, more than 20 req/s from one IP means Block / CAPTCHA
- 3.API Gateway: token bucket per user, 5 search/sec, burst 10; if the limit is crossed, 429 Too Many Requests + Retry-After: 5
- 4.Login / Search API: 5 wrong passwords = 15 min lock (423 Locked, TTL 900s); counters live in Redis (INCR user:42 + EXPIRE 60) so all servers see the same count
- 5.Payment service: only X TPS; when TPS is full, wait in the waiting room queue
- 6.If the client got a 429 it retries with exponential backoff: 5s, 10s, 20s...
The limit is applied in layers, not in one place: edge, gateway, app, and payment
- 1.Token Bucket (API Gateway, Envoy): bucket max 10, refills 10 tokens/sec, 1 req = 1 token; if a token is there it is OK, if none are left 429; bursts are allowed as long as tokens remain
- 2.Leaky Bucket (Nginx limit_req): requests sit in a fixed-size queue and go to the server at a rate of 10 r/s; smooth output, no burst; when the queue is full a new request is dropped
- 3.Fixed Window Counter: 12:00-12:01 max 100, 12:01-12:02 max 100; simple, but 200 requests can pass within 2 seconds around the boundary (2x burst problem)
- 4.Sliding Window Counter (popular): previous minute 80 req, current minute 30 req, 25% overlap -> 80 x 25% + 30 = 50, limit 100 -> OK; accurate + low memory
- Token Bucket: tokens fill the bucket at a fixed rate (10/sec), each request takes one token; bursts are allowed as long as tokens remain.
- Leaky Bucket: requests enter a queue and leave at a fixed rate. Smooth output, no burst; when the queue is full, requests are dropped.
- Fixed Window Counter: a counter for each minute (max 100 in 12:00-12:01). Simple, but a 2x burst can happen at the window boundary.
- Sliding Window Log: store each request's timestamp and count the last 60 sec. Accurate but uses more memory.
- Sliding Window Counter: a weighted count of the current + previous window. Accurate and low memory, very popular.
- Key on what: user ID, API key, IP address, endpoint. For example strict on /login, relaxed on /products.
- Distributed limit: with many servers keep the counter in a shared store (Redis), made atomic with INCR + EXPIRE or a Lua script.
- Response: HTTP 429 + headers like Retry-After and X-RateLimit-Remaining; the client retries with exponential backoff.
5.3 Which tools can be used
| Tool | Type | How it is used |
|---|---|---|
| Redis | In-memory store | A per-user counter with INCR + EXPIRE or a Lua script; all app servers share the same counter. |
| Nginx limit_req | Web server module | limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s; leaky bucket style, with a burst option. |
| AWS API Gateway | Managed gateway | Account/stage/method level throttling (token bucket) and per-API-key quota through usage plans. |
| Envoy | L7 proxy | A local rate limit filter (per-instance token bucket) and a global rate limit service (Redis backed). |
| Kong rate-limiting plugin | API Gateway plugin | Per consumer/IP limit, with a local/cluster/redis policy. |
| Cloudflare / AWS WAF rate rules | Edge rate limiting | IP-based rate rules right at the edge. Blocks the attack before it reaches the origin. |
5.4 Comparison: Redis vs Nginx vs API Gateway vs Envoy
| Point | Redis (custom) | Nginx limit_req | AWS API Gateway | Envoy |
|---|---|---|---|---|
| Where it sits | Inside the app code | At the proxy layer | On the managed gateway | On the proxy / service mesh |
| Speed | Sub-millisecond, but one extra network call | Very fast, local memory | Fast, managed | Local filter is very fast, global needs one extra call |
| Distributed (shared count across all servers) | Yes, by design | No, each Nginx has its own count | Yes, AWS manages it | Yes, through the global rate limit service |
| Flexibility | Highest: any logic/key | Limited: IP/variable based | API key, stage, method level | Flexible through descriptors |
| Algorithm | Whatever you want (token, sliding window) | Leaky bucket | Token bucket | Token bucket |
| Complexity | You must write code | 2 lines of config | A setting in the console | Needs mesh/config knowledge |
| Best for | Business rules (per-user OTP limit) | Simple IP-based protection | Serverless/public APIs with quotas | Microservices on Kubernetes |
5.5 When to choose which
- Redis: when you need a shared counter across multiple servers and have custom business logic (per user per action).
- Nginx limit_req: when you need quick IP-based protection, for one or two servers.
- API Gateway: public APIs with API keys, usage plans and paid tiers.
- Envoy: service-to-service limits in Kubernetes/service mesh.
- A layered approach is best: IP limit at the edge (Cloudflare/WAF) + user limit at the gateway + business limit in the app.
5.6 Real-world example
Scenario: IRCTC Tatkal booking at 10 AM
- The moment Tatkal opens at 10 in the morning, lakhs of users and bots log in and search at the same time.
- A rule at the Edge/WAF: more than 20 requests in 1 second from one IP -> block or CAPTCHA.
- A per-user token bucket at the gateway: each user gets 5 searches/second, burst 10.
- A strict limit on login: after 5 wrong passwords, a 15 minute lock (Redis counter with TTL 900s).
- Throttling on the booking API: the payment service can handle only X TPS, so extra requests go to a queue/waiting room.
- Anyone who crosses the limit gets HTTP 429 and 'Retry-After: 5'; the app retries after 5 seconds.
- Result: the system keeps working for genuine users, and the unfair advantage of bots and agents shrinks.
5.7 Interview tips
- Explain the difference between token bucket, leaky bucket and sliding window with a diagram, and mention 429 + Retry-After.
- Mistake: limiting only by IP. Thousands of users behind an office/college NAT share one IP, and you will block them by mistake.
- Mistake: keeping a local counter on each server when you have 10 servers. The actual limit becomes 10x. Use a shared Redis.
- What if Redis goes down? Decide beforehand whether to fail-open (allow) or fail-closed (deny).
- Number: a fixed window can allow a 2x burst at the boundary (200 requests on a limit of 100); a sliding window counter avoids this.