Load Testing, Performance Testing and Chaos Engineering

Before the real sale arrives, send fake traffic and inject deliberate failures to check how much load and breakage the system can take.

49.1 Why is it used?

• Finding out on sale day that the DB falls over at 20K QPS is far too late; a load test tells you beforehand.
• Finding bottlenecks: a slow query, a small connection pool, too few threads, or network bandwidth.
• Capacity planning gets real numbers: how many req/sec one pod handles.
• Chaos engineering shows whether the system handles a server, AZ or dependency failure gracefully.
• Catch performance regressions after every release (p99 was 200ms before, why 600ms now?).

49.2 How it works

• Load test: run the system at expected traffic (say 50K RPS) and check latency and error rate.
• Stress test: keep raising load until the system breaks, to learn the breaking point and the failure behaviour.
• Spike test: suddenly go from 0 to 10x traffic (sale at 12 midnight or an IPL wicket) and see how fast autoscaling reacts.
• Soak/endurance test: run at normal load for 8-24 hours to catch memory leaks and connection leaks.
• Metrics: throughput (RPS), p50/p95/p99 latency, error rate, CPU/RAM, DB connections; look at p99, not the average.
• Chaos engineering: form a hypothesis ('if one AZ goes down, checkout keeps working'), inject the failure with a small blast radius, and observe.
• Failure types: instance kill, network latency/packet loss, CPU spike, dependency timeout, a whole AZ/region down (game day).

49.3 Tools

49.4 Comparison: JMeter vs k6 vs Gatling vs Locust

49.5 When to choose which

• k6: the default for modern teams; a small perf test on every release in CI, and the pipeline goes red if thresholds fail.
• JMeter: when the QA team wants a GUI or needs to test non-HTTP protocols such as JDBC/JMS.
• Gatling: when you need to generate a very large load from one or two machines and want detailed reports.
• Locust: when you have a Python team and need to code a complex user journey (login -> search -> cart -> pay).
• Chaos: on AWS start with FIS; on Kubernetes use Chaos Mesh/Litmus; staging first, then controlled prod.

49.6 Real-world example

Netflix Chaos Monkey and a pre-sale load test (an illustrative scenario).

1. Netflix publicly built Chaos Monkey, which kills random instances in production so engineers build failure-tolerant services.
2. Using the same idea, an e-commerce team plans a 'game day' 3 weeks before the sale.
3. The last sale peaked at 80K RPS; the target is 2x = 160K RPS, with a real user journey in the k6 scripts (browse 70%, search 20%, checkout 10%).
4. Distributed k6 workers send load from multiple regions; at 100K RPS the DB connection pool fills up and p99 latency hits 3 sec.
5. Fix: a bigger connection pool, RDS Proxy, and hot product pages in Redis cache; on retest p99 is 300ms at 170K RPS.
6. Then AWS FIS shuts down the EC2/EKS nodes of one AZ; the load balancer sends traffic to the other 2 AZs and the error rate only reaches 0.2%.
7. 2 sec of latency is injected into the payment gateway; the circuit breaker falls back on timeout and the whole checkout does not hang.
8. All findings go into the runbook and sale day runs smoothly.

49.7 Interview tips

• Explain the difference between load, stress, spike and soak tests, and the chaos concept of 'hypothesis + blast radius'.
• Common mistake: the load generator machine itself becomes the bottleneck (CPU 100%) and you wrongly blame the server for being slow.
• Testing only one URL; if you do not test the real traffic mix and cache-miss scenarios, the result is false.
• Looking at average latency instead of p99, and running chaos in prod without a stop-condition or rollback plan. Number to quote: target 2x the last peak (80K -> 160K RPS).