Monitoring, Metrics & Alerting
Continuously measure the health numbers of the system (CPU, latency, errors), view them in graphs, and alert the team as soon as something goes wrong.
44.1 Why is it used?
- The team should know something is breaking before the customer complains.
- On traffic spikes like sale day, capacity and autoscaling decisions are taken from numbers.
- An SLO tells you whether the service is running "good enough" for users.
- If the error rate or latency rises after a deployment, you can roll back immediately.
- The on-call engineer is woken at night only for a real problem, not for noise.
44.2 How it works
- Metrics: time-series numbers like request count, latency, CPU; types: counter, gauge, histogram.
- Pull vs Push: Prometheus scrapes the /metrics endpoint every 15s (pull); the Datadog/CloudWatch agent pushes.
- Four Golden Signals (Google SRE): Latency, Traffic, Errors, Saturation.
- RED method (services): Rate, Errors, Duration. USE method (infra): Utilization, Saturation, Errors.
- SLI: what you measure (like 99th percentile latency). SLO: the internal target (99.9% requests < 300ms). SLA: the legal promise to the customer, with a penalty if broken.
- Error budget: a 99.9% SLO = ~43 min of downtime allowed per month; when the budget is exhausted, freeze new features.
- Alerting: rule (error rate > 5% for 5 min) -> Alertmanager/PagerDuty -> call/SMS to on-call; alert on symptoms, not on causes.
- Percentiles: average latency deceives, look at p95/p99.
- 1.Streaming pods expose a /metrics endpoint (counter, gauge, histogram)
- 2.Prometheus scrapes every 15s (pull) and stores in a time-series DB
- 3.Grafana dashboard: viewers, req/s, p99 latency, 5xx error rate, CPU
- 4.CPU > 80%: HPA autoscale adds new pods; error rate > 2% for 5 min: alert rule fires
- 5.Alertmanager -> PagerDuty -> call / SMS to the on-call SRE
- 6.The SRE shifts that region's traffic to another region
Alert on symptoms (errors, latency), not on causes, or you get alert fatigue.
- 1.Four Golden Signals: Traffic (requests / sec), Latency (p99 < 500ms), Errors (5xx rate), Saturation (CPU, memory)
- 2.SLI: what you measure (p99 latency)
- 3.SLO: the target, 99.9% requests p99 < 500ms
- 4.SLA: customer promise + penalty
- 5.Error budget: 99.9% SLO = ~43 min / month; when the budget is gone, freeze new features
Average latency deceives: always look at p95/p99.
44.3 Tools you can use
| Tool | Type | How it is used |
|---|---|---|
| Prometheus | Open-source metrics DB (CNCF) | Scrapes metrics of services and Kubernetes, PromQL queries, alerts through Alertmanager. |
| Grafana | Open-source dashboards | Builds graphs and dashboards from sources like Prometheus, Loki, CloudWatch, and does alerting too. |
| Datadog | Commercial SaaS observability | Install the agent: metrics, logs, APM, traces all in one UI, with "out-of-box" integrations. |
| AWS CloudWatch | AWS managed monitoring | Automatic metrics for EC2, RDS, ELB, Lambda; CloudWatch Alarms trigger SNS/autoscaling. |
| PagerDuty | Incident alerting / on-call | Alerts go as call/SMS/app notification according to the on-call schedule, with an escalation policy. |
| Azure Monitor / Google Cloud Monitoring | Cloud managed monitoring | Metrics, dashboards and alerts for Azure and GCP resources. |
44.4 Comparison: Prometheus + Grafana vs Datadog vs CloudWatch
| Point | Prometheus + Grafana | Datadog | AWS CloudWatch |
|---|---|---|---|
| Type | Open-source, self-hosted | Commercial SaaS | AWS managed |
| Collection | Pull (scrape /metrics) | Push via agent | Push, AWS services automatic |
| Cost | Free licence, infra + ops cost | Expensive, per host + custom metrics | Pay per metric/alarm, cheap to start on AWS |
| Setup effort | Medium-high, long-term storage (Thanos/Mimir) is separate | Very low | Low |
| Kubernetes support | Best, the de-facto standard | Very good | Container Insights, so-so |
| Coverage | Metrics only (logs/traces need other tools) | Metrics + logs + APM + traces in one place | Metrics + logs + basic traces (X-Ray) |
| Best for | Kubernetes, cost-conscious teams | Fast setup, multi-cloud, all-in-one | AWS-only workloads |
44.5 When to choose which
- Prometheus + Grafana: Kubernetes/microservices, a preference for open source, and the team has ops skills.
- Datadog: you want everything in one tool, quick setup, and you have budget.
- CloudWatch: the default for basic metrics and autoscaling alarms of AWS services (RDS, Lambda, ALB).
- PagerDuty / Opsgenie: for on-call routing and escalation on top of any monitoring tool.
44.6 Real-world example
Such a scenario: monitoring a streaming app during the IPL final
- Before the match the team sets an SLO: 99.9% of video segment requests in under 500ms.
- The Grafana dashboard shows concurrent viewers, requests/sec, p99 latency, 5xx error rate and CPU live.
- At the toss traffic suddenly rises many times over; as soon as CPU saturation crosses 80%, autoscaling adds new pods.
- In one region the error rate stays above 2% for 5 min and the alert rule fires.
- PagerDuty calls the on-call SRE, who looks at the dashboard and shifts that region's traffic to another region.
- After the match the error budget is reviewed and the capacity plan is updated in the postmortem.
44.7 Interview tips
- Alerting on every small thing: with alert fatigue the on-call starts ignoring real alerts.
- Looking at only average latency; p99 can be bad while the average looks fine.
- Putting high-cardinality labels (user_id, order_id) in Prometheus metrics blows up memory.
- Interview tip: explain the difference between SLI, SLO and SLA, with an error budget example.