Centralized Logging
Collect the logs of all servers and services in one central place so you can search, filter and analyse them.
43.1 Why is it used?
- With 500 servers, SSHing into each server to read logs is impossible; you need one place to search.
- When Kubernetes pods restart their local logs vanish, so shipping them out is essential.
- One user request passes through 10 services; a correlation ID shows the whole journey in a single search.
- For security audit and compliance, logs must be kept safe for months.
- From logs you can also build alerts on error spikes and dashboards.
43.2 How it works
- Pipeline: the app writes logs -> an agent/shipper collects them -> buffer/processing -> storage + index -> search/UI.
- Structured logs: write logs as JSON instead of plain text (timestamp, level, service, user_id, trace_id), which makes filtering easy.
- Correlation ID / Request ID: a unique ID is created at the gateway, and every service forwards it in a header and writes it in its log.
- Log levels: DEBUG, INFO, WARN, ERROR, FATAL; keep DEBUG off in production or cost and noise will grow.
- Agents: Filebeat, Fluent Bit, Fluentd, Promtail/Grafana Alloy: they run on the node and ship logs.
- Indexing: Elasticsearch builds a full-text index of every word (fast search, expensive); Loki indexes only labels (cheap).
- Retention: hot storage 7-15 days, then a cheap S3/Glacier archive; mask PII before logging.
- 1.API Gateway, Cart svc, Payment svc, Order svc: write JSON logs
- 2.Agent on every node (Fluent Bit / Filebeat / Alloy): logs survive a pod restart
- 3.Kafka: buffer, absorbs spikes
- 4.Logstash: parse + PII mask
- 5.Elasticsearch (full-text index) or Loki (labels only)
- 6.Kibana / Grafana: search + dashboard
- 7.Hot 7-15 days, then S3 / Glacier archive
Pod restart = local logs gone, so ship logs out.
- 1.Gateway creates X-Request-ID=abc123 (12:00:01.10 POST /checkout)
- 2.cart-svc: cart locked, total=Rs 450 (12:00:01.18)
- 3.payment-svc: gateway success, payment_id=pay_77 (12:00:01.95)
- 4.order-svc ERROR: DB timeout after 3000ms, order not created (12:00:04.97)
- 5.Searching request_id="abc123" in Kibana shows the whole journey in one place; alert: ERROR count > 100/min
Structured JSON log, e.g. {"ts":..., "level":"ERROR", "service":"order-svc", "request_id":"abc123", "user_id":42}. DEBUG off in production, PII masked.
43.3 Tools you can use
| Tool | Type | How it is used |
|---|---|---|
| ELK / Elastic Stack | Open-source logging stack | Beats/Logstash collect, Elasticsearch stores + indexes, search and dashboards in Kibana. |
| Grafana Loki | Open-source log aggregation | Indexes only labels and stores cheaply in object storage (S3), queried with LogQL in Grafana. |
| Splunk | Commercial log analytics / SIEM | For enterprise logs, powerful SPL queries, security monitoring (SIEM). |
| AWS CloudWatch Logs | AWS managed logging | Lambda, ECS, EC2 logs arrive directly, query with Logs Insights, alarms through metric filters. |
| Fluent Bit / Fluentd | Log shipper (CNCF) | Collects logs from Kubernetes nodes and ships them anywhere: Elasticsearch, Loki, S3, CloudWatch. |
| Azure Monitor Logs / GCP Cloud Logging | Cloud managed logging | Logs of Azure and GCP workloads with central query (KQL / Logs Explorer). |
43.4 Comparison: ELK vs Loki vs Splunk vs CloudWatch Logs
| Point | ELK Stack | Grafana Loki | Splunk | CloudWatch Logs |
|---|---|---|---|---|
| Indexing | Full-text index, everything searchable | Only labels, content is scanned like grep | Full index, very powerful query | Managed, query with Logs Insights |
| Search speed | Very fast | Label filter fast, free-text slow | Very fast | Okay, slow over large ranges |
| Storage cost | High (large index) | Low (S3 object storage) | Very high (ingest licence) | Medium, pay for ingest + storage |
| Ops effort | High, cluster tuning | Low-medium | Low (Cloud) / medium | Zero, fully managed |
| Query language | KQL / Lucene | LogQL | SPL | Logs Insights query |
| Best for | Heavy search, analytics | Kubernetes + Grafana users, cheap | Enterprise, security SIEM | AWS-native apps |
43.5 When to choose which
- ELK / OpenSearch: when you need a lot of free-text search and log analytics and the team can handle a cluster.
- Loki: when you already have Kubernetes + Prometheus + Grafana and want to keep a lot of logs at low cost.
- Splunk: a large enterprise where the security team needs SIEM, and there is compliance and budget.
- CloudWatch Logs: a small-to-medium AWS setup where you want to manage nothing.
43.6 Real-world example
Such a scenario: debugging a failed payment on sale day
- During the sale a customer complains: "payment was deducted, order was not created".
- The support team pulls out the order/request ID, which the API gateway created in the X-Request-ID header.
- An engineer searches request_id=abc123 in Kibana/Grafana.
- In a single query, the JSON logs of the gateway, cart, payment and order services appear in time order.
- It turns out the payment service got success but the order service logged a DB timeout ERROR.
- For this pattern an alert "ERROR count > 100/min" is set and the retry logic is fixed.
43.7 Interview tips
- Writing PII like passwords, card numbers, OTPs or Aadhaar into logs: masking is a must.
- Unstructured text logs and logs without a correlation ID; debugging a distributed system becomes hard.
- Leaving the DEBUG level ON in production makes the log bill and storage suddenly balloon.
- Interview tip: explain logs, metrics, traces: the three pillars of observability and how they differ.