Event Streaming with Kafka
Kafka is a distributed, replicated commit log where events are saved on disk in order and many consumers read at their own speed.
26.1 Why is it used?
- Companies like Netflix, Uber and LinkedIn handle trillions of events (clicks, logs, location) every day - an ordinary queue cannot take that load.
- One event is needed by multiple systems (analytics, search, fraud, billing), and all of them read at their own speed - in Kafka every consumer group is independent.
- Data retention and replay: after fixing a bug you can reprocess the last 3 days of data.
- The backbone of database change data capture (CDC), event sourcing and real-time analytics pipelines.
26.2 How it works
- Topic: a category of events, e.g. 'orders', 'payments'. The producer writes to a topic and the consumer reads from it.
- Partition: a topic is split into N partitions; each partition is an ordered append-only log. Parallelism = the number of partitions.
- Key based partitioning: the same key (e.g. orderId/userId) always goes to the same partition, so the order for that key is maintained.
- Offset: the serial number of each message in a partition; the consumer commits its last offset and continues from there after a restart.
- Consumer group: within a group each partition goes to only one consumer; 12 partitions = at most 12 active consumers per group. A different group = a separate full copy.
- Replication: each partition has one leader and followers (replication factor 3); acks=all + min.insync.replicas=2 prevents data loss.
- Retention: deletion by time (e.g. 7 days) or size; with log compaction only the latest value of each key is kept.
- KRaft: in newer Kafka (production ready from 3.3, ZooKeeper removed in 4.0) Kafka manages its metadata itself using Raft, with no need for ZooKeeper.
- 1.Player apps: 2-3 crore viewers, heartbeat + quality events
- 2.Ingestion API producer: acks = all, partition chosen by key = deviceId
- 3.Kafka cluster (KRaft): 200 partitions P0..P199, RF = 3, append-only log, offsets 0,1,2..., retention 3 days; Broker 1 (P0 leader), Broker 2 / Broker 3 (P0 followers)
- 4.Group qoe-analytics: Flink, buffering rate, switch CDN in regions with high buffering
- 5.Group ads-billing: same events, its own offset, ad impressions
- 6.A different group = a full copy, they do not slow each other down
- 1.P0, P1: Consumer 1 (2 partitions)
- 2.P2, P3: Consumer 2 (2 partitions)
- 3.P4, P5: Consumer 3 (2 partitions)
- 4.Consumer 7: idle, got no partition
- 5.A partition goes to only 1 consumer per group; max consumers = partitions
- 1.Log offsets 0 1 2 3 4 5 6 7 8 9; committed = 6, latest = 9, lag = 3
- 2.Lag rises: K8s starts more consumers, up to 200 (partition count), beyond that they sit idle
- 3.A bug is found in billing: deploy the fix -> move ads-billing's offset back (replay: offset reset)
- 4.Reprocess the 3 days of retained data
26.3 Tools you can use
| Tool | Type | How it is used |
|---|---|---|
| Apache Kafka | Open-source event streaming | A self-managed cluster; DB/S3 integration with Kafka Connect and stream processing with Kafka Streams. |
| Confluent Cloud / AWS MSK | Managed Kafka | Run Kafka without managing brokers; MSK sits in an AWS VPC, Confluent is multi-cloud with Schema Registry. |
| AWS Kinesis Data Streams | Managed streaming (AWS) | Data in shards; easy integration with Lambda/Firehose, default 24 hours retention (extendable up to 365 days). |
| Apache Pulsar | Open-source streaming + queuing | Compute (brokers) and storage (BookKeeper) are separate; multi-tenancy, geo-replication and tiered storage built in. |
| Azure Event Hubs | Managed streaming (Azure) | Partitions + consumer groups, a Kafka-protocol-compatible endpoint - Kafka apps can run without code changes. |
| Apache Flink / Spark Streaming | Stream processing | Read events from Kafka for real-time aggregation, windowing and fraud detection. |
26.4 Comparison: Kafka vs Kinesis vs Pulsar vs Azure Event Hubs
| Point | Kafka | AWS Kinesis | Apache Pulsar | Azure Event Hubs |
|---|---|---|---|---|
| Hosting | Self-managed or MSK/Confluent | Fully managed AWS | Self-managed or StreamNative cloud | Fully managed Azure |
| Scaling unit | Partitions + brokers | Shards (1 MB/s or 1000 rec/s write per shard) or on-demand mode | Partitions; storage scales separately | Partitions + throughput/processing units |
| Retention | Configurable, from days to forever (tiered storage) | 24 hours default, up to 365 days | Configurable + tiered storage (S3) built in | 1 day default, up to 90 days depending on tier |
| Throughput | Highest, millions of msg/sec | Depends on shards, good | Comparable to Kafka, high | High, depends on units |
| Ecosystem | Largest: Connect, Streams, ksqlDB, Schema Registry | Tight with AWS services (Lambda, Firehose) | Smaller but growing, Functions built in | Azure services + Kafka API compatible |
| Ops effort | High (lower with MSK) | Very low | High, more components | Very low |
| Best for | Large scale, multi-cloud, rich ecosystem | AWS-only streaming, quick setup | Multi-tenant, geo-replication, both queue + stream | Azure telemetry/IoT ingestion |
26.5 When to choose which
- Kafka (or MSK/Confluent): you need large scale, a rich ecosystem and cloud independence.
- Kinesis: you are on AWS with a small team and want to send straight to S3/Redshift through Lambda/Firehose.
- Pulsar: a multi-tenant platform, built-in geo-replication, or both queue + stream in one system.
- Azure Event Hubs: telemetry/IoT ingestion on Azure, and you want to reuse existing Kafka clients.
26.6 Real-world example
A hypothetical scenario: clickstream of a live cricket streaming app (IPL final)
- At the IPL final there are 2-3 crore concurrent viewers; every player app sends a 'heartbeat' and quality events every few seconds.
- Events go through the ingestion API into the Kafka topic 'player-events', which has 200 partitions, key = deviceId.
- Replication factor 3, acks=all - data stays safe even if one broker goes down.
- Consumer group 'qoe-analytics' (Flink) computes the buffering rate in real time; CDN is switched in regions with high buffering.
- Consumer group 'ads-billing' separately reads the same events and counts ad impressions.
- When consumer lag (latest offset - committed offset) rises, Kubernetes starts more consumers, up to 200 (the partition count).
- Retention is 3 days; when a bug was found in billing, they fixed it, moved the offset back and replayed the data.
26.7 Interview tips
- Common mistake: keeping too few partitions - adding consumers later will not raise parallelism; increasing partitions also changes the key mapping.
- Common mistake: expecting global ordering - Kafka orders only within a partition; choose the right key if you need order. A hot partition is also a problem - one very popular key (e.g. a celebrity userId) puts all the load on one partition.
- Interview tip: be sure to talk about consumer lag, acks=all, idempotent producer and consumer group rebalancing.
- Worth quoting: replication factor 3 + acks=all + min.insync.replicas=2; max consumers = partitions (e.g. 200); lag = latest offset - committed offset.