Pub/Sub vs Message Queue vs Event Streaming
In a queue one message goes to one worker, in Pub/Sub every subscriber gets a copy, and in Event Streaming messages stay saved in a log that anyone can replay at any time.
25.1 Why is it used?
- One event (order placed) is needed by email, inventory, analytics, fraud - all services; a queue gives it to only one, pub/sub/streaming gives it to all.
- Choose the wrong tool and you either lose data (Redis Pub/Sub) or get pointless complexity and cost (Kafka for a small job).
- In some use cases re-reading old data (replay) is essential - for example a new analytics service that must process the last 7 days of data.
- The most common interview question: 'When will you use Redis vs Kafka vs RabbitMQ?' - this topic clears up exactly that.
25.2 How it works
- Message Queue (point-to-point): producer -> queue -> one consumer; the message is deleted after the ack. For distributing work (RabbitMQ, SQS).
- Pub/Sub: the publisher sends to a topic and every subscriber gets its own copy (fan-out). The publisher does not know who is listening.
- Event Streaming: events are saved on disk in an append-only log (partition); the consumer remembers its offset and can replay (Kafka).
- Redis Pub/Sub: fire-and-forget - if the subscriber was not connected at that time the message is gone; no persistence or ack.
- Redis Streams: a log-like data type from Redis 5+ - XADD, consumer groups, XACK, a pending list; persistence depends on Redis RDB/AOF.
- Kafka: a distributed commit log - topics, partitions, consumer groups, retention (days/weeks), lakhs of msg/sec throughput.
- RabbitMQ: a smart broker - exchanges (fanout/topic/direct) give both queue and pub/sub patterns; the message is deleted once consumed.
- Cloud Pub/Sub (Google Pub/Sub, AWS SNS): managed fan-out; SNS pushes to SQS/Lambda/HTTP/SMS, and Google Pub/Sub has pull/push subscriptions and up to 7 days of retention.
- 1.Producer: sends a message
- 2.Queue: m3 m2 m1
- 3.Worker 1: only m1; Worker 2: only m2
- 4.Message deleted after ack; 1 msg -> 1 consumer; replay: no
- 1.Publisher: sends a message to a topic
- 2.Topic: order-events: m1
- 3.Subscribers: Email (m1 copy), Inventory (m1 copy), Analytics (offline -> misses it)
- 4.1 msg -> a copy to every subscriber
- 5.Redis: an offline subscriber's msg is lost; SNS: pushes to SQS/Lambda/SMS; replay: no
- 1.Producer: sends an event
- 2.Partition log on disk, 7 days retention: offsets 0 1 2 3 4 5
- 3.new ML service: replay from offset 0
- 4.analytics: offset 2
- 5.payment: offset 5
- 6.Msg stays in the log; each group has its own offset; replay: yes, order within a partition
- 1.OrderPlaced event: Kafka 'orders' (12 partitions, 7 days); Restaurant, Payment, Analytics, Fraud - separate consumer groups
- 2.Invoice PDF + SMS: RabbitMQ task queue, 10 workers, ack, retry on failure -> DLQ
- 3.Rider live location: Redis Pub/Sub, update every 2 sec; WebSocket server -> customer; missing 1 is fine
- 4.payment-success: AWS SNS topic, managed fan-out -> SQS queues + Lambda (loyalty points)
25.3 Tools you can use
| Tool | Type | How it is used |
|---|---|---|
| Redis Pub/Sub & Streams | In-memory data store | Pub/Sub for live chat/notification fan-out; Streams for a light durable queue with consumer groups. |
| Apache Kafka | Distributed event streaming platform | High-throughput event log; clickstream, logs, CDC, order events - multiple teams read the same data independently. |
| RabbitMQ | Message broker (AMQP) | Task queues and routing-based pub/sub; a fanout exchange puts one message into multiple queues. |
| Google Cloud Pub/Sub | Managed global pub/sub (GCP) | Serverless, auto-scale, at-least-once delivery, message retention and seek/replay support. |
| AWS SNS | Managed pub/sub (AWS) | Publish to a topic -> fan-out to SQS, Lambda, HTTP, email, SMS, mobile push; SNS + SQS is a common pattern. |
| AWS Kinesis / Azure Event Hubs | Managed event streaming | A Kafka-like managed stream; Event Hubs also supports the Kafka protocol. |
25.4 Comparison: Redis vs Kafka vs RabbitMQ vs Cloud Pub/Sub
| Point | Redis (Pub/Sub/Streams) | Kafka | RabbitMQ | Cloud Pub/Sub (Google Pub/Sub / AWS SNS) |
|---|---|---|---|---|
| Model | Pub/Sub: fire-and-forget fan-out; Streams: append-only log with consumer groups | Distributed partitioned log (event streaming) | Broker with queues + exchanges (both queue and pub/sub) | Managed topic + subscriptions fan-out |
| Durability | Pub/Sub: none, an offline subscriber misses it; Streams: memory + RDB/AOF | Replicated on disk (replication factor 3 is common), very durable | Durable queue + persistent message on disk | Managed, replicated; Google 7 days default retention, SNS does not store |
| Replay | Pub/Sub: no; Streams: yes, you can re-read by ID | Yes, reset the offset and replay anything within retention | No, the message is deleted after ack (Streams feature in newer versions) | Google: replay via seek/snapshot; SNS: no (archive in SQS) |
| Throughput | Very fast, lakhs of msg/sec, but limited to one node's memory | Highest - millions of msg/sec in a cluster | Good, tens of thousands of msg/sec per node | Auto scale, practically unlimited, the cloud manages it |
| Ordering | Ordered within a single stream in Streams | Strict order within a partition | Order in a single queue, can break with multiple consumers | Google: via ordering key; SNS: via FIFO topic |
| Delivery | Pub/Sub at-most-once; Streams at-least-once (XACK) | At-least-once by default, exactly-once (idempotent producer + transactions) | At-least-once with ack, or at-most-once | At-least-once (Google has an exactly-once option too) |
| Ops complexity | Simplest, free if you already have Redis | Heaviest - managing brokers, partitions, KRaft (or MSK/Confluent) | Medium - cluster, queues, monitoring | Zero ops, fully managed, pay per message |
| Best for | Live chat, notifications, cache invalidation, small queues | Clickstream, logs, analytics, event sourcing, CDC | Task queues, complex routing, RPC-style jobs | Serverless fan-out, cloud-native apps, cross-service events |
25.5 When to choose which
- Redis Pub/Sub: when you need real-time broadcast and missing a message or two is fine (typing indicator, live score).
- Redis Streams: small-medium load, Redis already in place, and you want a simple durable queue with consumer groups.
- Kafka: high-volume events, replay needed, multiple teams will read the same events (analytics + search + fraud).
- RabbitMQ: background jobs, per-message ack/retry, complex routing and low-latency task distribution.
- Google Pub/Sub / AWS SNS: you are on the cloud and want fan-out without managing servers; SNS + SQS for durable fan-out.
25.6 Real-world example
A hypothetical scenario: using all three in a food delivery app
- The user places an order; the Order Service publishes an 'OrderPlaced' event to the Kafka topic 'orders' (12 partitions, 7 days retention).
- Restaurant Service, Payment Service, Analytics and Fraud - each creates its own consumer group and reads the same event independently.
- A new ML recommendation service launches; it sets its offset to the start and replays the last 7 days of orders.
- Generating the invoice PDF and sending SMS goes into a RabbitMQ task queue; 10 workers process with ack, and failures go to the DLQ.
- The delivery partner's live location travels over Redis Pub/Sub to the customer's WebSocket server - if one update is missed the next arrives in 2 sec.
- On the cloud side, 'payment-success' goes to an AWS SNS topic that fans out to SQS queues and Lambda (loyalty points).
- Result: the right tool for each job - durability where needed, speed where needed, and lower cost.
25.7 Interview tips
- Common mistake: treating Redis Pub/Sub as a reliable queue - if the subscriber disconnects the message is lost forever.
- Common mistake: using Kafka like a simple task queue - no per-message retry/DLQ built in, and consumers beyond the partition count sit idle. Setting up a Kafka cluster from day one in a small startup is also wrong - high ops cost; start with SQS/SNS or Redis Streams.
- Interview tip: always state 3 things - does the message remain after being consumed (replay), how many consumers get it, and how ordering is achieved.
- Worth quoting: Kafka replication factor 3, retention 7 days; Google Pub/Sub default retention 7 days; Redis Pub/Sub is at-most-once.