Case Study 8: Notification System (Email, SMS, Push, In-App)

A central platform that sends messages on behalf of all services to the right user, on the right channel, at the right time, without duplicates.

C.1 Requirements

Functional requirements:

• Internal services send notifications through one API (to a single user, in bulk, or to a segment).
• Channels: Email, SMS, Mobile Push (FCM/APNs), In-App inbox.
• Templates with variables and multi-language support (Hindi, English, etc.).
• User preferences: respect which channel, which category and quiet hours (DND).
• Priorities: OTP/transactional high priority, marketing low priority.
• Retry with backoff on failure, and a fallback channel (if push fails, SMS).
• Deduplication: the user must not get two messages for the same event.
• Delivery status tracking: sent, delivered, opened, failed.

Non-functional requirements:

• OTP/transactional notification end-to-end < 5 sec (p99).
• High throughput for marketing campaigns: 1 crore (10 million) messages in an hour.
• At-least-once delivery + dedup = an effectively exactly-once user experience.
• 99.99% availability of the notification ingestion API.
• Vendor rate limits (SMS provider TPS, FCM quota) must never be breached.
• Horizontal scale: every channel worker scales independently.

C.2 Estimation (back-of-the-envelope)

C.3 API design

C.4 Data model

C.5 High-level architecture

• Order, Auth OTP and the Marketing Tool call the Notification API through the API Gateway; the API checks the Idempotency-Key and returns 202 Accepted immediately.
• Separate Kafka topics by priority: P0 OTP, P1 transactional, P2 marketing (Scheduler 10k chunks); the marketing backlog does not delay OTP.
• The Processor reads preference / DND from Redis, dedups with SET NX and renders the message from the versioned MySQL template.
• Channel workers send within the vendor quota using a token bucket; on a failure / 5xx the Retry + DLQ worker takes over.
• Vendor webhooks -> Status Tracker -> Cassandra notification_log, hot for 90 days, then S3.

C.6 Main flows

OTP SMS flow:

1. The Auth Service calls POST /notifications with priority=P0, template=otp_login and an Idempotency-Key.
2. The Notification API checks the key; if it is new, it publishes to the Kafka P0 topic and returns 202.
3. The Processor reads the preference from Redis (OTP is mandatory, DND is skipped).
4. Dedup: SET NX on hash(user+event+channel); if it already exists, drop.
5. Template render: "Your OTP is 123456".
6. The SMS Worker takes a vendor slot from the token bucket and calls the primary SMS vendor.
7. On a vendor error / timeout the circuit breaker switches to the secondary vendor.
8. The status DELIVERED is updated in Cassandra from the delivery report webhook.

Marketing campaign flow:

1. The marketing team schedules a campaign: segment = active users in Delhi, time = 6 PM.
2. The Campaign Scheduler splits the segment into chunks (for example 10,000 users).
3. Every chunk goes to the P2 Kafka topic.
4. The Processor checks every user's preference, quiet hours and frequency cap (for example max 3 promos/day).
5. For allowed users, the push/email workers send at a rate-limited speed.
6. Opens/clicks are tracked and go into the analytics pipeline.

Retry and fallback flow:

1. The Push Worker calls FCM and gets an invalid-token or unregistered error.
2. The invalid token is deleted from the device_tokens table.
3. If it is transactional and the user has no valid device, an SMS fallback event is published.
4. On a 5xx / timeout the message goes to a retry topic: 1 min, 5 min, 30 min backoff.
5. After max attempts: DLQ, an alert and manual review.

C.7 Deep dives

Problem: a marketing blast delays OTP. Separate Kafka topics + separate worker pools per priority. P0 workers are dedicated and over-provisioned. Weighted consumption keeps low priority from starving.

Problem: duplicate notifications (retry or a producer double send). An Idempotency-Key on the producer side, a Redis SET NX dedup key with a TTL on the consumer side. Kafka is at-least-once, so the consumer must be made idempotent.

Problem: vendor rate limits and outages. A distributed token bucket (Redis) for every vendor. Multiple SMS/email vendors with a circuit breaker and weighted routing; if one goes down, traffic moves to the other.

Problem: user preferences and quiet hours on every message. Preferences are cached in Redis and the cache is invalidated on update. During quiet hours low-priority messages are parked in a delayed queue and released according to the user's timezone.

Problem: storing the status of crores of messages. Cassandra with a user_id partition + time clustering, TTL 90 days. Old data is archived in S3, with a data lake for analytics.

C.8 Trade-offs

C.9 Interview tips

• Mention priority separation (OTP vs marketing) right at the start; interviewers check for exactly this.
• Explain dedup and idempotency at both layers: the API and the consumer.
• Rate limiting is of two kinds: one that protects users from spam and one for vendor quota.
• Draw the flow of retry with exponential backoff + DLQ + fallback channel clearly.
• Mentioning template versioning and i18n earns a bonus point.