Idempotency and Exactly-once Processing

Whether the same request arrives once or 10 times, the result is applied only once; that is idempotency.

34.1 Why is it used?

After a network timeout the client does not know whether the request reached the server, so it retries.
Without idempotency, a retry can cause a double payment, a double order or a duplicate SMS.
Queues like Kafka and SQS give at-least-once delivery, meaning the same message can reach the consumer twice.
A user's double-click or an app's auto-retry also sends a duplicate request.
Exactly-once is practically impossible at the network level, so at-least-once delivery + idempotent processing = effectively exactly-once.

34.2 How it works

Idempotency Key: the client generates a UUID for every unique operation and sends it in a header, for example Idempotency-Key: 7f3a...; the same key is used on retry.
The server checks the key first: if not found, process and store the response; if found, return the stored response.
Dedup table: a DB table (idempotency_key PRIMARY KEY, status, response, created_at). The unique constraint blocks a duplicate insert.
In-progress lock: if 2 parallel requests arrive with the same key, give the second a 409 Conflict until the first completes.
Naturally idempotent operations: SET balance=100, DELETE and PUT are idempotent; balance=balance+100 and POST are not.
Consumer dedup: save the message_id in a processed_messages table in the same DB transaction as the business update.
Kafka exactly-once: an idempotent producer (enable.idempotence=true) + transactions give exactly-once semantics in a Kafka-to-Kafka pipeline.
TTL: expire idempotency keys after 24 hours or a few days; Stripe keeps keys for at least 24 hours.

Payment retry: same key, only one charge

1.Swiggy app: Pay Rs 450, key = abc-123 (a UUID, generated on the client side)
2.Order Backend: POST /charges, header Idempotency-Key: abc-123 -> Payment Provider (Stripe API)
3.The provider charges the card, but a timeout occurs while the response is coming back
4.The backend retries with the same key abc-123; the provider checks its idempotency_keys table (abc-123 -> success, TTL 24h)
5.Key found -> the old response is returned and NO new charge is created; result: money deducted once, one order created (same key + different amount -> Stripe error)
6.Inside the server: Redis SET key NX EX 86400 (fast filter, the first writer wins) -> DB transaction (BEGIN ... COMMIT): INSERT idempotency_keys UNIQUE + INSERT order + charge + save response
7.Duplicate key: status done -> stored response 200; in-progress -> 409 Conflict until the first one completes

at-least-once delivery (Kafka / SQS) + idempotent consumer (save message_id in processed_messages in the same DB transaction) = effectively exactly-once

34.3 Tools

Tool	Type	How it is used
Redis (SET NX EX)	In-memory store	A fast dedup check with SET key value NX EX 86400; the first writer wins, the rest are treated as duplicates.
PostgreSQL / MySQL unique constraint	Relational DB	A UNIQUE index on the idempotency_keys table; the business row and the key are inserted in one transaction, the strongest guarantee.
Apache Kafka (idempotent producer + transactions)	Event streaming	Prevents duplicate writes from the producer and makes read-process-write atomic.
AWS SQS FIFO	Managed queue	Drops duplicate messages within a 5-minute window using MessageDeduplicationId.
DynamoDB conditional write	NoSQL DB	PutItem with the attribute_not_exists(pk) condition; on a duplicate you get ConditionalCheckFailed.
Stripe API Idempotency-Key	Payment API pattern	Stripe accepts an Idempotency-Key header on POST requests and returns the same result for the same key.

34.4 Comparison: Redis SETNX vs DB Unique Constraint vs SQS FIFO dedup

Point	Redis SET NX	DB Unique Constraint	SQS FIFO Dedup
Speed	Sub-millisecond	A few milliseconds	With queue latency
Durability	Memory; keys can be lost on failover	Durable on disk	AWS managed, durable
Atomic with business data	No, it is a separate system	Yes, same transaction	No, only at message level
Dedup window	As long as the TTL you set	Until the row is deleted	Fixed 5 minutes
Scaling	Easy with a cluster	DB write load increases	FIFO throughput limits apply
Best for	Fast first-level filter, API gateway	Payments, orders where correctness is critical	Stopping duplicate sends on the producer side

34.5 When to choose which

DB unique constraint: money, orders, ledger where you need 100% correctness.
Redis SET NX: a fast duplicate filter in a high-traffic API, the first layer in front of the DB check.
SQS FIFO / Kafka idempotent producer: to reduce duplicates at the queue level; still keep consumer dedup.
Best combo: client idempotency key + Redis fast check + DB unique constraint as the final guard.

34.6 Real-world example

Real world: a Stripe-style payment retry

In a Swiggy-like app the user taps Pay Rs 450, and the app creates a UUID idempotency key.
The backend calls POST /charges with the header Idempotency-Key: abc-123.
The payment provider charges the card, but a network timeout occurs while the response is coming back.
The backend does not know whether the charge happened, so it retries with the same key abc-123.
The provider sees that abc-123 was already processed, so it does not create a new charge and returns the old success response.
The user's money was deducted only once, and only one order was created.
If the retry sends a different amount with the same key, Stripe returns an error, because a key is bound to one specific request.

34.7 Interview tips

Generating the key on the server: a retry creates a new key and dedup fails. The key must be generated on the client side.
Doing the business action first and saving the key later: a crash in between causes a duplicate. Do both in the same transaction.
Assuming Kafka exactly-once is end-to-end: for an external DB or API call you still need consumer-side idempotency.
Interview tip: remember this line: 'at-least-once + idempotent consumer = effectively exactly-once'.