Distributed Locking and Leader Election

To make sure only one of many servers does a job at a time, or only one becomes the leader, we use a distributed lock and leader election.

A mutex works on one server, but for a shared resource across 50 servers you need a network-wide lock.
If a cron job runs on 10 pods, the daily report will be generated 10 times; only one pod should run it.
For roles like database primary, Kafka controller and scheduler you need exactly one leader, otherwise split-brain.
If 2 parallel requests update the same user's wallet, a race condition can produce a wrong balance.
If the leader crashes, a new leader must be chosen automatically, without manual intervention.

Lock with TTL/lease: take the lock with an expiry, so if the holder crashes the lock frees itself.
Redis simple lock: SET lock:order123 <random-id> NX PX 30000; release only when the value is your own random-id (via a Lua script).
Redlock: the lock must be acquired on a majority (3) of 5 independent Redis nodes. Martin Kleppmann has questioned its safety.
Fencing token: give an increasing number on every lock grant; storage rejects writes carrying an older token, protecting against a zombie client after a GC pause.
ZooKeeper: create ephemeral sequential znodes; the one with the smallest number is the lock holder/leader; when the session ends the node is deleted.
etcd: based on Raft consensus; lock and election APIs with lease + revision; the Kubernetes control plane uses this.
DB row lock: SELECT ... FOR UPDATE or an advisory lock (pg_advisory_lock); simple, and best when the data is already in the DB.
Leader election: candidates race to take the lock/lease; the winner becomes the leader and renews the lease with a heartbeat.

Distributed lock, leader election and fencing token

1.A Cron should run on only one pod: Pod-1..Pod-5 try Redis SET lock:settlement:2026-10-11 podId NX PX 600000 (TTL 10 min)
2.Pod-2 gets the lock and runs the daily settlement job; the others get nil and skip
3.Release: a Lua script, DEL only if value == podId; the Settlement DB accepts only writes with token 34 (only token >= last)
4.B Leader election (Kubernetes Lease, etcd): kube-controller-manager x 3; Replica 1 is LEADER (renews every ~2s), Replica 2 and 3 are standby; etcd Lease object holder = replica-1 (Raft consensus)
5.Leader crashes -> renewal stops -> lease expires -> Replica 2 takes the lease -> new leader, no manual work
6.C Fencing token (protection from a zombie write): Pod A gets the lock, token 33 -> Pod A has a GC pause, TTL expires -> Pod B gets the lock, token 34, write OK (storage last token = 34) -> Pod A wakes up, writes with token 33 -> 33 < 34, REJECT

Redis has no built-in fencing; use the etcd revision or the ZooKeeper zxid

Tool	Type	How it is used
Redis / Redisson	In-memory store + Java client	A fast lock with SET NX PX; Redisson provides RLock with auto-renew (watchdog).
Apache ZooKeeper / Curator	Coordination service	Strong locks and leader election via ephemeral sequential nodes; Curator has ready-made recipes.
etcd	Raft-based KV store	Lease, lock and election APIs; Kubernetes leader election (controller-manager) is based on etcd.
Consul	Service discovery + KV	Locks and leader election through sessions and KV acquire.
PostgreSQL advisory locks / SELECT FOR UPDATE	Relational DB	A row or application-level lock when you do not want extra infra.
DynamoDB lock client / Azure Blob lease	Cloud managed	A serverless-friendly distributed lock via conditional writes or a blob lease.

Point	Redis / Redlock	ZooKeeper	etcd
Consistency	No strong guarantee, async replication	Strong, ZAB consensus	Strong, Raft consensus
Speed	Fastest, sub-ms	Milliseconds	Milliseconds
Failure handling	Depends on TTL expiry	Ephemeral node auto-deleted when the session expires	Key auto-deleted when the lease expires
Fencing token	Not built in	You can use the zxid / znode version	You can use the revision number
Ops complexity	Low, you usually already have Redis	High, you must maintain a JVM cluster	Medium, a single Go binary
Best for	Efficiency locks, preventing duplicate work	Older Hadoop/Kafka ecosystems, strong locks	Kubernetes, cloud-native leader election

Redis lock: when the lock is only for efficiency (preventing duplicate emails/cron) and an occasional double run is acceptable.
ZooKeeper / etcd: when correctness is critical and you cannot allow split-brain at all.
DB row lock: when the data is already in one DB, the simplest and most reliable option.
If you are on Kubernetes, use client-go leader election (Lease object); do not add a separate system.

Real world: Kubernetes controller leader election and a cron job

Kubernetes kube-controller-manager runs as 3 replicas for HA, but only one should be active.
All three replicas try to take the Lease object (stored in etcd); whoever takes it first is the leader.
The leader renews the lease every few seconds; the others wait on standby.
If the leader pod crashes and stops renewing, the lease expires and a standby becomes the new leader.
E-commerce scenario: a daily settlement cron is deployed on 5 pods.
Each pod tries SET lock:settlement:2026-10-11 podId NX PX 600000 in Redis; only one gets the lock and only that pod runs the job.
The Settlement DB checks the fencing token, so that an old stuck pod cannot make a wrong write later.

Taking a lock without a TTL: if the holder crashes the lock stays stuck forever (deadlock).
Not checking the value on release: you delete someone else's lock. Use a compare-and-delete Lua script.
Work that runs longer than the TTL, or a GC pause: the lock expires, someone else gets it, and both write together. Add a fencing token.
Interview tip: bringing up the Redlock vs Kleppmann debate and the fencing token concept makes a strong impression.