Service Discovery and Configuration Management
Service discovery tells you 'on which IP:port is the payment service right now', and config management gives all services their settings/secrets from one central place.
32.1 Why is it used?
- In cloud/Kubernetes, instances keep coming and going and IPs change; a hardcoded IP will not work.
- With auto scaling, 5 instances become 50; callers must learn about new instances immediately and dead instances must be removed.
- DB URL, feature flags and timeouts differ in every environment (dev/stage/prod); if hardcoded in code, every change needs a redeploy.
- Passwords/API keys must not be in code or Git; you need a secure central store.
32.2 How it works
- Service registry: every instance registers its name + IP:port on start and keeps sending a heartbeat/health check.
- Client-side discovery: the caller gets the list from the registry and load balances itself (Netflix Eureka + Ribbon style).
- Server-side discovery: the caller calls one LB/DNS name and the LB picks an instance from the registry (AWS ALB, K8s Service).
- Kubernetes DNS: create a Service and you get a name like 'payment.prod.svc.cluster.local'; CoreDNS + kube-proxy route to healthy pods.
- Health checks: instances that fail are removed from the registry (Consul health checks, K8s readiness probe).
- Consensus stores: etcd (Raft), ZooKeeper (ZAB), Consul (Raft); strongly consistent key-value, used for leader election and distributed locks.
- Centralized config: the app reads config from a config server/parameter store at startup or at runtime; refresh/watch on change.
- Secrets: stored encrypted in Vault or AWS Secrets Manager with rotation support; the app fetches them using an IAM role.
- 1.A Checkout service: the code has only the name http://payment.prod.svc.cluster.local:8080, no IP
- 2.CoreDNS: name -> Service IP; kube-proxy carries the call to ready pods (K8s Service endpoints = ready pods)
- 3.Payment pods: HPA 5 -> 40; a new pod joins the endpoints once readiness passes; pod-7 hung -> readiness probe fails -> removed from endpoints, no traffic
- 4.B Config + Secrets without redeploy: ConfigMap (payment_timeout 2s -> 3s, rolling restart), Secrets Manager (DB password, fetched via IAM role IRSA, 30-day rotation), Parameter Store (new_upi_flow = 10% ON)
- 5.C Hybrid: a Consul registry holding the VM (legacy billing, resolves 'payment.service.consul') and K8s payment pods (register + heartbeat), with health checks
Mistake: hardcoded IP. Pods come and go and IPs change; always call by name. Client-side (Eureka): the caller fetches the list itself; server-side (ALB, K8s Service): the LB picks the instance
32.3 Tools
| Tool | Type | How it is used |
|---|---|---|
| HashiCorp Consul | Service discovery + KV + mesh | Service registry with health checks, DNS/HTTP interface, KV config, multi-datacenter support. |
| etcd | Distributed KV store (Raft) | The brain of Kubernetes; all cluster state lives in it; also used for leader election and config. |
| Apache ZooKeeper | Coordination service | Leader election, config and locks in Hadoop/HBase/older Kafka; tracks live instances through ephemeral nodes. |
| Kubernetes DNS (CoreDNS) | Built-in discovery | Automatic discovery by K8s Service name, config injected via ConfigMaps/Secrets. |
| Spring Cloud Config / Netflix Eureka | Java config server + registry | Config is served from a Git repo and updated at runtime via /actuator/refresh; services register in Eureka. |
| AWS Systems Manager Parameter Store / Secrets Manager | Managed config + secrets (AWS) | Hierarchical params (/prod/payment/db_url), KMS encryption; Secrets Manager provides auto rotation. Azure App Configuration/Key Vault are the equivalents. |
32.4 Comparison: Consul vs etcd vs ZooKeeper vs Kubernetes DNS
| Point | Consul | etcd | ZooKeeper | Kubernetes DNS |
|---|---|---|---|---|
| Main use | Service discovery + health checks + KV + mesh | Consistent KV store (K8s backing store) | Coordination: locks, leader election, config | Built-in service discovery |
| Consensus | Raft | Raft | ZAB | On top of etcd (K8s API) |
| Discovery interface | DNS + HTTP API | Watch API (you write the logic) | Client library, ephemeral znodes | DNS names (svc.cluster.local) |
| Health checks | Built-in (HTTP, TCP, script) | No, via leases/TTL | Via session timeout | Readiness/liveness probes |
| Multi-DC / hybrid | Strong, both VMs + K8s | Single cluster focus | Single cluster focus | Only inside the cluster |
| Ops | Medium | Medium (managed in K8s) | Heavy, Java, old | Zero extra, built-in |
| Best for | Hybrid VM + K8s, multi-datacenter | K8s internals, small critical config | Legacy Hadoop/older Kafka ecosystems | Pure Kubernetes apps |
32.5 When to choose which
- Kubernetes DNS + ConfigMaps/Secrets: if everything is on K8s, do not add anything extra.
- Consul: a mix of VMs and K8s, multiple datacenters, health checks + KV in one place.
- etcd: already inside K8s; you can use it if your apps need leader election/locks.
- ZooKeeper: only when an existing ecosystem (Hadoop, HBase, Solr, older Kafka) demands it; newer Kafka versions use KRaft, not ZooKeeper.
- AWS Parameter Store / Secrets Manager or Spring Cloud Config: centralized config; secrets always in Secrets Manager/Vault.
32.6 Real-world example
Scenario: E-commerce checkout on Kubernetes
- The checkout service has to call payment; the code only says 'http://payment.prod.svc.cluster.local:8080', no IP.
- When the sale starts, HPA scales payment pods from 5 to 40; new pods join the Service endpoints as soon as they pass the readiness probe.
- One pod hangs and its readiness probe fails; Kubernetes stops sending it traffic and the CoreDNS name stays the same.
- The payment timeout value lives in a ConfigMap; for the sale it was raised from 2s to 3s and a rolling restart made the pods pick up the new config.
- The DB password is in AWS Secrets Manager; the pod fetches it using an IAM role (IRSA); auto rotation every 30 days.
- Legacy billing runs on VMs; Consul keeps both (VM + K8s) services in one registry so that the VMs can also resolve 'payment.service.consul'.
- The feature flag 'new_upi_flow' was turned ON for 10% of users from Parameter Store, without a redeploy.
32.7 Interview tips
- Hardcoding IPs or caching a long DNS TTL: after scaling, traffic goes to dead instances.
- Keeping secrets in plain text in a ConfigMap, env file or Git.
- A health check that only verifies 'the process is alive' and not DB connectivity: a broken instance keeps receiving traffic.
- Interview tip: explain the difference between client-side and server-side discovery, and the role of consensus (Raft/ZAB) in etcd/ZooKeeper.