Log in
SkillPilotAI
Plans

Distributed Tracing

See the whole journey of one request in a single timeline: which services it went through and how much time it took at each place.

45.1 Why is it used?

  • In microservices one checkout request passes through 10-20 services; finding which is slow from logs is hard.
  • The latency bottleneck (a slow DB query, a slow third-party API) shows up directly on the timeline.
  • A service dependency map gets built: who calls whom.
  • You learn which service errors started in and how far the failures spread.
  • Together with logs and metrics it completes the third pillar of observability.

45.2 How it works

  • Trace: the journey of one whole request, with a unique trace_id.
  • Span: one step of a trace (like "DB query" or "payment API call"), with start time, duration and tags; spans form a parent-child tree.
  • Context propagation: trace_id and span_id travel to the next service in an HTTP header (W3C traceparent) or a Kafka message header.
  • Instrumentation: the OpenTelemetry SDK/auto-instrumentation creates spans by itself on HTTP, DB and gRPC calls.
  • Collector: the OTel Collector receives, processes and batches spans and sends them to a backend like Jaeger, Tempo, X-Ray or Datadog.
  • Sampling: tracing every request is expensive; head sampling (1-10% random) or tail sampling (keep only slow/error traces).
  • Put trace_id in logs so that the related logs open directly from a trace.
Food delivery "Place Order": trace context propagation
  1. 1.API Gateway: trace_id is created, header traceparent: 00-4bf92f...(trace_id)-00f067...(span_id)-01
  2. 2.Cart svc, Restaurant svc, Payment svc, Bank API: each service forwards the header and creates its own span (OTel SDK)
  3. 3.OTel Collector: batch + sampling (1-10% or tail)
  4. 4.Jaeger / Tempo: trace UI

If context propagation breaks (the header is not passed through a queue or thread pool), the trace is cut in the middle.

Jaeger timeline: trace_id 4bf92f, duration 3.65s
  1. 1.api-gateway: place-order 3.65s (gateway 20ms)
  2. 2.cart-svc 50ms
  3. 3.restaurant-svc 80ms
  4. 4.payment-svc 3.5s, with a child span inside: bank API call 3.3s (bottleneck)
  5. 5.Search logs by trace_id: found "retry 3 times"
  6. 6.Fix: timeout + circuit breaker + async bank call, p99 4s -> 600ms, verified in Grafana

Put trace_id in logs so that logs open directly from the trace.

45.3 Tools you can use

ToolTypeHow it is used
OpenTelemetry (OTel)Open standard + SDKs (CNCF)Generates and exports traces, metrics, logs in a vendor-neutral way; you can change the backend later.
JaegerOpen-source tracing backend (CNCF)Stores traces and shows the timeline and dependency graph in a UI; built by Uber.
ZipkinOpen-source tracing backendAn older, simple tracing system built by Twitter; easy for small setups.
AWS X-RayAWS managed tracingTraces and a service map for Lambda, API Gateway, ECS; data can also be sent via ADOT (AWS Distro for OTel).
Grafana TempoOpen-source tracing backendCheap trace storage in object storage, linked in Grafana with Loki logs and Prometheus metrics.
Datadog APMCommercial SaaS APMAutomatic traces through the agent, correlated in one UI with metrics and logs.

45.4 Comparison: Jaeger vs Zipkin vs AWS X-Ray (OpenTelemetry works with all)

PointJaegerZipkinAWS X-Ray
TypeOpen-source, self-hostedOpen-source, self-hostedAWS managed service
OriginUber, CNCF graduatedTwitter, inspired by the Google Dapper paperAWS
StorageCassandra, Elasticsearch/OpenSearch, BadgerCassandra, Elasticsearch, MySQL, memoryAWS managed, nothing to manage
OpenTelemetry supportNative OTLP supportVia an OTel exporterVia the ADOT collector
ScalingGood, but you handle the cluster yourselfFor small-to-medium setupsAWS auto scale
CostFree, infra costFree, infra costPay per trace recorded/scanned
Best forKubernetes microservices, large scaleSimple, legacy Java/Spring setupsAWS Lambda/serverless apps

45.5 When to choose which

  • Always instrument with OpenTelemetry so you can change the backend (Jaeger/X-Ray/Datadog) later without code changes.
  • Jaeger / Tempo: when you want a self-hosted, open-source stack on Kubernetes.
  • Zipkin: a small or older Spring Boot setup where it is already integrated.
  • AWS X-Ray: a serverless/AWS-native app where you want zero ops.
  • Datadog APM: when you already use paid all-in-one observability.

45.6 Real-world example

Such a scenario: "Place Order" is slow in a food delivery app

  1. Users say placing an order takes 4 seconds, and p99 latency is up on the dashboards.
  2. An engineer filters the slow traces of "place-order" in Jaeger (duration > 3s).
  3. The trace timeline shows: API gateway 20ms, cart service 50ms, restaurant service 80ms, payment service 3.5s.
  4. The child span inside the payment span shows that the third-party bank API call alone takes 3.3s.
  5. They copy the trace_id and search the logs, where they find "retry 3 times" logs.
  6. The team adds a timeout + circuit breaker and makes the bank call async; p99 comes back to 600ms.

45.7 Interview tips

  • Forgetting context propagation (not passing the header through an async queue or thread pool) makes the trace break in the middle.
  • Keeping 100% sampling in production makes storage and network cost balloon.
  • Not putting trace_id in logs means traces and logs cannot be linked.
  • Interview tip: explain the difference between trace, span and log, and the sampling strategy.