{
 "id": "epw5srg97m",
 "slug": "distributed-tracing",
 "title": "Distributed tracing",
 "updated": "2026-10-15",
 "topic_path": [
  {
   "id": "technology",
   "label": "Technology and the built world",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology"
  },
  {
   "id": "technology.computing",
   "label": "Computing and digital systems",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing"
  },
  {
   "id": "technology.computing.software",
   "label": "Software and programming",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.software"
  },
  {
   "id": "technology.computing.software.software_engineering",
   "label": "Software engineering and development process",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.software.software_engineering"
  },
  {
   "id": "technology.computing.software.software_engineering.software-testing-and-quality",
   "label": "Software testing and quality",
   "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.software.software_engineering.software-testing-and-quality"
  }
 ],
 "geo": [
  {
   "id": "geo.nongeo.t2001.technology.computing.software",
   "label": "Non-geographic · 2001 to 2020: Software and programming",
   "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.software",
   "path": [
    {
     "id": "geo.nongeo",
     "label": "Non-geographic",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo"
    },
    {
     "id": "geo.nongeo.t2001",
     "label": "Non-geographic · 2001 to 2020",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001"
    },
    {
     "id": "geo.nongeo.t2001.technology",
     "label": "Technology and the built world",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology"
    },
    {
     "id": "geo.nongeo.t2001.technology.computing",
     "label": "Computing and digital systems",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing"
    },
    {
     "id": "geo.nongeo.t2001.technology.computing.software",
     "label": "Software and programming",
     "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.software"
    }
   ]
  }
 ],
 "excerpt": "Distributed tracing is an observability technique that records the path of a request through a distributed system's services, capturing timing data to diagnose latency and failures.",
 "snippet": "Distributed tracing is an observability technique that records the path of a request through a distributed system's services, capturing timing data to diagnose latency and failures.",
 "node": "technology.computing.software.software_engineering.software-testing-and-quality",
 "markdown": "# Distributed tracing\n\nDistributed tracing is an observability technique that records the path of a request as it propagates through the services of a distributed system, capturing timing and causal data so that latency and failures can be diagnosed across service boundaries. It complements metrics and logs by preserving the causal structure of a single request's execution. Adoption remains uneven: a SIGCOMM 2024 paper reports that only about 20% of enterprise applications were instrumented for tracing as of 2021, because context propagation traditionally requires application-level instrumentation.<sup>[1](https://sachin.cs.illinois.edu/papers/traceweaver-ashok-sigcomm24.pdf)</sup>\n\n| Key fact | Detail |\n|---|---|\n| Data model | A trace is a directed acyclic graph of spans; each span has an operation name, start time, and duration, and spans nest to model causal relationships<sup>[2](https://www.jaegertracing.io/docs/1.76/architecture/)</sup> |\n| Identifiers | OTLP trace_id is a 16-byte array, span_id 8 bytes; all spans in a trace share the trace_id<sup>[3](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto)</sup> |\n| Propagation standard | W3C Trace Context defines the `traceparent` and `tracestate` headers with version, trace-id, parent-id, and trace-flags fields<sup>[4](https://www.w3.org/TR/trace-context-2/)</sup> |\n| In-band vs out-of-band | Only IDs and baggage travel with requests; timing, tags, and logs are exported asynchronously to the backend<sup>[2](https://www.jaegertracing.io/docs/1.76/architecture/)</sup> |\n| Sampling | Head-based coherent, tail-based coherent, and unitary sampling are the three fundamental options<sup>[5](https://www.pdl.cmu.edu/ftp/SelfStar/CMU-PDL-14-102.pdf)</sup> |\n| Scale example | Facebook's Canopy records 1.3 billion traces per day<sup>[6](https://cs.brown.edu/people/jcmace/papers/kaldor2017canopy.pdf)</sup> |\n\n## How it works\n\nA tracing framework assigns each incoming request a unique trace_id and a sampled flag.<sup>[7](https://arxiv.org/pdf/2502.06318)</sup> Each component that processes the request creates spans recording the work it did. In the [OpenTelemetry](https://www.edgepedia.org/opentelemetry) protocol, a span carries a span name, parent span ID, span ID, start and end timestamps in UNIX Epoch nanoseconds, events, and attributes; span kinds distinguish INTERNAL, SERVER, CLIENT, PRODUCER, and CONSUMER roles.<sup>[3](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto)</sup> When Service A calls Service B, A includes a trace ID and span ID in the context, and B creates a new span in the same trace with A's span as its parent.<sup>[8](https://opentelemetry.io/docs/concepts/context-propagation/)</sup> The backend then assembles spans dispersed across components into a coherent trace using trace_id, parent_span_id, and span_id.<sup>[7](https://arxiv.org/pdf/2502.06318)</sup>\n\nContext propagation is the mechanism that carries this identity across boundaries. The W3C Trace Context standard defines the `traceparent` HTTP header with four fields: version, trace-id, parent-id, and trace-flags. The trace-id is a 16-byte array and the parent-id an 8-byte array, with all-zero values invalid; the optional `tracestate` header carries vendor-specific name/value pairs.<sup>[4](https://www.w3.org/TR/trace-context-2/)</sup> A vendor receiving a `traceparent` request header must send it on outgoing requests and may mutate its value.<sup>[9](https://www.w3.org/TR/trace-context-1/)</sup> For gRPC, propagation uses gRPC metadata, either the `traceparent` header or a `grpc-trace-bin` metadata key.<sup>[10](https://cloud.google.com/trace/docs/trace-context)</sup> Beyond IDs, OpenTelemetry's baggage mechanism propagates arbitrary key-value pairs across services, with the caveat that credentials, API keys, or personal data should not be placed in it.<sup>[8](https://opentelemetry.io/docs/concepts/context-propagation/)</sup>\n\n## How it is done\n\nInstrumentation follows a common pattern. The Dapper design restricts instrumentation to a small set of common libraries (threading, control flow, and the RPC framework) rather than requiring developers to modify application code, and uses sampling to keep overhead low.<sup>[11](https://mwhittaker.github.io/papers/html/sigelman2010dapper.html)</sup> Tracers then propagate only IDs in-band with requests, while completed spans are reported out-of-band, asynchronously, to the backend; Zipkin's tracers work exactly this way.<sup>[12](https://zipkin.io/pages/architecture.html)</sup> Jaeger behaves the same: unsampled traces collect no data at all, and tracing API calls are short-circuited.<sup>[2](https://www.jaegertracing.io/docs/1.76/architecture/)</sup>\n\nCollection then moves spans to storage. Zipkin supports HTTP, Kafka, and Scribe as transports, and its collector validates, stores, and indexes trace data.<sup>[12](https://zipkin.io/pages/architecture.html)</sup> Jaeger accepts OTLP directly, can use Kafka as a persistent intermediary queue between collectors and storage, and offers a storage plugin framework.<sup>[2](https://www.jaegertracing.io/docs/1.76/architecture/)</sup>\n\nSampling decides what is recorded. A retrospective by tracing-infrastructure authors identifies three fundamentally different options: head-based coherent sampling, tail-based coherent sampling, and unitary sampling.<sup>[5](https://www.pdl.cmu.edu/ftp/SelfStar/CMU-PDL-14-102.pdf)</sup> The W3C specification describes probability sampling (for example, sampling 1 out of 100 traces), delayed decision based on duration or result, and deferred sampling where the callee decides, and notes that recording all requests becomes prohibitively expensive at load.<sup>[4](https://www.w3.org/TR/trace-context-2/)</sup> Dapper's production experience found that logging 1/1024 of traces was sufficient to debug most things, and 1/16 sufficient to avoid overheads.\n\n## Origin\n\nThe lineage of modern tracing runs through four pioneering systems: Magpie (Barham, Isaacs, Mortier, and Narayanan, 2003), Whodunit (Chanda, Cox, and Zwaenepoel, 2007, in ACM SIGOPS Operating Systems Review), X-Trace (Fonseca and colleagues, 2007), and Dapper (Sigelman and colleagues, 2010).<sup>[7](https://arxiv.org/pdf/2502.06318)</sup> X-Trace, published at NSDI 2007, is a cross-layer, cross-application framework that tags all network operations resulting from a task with the same task identifier and reconstructs the resulting task tree; its metadata carries a TreeInfo tuple (ParentID, OpID, EdgeType) in protocol extension fields such as IP options, TCP options, and HTTP headers.<sup>[13](https://www.usenix.org/legacy/event/nsdi07/tech/full_papers/fonseca/fonseca_html/index.html)</sup> Dapper's authors name Pinpoint, Magpie, and X-Trace as the systems most closely related to their work, and distinguish black-box statistical-inference schemes from annotation-based schemes that propagate a global identifier. Google states that the Dapper paper inspired open-source projects such as Zipkin and that Dapper-style tracing became an industry-wide standard.<sup>[14](https://cloud.google.com/blog/products/gcp/distributed-tracing-for-go)</sup> Industry implementations descended from this lineage include Google's Dapper, Cloudera's HTrace, and Twitter's Zipkin.<sup>[5](https://www.pdl.cmu.edu/ftp/SelfStar/CMU-PDL-14-102.pdf)</sup>\n\n## Variants\n\nThe instrumentation layer consolidated in stages. The OpenTracing initiative provided standard extensions and auto-instrumentation for various languages; Google open-sourced OpenCensus in 2018; and in 2019 the two projects were merged to form OpenTelemetry, with migration and predecessor-project transitions continuing afterward.<sup>[15](https://icpe2026.spec.org/preprint/Benchmarking_the_Overhead_of_Distributed_Tracing_Agents_2.pdf)</sup> OpenTelemetry standardizes APIs for telemetry collection, instrumentation libraries, and semantic conventions, and works with backends such as Jaeger, Zipkin, and [Prometheus](https://www.edgepedia.org/prometheus).<sup>[7](https://arxiv.org/pdf/2502.06318)</sup>\n\nA newer variant removes the instrumentation requirement. OpenTelemetry's eBPF-based OBI auto-instrumentation implements distributed tracing by propagating the W3C `traceparent` header, with outgoing network-level propagation disabled by default and enabled via `OTEL_EBPF_BPF_CONTEXT_PROPAGATION=all`.<sup>[17](https://opentelemetry.io/docs/zero-code/obi/distributed-traces/)</sup> The research system DeepFlow similarly establishes a network-centric tracing plane using eBPF ([Linux kernel](https://www.edgepedia.org/linux-kernel) technology running sandboxed programs in-kernel) in the kernel, and TraceWeaver reconstructs traces from eBPF-captured request and response data without application modification.<sup>[7](https://arxiv.org/pdf/2502.06318)</sup> Commercial backends named alongside Jaeger and Zipkin in the recent literature include Instana and Datadog.<sup>[1](https://sachin.cs.illinois.edu/papers/traceweaver-ashok-sigcomm24.pdf)</sup> Recent standardization work includes W3C Trace Context Level 2, which adds a random-trace-id flag requiring at least the right-most 7 bytes of the trace-id to be randomly generated with uniform distribution over \\( [0..2^{56}-1] \\).<sup>[4](https://www.w3.org/TR/trace-context-2/)</sup>\n\n## Applications\n\nTraces support latency diagnosis and aggregation. Context propagation enables per-path metrics, for example measuring `GET /checkout > GET /product` at 40 calls per second with a 1703 ms average response time, against 300 ms for all callers of the endpoint.<sup>[8](https://opentelemetry.io/docs/concepts/context-propagation/)</sup> At production scale, Canopy propagates identifiers through the system to correlate information across components and records 1.3 billion traces per day.<sup>[6](https://cs.brown.edu/people/jcmace/papers/kaldor2017canopy.pdf)</sup>\n\n## Limitations and alternatives\n\nRecording everything is expensive: the W3C specification states that recording all requests becomes prohibitively expensive at load, which is why sampling exists.<sup>[4](https://www.w3.org/TR/trace-context-2/)</sup> Sampling in turn discards data; the choice among head-based, tail-based, and unitary sampling is one of four design axes (causal relationships preserved, tracking method, sampling, and visualization) that determine trace utility.<sup>[5](https://www.pdl.cmu.edu/ftp/SelfStar/CMU-PDL-14-102.pdf)</sup> Propagating the sampled flag naively on a public API is a security risk: the W3C spec warns an attacker could overwhelm an application with tracing overhead, forge `trace-id` collisions that make monitoring data unusable, or inflate a SaaS tracing bill.<sup>[4](https://www.w3.org/TR/trace-context-2/)</sup>\n\nBroken propagation is the dominant failure mode for zero-code approaches. For HTTPS traffic, OBI cannot inject headers and instead injects trace information at the TCP/IP packet level, so propagation reaches only other OBI-instrumented services and is disrupted by L7 proxies and load balancers; network-level propagation generally requires Linux kernel 5.17 or newer.<sup>[17](https://opentelemetry.io/docs/zero-code/obi/distributed-traces/)</sup> In agentic-AI setups, Llama Stack 0.2.x and 0.3.x do not automatically propagate span context from the Llama Stack server to MCP servers, requiring manual injection of the parent context into HTTP headers.<sup>[16](https://developers.redhat.com/articles/2026/04/06/distributed-tracing-agentic-workflows-opentelemetry)</sup> On overhead, a [Red Hat](https://www.edgepedia.org/red-hat) article states manual instrumentation adds minimal overhead, often microseconds per span, but cautions against creating too many fine-grained spans for trivial operations.<sup>[16](https://developers.redhat.com/articles/2026/04/06/distributed-tracing-agentic-workflows-opentelemetry)</sup>\n\n## References\n\n1. [TraceWeaver: Distributed Request Tracing for Microservices Without Application Modification (SIGCOMM 2024)](https://sachin.cs.illinois.edu/papers/traceweaver-ashok-sigcomm24.pdf)\n2. [Jaeger Architecture (official documentation)](https://www.jaegertracing.io/docs/1.76/architecture/)\n3. [opentelemetry/proto/trace/v1/trace.proto (OTLP)](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto)\n4. [Trace Context Level 2 (W3C Candidate Recommendation Draft)](https://www.w3.org/TR/trace-context-2/)\n5. [So, you want to trace your distributed system? Key design insights from years of practical experience (CMU-PDL-14-102)](https://www.pdl.cmu.edu/ftp/SelfStar/CMU-PDL-14-102.pdf)\n6. [Canopy: An End-to-End Performance Tracing And Analysis System (Facebook, SOSP 2017)](https://cs.brown.edu/people/jcmace/papers/kaldor2017canopy.pdf)\n7. [Distributed tracing survey (arXiv preprint, Feb 2025)](https://arxiv.org/pdf/2502.06318)\n8. [Context propagation | OpenTelemetry](https://opentelemetry.io/docs/concepts/context-propagation/)\n9. [Trace Context (W3C Recommendation, Level 1)](https://www.w3.org/TR/trace-context-1/)\n10. [Trace context | Google Cloud documentation](https://cloud.google.com/trace/docs/trace-context)\n11. [Notes on Sigelman 2010 Dapper paper](https://mwhittaker.github.io/papers/html/sigelman2010dapper.html)\n12. [Zipkin Architecture (official documentation)](https://zipkin.io/pages/architecture.html)\n13. [X-Trace: A Pervasive Network Tracing Framework (NSDI 2007)](https://www.usenix.org/legacy/event/nsdi07/tech/full_papers/fonseca/fonseca_html/index.html)\n14. [Distributed tracing for Go (Google Cloud blog)](https://cloud.google.com/blog/products/gcp/distributed-tracing-for-go)\n15. [Benchmarking the Overhead of Distributed Tracing Agents (ICPE 2026 preprint)](https://icpe2026.spec.org/preprint/Benchmarking_the_Overhead_of_Distributed_Tracing_Agents_2.pdf)\n16. [Distributed tracing for agentic workflows with OpenTelemetry | Red Hat Developer](https://developers.redhat.com/articles/2026/04/06/distributed-tracing-agentic-workflows-opentelemetry)\n17. [Distributed traces with OBI | OpenTelemetry](https://opentelemetry.io/docs/zero-code/obi/distributed-traces/)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Software and programming › Software engineering and development process › Software testing and quality*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n",
 "same_as": [
  "https://cs.brown.edu/people/jcmace/papers/kaldor2017canopy.pdf"
 ],
 "url": "https://www.edgepedia.org/distributed-tracing",
 "markdown_url": "https://www.edgepedia.org/distributed-tracing.md",
 "license": {
  "name": "Edgepedia Community License 1.0",
  "url": "https://www.edgepedia.org/edgepedia/license",
  "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.",
  "spdx": "LicenseRef-Edgepedia-Community-1.0"
 },
 "credit": "\"Distributed tracing\", Edgepedia (EdgeChat), https://www.edgepedia.org/distributed-tracing. Edgepedia Community License 1.0.",
 "credit_md": "\"[Distributed tracing](https://www.edgepedia.org/distributed-tracing)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/distributed-tracing](https://www.edgepedia.org/distributed-tracing). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).",
 "credit_html": "\"<a href=\"https://www.edgepedia.org/distributed-tracing\">Distributed tracing</a>\", Edgepedia (EdgeChat), <a href=\"https://www.edgepedia.org/distributed-tracing\">https://www.edgepedia.org/distributed-tracing</a>. <a href=\"https://www.edgepedia.org/edgepedia/license\">Edgepedia Community License 1.0</a>.",
 "speakable": "Distributed tracing is an observability technique that records the path of a request through a distributed system's services, capturing timing data to diagnose latency and failures."
}