OpenTelemetry

Article.

Verified 2026-10

Verified 2026-10-15 against opentelemetry-sdk 1.45.1 on Python 3.14; the outputs below come from that run.

The CNCF-graduated standard for emitting observability data: traces, metrics and logs. It replaced OpenTracing and OpenCensus (merged in 2019) and is the vendor-neutral layer under Jaeger, Tempo, Datadog, New Relic and the rest: instrument once, ship anywhere.

The question worth answering properly is how one request’s context travels through a system, and where the telemetry goes missing. Traces rarely fail loudly. They arrive with holes, as orphans, or not at all, and each of those has a mechanism.

The architecture

app + OpenTelemetry SDK     spans, metrics, logs made here
        │ OTLP
        ▼
OTel Collector (optional)   batch, sample, redact, route
        │ OTLP or vendor protocol
        ▼
Jaeger, Tempo · Prometheus, Loki · Datadog, Honeycomb

Three concepts:

1. API — what your app code calls (tracer.start_as_current_span("...").
2. SDK — the implementation that creates spans, batches, exports.
3. Collector — optional intermediary process that routes, processes, samples.

App code depends only on the API; the SDK and collector can be swapped without changing application code.

The mechanism: the API does nothing until an SDK is installed

Libraries instrument against the API only, so that importing an instrumented library never forces a telemetry pipeline on you. Until the application installs an SDK TracerProvider, every tracer is a no-op:

tracer = trace.get_tracer("checkout")
with tracer.start_as_current_span("charge") as span:
    print(span.is_recording(), span.get_span_context().trace_id)

no SDK: recording=False, trace_id=00000000000000000000000000000000

No error, no warning, and a perfectly working program. “We added OpenTelemetry and see no traces” is usually this: the instrumentation is there, and nobody called trace.set_tracer_provider() in the process that runs it, often a worker started by a different entry point than the web app.

The current span lives in a contextvars.ContextVar. That is what makes nesting work inside one process, and it is also why context silently stays behind when work hops to a thread through run_in_executor, which does not copy context (run_in_executor and asyncio.to_thread).

OTLP — the wire protocol

OpenTelemetry Protocol. Two transports:

• gRPC (port 4317) — efficient binary. The spec says SDKs should default to http/protobuf; the Python distro defaults to gRPC, so set OTEL_EXPORTER_OTLP_PROTOCOL explicitly rather than relying on either.
• HTTP/JSON or HTTP/Protobuf (port 4318) — works through HTTP proxies, easier debugging.

Most backends (Jaeger 1.35+, Tempo, Datadog, Honeycomb) accept OTLP directly. Some still require translation via the Collector.

Python instrumentation — the minimum

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

resource = Resource(attributes={
    "service.name": "my-app",
    "service.version": "1.2.3",
    "deployment.environment.name": "production",
})

provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(endpoint="http://otel-collector:4317", insecure=True)
))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

@tracer.start_as_current_span("checkout")
def checkout(order):
    span = trace.get_current_span()
    span.set_attribute("order.id", order.id)
    span.set_attribute("order.total", float(order.total))
    process(order)

Pattern: get a tracer at module level; use start_as_current_span decorator or context manager around units of work.

Resource attributes — the constants

Resource attributes are attached to every span/metric/log from this process. Standard ones (from the OTel semantic conventions):

Set them once at startup. Don’t repeat per-span — they’re inherited.

Auto-instrumentation

For most popular Python libraries, instrumentation packages exist:

pip install opentelemetry-distro \
  opentelemetry-instrumentation-flask \
  opentelemetry-instrumentation-requests \
  opentelemetry-instrumentation-sqlalchemy \
  opentelemetry-instrumentation-redis \
  opentelemetry-instrumentation-celery

# Or auto-install for everything already in your env:
opentelemetry-bootstrap -a install

Run with auto-instrumentation:

opentelemetry-instrument \
  --service_name=my-app \
  --exporter_otlp_endpoint=http://otel-collector:4317 \
  python app.py

Spans are auto-created for HTTP requests, DB queries, cache calls, queue consumption — no code changes. You add custom spans only for app-specific work the instrumentation doesn’t cover.

Instrumentation packages

They follow one naming scheme, opentelemetry-instrumentation-<library>: flask, fastapi, django, requests, httpx, sqlalchemy, psycopg, redis, celery, kafka-python, grpc, botocore and more. opentelemetry-bootstrap -a install installs the ones matching what is already in the environment.

Spans, attributes, events, links

with tracer.start_as_current_span("process_order") as span:
    span.set_attribute("order.id", 42)
    span.set_attribute("order.items", 5)

    try:
        result = charge_card(order)
        span.set_attribute("payment.amount", result.amount)
        span.set_status(trace.Status(trace.StatusCode.OK))
    except PaymentError as e:
        span.record_exception(e)
        span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
        raise

    span.add_event("order_persisted", {"db.row_id": order.row_id})

Semantic conventions

OTel defines standard attribute names so backends can do consistent analysis:

Auto-instrumentation populates these. For manual spans, follow the conventions so dashboards / queries built for the standard work.

Metrics and logs, briefly

The same SDK emits metrics through a MeterProvider, with four instrument kinds:

meter = metrics.get_meter(__name__)
orders = meter.create_counter("orders.placed", unit="1")
orders.add(1, {"tier": "pro", "region": "us-east"})

For logs, most teams keep their structured logger and add opentelemetry-instrumentation-logging, which stamps trace_id and span_id on each record so a log line links to its trace. The logs data model and bridge API are stable; the Python logs SDK is still marked Development. See Prometheus and Structured logging in Python.

Sampling

Two independent decisions, and they are configured in different places:

from opentelemetry.sdk.trace.sampling import ParentBasedTraceIdRatio

provider = TracerProvider(
    resource=resource,
    sampler=ParentBasedTraceIdRatio(0.1),   # 10%, honouring the parent
)

ParentBased matters more than the ratio: without it each service samples independently, and a trace kept by one service and dropped by the next is a trace with holes in it rather than a trace you did not keep.

Tail-based sampling needs the Collector’s tail_sampling processor, because only something that has seen the whole trace can decide “keep it because it errored”. That costs memory and a decision_wait delay, which is the trade.

Policies, adaptive rates, per-endpoint rules and the statistical bias sampling introduces are in Sampling Strategies for Tracing.

The Collector — the routing layer

A standalone process that:

• Receives telemetry from apps (OTLP).
• Processes (sample, redact, transform, batch).
• Exports to one or more backends (Jaeger, Prometheus, Datadog, …).

A minimal config

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc: {}
      http: {}

processors:
  batch: {}
  resource:
    attributes:
      - key: deployment.environment.name
        value: production
        action: insert

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls: { insecure: true }
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlp/jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

Run as a sidecar (one per pod / host) or as a centralized service. Sidecar pattern is common in Kubernetes; centralized for VMs.

Why run one

• Switch backends without redeploying apps.
• Centralized sampling, redaction, batching.
• One process handles auth and TLS to the backend.
• Buffer when the backend is slow/down.
• Export to two backends at once (a vendor and a self-hosted Jaeger), at the cost of ingesting every trace twice, as a hedge against lock-in.

For small deployments: skip the collector; export directly. For production: run it.

Context propagation across boundaries

A trace only holds together if the parent’s span context reaches the child. Between processes that is the W3C traceparent header: inject writes it, the next service must extract it, or its span starts a new trace:

headers: dict[str, str] = {}
propagate.inject(headers)                       # service A, outgoing call

ctx = propagate.extract(request.headers)        # service B, incoming request
with tracer.start_as_current_span("payments: charge", context=ctx):
    ...

outgoing header traceparent: 00-b2e15dd5911bc4dc0df521b5830d6d86-a21aec364735f813-03
child in the next service shares the trace id: True
without extract(): shares the trace id: False (a new, orphan trace)

The header is version, 128-bit trace id, the parent span id, and flags: 03 here is sampled (01) plus the Level 2 “random trace id” bit (02). HTTP auto-instrumentation does both halves.

Across a queue or a thread

Across a queue, a thread pool or a custom protocol, something has to carry it:

from opentelemetry.propagate import inject, extract

carrier: dict[str, str] = {}
inject(carrier)                                    # producer: write context out
celery_app.send_task("work", kwargs={"data": ..., "_trace": carrier})

def work(data, _trace=None):                       # consumer: read it back
    with tracer.start_as_current_span("work", context=extract(_trace or {})):
        ...

The Celery, Kafka, RabbitMQ and gRPC instrumentations do this for you. The boundaries that silently break it — run_in_executor, a manually created thread, a task spawned without the current context — are the subject of Trace context in async Python, along with the contextvars mechanism underneath.

Where it breaks in production

The batch queue drops spans, by design

BatchSpanProcessor never blocks your request thread. When the exporter falls behind, its queue (2,048 spans by default) fills and new spans are discarded. A burst of 10,000 spans against an exporter taking 50 ms a batch:

Queue full, dropping Span.
10,000 spans in a burst, slow exporter: exported 4,608

More than half the burst was lost, announced by one log line. Traces from the busiest moment, exactly the ones you will want, are the ones with holes. Export to a Collector next to the app (a local hop that keeps up), size OTEL_BSP_MAX_QUEUE_SIZE for your burst, and alert on dropped-span counts.

A hard exit loses the last batch

exit 'normal': exported 100
exit 'kill': exported nothing

The SDK flushes through an atexit hook. SIGKILL after a failed graceful shutdown, an OOM kill, os._exit in a forked worker or a frozen Lambda skip it, so the spans of the crash itself never arrive. Call provider.force_flush() at the end of a job or Lambda invocation, and give the pod’s shutdown enough grace for the flush.

And the quieter ones

• No SDK in the worker process. The no-op API above: Celery workers and cron jobs started from a different entry point than the web app emit nothing.
• A broken hop. One proxy, queue or thread pool that does not carry traceparent splits every trace into orphans at that point.
• Service name drift. Dashboards built on service.name=my-app break when one host emits my_app; set OTEL_SERVICE_NAME in deployment config.
• High-cardinality metric labels. user.id is fine on a span and a cost explosion as a metric label.
• Noisy auto-instrumentation. Health checks and metrics scrapes as spans; exclude them (OTEL_PYTHON_FASTAPI_EXCLUDED_URLS and its equivalents).
• SimpleSpanProcessor in production. It exports synchronously on the request path; it is for tests.

The trade not taken: the vendor’s own agent

The vendor agent is the faster first week and the expensive third year: switching backends means re-instrumenting every service. Most vendors now ingest OTLP (Datadog, New Relic, Honeycomb, Splunk), so the default is to instrument with OpenTelemetry and route through a Collector, and add a vendor agent only for a feature that pays for the lock-in, such as a continuous profiler.

Two confusions worth correcting in an interview: OpenTelemetry does not replace Jaeger (one emits, the other stores and shows), and OTel logs are usable through the logging bridge even though the Python logs SDK itself is still marked Development.