Find the missing span before you blame context propagation

An AI agent's tool span is missing after the request crosses a queue. The worker's spans show up as a separate trace, or not at all. The tempting response is to add inject and extract calls everywhere and redeploy. That helps only when the evidence shows a propagation break, and only with the right carrier, propagator and context selection. When AI agent spans go missing from OpenTelemetry, check five things in order: was the span created, was it ended, was it sampled, which parent and links did it get, and did it reach your backend and become visible in a query. These are diagnostic categories, not a complete list of causes, and each leaves different evidence. Collect it one hop at a time before you change any instrumentation.

A trace that starts at the worker proves less than it looks#

Three different situations can look the same in a trace view: the worker's span appears at the top.

In the first, the worker's span has no parent. The tracing API specification says a span created from a context with no span in it becomes a root span, and every root span gets a new trace ID. An empty parent ID means no parent was selected for that span. No context may have arrived. An invalid context may have been extracted. A valid one may not have been passed to span creation. Or the code asked for a root on purpose: the API requires an option to create one.

In the second, the worker's span carries a parent ID, and the backend has no span with that ID. A backend may display a span whose parent is absent as the top visible span. Inspect the stored parent ID rather than its position in the UI. A missing parent record is a reason to inspect that parent's creation, sampling, completion and delivery. It is not proof the parent was lost in transit.

In the third, nothing is broken. The worker's span has a parent from its own process, such as an HTTP server span, and a link to the producer. The messaging semantic conventions, still marked Development, use span links as the default mechanism to correlate producers and consumers. One reason they give: message consumption can happen "in the scope of another ambient context". A view that only follows parents will show two traces.

Context reaching the worker does not prove the trace is complete, either. The SDK specification says a span ID is generated "independently of the sampling decision". A trace ID and span ID in a log line do not tell you the span was recorded, ended or exported.

The synthetic request#

One hypothetical path, test data only:

1. An entry service receives POST /tasks and publishes a message to a queue.
2. A worker consumes the message, calls a model, then calls a tool over HTTP.
3. Every service exports to a Collector, which forwards to your backend.

Mark every span with a test-only attribute, test.run_id = "hopcheck-0001". Do not use baggage for this, and do not use a real user or tenant ID. OpenTelemetry's propagation guidance warns that baggage crosses service boundaries and "might be logged or sent to untrusted downstream services". An attribute stays on the span.

Collect five kinds of evidence at every hop#

• Sender: the current span's trace ID, span ID and sampled flag when you call inject.
• Carrier: the traceparent written into the message headers, and the value read back before extract.
• Extracted context: whether it is valid (non-zero trace and span IDs, per the API spec) and remote, and its IDs.
• Span creation: the new span's IDs, sampled flag and whether it is recording; for a recording span, also the parent the SDK selected and its links.
• Completion and export: whether the span ended, and whether a span with that ID reaches the next pipeline stage.

The W3C traceparent header has the form version-traceid-parentid-traceflags, per the propagation docs. The parent-id field should equal the sender's span ID. The last byte carries the sampled flag.

The snippets use the OpenTelemetry Python SDK. The API package alone records nothing, so they assume an SDK TracerProvider with a span processor and an exporter, set up as below.

The setup adds a test-only processor, HopLog, that prints each recording span when it starts and ends. It does not see every span. The SDK specification says span processors receive only spans whose IsRecording is true, and a DROP sampling decision makes a span non-recording. For a dropped span, neither callback runs, so its parent and links cannot be read through HopLog. The caller-side created helper covers that gap: it prints the IDs, the sampled flag and is_recording() whatever the sampler decided. Call it as the first line inside the with block of every span you diagnose; the examples below do. The parent, links and instrumentation_scope fields HopLog reads are Python SDK attributes.

from opentelemetry import trace, propagate
from opentelemetry.sdk.trace import TracerProvider, SpanProcessor
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

def ids(sc):
    if sc is None:
        return None
    return {
        "trace_id": format(sc.trace_id, "032x"),
        "span_id": format(sc.span_id, "016x"),
        "remote": sc.is_remote,
    }

def created(span, label):
    sc = span.get_span_context()
    print({"event": "created", "label": label, "span": ids(sc),
           "sampled": sc.trace_flags.sampled,
           "recording": span.is_recording()})

class HopLog(SpanProcessor):
    def on_start(self, span, parent_context=None):
        sc = span.get_span_context()
        print({
            "event": "start",
            "name": span.name,
            "scope": span.instrumentation_scope.name if span.instrumentation_scope else None,
            "span": ids(sc),
            "sampled": sc.trace_flags.sampled,
            "parent": ids(span.parent),
            "links": [ids(link.context) for link in span.links],
        })

    def on_end(self, span):
        print({"event": "end", "name": span.name, "span": ids(span.context)})

provider = TracerProvider()
provider.add_span_processor(HopLog())
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("hopcheck")

On the producer side, log the new span first, then inject inside the span you want as the parent:

with tracer.start_as_current_span(
    "send tasks",
    kind=trace.SpanKind.PRODUCER,
    attributes={"test.run_id": "hopcheck-0001"},
) as span:
    created(span, "send tasks")
    headers = {}
    propagate.inject(headers)
    print({"event": "sender", "traceparent": headers.get("traceparent")})
    queue_client.publish(body=payload, headers=headers)  # your client here

The created line gives the sender's trace ID, span ID and sampled flag; the sender line gives what was written into the carrier.

On the consumer side, log the carrier, then the extracted context, then pass that context explicitly to span creation and log what was created:

def handle(message):
    print({"event": "carrier", "traceparent": message.headers.get("traceparent")})
    ctx = propagate.extract(message.headers)
    remote = trace.get_current_span(ctx).get_span_context()
    print({"event": "extracted", "valid": remote.is_valid, "span": ids(remote)})
    with tracer.start_as_current_span(
        "process tasks",
        context=ctx,
        kind=trace.SpanKind.CONSUMER,
        attributes={"test.run_id": "hopcheck-0001"},
    ) as span:
        created(span, "process tasks")
        run_agent_step(message)  # model call, then tool call

If process tasks is recording, its start line shows the parent the SDK actually selected. Carrier present, extraction valid and remote, and a parent equal to it: this hop propagated. Extraction invalid: look at the carrier and the propagators on each side. Extraction valid but a different parent: look at what context reached span creation, and whether the producer appears under links.

queue_client and message.headers stand in for your own client. Whether your broker and client library keep custom headers end to end is a property of that system: verify it with the carrier line.

Print these values only for synthetic runs. Real headers can carry tracestate, baggage and internal identifiers, which the propagation docs say "might reveal sensitive information about your internal architecture or business logic".

Read the evidence#

Parent and link evidence comes from HopLog, so it exists only for recording spans. Every created check below assumes the helper is called immediately after that span's creation and that you have confirmed the process's output is captured.

Sampling is decided in more than one place. The SDK specification defines three decisions. DROP: the span is not recording, processors and exporters do not see it, and only the caller-side created line shows it existed. RECORD_ONLY: processors see it, exporters do not, so it prints start and end lines and reaches no backend. RECORD_AND_SAMPLE: recorded and exported. These are head decisions, made as early as possible without seeing the whole trace. Tail sampling decides later, in the pipeline. The Collector troubleshooting page notes that processors for trace data, except tail sampling, "work only on individual spans". A filter that removes spans by attribute can remove a parent and leave its children: the "same trace, parent not found" row.

A separate trace ID can be deliberate. The propagation docs suggest you might ignore or sanitize incoming context from untrusted sources, because forged trace headers can manipulate your tracing data. A boundary that does this starts a new trace by design. Read the links before you call it broken.

Inside the worker: check the active context, then the tool span#

Agents add a hop that has nothing to do with the queue: the model call and tool call may run in a callback, a background task or a thread pool. Correct extraction at the consumer does not help if the tool runs where that context is not current.

def call_tool(args):
    active = trace.get_current_span().get_span_context()
    print({"event": "tool-entry", "valid": active.is_valid, "span": ids(active)})
    with tracer.start_as_current_span(
        "call tool", attributes={"test.run_id": "hopcheck-0001"}
    ) as span:
        created(span, "call tool")
        return tool_client.call(args)  # your tool here

The tool-entry line describes the active context. The created line describes the tool span, and for a recording span the HopLog start line adds the parent it received.

• valid: False at entry: no span is active where the tool runs. The tool span becomes a root with a new trace ID.
• Valid, with the trace ID of process tasks: the context reached the tool. If the tool span is still missing, go back to the table.
• Valid, with another trace ID: a different ambient context is active there.

Before you call the first or last case an unintended break, rule out a deliberate root, a link, or a design that runs the tool under another context. Then carry the context into that execution unit, for example by capturing opentelemetry.context.get_current() before the handoff and attaching it on the other side. The message headers are not involved.

Verify delivery with the Collector debug exporter#

When the span exists, has ended, is sampled and has the parent you expect, but does not arrive, look at delivery, hop by hop.

End, then flush, before exit. The batch span processor exports on a schedule. Its defaults are a 5,000 ms delay between exports, a queue of 2,048 spans and batches of 512. A worker that handles one message and exits can end before the next export. In the Python SDK, force_flush exports "all ended spans" not yet exported and returns False if the timeout is exceeded. Let every with block close, then call provider.force_flush() or provider.shutdown() on the way out, and log the return value.

Inspect each Collector hop. The troubleshooting guide recommends the debug exporter to confirm that the Collector is receiving, processing and exporting data. For the synthetic run, point the services at a local Collector:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 127.0.0.1:4318
processors:
  # copy every processor definition from your real config here, unchanged
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: []   # second run: your real processors list, same order
      exporters: [debug]

With verbosity: detailed, each span prints with its Trace ID, Parent ID and ID, so you can match it against your hop log.

Run the same synthetic path twice: once with processors: [], once with your real list in its real order. In both runs, confirm the producer's flush returned True, and wait longer than any batch timeout or tail-sampling decision window you configured before reading the output. A span present in the first run and absent in the second implicates the processor-enabled pipeline, without saying why: it may be delayed, deliberately filtered or sampled, or dropped by a failing component. Read the Collector's logs and internal telemetry, and repeat the guide's per-hop checklist for every Collector up to the backend.

Keep detailed output away from real traffic. It prints the entire payload, including every attribute: on a production pipeline, prompts, tool arguments and identifiers in Collector logs. The local Collector above binds to 127.0.0.1 and should only see hopcheck-0001 data.

When the backend is the destination#

The same isolation applies whichever backend receives the spans. If that backend is Worksi, the OpenTelemetry integration accepts OTLP/HTTP in JSON or protobuf and maps spans that follow the GenAI semantic conventions to model calls, tool calls and agent steps. It can only show the spans that arrive: a span your worker did not create, record, end or export will not appear. OTLP ingestion is also observation only: allowlists, budgets, pause and stop apply only to traffic that goes through the Worksi gateway.

A span that arrives intact is still only evidence that the operation was recorded. Whether the tool did the right thing is a different question, covered in why a green tool span does not prove the right outcome.