GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — feat(tracing): correlate business spans with active observability trace by NiteshDhanpal · Pull Request #465 · scaleapi/scale-agentex-python

Conversation

NiteshDhanpal commented Jul 21, 2026 •

edited by greptile-apps Bot

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

Stack (1/3)

feat(tracing): correlate business spans with active observability trace ← this PR
feat(tracing): establish W3C trace context at FastACP ingress
feat(tracing): propagate trace context into Temporal via OTel interceptor

What this PR does

Tags each adk business span, at creation time, with the active observability trace_id/span_id (OpenTelemetry preferred, ddtrace fallback), so a persisted span can be pivoted to its per-turn Tempo/Datadog trace.

New core/tracing/obs_ids.py: get_obs_mode() (reads SGP_OBS_MODE; unset/empty/unrecognized → dd_only) and obs_correlation() returning {"obs.trace_id", "obs.span_id"} for the active obs context, or {} if none.
core/tracing/trace.py: both start_span (sync + async) merge obs_correlation() into the span's data.

Design notes

Business trace_id is unchanged — it stays the run-level task id. We use the OTel span-link pattern (correlate across granularities) instead of overwriting the trace_id, which would shatter one agent run across N per-request obs traces.
Emit-time capture, never at ingest. Correlation is taken from the context active when the span is created, so batched /v5/spans/batch flushing cannot mis-group spans.
Safe no-op by default. SGP_OBS_MODE unset → dd_only; returns no tag when no obs context is active. Behavior only changes once an obs context exists (see PRs 2 & 3).

Verification

py_compile + unit smoke: default mode dd_only, empty dict when no active context, correct id widths (32-hex / 16-hex).

Greptile Summary

This PR introduces obs-correlation tagging: at span creation time, each business span is stamped with the active observability trace_id/span_id (OTel preferred, ddtrace fallback) so a persisted span can be pivoted to its per-turn Tempo/Datadog trace without overwriting the run-level business trace_id.

obs_ids.py (new): get_obs_mode() reads SGP_OBS_MODE (default dd_only) and obs_correlation() returns {\"obs.trace_id\", \"obs.span_id\"} or {} when no context is active.
trace.py: Both Trace.start_span and AsyncTrace.start_span now merge obs_correlation() into serialized_data at emission time, ensuring the correct per-request context is captured rather than waiting for batch flush.
CI: Runner condition broadened from an exact repo-name match to a startsWith('stainless-sdks/') prefix check.

Confidence Score: 4/5

The obs-correlation logic runs in the hot path of every span creation; an unhandled exception from ddtrace or OTel would silently crash all span creation for that agent run.

Two issues in obs_ids.py directly affect span creation reliability: the narrow except ImportError leaves runtime errors from ddtrace/OTel unhandled, and the span_id or 0 null-sentinel path (flagged in a prior thread). Both live in the critical start_span path shared by every caller of Trace and AsyncTrace.

src/agentex/lib/core/tracing/obs_ids.py and src/agentex/lib/core/tracing/trace.py need attention before merge.

Important Files Changed

Filename	Overview
src/agentex/lib/core/tracing/obs_ids.py	New correlation helper — correctly handles ImportError and OTel is_valid checks, but exception handling is too narrow: any runtime error from ddtrace/OTel propagates into start_span. Also emits the OTel null-sentinel span_id (already flagged in a previous thread).
src/agentex/lib/core/tracing/trace.py	Merges obs_correlation() into serialized_data at span creation. The dict-unpack pattern crashes with TypeError when data is list[dict] (already flagged in previous thread); otherwise the logic is sound.
.github/workflows/ci.yml	Runner condition broadened from an exact repo-name match to startsWith('stainless-sdks/'), allowing the depot runner for any fork under that org prefix. Change is intentional and safe.
.stats.yml	Auto-generated openapi spec URL and hash update — no logic changes.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Caller
    participant Trace/AsyncTrace
    participant obs_ids
    participant OTel/ddtrace

    Caller->>Trace/AsyncTrace: "start_span(name, data=...)"
    Trace/AsyncTrace->>obs_ids: obs_correlation()
    obs_ids->>obs_ids: get_obs_mode() reads SGP_OBS_MODE
    alt "mode == lgtm"
        obs_ids->>OTel/ddtrace: _lgtm_ids() get_current_span()
    else "mode == dual"
        obs_ids->>OTel/ddtrace: _lgtm_ids() or _ddtrace_ids()
    else "mode == dd_only default"
        obs_ids->>OTel/ddtrace: _ddtrace_ids() current_trace_context()
    end
    OTel/ddtrace-->>obs_ids: (trace_id, span_id) or None
    obs_ids-->>Trace/AsyncTrace: obs.trace_id and obs.span_id or empty dict
    Trace/AsyncTrace->>Trace/AsyncTrace: merge obs into serialized_data
    Trace/AsyncTrace->>Trace/AsyncTrace: create Span with merged data
    Trace/AsyncTrace-->>Caller: Span

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Caller
    participant Trace/AsyncTrace
    participant obs_ids
    participant OTel/ddtrace

    Caller->>Trace/AsyncTrace: "start_span(name, data=...)"
    Trace/AsyncTrace->>obs_ids: obs_correlation()
    obs_ids->>obs_ids: get_obs_mode() reads SGP_OBS_MODE
    alt "mode == lgtm"
        obs_ids->>OTel/ddtrace: _lgtm_ids() get_current_span()
    else "mode == dual"
        obs_ids->>OTel/ddtrace: _lgtm_ids() or _ddtrace_ids()
    else "mode == dd_only default"
        obs_ids->>OTel/ddtrace: _ddtrace_ids() current_trace_context()
    end
    OTel/ddtrace-->>obs_ids: (trace_id, span_id) or None
    obs_ids-->>Trace/AsyncTrace: obs.trace_id and obs.span_id or empty dict
    Trace/AsyncTrace->>Trace/AsyncTrace: merge obs into serialized_data
    Trace/AsyncTrace->>Trace/AsyncTrace: create Span with merged data
    Trace/AsyncTrace-->>Caller: Span

_{Reviews (2): Last reviewed commit: "feat(tracing): correlate business spans ..." | Re-trigger Greptile}

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

greptile-apps[bot] reacted with thumbs up emoji

All reactions

👍 1 reaction

This was referenced Jul 21, 2026

feat(tracing): establish W3C trace context at FastACP ingress #466

Closed

feat(tracing): propagate trace context into Temporal via OTel interceptor #467

Open

NiteshDhanpal changed the title ~~feat(tracing): correlate business spans with active observability trace~~ [WIP] feat(tracing): correlate business spans with active observability trace

Jul 21, 2026

greptile-apps Bot reviewed

Jul 21, 2026

View reviewed changes

Comment thread

src/agentex/lib/core/tracing/trace.py

Comment on lines +86 to +88

		obs = obs_correlation()
		if obs:
		serialized_data = {(serialized_data or {}), obs}

greptile-apps Bot Jul 21, 2026

Copy link

Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

serialized_data can be a list when data is passed as list[dict[str, Any]] — recursive_model_dump returns a list for that case, and a non-empty list is truthy, so serialized_data or {} evaluates to the list. {**<list>, **obs} then raises TypeError: argument after ** must be a mapping, not list at span-creation time whenever an obs context is active. The same pattern is duplicated in AsyncTrace.start_span (line 243–244). A safe guard would be: if obs and isinstance(serialized_data, dict): serialized_data = {**serialized_data, **obs} (or similarly initialise to {} when None).

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/agentex/lib/core/tracing/trace.py
Line: 86-88

Comment:
`serialized_data` can be a list when `data` is passed as `list[dict[str, Any]]` — `recursive_model_dump` returns a list for that case, and a non-empty list is truthy, so `serialized_data or {}` evaluates to the list. `{**<list>, **obs}` then raises `TypeError: argument after ** must be a mapping, not list` at span-creation time whenever an obs context is active. The same pattern is duplicated in `AsyncTrace.start_span` (line 243–244). A safe guard would be: `if obs and isinstance(serialized_data, dict): serialized_data = {**serialized_data, **obs}` (or similarly initialise to `{}` when `None`).

How can I resolve this? If you propose a fix, please make it concise.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

src/agentex/lib/core/tracing/obs_ids.py

Comment on lines +61 to +64

		ctx = tracer.current_trace_context()
		if ctx and ctx.trace_id:
		return format(ctx.trace_id, "032x"), format(ctx.span_id or 0, "016x")
		return None

greptile-apps Bot Jul 21, 2026

Copy link

Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ctx.span_id or 0 falls back to 0 when span_id is None or zero, which formats to "0000000000000000" — the OTel invalid/null sentinel. This produces a correlation pair where obs.trace_id is populated but obs.span_id is the null sentinel, which consumers may interpret as "no active span." If span_id is absent, skipping the correlation entirely is more correct.

Suggested change

	ctx = tracer.current_trace_context()
	if ctx and ctx.trace_id:
	return format(ctx.trace_id, "032x"), format(ctx.span_id or 0, "016x")
	return None
	ctx = tracer.current_trace_context()
	if ctx and ctx.trace_id and ctx.span_id:
	return format(ctx.trace_id, "032x"), format(ctx.span_id, "016x")
	return None

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/agentex/lib/core/tracing/obs_ids.py
Line: 61-64

Comment:
`ctx.span_id or 0` falls back to `0` when `span_id` is `None` or zero, which formats to `"0000000000000000"` — the OTel invalid/null sentinel. This produces a correlation pair where `obs.trace_id` is populated but `obs.span_id` is the null sentinel, which consumers may interpret as "no active span." If `span_id` is absent, skipping the correlation entirely is more correct.

```suggestion
    ctx = tracer.current_trace_context()
    if ctx and ctx.trace_id and ctx.span_id:
        return format(ctx.trace_id, "032x"), format(ctx.span_id, "016x")
    return None
```

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions


          feat(tracing): correlate business spans with active observability trace

cd91ba7

Tag each adk business span at creation time with the active observability
trace_id/span_id (OpenTelemetry preferred, ddtrace fallback), selected via
SGP_OBS_MODE. The business trace_id stays the run-level task id; the obs ids
are attached as span data (obs.trace_id/obs.span_id) using the span-link
pattern so a span can be pivoted to its per-turn Tempo/Datadog trace without
collapsing the agent-run grouping.

Capture is emit-time (in start_span), never at ingest, so batched
/v5/spans/batch flushing cannot shatter one run across N obs traces. Returns
no tag when no obs context is active (safe no-op; default SGP_OBS_MODE=dd_only).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

NiteshDhanpal force-pushed the obs/01-span-obs-correlation branch from a3a7ae1 to cd91ba7 Compare

July 21, 2026 20:46

NiteshDhanpal changed the base branch from main to next

July 21, 2026 20:46

mohammadatallah-scale mentioned this pull request

Jul 23, 2026

feat(tracing): adopt ambient W3C/OTel trace id for agent spans #470

Closed

2 tasks done

mohammadatallah-scale approved these changes

Jul 23, 2026

View reviewed changes

NiteshDhanpal mentioned this pull request

Jul 24, 2026

feat(tracing): make ddtrace adopt the OTel trace id in dual mode #471

Closed

NiteshDhanpal merged commit c67e177 into next

Jul 24, 2026

46 of 51 checks passed

Uh oh!

There was an error while loading. Please reload this page.

NiteshDhanpal deleted the obs/01-span-obs-correlation branch

July 24, 2026 06:18

NiteshDhanpal changed the title ~~[WIP] feat(tracing): correlate business spans with active observability trace~~ feat(tracing): correlate business spans with active observability trace

Jul 24, 2026

NiteshDhanpal mentioned this pull request

Jul 26, 2026

revert: correlate business spans with active observability trace (#465) #472

Closed

Reviewers

greptile-apps[bot]

greptile-apps[bot] left review comments

mohammadatallah-scale

mohammadatallah-scale approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

NiteshDhanpal commented Jul 21, 2026 • edited by greptile-apps Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Stack (1/3)

What this PR does

Design notes

Verification

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps Bot Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

NiteshDhanpal commented Jul 21, 2026 •

edited by greptile-apps Bot

Loading

Uh oh!

There was an error while loading. Please reload this page.