GeezerVison is showing github.com — feat(tracing): write the span attributes the failure checks read by mohammadatallah-scale · Pull Request #555 · scaleapi/scale-agentex-python
Conversation
Important Pending Mohammad's self-review. This note is removed by a human, not automation.
Offline failure checks read attributes an agent declares on its spans, and AgentEx agents hand-build every key with nothing checking the values (OVE-1375). This adds a helper that writes and validates them from one spec, ported to scale-gp-beta in scaleapi/sgp-dev-python-staging#10.
- Spec: one validated object writes the root span's declarations as flat attribute keys, with the output schema as JSON text that is checked again when it renders.
- Per-span helpers: one call each for a required output, a verification result and an escalation that may name several failures, passed as a span's
data.
- Spans: harness tool spans carry the tool keys, OpenAI hook results are JSON, model calls name their model, and a span cancelled mid-call reads failed.
- OpenTelemetry:
otel_attributes turns any of these into span attributes, writing the verification rules as JSON text, and otel_span_id names a span to escalates.
Merge waits until public scale-agentex-python merges reopen after the vulnerability work, about end of November. Until then the Dogfood agent carries its own copy of the helper, as the end-to-end harness does.
Test plan
- The two-failure escalation, cancelled span and model id tests fail on the previous head and pass now.
- OpenTelemetry SDK spans written through the adapter pass the failure checks' escalation and verification_ran checks. Without it the SDK drops the verification rules and the check cannot judge, and the adapter tests fail against a pass-through adapter.
- Same results and error messages as the scale-gp-beta port over 160,000 generated inputs, including schema edits made after construction and random non-string keys.
- The failure checks' own span reader (OVE-1371 branch) reads Claude Code and Codex tool spans, passed or failed, as tool calls and decodes an object OpenAI result.
- gap: not exported to a live SGP spans API from an AgentEx agent, the span request tests cover the payload and the port's dev-sgp round trip covers the store.
- gap: a traced example whose spans pass the failure checks is not shown here, the end-to-end harness in OVE-1400 proves it.
🤖 Generated with Claude Code
  Confidence Score: 5/5
The PR appears safe to merge; no actionable issues remain from this review.
Summary
This PR adds validated attributes for offline failure checks and records tool names, model names, JSON results, and interrupted calls on spans.
- Agents can declare failure-check facts directly on their spans.
- Tool and model spans identify the calls they represent.
- OpenAI tool spans keep structured results in JSON text.
- Cancelled calls leave their spans marked as failed.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
A[FailureCheckSpec and per-span helpers] --> B[Attribute dictionary]
B --> C[AgentEx span data]
C --> D[SGP metadata]
B --> E[otel_attributes]
E --> F[OpenTelemetry span attributes]
G[OpenTelemetry span] --> H[otel_span_id]
H --> I[escalates]
I --> B
Loading
Reviews (7) · Last reviewed commit: "feat(tracing): write failure-check attri..." · Reviewed by Greptile
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
greptile-apps[bot] reacted with thumbs up emoji
Offline failure checks read sgp.agent.* attributes the agent declares on its
spans. FailureCheckSpec validates the root span declarations on construction
and renders them as flat dotted keys holding native JSON values, which the SGP
processor ships verbatim as span metadata. output_name, verification_result
and escalates write the per-span attributes.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The span store keeps metadata in a ClickHouse JSON column, where a schema
object spreads into many dynamic paths whose keys can collide after
dot-splitting. FailureCheckSpec still takes a JSON object and validates it,
then renders sgp.agent.output_schema as a JSON string. Every other attribute
stays a native value.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
output_schema is a mutable dict on a frozen spec, so an edit after
construction skipped the checks construction applies and could write NaN or
invalid JSON text. attributes() now runs the same check before writing the
schema, so an edited schema fails with the error construction raises.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
|
P3 tests/lib/core/tracing/test_failure_attributes.py:154 - the "error names the attribute" contract is pinned only for step_budget, stop_reason and prompt_version. Relabelling escalation_channels as quote_arguments in _NORMALIZERS, or renaming the required_outputs, verification and VerificationRule labels, leaves all 118 tests green.
P3 failure_attributes.py:89 - json.dumps coerces non-string keys, so output_schema={1: "a", "1": "b"} is accepted and silently stored as {"1": "b"} instead of being rejected.
Refuted: __agent_version__ and __commit_sha__ README claims hold at this head (_add_source_to_span, code_revision), and the port's logic matches line for line apart from 3.9 syntax.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
adk.tracing.span yields None when trace_id is empty or the span fails to
start, so the example reading root.data raised AttributeError on that path.
It now guards the span the way the tutorial's acp.py does.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Only step_budget, stop_reason and prompt_version errors were checked for
the attribute they name, so relabelling escalation_channels as
quote_arguments still passed. Every root field and VerificationRule now
asserts the name its error starts with.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
P3 Follow-up: the attribute-naming pins landed in 815dad0, and each label swap above now fails 2 to 11 tests. The non-string key coercion is left open because fixing it changes behavior the scale-gp-beta port must match, so both helpers need it together.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
The depth was an inline number, so a reader could not tell what it controls.
It is now a named class constant.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
A path like send_email.to.cc was rejected with a message that did not say
why. Tool names cannot contain dots, so an entry is a tool and one top-level
argument, and the error now says nested argument paths are not supported.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
escalates("") failed with a message about a verification span id, which did
not name the attribute being written. The message now starts with escalates,
as the other validation errors start with their attribute.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
JSON writes every object key as a string, so the encoder quietly turned an
int, bool, float or None key into text and let {1: "a", "1": "b"} collapse
into one entry. Any non-string mapping key in the schema is now rejected, at
construction and again when attributes render.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
A successful Claude Code or Codex tool call reports no is_error, so the
failure checks could not tell its span was a tool call. Every harness tool
span now carries gen_ai.operation.name=execute_tool and gen_ai.tool.name.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
The hooks wrote str(result), a Python repr the failure checks cannot decode.
Objects are now written as JSON text, strings stay as they are, and str is
the fallback only when the result cannot be serialized.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
mohammadatallah-scale
changed the title
feat(tracing): add failure-check span attribute helper
feat(tracing): write the span attributes the failure checks read
Oct 28, 2026
The escalation check reads a list of span ids, so escalates() now takes several and writes them as a list.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The span context managers caught only Exception, so a cancelled or interrupted call ended its span as SUCCESS. They now record any BaseException and re-raise it.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The LiteLLM completion spans and the OpenAI Agents streaming model span now carry gen_ai.request.model, the attribute the failure checks read as the model a call was made to.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
otel_attributes turns the helpers' data into OpenTelemetry attribute values, writing the
verification rules and any other object as JSON text, and otel_span_id gives the id escalates
takes for an OpenTelemetry span.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This branch has not been deployed
No deployments
Reviewers
At least 1 approving review is required to merge this pull request.
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|