GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | ||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | ||||||||||||||||||||||
GeezerVison is showing github.com — feat(harness): unified harness surface — foundation (span derivation, delivery adapters, emitter) by declan-scale · Pull Request #412 · scaleapi/scale-agentex-python Conversation
|
| Filename | Overview |
|---|---|
| src/agentex/lib/core/harness/types.py | Defines core types: StreamTaskMessage union, OpenSpan/CloseSpan/SpanSignal, TurnUsage, TurnResult, HarnessTurn protocol. Clean dataclass/Pydantic design; runtime-checkable protocol for structural duck-typing. |
| src/agentex/lib/core/harness/span_derivation.py | Pure stateful reducer converting StreamTaskMessage events to span open/close signals. Handles both Start+Delta+Done (streamed) and Full (LangGraph-style) tool-request paths. Minor: reasoning detection uses string comparison (content.type == "reasoning") rather than isinstance, which is fragile if the type string changes. |
| src/agentex/lib/core/harness/auto_send.py | Async delivery adapter: index-keyed ctx_map routing, per-context error-guarded _close_all(), last-segment final_text semantics, Full(TextContent) contributing to final_text, and correct streaming of Start(ToolRequestContent). All previously flagged gaps addressed in current HEAD. |
| src/agentex/lib/core/harness/emitter.py | UnifiedEmitter facade wiring trace context + delivery mode. auto_send_turn correctly calls turn.usage() after auto_send consumes the stream; the previously flagged eager-evaluation bug is resolved. |
| src/agentex/lib/core/harness/tracer.py | Best-effort span adapter to adk.tracing; correctly swallows tracing errors, records is_error via span.data mutation. Duplicate-open span orphan behavior is documented and tracked in AGX1-376. |
| src/agentex/lib/core/harness/yield_delivery.py | Straightforward passthrough async generator with try/finally flush; correctly handles early aclose() via GeneratorExit triggering the finally block. |
| tests/lib/core/harness/conformance/test_conformance.py | Conformance test asserts only idempotency (derive_all(x)==derive_all(x)), not correctness of the actual span signals. The registered builtin-single-tool fixture never has its expected signals verified. Tracked in AGX1-373 but currently provides near-zero regression value. |
| .github/workflows/harness-integration.yml | New CI job delegating to ./scripts/test (preserving exact multi-Python-version invocation); placeholder live-matrix job with if:false is intentional and harmless. |
%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Harness as Harness tap
participant Emitter as UnifiedEmitter
participant YD as yield_events
participant AS as auto_send
participant SD as SpanDeriver
participant ST as SpanTracer
participant Streaming as adk.streaming
Note over Emitter: yield_turn (sync HTTP ACP)
Harness->>Emitter: yield_turn(turn)
Emitter->>YD: yield_events(turn.events, tracer)
loop each event
YD->>SD: observe(event)
SD-->>YD: [SpanSignals]
YD->>ST: handle(signal)
YD-->>Emitter: yield event
end
YD->>SD: flush()
SD-->>YD: [unclosed signals]
YD->>ST: handle(signal)
Note over Emitter: auto_send_turn (async/Temporal)
Harness->>Emitter: auto_send_turn(turn)
Emitter->>AS: auto_send(turn.events, ...)
loop each event
AS->>SD: observe(event)
SD-->>AS: [SpanSignals]
AS->>ST: handle(signal)
AS->>Streaming: streaming_task_message_context(...)
Streaming-->>AS: ctx
AS->>Streaming: ctx.stream_update(delta) / ctx.close()
end
AS->>SD: flush()
AS-->>Emitter: TurnResult(final_text)
Emitter->>Harness: "result (with result.usage = turn.usage())"
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Harness as Harness tap
participant Emitter as UnifiedEmitter
participant YD as yield_events
participant AS as auto_send
participant SD as SpanDeriver
participant ST as SpanTracer
participant Streaming as adk.streaming
Note over Emitter: yield_turn (sync HTTP ACP)
Harness->>Emitter: yield_turn(turn)
Emitter->>YD: yield_events(turn.events, tracer)
loop each event
YD->>SD: observe(event)
SD-->>YD: [SpanSignals]
YD->>ST: handle(signal)
YD-->>Emitter: yield event
end
YD->>SD: flush()
SD-->>YD: [unclosed signals]
YD->>ST: handle(signal)
Note over Emitter: auto_send_turn (async/Temporal)
Harness->>Emitter: auto_send_turn(turn)
Emitter->>AS: auto_send(turn.events, ...)
loop each event
AS->>SD: observe(event)
SD-->>AS: [SpanSignals]
AS->>ST: handle(signal)
AS->>Streaming: streaming_task_message_context(...)
Streaming-->>AS: ctx
AS->>Streaming: ctx.stream_update(delta) / ctx.close()
end
AS->>SD: flush()
AS-->>Emitter: TurnResult(final_text)
Emitter->>Harness: "result (with result.usage = turn.usage())"
General comment
UnifiedEmitter ignores duck-typed tracer override and constructs real SpanTracer
handle() method to UnifiedEmitter(..., tracer=custom_tracer) did not use that object. Instead, the emitter fell through to default construction of SpanTracer, which attempted to import the real ADK stack and failed in this environment with ModuleNotFoundError: No module named 'temporalio'. This contradicts the requested override behavior for tracing in the unified emitter surface.src/agentex/lib/core/harness/emitter.py only accepts overrides when isinstance(tracer, SpanTracer) is true. A valid injected/duck-typed tracer object is ignored, causing fallback to SpanTracer(...) whenever trace_id is truthy.handle(signal) contract, or define and use a runtime-checkable tracer Protocol instead of requiring isinstance(tracer, SpanTracer).Reviews (12): Last reviewed commit: "feat(harness): mark derived tool spans a..." | Re-trigger Greptile
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
d21c54a to
ebc468d
Compare
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
|
@greptile review |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
|
@greptile review |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
b4b8b33 to
da780a1
Compare
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
|
On AGX1-371 (no Golden agent's Since it's an additive optional ( |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
Will land this in scale-agentex and then have a follow up here. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
Approach A (Agentex event stream as canonical source of truth): one tap per harness feeds shared yield/auto-send delivery adapters and a span-deriving tracing tap. Additive backwards-compat, stacked PRs <1000 lines, conformance + live-matrix testing (3 test agents per harness: sync/async/temporal). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
… golden-agent integration - Make tracing-tap span derivation explicit (tool open on Done of a ToolRequestContent index, close on matching ToolResponseContent by tool_call_id; parallel-safe; reasoning start->done). Flag missing is_error on ToolResponseContent as an additive upstream decision. - Add first-class TurnUsage/TurnResult shape (aligned to llm_metrics token taxonomy) attached to the turn span via span(data=) and reused for metrics. - Document golden-agent integration: all SGP/sandbox/secret/MCP coupling stays in the agent; only parsing/streaming/tracing/usage move to SDK taps + emitter; sandbox-setup events chain before the harness stream. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
… 1-3) Bite-sized TDD tasks: foundation types, pure SpanDeriver, SpanTracer adapter, yield + auto_send delivery, UnifiedEmitter facade, conformance scaffold + CI job. Migration/parser PRs (4-9) listed as follow-on plans. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
… signals Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
… handling in SpanDeriver Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…sts for SpanTracer Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…on early close Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…reaming + tracing) Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
… + cover error/finally paths in auto_send Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…send_turn + doc tracer modes Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…egistry semantics Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…he package Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…or consistency Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…reportImplicitOverride) Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…MissingImports) Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…ast-segment, created_at (AGX1-377, AGX1-378) auto_send.py: - Replace single current_ctx with ctx_map[index] so parallel streams route correctly - Open a streaming context for ALL content types on Start (not just text/reasoning), fixing tool_request/tool_response stream delivery (AGX1-377) - Reset final_text_parts on each new Start(TextContent) and on Full(TextContent) so multi-step turns return the LAST text segment, not the full accumulation - Add created_at: datetime | None param; forward to every streaming_task_message_context call (AGX1-378) span_derivation.py: - _on_full: handle Full(ToolRequestContent) by opening a tool span keyed by tool_call_id if not already open; adds LangGraph full-event harness support Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…ting, last-segment, created_at, Full ToolRequest spans test_auto_send.py: - Fix test 2: remove bare Start(ToolRequestContent) from events (old behavior was that Start did not open a ctx; new behavior does, so test was updated to use Full-only events that still verify the two-context behavior) - Extend _FakeStreaming to record created_at on each context call - Add test 6: streamed tool_request opens a ctx + routes deltas (AGX1-377 core) - Add test 7: interleaved indexes route deltas to correct per-index contexts - Add test 8: multi-step turns return the LAST text segment only - Add test 9: Full(TextContent) contributes its content to final_text - Add test 10: created_at is forwarded to every streaming context call (AGX1-378) test_span_derivation.py: - Add test_full_tool_request_opens_span: Full(ToolRequestContent) opens a span - Add test_full_tool_request_and_response_paired: paired Full request+response produces a complete OpenSpan+CloseSpan - Add test_full_tool_request_does_not_double_open: idempotent; a Full for an already-open tool_call_id is a no-op Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…n (AGX1-378) So migration helpers can restore the deterministic first-message timestamp on the temporal path. Default None preserves current behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…turn Bug: auto_send_turn passed usage=turn.usage() eagerly, capturing the empty default before the stream was consumed — so TurnResult.usage came back empty for every auto_send harness (real turns populate usage during consumption). Now drives delivery first, then reads usage. Adds a contract-honoring regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
_close_all() iterated open contexts and closed them sequentially with no per-context error guard, so a failure on the first context (e.g. a backend hiccup during teardown) abandoned every remaining open context and their task messages were never finalized. Wrap each close() in its own try/except (warn and continue) so all contexts are attempted regardless of individual failures. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
These were brainstorming/spec artifacts not meant to ship; per review feedback they don't belong in the foundation PR. Also de-references the now-removed docs path in the harness-integration TODO comment. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…ntent.is_error Now that the SDK carries ToolResponseContent.is_error (AGX1-371), thread it through the harness surface: CloseSpan gains an is_error field, the span deriver populates it when closing a tool span on a ToolResponseContent, and the tracer records the status on span.data (Span has no dedicated error field; None means the harness reported no status, so data is left untouched). This preserves tool-error tracing fidelity for harnesses that report tool failures (e.g. golden agent's ToolCompleted.is_error) once they move onto the unified surface, instead of silently dropping the status. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
3384505 to
41b1a32
Compare
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.