GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fix(harness): assert cross-channel (yield vs auto-send) conformance equivalence [AGX1-373] by declan-scale · Pull Request #414 · scaleapi/scale-agentex-python

Conversation

@declan-scale

@declan-scale declan-scale commented Jun 18, 2026 •
edited by greptile-apps Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Summary

Fast-follow on the unified harness surface foundation. Upgrades the conformance runner to actually assert cross-channel equivalence between yield_events and auto_send, replacing the prior determinism-only test that merely ran the same deriver twice.

Equivalence approach

Both channels are driven over each fixture using in-test fakes (mirroring patterns from test_yield_delivery.py and test_auto_send.py). The results are normalised to LogicalDelivery(content_type, identity) tuples that strip the streaming-envelope difference:

  • yield channel delivers StreamTaskMessageFull(ToolResponseContent) verbatim.
  • auto_send channel delivers the same content by opening a streaming context with initial_content and closing it immediately (no deltas).

Both collapse to LogicalDelivery("tool_response", frozenset({("tool_call_id", ...), ("name", ...)})) and compare equal.

Text/reasoning deliveries are normalised to sequential position within their type (since auto_send has no event index in its streaming sink).

Span signals are asserted identical: both channels call SpanDeriver.observe() on the same event sequence, so the derived signals must match.

Full-message decision: keep open+immediate-close

auto_send retains the existing approach of posting a StreamTaskMessageFull (tool_request/tool_response) via streaming_task_message_context(...).__aenter__() + immediate close(). Rationale:

  • StreamingTaskMessageContext.close() persists initial_content when the accumulator is empty, so the message is correctly written.
  • This mirrors the _langgraph_async.py pattern already in production.
  • Switching to adk.messages.create would require a new injectable dependency for no observable benefit.

The envelope difference (Full vs Start+Done on the wire) is documented as an acceptable design choice in runner.py alongside the decision rationale.

Fixtures

  • builtin-single-tool — retained (existing fixture, tool request+response cycle)
  • streaming-text — new: text Start/delta/delta/Done path
  • reasoning-block — new: reasoning Start/delta/Done (exercises reasoning span open/close)

Results

  • ./scripts/test tests/lib/core/harness/ — 35 passed on Python 3.12 and 3.13
  • uv run pyright src/agentex/lib/core/harness/ — 0 errors

🤖 Generated with Claude Code

Greptile Summary

This PR upgrades the harness conformance runner from a simple determinism check (deriving spans twice from the same events) to a genuine cross-channel equivalence test: both yield_events and auto_send are driven over each fixture with in-test fakes, their outputs are normalised to LogicalDelivery tuples (including payloads for initial content, deltas, and tool arguments/response bodies), and both their recorded span signals and logical delivery sequences are asserted equal.

  • runner.py: introduces LogicalDelivery, _RecordingTracer (records what each channel's tracer actually receives rather than re-deriving), _FakeStreaming/_FakeCtx backends, and run_cross_channel_conformance() — directly addressing all five previously flagged gaps (tautological spans, omitted stream/tool payloads, suppressed tool-request deliveries, and dropped initial_content).
  • test_conformance.py: adds four new fixtures (streaming-text, reasoning-block, streamed-tool-request, parallel-tools-with-error) and an async test_cross_channel_equivalence parametrized over all five, alongside the retained idempotency regression test.
  • agentex-tutorials-test.yml: adds a conditional CI step that installs the Claude Code and Codex CLIs only for the relevant tutorial matrix entries, with CLAUDE_CODE_VERSION pinned and ANTHROPIC_API_KEY/live-test gate vars threaded through.

Confidence Score: 5/5

All changes are confined to the conformance test harness and CI workflow; no production code paths are modified.

The conformance runner now drives both delivery channels with real injected fakes and compares what they actually emitted, closing every structural gap the previous review identified. The five new fixtures cover text streaming, reasoning, streamed tool-request delivery, and parallel/error tool scenarios. The CI step is gated by matrix tutorial name and uses a pinned Claude Code version.

No files require special attention.

Important Files Changed
Filename Overview
tests/lib/core/harness/conformance/runner.py Major rewrite: adds LogicalDelivery normalisation, _RecordingTracer for real span capture, fake streaming backend, and run_cross_channel_conformance() that drives both channels and returns their actual deliveries and span signals for comparison.
tests/lib/core/harness/conformance/test_conformance.py Adds 5 parametrized fixtures (builtin-single-tool, streaming-text, reasoning-block, streamed-tool-request, parallel-tools-with-error) and a new async test_cross_channel_equivalence that asserts both logical deliveries and recorded span signals are identical across channels.
.github/workflows/agentex-tutorials-test.yml Adds a conditional step that installs @anthropic-ai/claude-code (pinned to 2.1.142) and @openai/codex (unpinned, per matching sandbox Dockerfile policy) for the relevant tutorial matrix entries, and threads ANTHROPIC_API_KEY / CLAUDE_LIVE_TESTS / CODEX_LIVE_TESTS through to the tutorial test run step.

Reviews (14): Last reviewed commit: "test(harness): exercise ToolResponseCont..." | Re-trigger Greptile

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
greptile-apps[bot] reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@declan-scale
declan-scale force-pushed the declan-scale/unified-harness-surface branch from d21c54a to ebc468d Compare June 18, 2026 17:29
greptile-apps[bot]
greptile-apps Bot reviewed Jun 18, 2026
View reviewed changes
Comment thread tests/lib/core/harness/conformance/runner.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread tests/lib/core/harness/conformance/runner.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread tests/lib/core/harness/conformance/runner.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread tests/lib/core/harness/conformance/runner.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@declan-scale
declan-scale force-pushed the declan-scale/agx1-373-conformance-equivalence branch 3 times, most recently from b4c53ca to cae14d4 Compare June 18, 2026 19:24
@declan-scale

declan-scale commented Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

@greptile review
greptile-apps[bot] reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
greptile-apps[bot]
greptile-apps Bot reviewed Jun 18, 2026
View reviewed changes
Comment thread tests/lib/core/harness/conformance/runner.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@declan-scale
declan-scale force-pushed the declan-scale/agx1-373-conformance-equivalence branch from 8cd851c to 2e820c7 Compare June 18, 2026 21:08
@declan-scale

declan-scale commented Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

@greptile review
greptile-apps[bot] reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@declan-scale
declan-scale force-pushed the declan-scale/unified-harness-surface branch from b4b8b33 to da780a1 Compare June 22, 2026 13:48
@declan-scale
declan-scale force-pushed the declan-scale/agx1-373-conformance-equivalence branch from 2e820c7 to 37421b6 Compare June 22, 2026 13:48
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(openai-conformance): adapt to cross-channel runner on #414 found…
cbc2a9b 
…ation

Replace the old determinism-only test (derive_all) with the full
cross-channel assertion pattern: register fixtures with per-module
_OPENAI_FIXTURES, call run_cross_channel_conformance, and assert
logical-delivery and span-signal equivalence across yield_events and
auto_send — matching the pattern in test_conformance.py.

Swap ReasoningSummaryDelta for ReasoningContentDelta so the runner's
payload accumulator recognises the delta type and the payload comparison
exercises the reasoning seeding path. Remove derive_all import.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(langgraph): adapt conformance test to cross-channel runner (AGX1…
40b4cbe 
…-373)

Rewrites test_langgraph_conformance.py to use the cross-channel runner
from PR #414 (run_cross_channel_conformance, LogicalDelivery) instead of
the simpler derive_all-only API it was written against.

The four fixtures (text-only, single-tool, reasoning, multi-step) are
retained as canonical StreamTaskMessage* sequences. Each is now exercised
by test_cross_channel_equivalence (yield_events vs auto_send logical
deliveries and span signals) plus the backward-compat
test_span_derivation_is_deterministic guard.

LangGraph tool requests arrive as Full events from the "updates" stream;
auto_send handles them via open+close, yielding the same LogicalDelivery
on both channels. No coalesce_tool_requests option is needed.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@declan-scale
declan-scale force-pushed the declan-scale/agx1-373-conformance-equivalence branch from 37421b6 to df3461c Compare June 22, 2026 14:13
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(openai-conformance): adapt to cross-channel runner on #414 found…
dcc0b33 
…ation

Replace the old determinism-only test (derive_all) with the full
cross-channel assertion pattern: register fixtures with per-module
_OPENAI_FIXTURES, call run_cross_channel_conformance, and assert
logical-delivery and span-signal equivalence across yield_events and
auto_send — matching the pattern in test_conformance.py.

Swap ReasoningSummaryDelta for ReasoningContentDelta so the runner's
payload accumulator recognises the delta type and the payload comparison
exercises the reasoning seeding path. Remove derive_all import.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(langgraph): adapt conformance test to cross-channel runner (AGX1…
4fd2ff4 
…-373)

Rewrites test_langgraph_conformance.py to use the cross-channel runner
from PR #414 (run_cross_channel_conformance, LogicalDelivery) instead of
the simpler derive_all-only API it was written against.

The four fixtures (text-only, single-tool, reasoning, multi-step) are
retained as canonical StreamTaskMessage* sequences. Each is now exercised
by test_cross_channel_equivalence (yield_events vs auto_send logical
deliveries and span signals) plus the backward-compat
test_span_derivation_is_deterministic guard.

LangGraph tool requests arrive as Full events from the "updates" stream;
auto_send handles them via open+close, yielding the same LogicalDelivery
on both channels. No coalesce_tool_requests option is needed.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(openai-conformance): adapt to cross-channel runner on #414 found…
e3c14a8 
…ation

Replace the old determinism-only test (derive_all) with the full
cross-channel assertion pattern: register fixtures with per-module
_OPENAI_FIXTURES, call run_cross_channel_conformance, and assert
logical-delivery and span-signal equivalence across yield_events and
auto_send — matching the pattern in test_conformance.py.

Swap ReasoningSummaryDelta for ReasoningContentDelta so the runner's
payload accumulator recognises the delta type and the payload comparison
exercises the reasoning seeding path. Remove derive_all import.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(langgraph): adapt conformance test to cross-channel runner (AGX1…
e03a584 
…-373)

Rewrites test_langgraph_conformance.py to use the cross-channel runner
from PR #414 (run_cross_channel_conformance, LogicalDelivery) instead of
the simpler derive_all-only API it was written against.

The four fixtures (text-only, single-tool, reasoning, multi-step) are
retained as canonical StreamTaskMessage* sequences. Each is now exercised
by test_cross_channel_equivalence (yield_events vs auto_send logical
deliveries and span signals) plus the backward-compat
test_span_derivation_is_deterministic guard.

LangGraph tool requests arrive as Full events from the "updates" stream;
auto_send handles them via open+close, yielding the same LogicalDelivery
on both channels. No coalesce_tool_requests option is needed.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@declan-scale
declan-scale force-pushed the declan-scale/agx1-373-conformance-equivalence branch from ccbd5cf to e3fa1cc Compare June 22, 2026 15:14
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(openai-conformance): adapt to cross-channel runner on #414 found…
390c41b 
…ation

Replace the old determinism-only test (derive_all) with the full
cross-channel assertion pattern: register fixtures with per-module
_OPENAI_FIXTURES, call run_cross_channel_conformance, and assert
logical-delivery and span-signal equivalence across yield_events and
auto_send — matching the pattern in test_conformance.py.

Swap ReasoningSummaryDelta for ReasoningContentDelta so the runner's
payload accumulator recognises the delta type and the payload comparison
exercises the reasoning seeding path. Remove derive_all import.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(langgraph): adapt conformance test to cross-channel runner (AGX1…
eab5388 
…-373)

Rewrites test_langgraph_conformance.py to use the cross-channel runner
from PR #414 (run_cross_channel_conformance, LogicalDelivery) instead of
the simpler derive_all-only API it was written against.

The four fixtures (text-only, single-tool, reasoning, multi-step) are
retained as canonical StreamTaskMessage* sequences. Each is now exercised
by test_cross_channel_equivalence (yield_events vs auto_send logical
deliveries and span signals) plus the backward-compat
test_span_derivation_is_deterministic guard.

LangGraph tool requests arrive as Full events from the "updates" stream;
auto_send handles them via open+close, yielding the same LogicalDelivery
on both channels. No coalesce_tool_requests option is needed.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(openai-conformance): adapt to cross-channel runner on #414 found…
84f99f4 
…ation

Replace the old determinism-only test (derive_all) with the full
cross-channel assertion pattern: register fixtures with per-module
_OPENAI_FIXTURES, call run_cross_channel_conformance, and assert
logical-delivery and span-signal equivalence across yield_events and
auto_send — matching the pattern in test_conformance.py.

Swap ReasoningSummaryDelta for ReasoningContentDelta so the runner's
payload accumulator recognises the delta type and the payload comparison
exercises the reasoning seeding path. Remove derive_all import.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(langgraph): adapt conformance test to cross-channel runner (AGX1…
188b039 
…-373)

Rewrites test_langgraph_conformance.py to use the cross-channel runner
from PR #414 (run_cross_channel_conformance, LogicalDelivery) instead of
the simpler derive_all-only API it was written against.

The four fixtures (text-only, single-tool, reasoning, multi-step) are
retained as canonical StreamTaskMessage* sequences. Each is now exercised
by test_cross_channel_equivalence (yield_events vs auto_send logical
deliveries and span signals) plus the backward-compat
test_span_derivation_is_deterministic guard.

LangGraph tool requests arrive as Full events from the "updates" stream;
auto_send handles them via open+close, yielding the same LogicalDelivery
on both channels. No coalesce_tool_requests option is needed.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@danielmillerp

danielmillerp commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Nice work on the equivalence fix — recording what each channel actually emits is the right move.

One thing I want to make sure we've got covered before relying on this: conformance proves yield ≡ auto-send, but only over the 3 fixtures (single-tool / streaming-text / reasoning-block). That tells us the two delivery paths agree — it doesn't tell us each harness's tap produces the correct canonical stream on the harder cases (parallel tool calls, interleaved reasoning+text, a tool that errors). My worry is the two channels agreeing with each other while both being wrong on an edge case.

Could we add a gnarlier fixture (parallel tools + a tool error) so the equivalence is over something representative? Happy to help write it.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
danielmillerp
danielmillerp approved these changes Jun 22, 2026
View reviewed changes
@declan-scale
declan-scale force-pushed the declan-scale/agx1-373-conformance-equivalence branch from c8c63d1 to 05120f3 Compare June 22, 2026 18:47
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(openai-conformance): adapt to cross-channel runner on #414 found…
430e83d 
…ation

Replace the old determinism-only test (derive_all) with the full
cross-channel assertion pattern: register fixtures with per-module
_OPENAI_FIXTURES, call run_cross_channel_conformance, and assert
logical-delivery and span-signal equivalence across yield_events and
auto_send — matching the pattern in test_conformance.py.

Swap ReasoningSummaryDelta for ReasoningContentDelta so the runner's
payload accumulator recognises the delta type and the payload comparison
exercises the reasoning seeding path. Remove derive_all import.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(langgraph): adapt conformance test to cross-channel runner (AGX1…
ab871c4 
…-373)

Rewrites test_langgraph_conformance.py to use the cross-channel runner
from PR #414 (run_cross_channel_conformance, LogicalDelivery) instead of
the simpler derive_all-only API it was written against.

The four fixtures (text-only, single-tool, reasoning, multi-step) are
retained as canonical StreamTaskMessage* sequences. Each is now exercised
by test_cross_channel_equivalence (yield_events vs auto_send logical
deliveries and span signals) plus the backward-compat
test_span_derivation_is_deterministic guard.

LangGraph tool requests arrive as Full events from the "updates" stream;
auto_send handles them via open+close, yielding the same LogicalDelivery
on both channels. No coalesce_tool_requests option is needed.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@declan-scale
declan-scale force-pushed the declan-scale/unified-harness-surface branch from 3384505 to 41b1a32 Compare June 22, 2026 19:54
@declan-scale
declan-scale force-pushed the declan-scale/agx1-373-conformance-equivalence branch from 05120f3 to c9a907c Compare June 22, 2026 19:54
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(openai-conformance): adapt to cross-channel runner on #414 found…
075e812 
…ation

Replace the old determinism-only test (derive_all) with the full
cross-channel assertion pattern: register fixtures with per-module
_OPENAI_FIXTURES, call run_cross_channel_conformance, and assert
logical-delivery and span-signal equivalence across yield_events and
auto_send — matching the pattern in test_conformance.py.

Swap ReasoningSummaryDelta for ReasoningContentDelta so the runner's
payload accumulator recognises the delta type and the payload comparison
exercises the reasoning seeding path. Remove derive_all import.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(langgraph): adapt conformance test to cross-channel runner (AGX1…
f78546a 
…-373)

Rewrites test_langgraph_conformance.py to use the cross-channel runner
from PR #414 (run_cross_channel_conformance, LogicalDelivery) instead of
the simpler derive_all-only API it was written against.

The four fixtures (text-only, single-tool, reasoning, multi-step) are
retained as canonical StreamTaskMessage* sequences. Each is now exercised
by test_cross_channel_equivalence (yield_events vs auto_send logical
deliveries and span signals) plus the backward-compat
test_span_derivation_is_deterministic guard.

LangGraph tool requests arrive as Full events from the "updates" stream;
auto_send handles them via open+close, yielding the same LogicalDelivery
on both channels. No coalesce_tool_requests option is needed.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Base automatically changed from declan-scale/unified-harness-surface to next June 22, 2026 19:59
declan-scale and others added 7 commits June 22, 2026 16:00
@declan-scale @claude
fix(harness): assert cross-channel (yield vs auto-send) conformance e…
7a4083c 
…quivalence [AGX1-373]

Rebased on the pyright-clean foundation. Includes @OverRide on _RecordingTracer.handle
and relative conformance imports so the whole-repo pyright (scripts/lint) passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@declan-scale @claude
test(harness): address greptile — strengthen conformance payload comp…
0ae359a 
…arison

- Add `payload: str` field to LogicalDelivery (NamedTuple, default "").
- _yield_logical_deliveries: track TextDelta / ReasoningContentDelta
  accumulation per-index; include "".join(deltas) as payload for text/
  reasoning deliveries. Include json.dumps(arguments, sort_keys=True) as
  payload for tool_request; str(content) for tool_response.
- _auto_send_logical_deliveries: collect ("update", delta) entries from
  the _FakeCtx sink between open and close; extract TextDelta /
  ReasoningContentDelta text and accumulate. Carry same tool payload
  fields.
- _yield_text_reasoning_seq: forward payload through when re-keying
  index → seq.
- All 35 harness tests pass; ruff + pyright clean.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@declan-scale @claude
test(harness): propagate AGX1-377/378 fix into conformance — un-suppr…
65c7d0a 
…ess streamed tool-request delivery, include initial_content in payload

- Remove the Start(tool_request)+Done suppression in _yield_logical_deliveries:
  auto_send now delivers streamed tool-request messages (AGX1-377 fix), so both
  channels emit a LogicalDelivery for a streamed tool_request. The cross-channel
  assertion verifies delivery on both sides.

- Include StreamTaskMessageStart.content in payload comparison for text and
  reasoning types: TextContent.content is prepended to accumulated deltas;
  ReasoningContent.summary items are prepended. This catches a channel that
  drops initial_content or reasoning summary (Greptile id 3438655533, P1).
  _auto_send_logical_deliveries mirrors the same seeding from ctx initial_content.

- Add "streamed-tool-request" fixture (Start + Done, no Full) to confirm
  delivery on both channels under the new auto_send behaviour.

- Update "streaming-text" fixture to use non-empty initial_content ("Init") so
  the initial_content seeding is actually exercised by the test.

- Update module/docstring comments that referenced the AGX1-377 suppression.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@declan-scale @claude
ci(tutorials): conditionally install claude-code/codex CLI + enable t…
dc01f80 
…heir gated live tests

Only for *claude_code* / *codex* tutorials (no-op for others). Installs the
CLI via npm (mirrors the golden-agent sandbox image) and sets
CLAUDE_LIVE_TESTS/CODEX_LIVE_TESTS so the gated live agent tests run.
Requires the TUTORIAL_ANTHROPIC_API_KEY secret.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@declan-scale @claude
ci(tutorials): pin claude-code CLI to the golden-agent sandbox versio…
410863c 
…n [greptile]

Install @anthropic-ai/[email protected] (CLAUDE_CODE_VERSION env) to mirror
the golden agent's sandbox image (teams/sgp/agents/golden_agent/sandbox/Dockerfile)
for reproducible CI. Codex is left unpinned to match the sandbox, which also
installs @openai/codex unpinned; documented inline so both stay in lockstep.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@declan-scale @claude
test(harness): add parallel-tools + tool-error conformance fixture
a6ee42d 
Per review (AGX1-373): the existing fixtures only exercise one tool at a
time, so cross-channel equivalence was proven over trivially-orderable
streams — it could not catch the two channels agreeing with each other
while both mishandling parallel/interleaved spans or a failing tool.

Adds a fixture where two tool spans are open simultaneously (one opened
via the streamed Start+Done path, one via Full), close in a different
order than they opened, and one returns an error. The tool error uses
today's "Error: ..." content convention; once ToolResponseContent.is_error
lands (AGX1-371) the fixture should also assert error status on the span.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@declan-scale @claude
test(harness): exercise ToolResponseContent.is_error in parallel-tool…
a04bf5e 
…s fixture

The field landed (AGX1-371), so the failing tool in the parallel-tools
fixture now sets is_error=True instead of relying on the "Error: ..."
string convention. The deriver threads it onto CloseSpan.is_error and the
cross-channel span-signal equivalence covers the error status.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@declan-scale
declan-scale force-pushed the declan-scale/agx1-373-conformance-equivalence branch from c9a907c to a04bf5e Compare June 22, 2026 20:01
@declan-scale
declan-scale merged commit 694960f into next Jun 22, 2026
40 checks passed

Uh oh!

There was an error while loading. Please reload this page.
@declan-scale
declan-scale deleted the declan-scale/agx1-373-conformance-equivalence branch June 22, 2026 20:09
@stainless-app stainless-app Bot mentioned this pull request Jun 22, 2026
chore: release main #424
Merged
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(openai-conformance): adapt to cross-channel runner on #414 found…
73c2dd1 
…ation

Replace the old determinism-only test (derive_all) with the full
cross-channel assertion pattern: register fixtures with per-module
_OPENAI_FIXTURES, call run_cross_channel_conformance, and assert
logical-delivery and span-signal equivalence across yield_events and
auto_send — matching the pattern in test_conformance.py.

Swap ReasoningSummaryDelta for ReasoningContentDelta so the runner's
payload accumulator recognises the delta type and the payload comparison
exercises the reasoning seeding path. Remove derive_all import.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(langgraph): adapt conformance test to cross-channel runner (AGX1…
459bbcf 
…-373)

Rewrites test_langgraph_conformance.py to use the cross-channel runner
from PR #414 (run_cross_channel_conformance, LogicalDelivery) instead of
the simpler derive_all-only API it was written against.

The four fixtures (text-only, single-tool, reasoning, multi-step) are
retained as canonical StreamTaskMessage* sequences. Each is now exercised
by test_cross_channel_equivalence (yield_events vs auto_send logical
deliveries and span signals) plus the backward-compat
test_span_derivation_is_deterministic guard.

LangGraph tool requests arrive as Full events from the "updates" stream;
auto_send handles them via open+close, yielding the same LogicalDelivery
on both channels. No coalesce_tool_requests option is needed.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
declan-scale added a commit that referenced this pull request Jun 22, 2026
@declan-scale @claude
test(langgraph): adapt conformance test to cross-channel runner (AGX1…
c6c5bb8 
…-373)

Rewrites test_langgraph_conformance.py to use the cross-channel runner
from PR #414 (run_cross_channel_conformance, LogicalDelivery) instead of
the simpler derive_all-only API it was written against.

The four fixtures (text-only, single-tool, reasoning, multi-step) are
retained as canonical StreamTaskMessage* sequences. Each is now exercised
by test_cross_channel_equivalence (yield_events vs auto_send logical
deliveries and span signals) plus the backward-compat
test_span_derivation_is_deterministic guard.

LangGraph tool requests arrive as Full events from the "updates" stream;
auto_send handles them via open+close, yielding the same LogicalDelivery
on both channels. No coalesce_tool_requests option is needed.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@declan-scale declan-scale mentioned this pull request Jun 23, 2026
refactor(harness)!: post-merge cleanup + init templates for the unified harness surface (PR 10) #425
Closed
3 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@greptile-apps greptile-apps[bot] greptile-apps[bot] left review comments
@danielmillerp danielmillerp danielmillerp approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@declan-scale @danielmillerp
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.