Description

mythical-fred-oss

bot

opened on Oct 28, 2026

Issue body actions

Verdict: infrastructure (GKE amd64 runner slowness), not a PR bug. Confidence: high.

What happened

Merge-queue run for PR #7681 failed Integration Tests / Runtime Integration Tests (gke-runners-amd64):
https://github.com/feldera/feldera/actions/runs/40756229796/job/123058212483

tests.runtime_aggtest suite aggregate_tests4 (aggtst_varchar_arg_min_gby, defined in python/tests/runtime_aggtest/aggregate_tests4/test_varchar_argmin.py) failed its result assertion (log line 772):

ASSERTION ERROR: failed view: varchar_arg_min_gby
Expected: [{'id': 0, 'c1': None, 'c2': 'fred'}, {'id': 1, 'c1': 'hello', 'c2': 'variable-length'}]
Received: []

The job then hit its own timeout: ##[error]Process completed with exit code 124 (line 833), with several other test modules (complex_type_tests, asof_tests) never finishing.

The pipeline backing this suite (458f4_aggregate_tests4_6a03df32) started at 19:58:24 (line 469) and took 651.7 seconds just to compile (line 643) — normal compiles in the same log finish in 5-25s. Several sibling pipelines in the same run show the same pattern: aggregate_tests3_ac90aa37 took 482.2s (line 611), aggregate_tests6_e885b154 took 168.8s (line 639), aggregate_tests2_4c437511 took 206.8s (line 549). By the time varchar_arg_min_gby was queried, the suite had burned almost the entire 20-minute job budget on compile queuing/contention alone, leaving no slack for the rest of the suite.

Why not the PR

git diff origin/main...HEAD --stat for PR #7681 touches only crates/connectors/src/integrated/delta_table/test.rs (2 lines, a wait-timeout tweak for Delta CDC test cases). It has no relation to the Python runtime_aggtest SQL suite, the ARG_MIN aggregate, or pipeline compilation/provisioning. aggtst_varchar_arg_min_gby was last touched in a [py] test-harness refactor (210e019e4), unrelated to this PR.

Same symptom family

Same job, same multi-minute compile-time blowouts already tracked in #7654 (10+ min compiles, exit 124) and #7465 (GKE control-plane ENHANCE_YOUR_CALM throttling). This run adds a new data point: the slowness is severe enough (10+ minutes to compile a trivial unoptimized test pipeline) to make an otherwise-correct test fail outright, not just time out on a wait.

Next step

Re-queueing is likely to pass — the signature here is runner/compile-queue contention, not a deterministic bug in ARG_MIN or the test. Worth escalating the GKE amd64 runner-pool capacity/scheduling investigation already open under #7654/#7465, since 10+ minute compiles for a one-table aggregate test suggest the compile step itself is being starved of CPU, not just waiting in a queue.

Activity

1. mythical-fred-oss

   added

   CI/CD

   on Oct 28, 2026
2. mythical-fred-oss

   assigned

   bestouff

   on Oct 28, 2026
3. mythical-fred-oss

   added

   fred-observeTest failure diagnosed by the fred-observe botTest failure diagnosed by the fred-observe bot

   on Oct 28, 2026
4. mythical-fred-oss commented on Oct 30, 2026

   

   mythical-fred-ossBot

   on Oct 30, 2026

   ContributorAuthor

   More actions

   Same symptom family, new data point.

   Run: https://github.com/feldera/feldera/actions/runs/40997977537/job/123876997300 (PR #7698, merge queue — PR touches only js-packages/web-console, no relation to the SQL runtime test suite)

   Compile-queue contention, same GKE amd64 runner:

   • baeed_aggregate_tests_ac2c7dd6 compiled in 597.8s (log line 631)
   • baeed_arithmetic_tests_4f59dcf2: 348.4s (line 699)
   • baeed_aggregate_tests4_6a03df32: 365.9s (line 596)
   • several more in the 90–300s range throughout the run

   Resulting false failure (log lines 793-799, the aggregate_tests6 suite):

   ASSERTION ERROR: failed view: varbinary_min_distinct_gby
   Expected: [{'id': 0, 'c1': b'\x0c\x16 ', 'c2': b'7BMX'}, {'id': 1, 'c1': b'\x178!\x15', 'c2': b' "'}]
   Received: []
   

   followed by the job's own timeout, ##[error]Process completed with exit code 124 (line 847), with illarg_tests/complex_type_tests/aggregate_tests7 still mid-flight.

   Different test (varbinary_min_distinct_gby vs varchar_arg_min_gby) but identical shape: compile times 10-50x normal burn the 20-minute job budget, a query against a pipeline races ahead empty, then the suite times out. aggtst_varbinary_min_distinct_gby wasn't touched by this PR and has no relation to js-packages/web-console.

   Re-queueing is likely to pass.
5. bestouff commented on Nov 3, 2026

   

   bestouff

   on Nov 3, 2026

   Contributor

   More actions

   I looked at both runs. The "slow compile, then the query raced ahead of ingestion" story does not match the logs. This is the post-ingestion face of #6986, so I am closing this one into it.

   Why not a race

   • Every table load is acknowledged by a completion token plus wait_for_completion before any view is queried (python/tests/runtime_aggtest/aggtst_base.py lines 309-313). The last "inputs processed" line comes 3 minutes before the failure.
   • Views are validated in order (line 315). The failing view sits mid-list: module 17 of 45 in aggregate_tests4, module 28 of 37 in aggregate_tests6. Sibling views queried seconds earlier, on the same tables with the same expected rows, returned correct data.
   • Exit code 124 is not a job timeout. Each run.sh child does || exit 255, and xargs reports 124 for that. In both runs the step ended minutes after the assertion, when the other children finished (same correction as on Runtime Integration Tests (amd64): pipeline stopped mid-test, RuntimeExecutionError "Pipeline must be running to wait for completion" #6986).

   Timeline of the first run (UTC, 2026-10-28):

   Time	Event
   20:11:49	aggregate_tests4 pipeline finishes compiling after 651.7 s
   20:12:25	pipeline reaches Running
   20:12:28 to 20:13:41	15 table loads, each acknowledged
   20:13:41 to about 20:16:38	view queries; mid-list, varchar_arg_min_gby returns []
   20:16:38 to 20:17:08	the finally block stops the pipeline, traceback, child exits 255
   20:20:26	last sibling child finishes, xargs exits 124

   The second run has the same shape: Running at 04:13:21, last input at 04:13:34, failure at about 04:16:55, exit 124 at 04:21:12.

   What fits

   A pipeline process that restarts in place between two queries comes back with empty tables, reports Running, and answers ad-hoc queries with empty results. The PipelineRestarted 410 errors in #6986 prove such in-place restarts happen on this instance; they surfaced there during ingestion because completion tokens carry the incarnation. Here the restart came after ingestion, where nothing checked it. The Kubernetes runner does not track container restart counts, so a kubelet restart is invisible to the manager. The memory limit is the 64 GB upper limit (the test only sets a minimum), so a cgroup OOM is unlikely; a node-level kill or a process crash on overloaded spot nodes remains open. Not confirmed from pod logs.

   #7816 makes the harness compare the incarnation UUID when a view check fails, so the next occurrence reports the restart instead of an empty view.
6. bestouff commented on Nov 3, 2026

   

   bestouff

   on Nov 3, 2026

   Contributor

   More actions

   Duplicate of #6986.
7. bestouff

   closed this as not plannedon Nov 3, 2026
8. yermakoffivan

   added a commit that references this issue on Nov 4, 2026

   [python] Report a pipeline restart instead of an empty view in the ag…

   02923be

Issue actions