GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — [connectors] Fix the flaky DynamoDB retry tests (#7828) by bestouff · Pull Request #7838 · feldera/feldera

Conversation

@bestouff

@bestouff bestouff commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Fixes #7828, on top of #7736.

Two commits, one per failure in the issue:

Failure Cause Fix
a_long_retry_loop_turns_unhealthy_after_the_threshold asserts no report The 500 ms unhealthy_after timer is armed before the gate fires, so a slow runner can pass it before the assertion runs Widen the margin to 3 s (cherry-pick of fred's 0df319db0)
the_retry_budget_is_exact sees 2 requests instead of 1 The first test's panic drops its Gate before its MockServer. The chunk task on the shared runtime wakes and sends one more request, which lands on the recycled pooled server that the second test picked up 80 ms later Use MockServer::builder().start(), which is not pooled

Reproduction of the second failure, with a panic injected after the chunk enters its retry and the released task delayed by 100 ms:

Server Runs Other retry tests broken
Pooled 10 5
Unpooled 10 0

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
mythical-fred-oss Bot and others added 2 commits November 4, 2026 17:14
@mythical-fred-oss @bestouff
[connectors] Widen the unhealthy_after margin in a_long_retry_loop_tu…
aaf80cf
…rns_unhealthy_after_the_threshold

The test asserted that reports() was still empty right after the chunk
entered its retry wait, racing a 500ms real-wall-clock threshold. On a
busy CI runner the test thread can lose the scheduler for that long
before it gets to the assertion, so the health report had already
landed. Raise the threshold to 3s to give the check a realistic margin;
the test's own 10s bound on the opposite wait still leaves plenty of
room for the report to show up.

Seen twice in the merge queue for #7736 (runs 41750935279 and
41850195763), tracked in #7828.

Signed-off-by: mythical-fred-oss[bot] <319348225+mythical-fred-oss[bot]@users.noreply.github.com>
@bestouff
[connectors] Give each DynamoDB mock test its own server, not a poole…
b3f5c36
…d one

`MockServer::start()` hands out servers from a pool and recycles them
when a test drops its server. A chunk writer that a failed test leaves
behind on the shared runtime can send one more request, which then lands
on the recycled server of whichever test picked it up next, and that
test counts it as its own. In CI, `the_retry_budget_is_exact` saw two
requests instead of one right after
`a_long_retry_loop_turns_unhealthy_after_the_threshold` failed.

`MockServer::builder().start()` returns a server that is not pooled, so
a stray request only ever reaches the server of the test that made it.

Reproduced with a panic injected after the chunk entered its retry and
the released task delayed by 100 ms: 5 of 10 runs broke another retry
test with the pooled server, 0 of 10 with this change.

Refs #7828.
mythical-fred-oss[bot]
mythical-fred-oss Bot reviewed Nov 4, 2026
View reviewed changes
Comment thread crates/connectors/src/integrated/dynamodb/test.rs Outdated
// its retry wait, not that this thread resumes promptly enough
// to see `reports()` still empty against a tight threshold on a
// busy CI runner (see #7828).
RetryPolicy::with_delay(None, delay).unhealthy_after(Duration::from_secs(3)),

@mythical-fred-oss mythical-fred-oss Bot Nov 4, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pitfall (medium): The 3 s threshold makes a flake much less likely, but the reports().is_empty() assertion below still races a wall clock. You can make it deterministic: the timer starts at the first retry, which happens after write_batch is called. So take an Instant just before write_batch and only assert when less time than the threshold has passed:

let start = std::time::Instant::now();
// ... write_batch / gate.entered() ...
if start.elapsed() < Duration::from_secs(3) {
    assert!(harness.publisher.reports().is_empty());
}

The threshold check stays correct, and a stalled runner can no longer cause a false failure.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mythical-fred-oss

mythical-fred-oss Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

What I ran (debug, -p dbsp_connectors):

Run Result
dynamodb::test::retry_tests once 18/18 pass (3.07 s)
All 28 tests in dynamodb::test, --test-threads=32, 15 runs the 27 mock-based tests passed in every run; dynamodb_progress_counter_batch_start_resets failed every run because it needs a live DynamoDB endpoint, which this sandbox doesn't have (not caused by this PR)
retry_tests while 160 busy-loop processes kept 80 cores at 2× oversubscription, 20 runs 20/20 pass
  • Unpooled MockServer: a correct fix for the request that leaked from one test into the next. The same helper backs every mock test (server as mock_server), so this covers the whole module. I did not re-run the injected-panic reproduction.
  • Flaky-tests gate: the threshold change only widens the margin, so the test is still timing-based. See the inline comment for a way to make it deterministic.

Otherwise LGTM. This PR only changes tests, so the docs, type-coverage and breaking-change gates don't apply.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bestouff
[connectors] Assert the early health check only before the threshold
bfabcc0
The timer that reports a long retry loop unhealthy starts at the chunk's
first retry, which is after the test calls `write_batch`. So an instant
taken before that call bounds the timer: while less than the threshold
has passed since then, no report can exist yet, and the assertion holds
by construction. A runner that stalls past the threshold skips the check
instead of failing it, so the test no longer races a wall clock.

Refs #7828.
gz
gz approved these changes Nov 4, 2026
View reviewed changes
@bestouff
bestouff merged commit 0cb9f7a into issue-7595 Nov 4, 2026
1 check passed

Uh oh!

There was an error while loading. Please reload this page.

@bestouff
bestouff deleted the issue-7828 branch November 4, 2026 19:09
@bestouff bestouff mentioned this pull request Nov 5, 2026
Flaky: dynamodb retry_tests fail intermittently in merge queue (a_long_retry_loop_turns_unhealthy_after_the_threshold, the_retry_budget_is_exact) #7828
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mythical-fred-oss mythical-fred-oss[bot] mythical-fred-oss[bot] left review comments
@gz gz gz approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@bestouff @gz
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.