GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |||||||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |||||||||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — [connectors] Fix the flaky DynamoDB retry tests (#7828) by bestouff · Pull Request #7838 · feldera/feldera Conversation
Contributor
Two commits, one per failure in the issue:
Reproduction of the second failure, with a panic injected after the chunk enters its retry and the released task delayed by 100 ms:
Sorry, something went wrong. Uh oh!
There was an error while loading. Please reload this page. All reactions…rns_unhealthy_after_the_threshold The test asserted that reports() was still empty right after the chunk entered its retry wait, racing a 500ms real-wall-clock threshold. On a busy CI runner the test thread can lose the scheduler for that long before it gets to the assertion, so the health report had already landed. Raise the threshold to 3s to give the check a realistic margin; the test's own 10s bound on the opposite wait still leaves plenty of room for the report to show up. Seen twice in the merge queue for #7736 (runs 41750935279 and 41850195763), tracked in #7828. Signed-off-by: mythical-fred-oss[bot] <319348225+mythical-fred-oss[bot]@users.noreply.github.com> …d one `MockServer::start()` hands out servers from a pool and recycles them when a test drops its server. A chunk writer that a failed test leaves behind on the shared runtime can send one more request, which then lands on the recycled server of whichever test picked it up next, and that test counts it as its own. In CI, `the_retry_budget_is_exact` saw two requests instead of one right after `a_long_retry_loop_turns_unhealthy_after_the_threshold` failed. `MockServer::builder().start()` returns a server that is not pooled, so a stray request only ever reaches the server of the test that made it. Reproduced with a panic injected after the chunk entered its retry and the released task delayed by 100 ms: 5 of 10 runs broke another retry test with the pooled server, 0 of 10 with this change. Refs #7828.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. pitfall (medium): The 3 s threshold makes a flake much less likely, but the let start = std::time::Instant::now(); // ... write_batch / gate.entered() ... if start.elapsed() < Duration::from_secs(3) { assert!(harness.publisher.reports().is_empty()); } The threshold check stays correct, and a stalled runner can no longer cause a false failure.
Sorry, something went wrong. Uh oh!
There was an error while loading. Please reload this page. All reactions
Contributor
All reactions
Sorry, something went wrong. Uh oh!
There was an error while loading. Please reload this page. The timer that reports a long retry loop unhealthy starts at the chunk's first retry, which is after the test calls `write_batch`. So an instant taken before that call bounds the timer: while less than the threshold has passed since then, no report can exist yet, and the assertion holds by construction. A runner that stalls past the threshold skips the check instead of failing it, so the test no longer races a wall clock. Refs #7828.
gz
approved these changes
Uh oh!
There was an error while loading. Please reload this page.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
ReviewersUh oh!
There was an error while loading. Please reload this page. AssigneesNo one assignedLabels
None yet
ProjectsMilestoneNo milestoneDevelopmentSuccessfully merging this pull request may close these issues. Uh oh!
There was an error while loading. Please reload this page. Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later. | |||||||||||||||||||||||||||||||||||||||
| You are visitor 01402503 — GeezerVison does not store the pages you look up. Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required. |