GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — [connectors] NATS: wait on the server for a library-recreated consumer. by blp · Pull Request #7824 · feldera/feldera

Conversation

@blp

@blp blp commented Nov 3, 2026

Copy link
Copy Markdown
Member

Fixes the flaky library_recreated_consumer_is_deleted_on_pause test.

Cause

The test deletes the consumer itself, then uses WaitForConsumerCount to wait for async-nats to recreate it. WaitForConsumerCount polls the connector's own created - deleted count:

Step Server consumers Connector count
CreatePipeline, Extend 1 1
DeleteConsumers (by the test) 0 1 (the connector does not see this delete)
WaitForConsumerCount{1} poll 0 1, so the poll returns at once
single server check 0 or 1 (race with the library's recreate)

On a loaded CI runner, the server check came first:

Connector's own count reached 1, but the server disagrees: found 0 consumers on stream 'str': []

Fix

Add WaitForServerConsumerCount, which polls the server's consumer list, and use it for this wait. WaitForConsumerCount stays the same where the connector makes every create and delete.

Testing

  • Delayed the library's recreate by 2 s in the test proxy (temporary change): the old wait fails with the error above, and the new wait passes.
  • NATS engine suite: 70 cases pass.

Closes #7820

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@blp
[connectors] NATS: wait on the server for a library-recreated consumer.
080c391 
library_recreated_consumer_is_deleted_on_pause deletes the consumer from
the test, then uses WaitForConsumerCount to wait for the client library
to recreate it. WaitForConsumerCount polls the connector's own
created - deleted count. The connector does not count a delete from
outside or a recreate by the library, so that count stays at 1 and the
wait returns at once. The single server check after it then raced the
recreate, and lost on a loaded CI runner:

    Connector's own count reached 1, but the server disagrees: found 0
    consumers on stream 'str': []

Add WaitForServerConsumerCount, which polls the server's consumer list,
and use it for this wait.

Tested by delaying the library's recreate by 2 s in the test proxy: the
old wait fails with the error above and the new wait passes. The NATS
engine suite (70 cases) passes.

Closes #7820

Signed-off-by: Ben Pfaff <[email protected]>
@blp
blp requested a review from bestouff November 3, 2026 23:51
@blp blp self-assigned this Nov 3, 2026
@blp blp added bug Something isn't working connectors Issues related to the adapters/connectors crate CI/CD QA Testing and quality assurance rust Pull requests that update Rust code labels Nov 3, 2026
@mythical-fred-oss

mythical-fred-oss Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

LGTM. This is a test-only fix: the new wait polls the server's consumer list, which is the right source of truth after an external delete. The 50 ms poll has a deadline, so it is justified and does not add flakiness.

Ran Result
cargo test -p static-connectors --test connector_suites -- library_recreated_consumer pass
Same test 30× with 8 CPU-burning yes processes running 30/30 pass
connector_suites -- transport::nats:: 70/70 pass
rustfmt --check --edition 2024 on the changed files clean
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
bestouff
bestouff approved these changes Nov 4, 2026
View reviewed changes
@bestouff
bestouff added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit 490e56f Nov 4, 2026
3 checks passed

Uh oh!

There was an error while loading. Please reload this page.
@bestouff
bestouff deleted the nats-consumer-count-race branch November 4, 2026 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bestouff bestouff bestouff approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@blp blp

Labels

bug Something isn't working CI/CD connectors Issues related to the adapters/connectors crate QA Testing and quality assurance rust Pull requests that update Rust code

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Flaky: nats::hardening_tests::library_recreated_consumer_is_deleted_on_pause — consumer count race

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@blp @bestouff
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.