GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — test: Retry CliRunner apply subprocess on timeout to reduce flakiness by nikolauspschuetz · Pull Request #6592 · feast-dev/feast

Conversation

@nikolauspschuetz

@nikolauspschuetz nikolauspschuetz commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

What changed

CliRunner.run() (the helper that runs feast apply in a fresh child interpreter for unit tests) now retries once on subprocess.TimeoutExpired instead of treating the first timeout as terminal. The per-attempt timeout is unchanged (120s); the timeout message now includes the attempt counter.

Why

The unit-test workflow flaked on unit-test-python (3.11, macos-14) while setting up test_feature_server.py::test_push_and_get:

ERROR sdk/python/tests/unit/test_feature_server.py::test_push_and_get
  AssertionError: stdout: b''
  stderr: b"Command timed out after 120s: [\x27apply\x27]"
2259 passed, 47 skipped, 1 error

run() spawns a new Python interpreter (sys.executable feast.cli apply) — deliberately, for feature-repo import isolation — which has to import Feast (pandas/pyarrow/protobuf/...) from cold. On a loaded macOS runner that cold start can occasionally exceed the 120s guard. The same commit passed on macos-14 / 3.12 and on all three Ubuntu legs, which is the signature of an environmental cold-start stall rather than a code fault.

Crucially, commands routed through run() (e.g. apply) do not materialize, so they cannot trigger the Dask atexit hang that run_with_output() and the module docstring describe. A timeout here is therefore almost always transient slowness, not a deadlock — so a single retry with warm OS/page caches clears it, while a genuine repeated stall still fails once the attempts are exhausted. Retries are intentionally not added to run_with_output(), whose commands can legitimately hang and whose existing machinery recovers partial output rather than retrying.

This builds directly on the CliRunner subprocess stabilization in #6560.

Validation

  • uv run ruff check sdk/python/tests/utils/cli_repo_creator.py — passed
  • uv run ruff format --check sdk/python/tests/utils/cli_repo_creator.py — already formatted
  • Retry logic verified by monkeypatching subprocess.run:
    • transient timeout on attempt 1 → recovered on attempt 2 (returncode == 0)
    • always timing out → returncode == -1, message ... (attempt 2/2): [...], no exception raised
    • success on first attempt → exactly one subprocess call (no behavior change for the passing path)

Note: the full unit suite could not be collected on my machine (local venv lacks the google.cloud test extra pulled by conftest.py); the change is confined to the retry loop, which is covered by the targeted verification above.

Separate from #6591 (which surfaced this flake).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@nikolauspschuetz
nikolauspschuetz requested a review from a team as a code owner July 9, 2026 07:09
@ntkathole
ntkathole force-pushed the test/stabilize-cli-runner-apply-retry branch from aadd2bb to e74df8e Compare July 14, 2026 06:04
@ntkathole ntkathole added the ok-to-test label Jul 14, 2026
@ntkathole
ntkathole force-pushed the test/stabilize-cli-runner-apply-retry branch from e74df8e to 9aca1af Compare July 30, 2026 06:51
@codecov-commenter

codecov-commenter commented Jul 30, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 46.36%. Comparing base (0f149a9) to head (0860438).
⚠️ Report is 6 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6592      +/-   ##
==========================================
+ Coverage   46.35%   46.36%   +0.01%     
==========================================
  Files         414      414              
  Lines       50052    50072      +20     
  Branches     7151     7154       +3     
==========================================
+ Hits        23201    23218      +17     
- Misses      25229    25231       +2     
- Partials     1622     1623       +1     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 47.66% <ø> (+0.01%) ⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a35fba...0860438. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ntkathole
ntkathole approved these changes Jul 30, 2026
View reviewed changes
@nikolauspschuetz @ntkathole
test: Retry CliRunner apply subprocess on timeout to reduce flakiness
0860438
CliRunner.run() (used for 'feast apply' in unit tests) spawns a fresh child
interpreter that must import Feast (pandas/pyarrow/...) from cold. On loaded
CI runners — observed on unit-test-python (3.11, macos-14) — this can exceed
the 120s timeout, failing the test with:

    AssertionError: Command timed out after 120s: ['apply']

Unlike the materializing paths guarded by run_with_output(), commands routed
through run() do not materialize and so cannot trigger the Dask atexit hang the
timeout was added for. A timeout here is therefore almost always transient cold
-start slowness, not a genuine deadlock, so retrying once (with warm OS/page
caches) clears it while a real repeated stall still fails after the attempts
are exhausted.

Builds on the CliRunner subprocess stabilization in feast-dev#6560.

Signed-off-by: Nikolaus Schuetz <[email protected]>
@ntkathole
ntkathole force-pushed the test/stabilize-cli-runner-apply-retry branch from 9aca1af to 0860438 Compare July 30, 2026 08:38
@ntkathole
ntkathole merged commit 6019bd6 into feast-dev:master Jul 30, 2026
18 of 22 checks passed

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ntkathole ntkathole ntkathole approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@nikolauspschuetz @codecov-commenter @ntkathole
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.