GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — ci: run CodSpeed benchmarks in parallel with pytest-xdist by invictustitan2 · Pull Request #668 · commit-check/commit-check

Conversation

@invictustitan2

@invictustitan2 invictustitan2 commented Nov 8, 2026 •
edited by coderabbitai Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Closes #663.

The Run benchmarks CodSpeed job runs the whole benchmark set serially under Valgrind (mode: simulation), which is why it dominates PR time (~54 min on the #657 run). This splits that one job across processes with pytest-xdist, which CodSpeed documents for exactly this case.

Change

  • .github/workflows/codspeed.yml: run: pytest tests/ --codspeed -n auto.
  • pyproject.toml: add pytest-xdist to the [test] extra — the benchmark job installs .[test], so the plugin has to be declared there. uv.lock regenerated (adds pytest-xdist, execnet).

Every benchmark still runs; CodSpeed aggregates the workers into one report. The CodSpeed Python docs give this exact command (pytest tests/ --codspeed -n auto) with mode: simulation for "running benchmarks in parallel processes."

Verification
pytest tests/ --codspeed -n auto runs green locally (full suite, exit 0) — the suite is parallel-safe, so no benchmark depends on serial ordering.

README pin: unchanged — it already pins v2.20.0, which matches the current PyPI release, so nothing to bump here.

The bigger lever (your call). The CodSpeed docs actually present process-parallelism as the fallback; their first-choice is sharding across CI jobs with pytest-test-groups (a matrix: shard: [1..N] with --test-group/--test-group-count), which parallelizes across runners rather than within one. That's a larger change to codspeed.yml and it's your CI infra, so I've kept this PR to the minimal in-job -n auto. Happy to follow up with the sharded-matrix version (or swap this PR to it) if you'd prefer that route.

Summary by CodeRabbit

  • Chores
    • Benchmark tests now run in parallel, with the required test dependency added.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@invictustitan2
ci: run CodSpeed benchmarks in parallel with pytest-xdist
8196c6b
Split the benchmark run across processes with pytest-xdist (-n auto) so
the single CodSpeed job stops serializing every benchmark under Valgrind.

- codspeed.yml: run 'pytest tests/ --codspeed -n auto' (simulation mode).
- pyproject.toml: add pytest-xdist to the [test] extra (the benchmark job
  installs .[test]); uv.lock regenerated (adds pytest-xdist, execnet).

CodSpeed documents this exact command for parallel runs in one job.
Full suite passes locally under the command.
@invictustitan2
invictustitan2 requested a review from a team as a code owner November 8, 2026 16:05
@invictustitan2
invictustitan2 requested review from shenxianpeng and removed request for a team November 8, 2026 16:05
@github-actions github-actions Bot added the chore label Nov 8, 2026
@coderabbitai

coderabbitai Bot commented Nov 8, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The test dependency group now includes pytest-xdist. The CodSpeed benchmark command uses -n auto to run tests with automatically selected workers.

Changes: CodSpeed benchmark parallelism

Layer / File(s) Summary
Enable parallel benchmark execution
pyproject.toml, .github/workflows/codspeed.yml
The test dependency group adds pytest-xdist. The CodSpeed command runs pytest tests/ --codspeed -n auto.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Change: Other

Merge Risk: Minimal · up to 8196c

The benchmark job runs in parallel. No merge-blocking issue is established; normal CI verification remains appropriate.

Pre-merge checks: Passed 5

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title "ci: run CodSpeed benchmarks in parallel with pytest-xdist" matches the main change. The CodSpeed workflow now runs pytest with the -n auto option. The test extra in pyproject.toml now inclu…
Linked Issues check Passed [#663] The issue requires running the CodSpeed benchmark job in parallel under simulation mode, without dropping benchmarks. It proposes pytest tests/ --codspeed -n auto and adding pytest-xdist to…
Out of Scope Changes check Passed The whole-PR summary lists two reviewed changes. Both implement #663. .github/workflows/codspeed.yml changes only the benchmark run command to add -n auto. pyproject.toml changes only the test…
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Full details: Title check

Explanation

The title "ci: run CodSpeed benchmarks in parallel with pytest-xdist" matches the main change. The CodSpeed workflow now runs pytest with the -n auto option. The test extra in pyproject.toml now includes pytest-xdist, which provides that option. The title is specific, short, and describes the primary intent from the linked issue (reducing the benchmark job runtime by parallelizing). The title does not cover the uv.lock regeneration, but that is a expected omission and the lock file is generated output.

Full details: Linked Issues check

Explanation

[#663] The issue requires running the CodSpeed benchmark job in parallel under simulation mode, without dropping benchmarks. It proposes pytest tests/ --codspeed -n auto and adding pytest-xdist to the test dependencies. The whole-PR summary shows both changes. .github/workflows/codspeed.yml changes the run command to pytest tests/ --codspeed -n auto (+1/-1). pyproject.toml adds pytest-xdist to the test optional dependency group (+1/-1). The benchmark job installs .[test], so the plugin is available to the job. No benchmark is removed or excluded. Sharding across a job matrix is a possible follow-up, not a requirement of #663. The issue does not require new tests. The parallel-safety claim is supported only by the author's local run and the issue text. The PR's own CodSpeed run provides the measurement, as the issue expects. uv.lock is excluded from review (ignored by the !**/*.lock rule). The lock file is a derived artifact of the pyproject.toml change, so the result does not depend on its contents.

Full details: Out of Scope Changes check

Explanation

The whole-PR summary lists two reviewed changes. Both implement #663. .github/workflows/codspeed.yml changes only the benchmark run command to add -n auto. pyproject.toml changes only the test extra to add pytest-xdist. Neither change touches benchmark content, the README, or unrelated CI jobs. The README pin is explicitly unchanged. uv.lock is excluded from review. Its changes are the lock-file result of adding pytest-xdist and execnet, which is a direct consequence of the in-scope pyproject.toml change. No out-of-scope change is identified.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

  • X
  • Mastodon
  • Reddit
  • LinkedIn
Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 619ed613-2126-4331-88a6-2955c4c7bb96
Commits

Reviewing files that changed from the base of the PR and between 880430d and 8196c6b.

Files ignored due to path filters: 1

  • uv.lock is excluded by !**/*.lock
Files selected for processing: 2

  • .github/workflows/codspeed.yml
  • pyproject.toml

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sonarqubecloud

sonarqubecloud Bot commented Nov 8, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@codecov

codecov Bot commented Nov 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (880430d) to head (8196c6b).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #668   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           14        14           
  Lines         2495      2495           
  Branches       484       484           
=========================================
  Hits          2495      2495           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@codspeed

codspeed Bot commented Nov 8, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 11.1%

3 warnings

⚠️ 136 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 1 benchmark with a significant change ran on different hardware

Its base and head results were measured on machines with a different CPU, core count or memory, so the change might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark with a significant change ran in a different software environment

Its base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the change might come from this difference rather than from the code. Open the benchmark to see what changed.

See affected benchmarks

❌ 1 regressed benchmark
✅ 660 untouched benchmarks
⏩ 131 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ test_no_terminal_output_produced 946.8 µs 1,065 µs -11.1%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing invictustitan2:chore/parallel-codspeed-benchmarks (8196c6b) with main (bd2ed6e)2

Open in CodSpeed

Footnotes

  1. 131 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on main (880430d) during the generation of this report, so bd2ed6e was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@shenxianpeng

shenxianpeng commented Nov 8, 2026

Copy link
Copy Markdown
Member
image

5x speed, a very impressive improvement. Thank you!

invictustitan2 reacted with rocket emoji
All reactions
  • 🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

shenxianpeng
shenxianpeng approved these changes Nov 8, 2026
View reviewed changes
@shenxianpeng
shenxianpeng merged commit 2d78e79 into commit-check:main Nov 8, 2026
26 of 27 checks passed

Uh oh!

There was an error while loading. Please reload this page.

This was referenced Nov 8, 2026
ci: shard CodSpeed benchmarks across jobs instead of xdist processes #669
Merged
Many CodSpeed benchmarks are dominated by system calls — simulation can't measure them, so they add noise #670
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@shenxianpeng shenxianpeng shenxianpeng approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

chore

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Speed up the CodSpeed benchmark job (~54 min/PR) by running benchmarks in parallel

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@invictustitan2 @shenxianpeng
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.