GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Run the pyperformance suite on a CodSpeed Macro runner by moreal · Pull Request #9623 · RustPython/RustPython

Conversation

@moreal

@moreal moreal commented Nov 3, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor
  • This PR follows our AI policy

Summary

Replace the CodSpeed job with one that runs the upstream pyperformance benchmark programs, unmodified, in walltime mode on a CodSpeed Macro runner. Executables are built on ubuntu-latest and only run on codspeed-macro.

  • benches/pyperformance/ vendors 43 bm_* directories (65 benchmarks with variants) verbatim from pyperformance 1.14.0; scripts/vendor-pyperformance.py pins the tag and archive hash and --check runs in the lint job.
  • benches/pyperformance/pyperf/ is a minimal stand-in for pyperf: each bench_* call runs the body once, Criterion does the repetition.
  • benches/pyperformance.rs discovers benchmarks from [tool.pyperformance] metadata and runs each run_benchmark.py as __main__ in its own interpreter. It does not link pyo3, so the binary is portable to the Macro runner. execution and microbenchmarks stay for cargo bench only.
  • The 13 hand-ported copies in benches/benchmarks/ and scripts/codspeed-environment.py are removed. benches/pyperformance/README.md accounts for all 73 upstream directories and why 30 are excluded.

What to review

Three of the four commits are hand-written. The vendoring commit is tool output and can be skipped:

Commit Review? Content
benches: add a pyperf shim … yes shim + its unittest file
scripts: add vendor-pyperformance.py yes vendoring script
benches: vendor pyperformance 1.14.0 benchmarks no (generated, ~13k lines) verbatim upstream files; verified by vendor-pyperformance.py --check in CI
Run the pyperformance suite under CodSpeed … yes Criterion target, tests, workflow, docs, deletions

Verified locally

cargo bench --bench pyperformance -- --test (65/65), cargo test --test pyperformance, the shim tests under CPython and RustPython, vendor-pyperformance.py --check, actionlint, shellcheck.

AI assistance

Written with Claude Code (claude-fable-5-1); every commit carries an Assisted-by trailer. The design decisions (Macro runner, walltime mode, running upstream pyperformance unmodified, build on a hosted runner and run only the executables on codspeed-macro, which benchmarks to include) were made by the author in dialogue; the Rust harness, the pyperf shim, the vendoring script, the workflow, the documentation and this description were drafted by the AI under that direction. The benchmark selection runs and the local verification listed above were executed on the author's macOS machine and reviewed by the author.

Needs on the CodSpeed side

The codspeed-macro runner must be enabled for this repository, or the run job waits for a runner. Benchmark names change to pyperformance/<name>, so the CodSpeed history starts fresh.

🤖 Generated with Claude Code

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

  • X
  • Mastodon
  • Reddit
  • LinkedIn
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@codspeed

codspeed Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 62 untouched benchmarks
⏩ 4 skipped benchmarks1


Comparing moreal:claude/codspeed-benchmark-testing-115155 (9fe6257) with main (18fc82b)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

moreal added 3 commits November 8, 2026 10:24
@moreal
benches: add a pyperf shim for running pyperformance workloads in-pro…
b5d47e2
…cess

The upstream pyperformance run_benchmark.py files will be vendored
unmodified under benches/pyperformance/ and executed by a Criterion
harness inside a RustPython interpreter. The one thing they need that
is not stdlib is `pyperf`; this module stands in for it.

Each bench_* method runs the benchmark body exactly once and does no
timing: repetition and measurement belong to Criterion, which runs the
whole script per iteration. parse_args goes through real argparse so
benchmark variants (extra_opts placed in sys.argv) take the upstream
code path. Only the API surface the vendored benchmarks use exists;
anything else raises AttributeError instead of silently measuring
nothing.

Tests run under both CPython and RustPython with unittest.

Assisted-by: Claude Code:claude-fable-5-1
@moreal
scripts: add vendor-pyperformance.py
f5e2a80
Copies a fixed list of bm_* directories from a pinned pyperformance
release tag into benches/pyperformance/, verbatim. The release archive
is checked against a SHA-256 hash before extraction, unlisted bm_*
directories already in the tree are removed, and a few files that are
listed but unusable here (the asyncio_tcp SSL variant and its
certificates) are dropped. `--check` compares the tree with the archive
without writing and exits non-zero on any difference; CI runs it so the
vendored copy cannot drift from upstream.

Assisted-by: Claude Code:claude-fable-5-1
@moreal
benches: vendor pyperformance 1.14.0 benchmarks
51d59fe
Generated by `python3 scripts/vendor-pyperformance.py`; nothing in this
commit is hand-written. 43 bm_* directories (65 benchmarks counting
variants) are copied unmodified from the pyperformance 1.14.0 release
archive, including their data files.

Assisted-by: Claude Code:claude-fable-5-1
@moreal
moreal force-pushed the claude/codspeed-benchmark-testing-115155 branch from f6e4f2e to ecb22c3 Compare November 8, 2026 01:34
@moreal
Run the pyperformance suite under CodSpeed on a Macro runner
46683eb
CodSpeed now measures wall time on a CodSpeed Macro runner instead of
simulated CPU on a hosted runner, and what it measures is the vendored
upstream pyperformance programs, unmodified.

benches/pyperformance.rs is a new Criterion target that discovers the
benchmarks from pyperformance's own [tool.pyperformance] metadata, runs
each run_benchmark.py as __main__ in its own interpreter with the
variant's extra_opts in sys.argv, and does not link pyo3, so the built
executable can be moved between machines. tests/pyperformance.rs covers
discovery and runs two benchmarks end to end; it replaces
tests/benchmark_scripts.rs, which exercised the hand-ported files.

The 13 hand-ported pyperformance workloads in benches/benchmarks/ go
away in favour of the originals. Every one of the 73 upstream
directories is accounted for in benches/pyperformance/README.md:
third-party dependencies, start-up benchmarks, two that fail under
RustPython (decimal_factorial, sqlite_synth), one that needs child
interpreters (concurrent_imap) and three that take 13 to 46 seconds per
iteration (bpe_tokeniser, decimal_pi, asyncio_tcp_ssl) are excluded.

The workflow is split in two: `build` compiles the benchmark executables
on ubuntu-latest with `cargo codspeed build --measurement-mode walltime`
and uploads target/codspeed/walltime/; `run` on the codspeed-macro
runner downloads them and runs `cargo codspeed run`, which in walltime
mode also aggregates the results the compat layer writes. cargo-codspeed
is installed from its released musl executable (v5.0.3, matching
codspeed-criterion-compat 5.x) by .github/scripts/install-cargo-codspeed.sh.
scripts/codspeed-environment.py and its guard steps are removed: the
Macro runner is fixed hardware. The execution and microbenchmarks
targets stay available to `cargo bench` but leave CodSpeed, since they
link libpython through pyo3 for the CPython comparison. The CI lint job
gains a `vendor-pyperformance.py --check` step.

Assisted-by: Claude Code:claude-fable-5-1
@moreal
moreal force-pushed the claude/codspeed-benchmark-testing-115155 branch from e3b51f2 to 46683eb Compare November 8, 2026 08:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@moreal
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.