GeezerVison is showing github.com — Guard raw and inspected signature observations in CI by youknowdot · Pull Request #9628 · RustPython/RustPython
Conversation
youknowdot
commented
Nov 3, 2026
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
Purpose
Turn the existing raw text-signature/result-persistence follow-up into a real CI regression gate, without another RustPython build or scanner pass. This remains separate from the merged non-raw reporting prerequisite (#9726).
Rebased onto main b66534d448. Current head 9671f56cee.
Changes
- Keep native raw
__text_signature__ observations independent of inspect.signature comparisons; preserve type-document and unknown distinctions.
- Reuse the existing three-OS scan. Compare exact individual observed failures and selected-scope fingerprints against platform-specific baselines.
- Reject new/changed failures, unknown/coverage regressions, missing required observation metadata, and stale resolved exceptions. Requiring resolved exceptions to disappear prevents later reintroduction under an old allowance.
- Preserve full scan/context artifacts even if the gate fails. Pin the CPython oracle through
.python-version; check exact OS/architecture/version/build-feature context.
- Keep the baseline generation command explicit and outside CI.
Baseline provenance and remaining work
The initial three baselines were generated from the existing verified Linux/macOS/Windows artifacts of CI run 42421287760, merge 5da7d5b1d1d56467fb60997bbdb66676e0c6c33d, CPython 3.15.0. Downloaded ZIP SHA256 values match GitHub's artifact metadata.
Baseline refresh completed: exact-head run 42491705325, head 4a6ddde862, merge 3b4735987efcc7ba59d454a1f6cf9d667a631778, produced all three platform artifacts. Removed 30 resolved observations on Linux/macOS and 28 on Windows; reviewed retained pre-existing main differences. Normalized only process addresses in exception repr diagnostics, not raw signatures or default literals. The gate now runs immediately after the existing build to avoid expensive tests after a detected signature regression.
Final-head validation passed: CI run 42496222992 passed on head 9671f56cee21682a26cc4c52f44924cb72114c5b, including the signature gate and native tests on Linux, Windows and macOS. This PR remains draft for maintainer review. Do not interpret old green CI or self-comparison as validation of current main. This gate is an observed-surface ratchet, not a claim of zero signature mismatches, complete discovery, argument behavior parity or CPython compatibility. Unknowns remain unverified. No runtime/Lib changes are included.
Validation
- 59 focused whatsleft and gate tests pass under CPython 3.15.1.
- All three current hosted platform artifacts pass the refreshed checker and matching-context CLI; deliberate reintroduction of a resolved queue mismatch fails on every platform.
- Independent review reproduced the important negative paths: missing provenance is rejected; resolved exceptions require deletion; unchanged observations pass.
- Repository hooks and
git diff --check pass after rebase.
- Previous head hosted CI passed Rust/C API tests, Clippy, WASI, lint, and native snippets/CPython tests on all three OSes; only the expected baseline gate failed. Final-head changes are checker/tests/baselines and step ordering only. Final-head hosted integration subsequently passed (run 42496222992), including all three signature gates, native tests, Rust tests, Clippy, WASI and lint; no additional local Rust build was performed.
AI assistance
Implementation and independent review assisted by dot; exact model version unavailable in this runtime.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
coderabbitai
Bot
commented
Nov 3, 2026
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
Important
Draft PR not reviewed
Draft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration:
reviews:
auto_review:
drafts: true
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Share
-
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
youknowdot
changed the title
Record native raw text signatures separately
Record native raw signatures and scan coverage
Nov 3, 2026
Merging this PR will not alter performance
✅ 62 untouched benchmarks
⏩ 4 skipped benchmarks1
Comparing youknowdot:fix/whatsleft-raw-text-observations-20261103 (5654448) with main (e9d1d9f)
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
youknowone
changed the title
Record native raw signatures and scan coverage
Consolidate whats_left reporting and comparison artifacts
Nov 5, 2026
This was referenced Nov 5, 2026
youknowone
changed the title
Consolidate whats_left reporting and comparison artifacts
Record raw text signatures after the reporting prerequisite
Nov 5, 2026
youknowone
changed the title
Record raw text signatures after the reporting prerequisite
Add raw text signatures and persist comparison results
Nov 5, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
tell me what's the benefit by doing this
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
youknowdot
changed the title
Add raw text signatures and persist comparison results
Guard raw and inspected signature observations in CI
Nov 8, 2026
This was referenced Nov 8, 2026
Keep the raw/persistence follow-up together, with exact per-platform observations and selected-scope coverage. Reject stale resolved exceptions to prevent reintroduction.
Assisted-by: dot: model version unavailable
Remove resolved observations, normalize only exception repr addresses, and fail before costly CPython tests when the signature gate detects a regression.
Assisted-by: dot: model version unavailable
Keep raw observations and coverage while accepting extra text when CPython has no signature. Preserve unknown/context and reverse missing-signature checks. Reclassify the existing three-platform artifacts without widening their scopes.
Assisted-by: OpenAI:ChatGPT
Keep reference diagnostics in JSON while excluding allowed extra signatures from human warning output and warning counts.
Assisted-by: OpenAI:ChatGPT
Keep signature mismatches and four checked-surface fingerprints instead of freezing unrelated unknowns, imports, missing APIs and class-selection decisions. Preserve missing or invalid signatures on existing comparable APIs and reject ambiguity or loss of checked coverage. Reclassify the recorded three-platform artifacts without changing interpreter behavior.
Assisted-by: OpenAI:ChatGPT
Rebase on a8bd43e and remove the four platform-independent raw IO default mismatches fixed by RustPython#9744. The stored provenance remains the original three-platform artifacts, with this source-checked adjustment recorded explicitly.
Assisted-by: OpenAI:ChatGPT
Reviewers
Requested changes must be addressed to merge this pull request.
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|