GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Fix per-patch Poetry/PDM lock re-parse (#760, #762) by mikolalysenko · Pull Request #877 · SocketDev/socket-patch

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 5, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Collaborator

LLM Description written by Claude Code:claude-opus-5-5

Fixes #760
Fixes #762

Summary

Hosted Poetry and PDM scans now rewrite poetry.lock / pdm.lock with one parse and one render per lock, not one of each per patched package. On the bench fixture (400 packages, 12 patched), the hosted scan's median wall time drops by about half:

scenario main (2 runs) this PR (2 runs)
poetry/hosted 320.4 / 310.6 ms 163.5 / 162.3 ms
pdm/hosted 263.5 / 280.6 ms 138.6 / 146.5 ms

(socket-patch-bench run -f '^<pm>/hosted', perf profile, both binaries alternated on the same sandbox runner.)

Root cause

Both rewriters go through the shared fragment-splice engine (utils/lock_fragments.rs::finish). For each patched dep, it mutated the parsed document, rendered the whole lock (DocumentMut::to_string), and parsed the rendering again to take the package's new fragments. LockParse only saved the input parse, so every dep still paid one full render and one full parse: O(patches × lock size). Callgrind put 77% of the hosted scan in this loop for both formats.

Fix

  • lock_fragments::rewrite_batch (shared by both formats) handles every dep in one pass. It plans each dep against the document as the earlier deps left it, so refusal and not-found verdicts match the step-by-step ones. It then applies all mutations, renders and re-parses once, and splices each dep's changed bytes into the original text. The splice must reproduce the rendering byte for byte.
  • The batch steps aside, and the old step-by-step path runs unchanged, when it can't vouch for an identical result: a lock with mixed line endings (the majority terminator that spells each spliced fragment can shift as deps land), a package rewritten twice in one run, an ambiguous fragment, or a failed check.
  • poetry_lock / pdm_lock split their rewrite into checks, plan and mutate steps, and add rewrite_*_lock_all (batch or steps). They return a LockStep per dep: Rewritten(edits), Unchanged, NotFound or Refused.
  • The hosted callers (patch/redirect/poetry.rs, patch/redirect/pdm.rs) consume those steps. Warnings, uuid sets and FileEdits come out in the same order as before. Vendored callers (one dep at a time) are unchanged.
  • No wrapper changes: npm/, pypi/ and gem/ only dispatch to the binary.

Out of scope: both issues also note a secondary cost, vex::discover::pypi_locks::extract re-parsing the lock for VEX (~10%). It's untouched here and can be a follow-up if it still shows up in the bench.

Tests

issue regression test red → green
#760 patch::redirect::poetry::equivalence_tests::many_patches_render_the_lock_once (every lock generation 1.0+, LF and CRLF, 12 patched packages) 29735ea: left: 12, right: 1 → passes
#762 patch::redirect::pdm::parse_reuse_equivalence_tests::many_patches_render_the_lock_once (every supported PDM generation, LF and CRLF) 29735ea: left: 12, right: 1 → passes

Equivalence:

  • utils::poetry_lock::batch_equivalence_tests::batch_matches_the_step_by_step_rewrite and utils::pdm_lock::batch_equivalence_tests::…: across every lock generation, LF / CRLF / mixed, with adjacent and sparse packages, reversed order, interleaved refusals and not-found deps, a package rewritten twice, and a re-run over the output, the batch's text and every dep's step equal the step-by-step result. The tests also assert the batch actually answers in every clean case and hands back in the mixed / rewritten-twice cases.
  • The existing goldens (poetry_rewrite, pdm_plan, pdm_rewrite_shared_parse, pdm_lock_reused_parse, …) pass unchanged, so there are no output changes.

Local runs (on 6e45592):

  • cargo clippy --workspace --all-features -- -D warnings: clean. Formatting follows the surrounding code. main itself isn't rustfmt-clean and CI doesn't check formatting, so I didn't run cargo fmt --all. 29735ea had run it and swept 117 unrelated files; 6e45592 restores them to main.
  • cargo test -p socket-patch-core --all-features --lib: 5250 passed. The 4 failures (copy_tree, vlt_heal, pypi_poetry, pypi_requirements write-failure tests) are chmod-based. They can't fail when the sandbox runs as root, and they fail the same way on main. All 35 core integration targets pass.
  • CLI: --lib (847), in_process_redirect_poetry, in_process_redirect_pdm, hosted_memory_engine, hosted_memory_parity, mode_migration_pypi, covgap_commands_vex, e2e_vex_vendor, in_process_vendor, in_process_python_envs: all pass.

CI on 6e45592: all green. All 12 workflows pass (CI, Poetry, PDM, vlt, Gradle, Go, Bun, Composer, npm, pnpm, Benchmarks, Audit GHA), including every e2e_vex_build Poetry leg (1.0.10–2.4.3) and PDM leg (1.4.5–2.29.2). Jobs that the runner queue had cancelled before any step ran were re-run once. Bugbot found no issues; no open review threads.

Also in this PR

c25dd8e cherry-picks #878 (Route Gradle digests through utils::digest). main @ 9c43dfc fails utils::digest::tests::production_digests_go_through_the_helpers on every PR. The cherry-pick becomes a no-op once #878 merges.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HBf8J4E4vApc1bRJfAUDu8


Generated by Claude Code

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@claude
Start fix for #760, #762
42c51ee
Assisted-by: Claude Code:claude-opus-5-5
This was referenced Oct 5, 2026
Slow scan: poetry hosted 4.5x median ms/pkg (per-patch poetry.lock re-parse) #760
Closed
Slow scan: pdm hosted 3.6x median ms/pkg (per-patch pdm.lock re-parse) #762
Closed
Claude (claude) added 4 commits October 5, 2026 18:34
@claude
Test that a Poetry/PDM lock renders once per scan
29735ea
Hosted scans rewrite poetry.lock and pdm.lock once per patched
package, rendering and re-parsing the whole lock each time. Count the
engine's whole-lock renders and require one per lock for a dozen
patched packages. Both tests fail today with 12 renders.

Refs #760, #762

Assisted-by: Claude Code:claude-opus-5-5
@claude
Rewrite each Poetry/PDM lock in one pass
134bea9
A hosted scan rewrote poetry.lock and pdm.lock once per patched
package, and every rewrite rendered and re-parsed the whole lock. A
project with a dozen patches paid for a dozen full parses, which made
Poetry and PDM scans 3.5-4.5x slower per package than other managers.

The shared lock-splice engine now plans every package against one
parsed lock, applies all the changes, renders and re-parses once, and
splices each package's changed fragments into the original text. The
result is checked against the rendering byte for byte. When a lock
mixes line endings, a package is rewritten twice, or any check fails,
the rewrite falls back to the old package-by-package path, so output
and recorded edits never change.

Differential tests run both paths over every Poetry and PDM lock
generation, LF, CRLF and mixed, with refusals, missing packages and
re-runs mixed in, and require identical text and per-package results.

Fixes #760, #762

Assisted-by: Claude Code:claude-opus-5-5
@claude
Read PDM lock_version from the parsed original
e40eed8
The rewrite never changes [metadata] lock_version, so read it from the
parse the presence probe already took instead of parsing the output.

Assisted-by: Claude Code:claude-opus-5-5
@claude
Route Gradle digests through utils::digest
c25dd8e
main has failed socket-patch-core's lib tests since Gradle support
(#646) and the digest helpers (#865) both landed. The guard test
production_digests_go_through_the_helpers flags three files #646 added
that still hash inline: crawlers/gradle_cache.rs, patch/jvm_jar.rs and
patch/sidecars/maven.rs. That breaks test, test-release and coverage on
every open PR.

Each inline sha1/sha256 call now goes through sha1_hex_of or
sha256_hex_of, which compute the same lowercase hex. Behaviour is
unchanged.

Assisted-by: Claude Code:claude-opus-5-5
@mikolalysenko

Mikola Lysenko (mikolalysenko) commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

[agent] test (ubuntu/macos/windows), test-release and coverage failed on the start commit 42c51ee (identical to main @ 9c43dfc). The failing test is utils::digest::tests::production_digests_go_through_the_helpers. main is red because Gradle support (#646) left three files hashing inline, and #865's guard test rejects that. This PR's change didn't cause it. #878 fixes it; I cherry-picked its commit as c25dd8e, and it becomes a no-op once #878 lands.


Generated by Claude Code

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mikolalysenko
Mikola Lysenko (mikolalysenko) marked this pull request as ready for review October 5, 2026 19:41
@mikolalysenko

Mikola Lysenko (mikolalysenko) commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

BugBot review


Generated by Claude Code

cursor[bot] reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

cursor[bot]
cursor Bot reviewed Oct 5, 2026
View reviewed changes

@cursor cursor Bot left a comment •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Tanmay182003
Tanmay Singla (Tanmay182003) approved these changes Oct 5, 2026
View reviewed changes
@mikolalysenko

Mikola Lysenko (mikolalysenko) commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

[agent] gradle 9.8.0 / jdk 21 / vendor / windows-latest (Gradle patch compatibility, run 37362181229) failed after 50 minutes inside "Run the vendor suites". The Gradle workflow runs on this PR only because of the #878 cherry-pick (c25dd8e), which routes three Gradle-path hashes through utils::digest with byte-identical output. This PR's Poetry/PDM change touches no Gradle code. Most other jobs in that run show cancelled and the run is still queued, and GitHub returns 404 for the failed job's log, so I can't root-cause it yet. I'll read the log when it's available, then either fix the cause or re-run the job once if it died before any test ran.


Generated by Claude Code

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@claude
Drop the unrelated formatting sweep
6e45592
Running cargo fmt over the whole workspace reformatted 117 files this
PR does not otherwise touch, because main is not rustfmt-clean and CI
does not check formatting. Restore those files to main and keep the
diff to the Poetry/PDM rewrite and the ported digest fix.

Assisted-by: Claude Code:claude-opus-5-5
@mikolalysenko

Mikola Lysenko (mikolalysenko) commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

BugBot review


Generated by Claude Code

cursor[bot] reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

cursor[bot]
cursor Bot reviewed Oct 5, 2026
View reviewed changes

@cursor cursor Bot left a comment

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 6e45592. Configure here.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mikolalysenko

Mikola Lysenko (mikolalysenko) commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

[agent] lock-diff (vlt patch compatibility, run 37371659871) failed on 6e45592 with no cross-OS lock set … locks from no OS for every cell, and 0 cells compared. The job compares the vlt lock artifacts that the native (<os>, <vlt>) jobs upload. In this run those jobs were cancelled after sitting in the runner queue, so it had no input. This PR changes no vlt code (only the Poetry/PDM lock rewriters and the ported #878 digest routing). I've re-run the failed and cancelled jobs once.


Generated by Claude Code

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mikolalysenko

Mikola Lysenko (mikolalysenko) commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

[agent] CI on 6e45592: nine workflows show red (CI, Gradle, Go, Bun, Composer, npm, pnpm, Benchmarks, Audit GHA), but none of them has a failed job. In each one, jobs were cancelled before they ran a single step. Most were cancelled together at around 21:00 UTC after sitting in the runner queue, e.g. CI's test (windows-latest), dispatch-tests, coverage-docker (npm/maven). Every job that did run passed: 174 in CI, including all the Poetry/PDM e2e_vex_build legs. The Poetry and PDM compatibility workflows and the vlt re-run (lock-diff) are green. I've re-run the cancelled jobs in each of those nine runs once.


Generated by Claude Code

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Oct 6, 2026
@mikolalysenko

Mikola Lysenko (mikolalysenko) commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

[agent] Ready for review at 6e45592. CI: every check on this head is green (the jobs a runner outage cancelled earlier were re-run once and passed); 0 failing. Bugbot reviewed 6e45592 with no findings. No open review threads.


Generated by Claude Code

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

This was referenced Oct 6, 2026
Vendored Poetry wires a 2.x lock through two different splicers depending on its line endings #936
Closed
Share the single-lock wire and revert envelope across the Poetry, PDM and Pipenv vendored backends #937
Closed
@mikolalysenko
Mikola Lysenko (mikolalysenko) merged commit 79c2597 into main Oct 7, 2026
765 of 785 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@mikolalysenko
Mikola Lysenko (mikolalysenko) deleted the agent/fix-python-lock-batch-rewrite branch October 7, 2026 12:02
This was referenced Oct 9, 2026
Slow scan: pipenv hosted 2.6x median ms/pkg (per-patch Pipfile.lock re-tokenize) #1246
Open
Slow scan: poetry hosted 2.6x median ms/pkg (lock_fragments::rewrite_batch + VEX lock re-parse) #1473
Closed
@mikolalysenko Mikola Lysenko (mikolalysenko) mentioned this pull request Oct 17, 2026
Slow scan: pdm hosted 2.7x median ms/pkg (toml_edit parse of pdm.lock 14x: rewrite_batch + VEX extract) #1624
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cursor cursor[bot] cursor[bot] left review comments
@Tanmay182003 Tanmay Singla (Tanmay182003) Tanmay182003 approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Slow scan: pdm hosted 3.6x median ms/pkg (per-patch pdm.lock re-parse) Slow scan: poetry hosted 4.5x median ms/pkg (per-patch poetry.lock re-parse)

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@mikolalysenko @Tanmay182003 @claude
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.