GeezerVison

LLM Description written by Claude Code:claude-opus-5-5

Fixes #1423

Summary

A hosted scan of a yarn classic project got about 2.6x slower after #1328. Two
checks that #1328 added re-read the whole yarn.lock once for every patched
package. Each lock is now read once per scan. On the 3000-package / 60-patch
bench, yarn-classic/hosted and yarn-classic/rescan are both about 56%
faster than current main. Rescan is back to its pre-#1328 time.

Root cause

Both are O(patched packages × lock size):

1. Choosing which served tarballs to fetch
   (hosted::engine::yarn_classic_artifact_targets). For each candidate it ran
   classic_locks_registry_copy, which re-ran scan_blocks over the whole lock
   and split every key. It also ran two whole-lock substring searches per
   candidate: one for the pinned URL and one for the name.
2. The Yarn classic: a patch that adds a dependency to the package's own package.json leaves vendored yarn.lock with a dangling dependency (offline frozen install fails, lock churns), and hosted silently installs without it #591 served-dependency check in the yarn classic rewriter
   (classic_deps::unlocked_descriptors). For each patched package it split
   every key of the lock into a fresh BTreeSet, even when the package has no
   dependencies.

Changes

• select_classic_artifacts collects the lock's registry copies
  (classic_registry_copies) once, and only if some candidate still needs
  them. It also collects every URL the lock already pins with a # fragment
  (quoted_fragment_urls) in one lazy pass. The cheap "already pinned" test
  runs first, so a rescan whose grants all carry a sha1 never reads the blocks.
• The rewriter already splits each key once (heads). The Yarn classic: a patch that adds a dependency to the package's own package.json leaves vendored yarn.lock with a dangling dependency (offline frozen install fails, lock churns), and hosted silently installs without it #591 check now
  looks descriptors up in a HashSet<&str> built from those patterns the first
  time a package needs it (classic_deps::unlocked_descriptors_in). The
  vendored path keeps unlocked_descriptors, because it calls it once per
  package.
• Selection is unchanged: the same tarballs are fetched and the same
  descriptors are reported as unlocked.

The history has one reverted experiment: a commit that fetched the tarballs
concurrently, then its revert. Code review found that it could hold up to 32
whole tarballs in memory, and that retries would hit the server together.
The wheel fetch window was designed specifically to avoid both. It also gained
almost nothing on the bench, so it was backed out.

Bench (socket-patch-bench compare --runs 20, same 4 vCPU sandbox, bench fixture from #1422)

Against main (1779317):

Against the commit before #1328 (f864148):

Most of the remaining hosted delta comes from the 60 served-tarball downloads
(plus gunzip and hashing) that #1328 added deliberately, for the #558 sha1 and
the #591 dependency check. #1423 itself calls those downloads intentional, so
this PR leaves them in place. Callgrind on the hosted scan: total instructions
dropped from 1.74G to about 1.1G. yarn_classic_artifact_targets went from
68% inclusive to under 4%, and unlocked_among went from 29% to 0%.

Tests

Commands run locally:

• cargo fmt --all -- --check: clean
• cargo clippy --workspace --all-features -- -D warnings: clean
• cargo test --workspace --all-features: 14229 passed, 13 failed. All 13
  also fail on main in this sandbox and none touch yarn code. Ten are
  permission-failure tests that can't fail as root (chmod is ignored), for
  example covgap_commands_vendor::*_state_write_failure_* and
  vendor::pypi_poetry::tests::wire_write_failure_*. The others need
  patches-api.socket.dev, which the proxy blocks, for example
  e2e_bun_lockb::binary_shared_bundled_record_hosted_pin_is_managed.
• scripts/yarn-classic-vex-matrix.sh 1.22.22 1.0.2 (real yarn,
  SOCKET_PATCH_YARN_E2E_REQUIRED=1): exit 0, every suite passed (110 PASS
  cells)
• cargo test -p socket-patch-cli --test scan --test hosted_memory_engine --test e2e_redirect_yarn_classic_build --test e2e_vendor_yarn_classic_build --test e2e_vendor_yarn_classic_dev_flow:
  all pass

Review notes (/code-review high, findings not fixed)

• A grant for a package missing from the lock still forces one block read.
  The read is bounded at once per lock. The alternative was a whole-lock
  substring search per dep, which is what this PR removes.
• yarn-berry's berry_lock_locks scans per dep too. That is a different
  backend and not part of this regression; the berry bench is unchanged. Left
  for a separate issue.

Follow-ups

• The 60 served-tarball GETs on a first hosted scan run serially. To make them
  concurrent, they need the wheel window's design: a small cap, a first-attempt
  window, then serial resumption.

🤖 Generated with Claude Code

https://claude.ai/code/session_018G3cdPFQuuBbZahpQSKpoB

Generated by Claude Code