GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — feat(vendor,scan): lockfile-driven discovery + verified auto-fetch of missing packages by mikolalysenko · Pull Request #113 · SocketDev/socket-patch

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #112 (fix/vendor-ux) — re-target to main after it merges.

Summary

Two capabilities that make scan / scan --vendor / vendor work on a completely fresh clone (no installed packages at all), across every vendorable ecosystem:

1. Vendor auto-fetch (no flag — automatic)

A manifest patch whose package has no installed copy no longer fails with package_not_installed:

  • Already-vendored purls stage from their own committed artifact, sha256-verified against the vendor ledger (offline-safe — fresh-clone re-vendor and already_vendored in-sync runs need no network).
  • Lockfile-resolved packages are fetched pristine and verified against the lock's recorded integrity fail-closed, before any disk write:
ecosystem source verifier
npm / pnpm / yarn classic / bun lock resolved URL, else conventional registry URL SRI (strongest of multi-hash); yarn classic sha1 fragment fallback
yarn berry conventional registry URL the lock's 10c0/ cache-zip checksum, rebuilt from the fetched tarball via the same deterministic berry_zip recipe the wiring uses
cargo static.crates.io Cargo.lock checksum (sha256 of the .crate)
golang module proxy (SOCKET_GOPROXY → GOPROXY → proxy.golang.org), !case path escaping go.sum h1: dirhash computed in memory over the zip — algorithm validated against a live sum.golang.org lookup
composer dist.url (zipball top dir stripped) dist.shasum (sha1; empty → never fetched)
gem {remote}/downloads/{gem} Gemfile.lock CHECKSUMS sha256 of the whole .gem (bundler ≥ 2.6; older locks never fetched)
pypi uv.lock wheel URL (pure py3-none-any only) uv.lock wheel sha256; the unzipped wheel is a site-packages-shaped stage

Anything the lock can't verify is never fetched (vendor_fetch_unverifiable warning + the calm package_not_installed skip). An attempted-and-failed fetch is a distinct vendor_fetch_failed failure. Successful fetches surface as vendor_fetched_missing warnings beside the normal Applied event. --offline refuses with the calm skip, naming the lockfile resolution. The fetch stages into a private tempdir — node_modules/site-packages/etc. are never touched.

Hardening: http(s)-only, download/decompression/entry-count/entry-size caps, regular-files-only extraction with traversal guards (fail-closed, nothing half-extracts), exec bits preserved (the deterministic re-pack reads modes from disk).

2. Scan lockfile supplement (default-on)

scan discovery now reads the project's lockfiles — npm-family (all five flavors), Cargo.lock, go.sum, composer.lock, Gemfile.lock, uv.lock/poetry.lock/pinned requirements.txt — and adds dependencies with no installed copy to discovery:

  • table rows flagged [NOT INSTALLED] + a stderr note; JSON gains lockfileOnlyPackages + packages[].notInstalled
  • lockfile-only purls count as scanned for --prune (a wiped node_modules no longer prunes lockfile-listed manifest entries)
  • --apply partitions them out before download: calm skipped/package_not_installed records, exit 0, no manifest writes — --vendor passes them to the auto-fetch instead
  • vendored-ledger entries also stay discoverable on a fresh clone (the committed artifact IS the dependency), so updates[] and re-vendor keep working pre-install

Test plan

  • ~50 new tests: per-flavor/per-ecosystem inventory units (incl. pnpm peer-suffix keys, yarn npm: aliases + sha1 fragments, bun tuples, berry resolutions, Cargo.lock sources, go.sum /go.mod skipping, composer dists, Gemfile CHECKSUMS, uv wheels, PEP 503 normalization); fetch units with wiremock (SRI/sha1/sha256/berry-zip/dirhash verification, mismatch-before-extraction, unverifiable-without-network, traversal + oversize fail-closed, exec-bit preservation, scoped URL forms, ledger-sha256 artifact staging); e2e (standalone vendor auto-fetch, integrity mismatch → vendor_fetch_failed with nothing written, offline calm skip, unverifiable entry, fresh-clone scan --vendor end-to-end + already_vendored rerun, read-only discovery flags, --apply calm skip)
  • cargo test --workspace --all-features: 101/102 suites green (the one red is the pre-existing non-blocking setup_matrix_composer BASELINE GAP); clippy clean
  • Go dirhash validated against the real checksum DB (golang.org/x/[email protected] ↔ sum.golang.org)
  • Live fresh-clone proof: a scratch project with only package.json + package-lock.json ([email protected], real registry integrity) → scan --vendor --yes against the live proxy discovers from the lockfile, fetches from registry.npmjs.org (integrity verified), vendors with the expected vendor_fetched_missing + auto-force warnings, node_modules never created; npm ci then installs the vendored artifact with 6/6 files at afterHash and the module works

Out of scope (documented)

.npmrc/auth registry config (env overrides + lock-recorded URLs are the escape hatches); pipenv/pdm lock parsing; platform-specific wheels and platform gems; npm v1 legacy locks.

🤖 Generated with Claude Code

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Mikola Lysenko (mikolalysenko) and others added 7 commits June 11, 2026 18:08
@mikolalysenko @claude
feat(vendor): lockfile inventory module for npm-family locks
1a2dc4a
Read-only inventories of the dependency set a lockfile resolves,
independent of what is installed: name/version/purl plus the lock's
artifact URL and content verifier (typed LockIntegrity: SRI, yarn sha1
fragment, berry cache-zip checksum, sha256 hex, go.sum h1 — the latter
two for the ecosystems that follow). Powers scan's lockfile supplement
and vendor's missing-package fetch.

Covers all five npm flavors via detect_npm_lock_flavor (package-lock/
shrinkwrap, pnpm v9, yarn classic, yarn berry, bun). Fail-soft per
entry, fail-closed per value (names/versions path-guarded; git/file/
link/workspace specs and our own vendored entries excluded; duplicate
instances dedup preferring a verifier). lookup() bridges percent-
encoded manifest purls. Reuses the wiring backends' parsers via
pub(super) visibility bumps.

Co-Authored-By: Claude Fable 5 <[email protected]>
@mikolalysenko @claude
feat(vendor): registry_fetch — verified pristine-artifact fetching
9c93b90
Downloads the artifact a lockfile entry resolves (lock-recorded URL,
else the conventional npm registry URL; SOCKET_NPM_REGISTRY override),
verifies it against the lock-recorded integrity FAIL-CLOSED before any
disk write (strongest hash of a multi-hash SRI; yarn sha1 fragment;
sha256 hex), and extracts to a private tempdir the vendor pipeline can
stage from. Entries with no verifier are refused without any network
I/O (Unverifiable).

Hardening: http(s)-only, download/decompression/entry-count/entry-size
caps, regular-files-only extraction with first-component strip +
is_safe_relative_subpath (fail-closed on traversal-bearing tarballs,
nothing half-extracts), exec bits preserved so the deterministic
re-pack keeps bin scripts executable.

Co-Authored-By: Claude Fable 5 <[email protected]>
@mikolalysenko @claude
feat(vendor,scan): auto-fetch missing packages + lockfile/ledger disc…
18822dc
…overy

vendor: a manifest patch whose package has no installed copy is now
satisfied automatically (no flag) instead of failing with
package_not_installed:
- already-vendored purls stage from their own committed artifact,
  sha256-verified against the vendor ledger (fresh-clone re-vendor and
  in-sync runs work offline, no registry traffic);
- otherwise the lockfile-resolved pristine artifact is fetched
  (lock-recorded URL else conventional registry URL), verified against
  the lock's integrity FAIL-CLOSED, and staged from a private tempdir —
  the project tree is never touched.
Reason codes: vendor_fetched_missing (skip-warning beside the Applied
event), vendor_fetch_failed (distinct Failed, suppresses the duplicate
not-installed skip), vendor_fetch_unverifiable (no lock integrity →
calm skip). --offline keeps the calm skip and names the lockfile as
the would-be source.

scan: discovery now supplements the installed-tree crawl with
(a) lockfile-only dependencies — warned '[NOT INSTALLED]' in the table
    + a stderr note, JSON lockfileOnlyPackages + packages[].notInstalled,
    counted as scanned so a wiped node_modules no longer prunes
    lockfile-listed entries, partitioned out of --apply BEFORE download
    (calm skipped/package_not_installed records, exit 0, no manifest
    writes) while --vendor passes them to the auto-fetch; and
(b) vendored-ledger entries — the committed artifact IS the dependency,
    so updates[] detection and scan --vendor keep working on a fresh
    clone before any install.

scan --json --vendor now vendors a completely fresh clone end-to-end
(e2e-proven, second run already_vendored).

Co-Authored-By: Claude Fable 5 <[email protected]>
@mikolalysenko @claude
feat(vendor): yarn berry checksum-verified fetch + ledger artifact st…
430145a
…aging tests

Berry locks never hash the tarball — the checksum is sha512 of the
deterministic cache zip. The fetch rebuilds that zip from the fetched
bytes via the same spike-pinned berry_zip recipe the wiring uses and
compares the 10c0/<hex> value fail-closed (foreign cacheKeys are
Unverifiable). Plus unit coverage for stage_local_artifact's
ledger-sha256 gate.

Co-Authored-By: Claude Fable 5 <[email protected]>
@mikolalysenko @claude
feat(vendor): cargo + golang lockfile inventory and verified fetch
16e7e54
- Cargo.lock [[package]] inventory: crates.io-sourced entries carry
  their sha256 .crate checksum (Sha256Hex); workspace members skipped,
  git/custom-registry sources discovery-only. Fetch from
  static.crates.io (SOCKET_CRATES_REGISTRY override), verify, extract
  ({name}-{version}/ top dir) — feeds vendor_cargo_crate's pristine_src.
- go.sum inventory: module-zip h1: lines (the /go.mod manifests-only
  lines skipped). Fetch from the module proxy (SOCKET_GOPROXY, else the
  standard GOPROXY's first non-direct element, else proxy.golang.org)
  with Go's !uppercase path escaping; verify the dirhash Hash1/HashZip
  in memory BEFORE extraction (algorithm validated against a live
  sum.golang.org lookup for golang.org/x/[email protected]); extract with
  the explicit module@version/ prefix (module paths contain slashes, so
  a first-component strip would be wrong) — feeds vendor_go_module.
- lookup() generalized across ecosystems; inventory_project() returns
  the union the scan supplement and vendor auto-fetch consume.

Co-Authored-By: Claude Fable 5 <[email protected]>
@mikolalysenko @claude
feat(vendor): composer + gem + pypi lockfile inventory and verified f…
b5e9f63
…etch

- composer.lock packages[]/packages-dev[]: zip dists with their sha1
  shasum (frequently empty → discovery-only); names lowercased to the
  packagist form, pretty leading v dropped; path dists (ours) excluded.
  Fetch verifies sha1 and strips the variable zipball top dir.
- Gemfile.lock GEM/specs + bundler 2.6 CHECKSUMS sha256 (older locks
  discovery-only); the GEM remote drives the /downloads/<gem> URL.
  Platform-suffixed specs skipped (unsupported for vendoring). The
  fetched .gem (plain tar) is sha256-verified whole, then data.tar.gz
  extracts at the root (no prefix strip).
- pypi: uv.lock registry packages with a pure py3-none-any wheel carry
  a fetchable URL + sha256; poetry.lock and ==-pinned requirements.txt
  contribute discovery-only entries (PEP 503-normalized names). The
  unzipped wheel is a site-packages-shaped stage for the pypi backend.

Co-Authored-By: Claude Fable 5 <[email protected]>
@mikolalysenko @claude
feat(scan): all-ecosystem lockfile supplement + docs
772f98d
scan's lockfile supplement now consumes inventory_project (npm-family,
Cargo.lock, go.sum, composer.lock, Gemfile.lock, uv/poetry/requirements)
with per-ecosystem counts; the vendor auto-fetch pass likewise serves
every inventoried ecosystem. CLI_CONTRACT.md gains the lockfile-
supplement and vendor-auto-fetch sections + the three reason codes;
README notes the fresh-clone flow; the exact-shape empty-scan contract
test pins the additive lockfileOnlyPackages field; the cargo build e2e
scrubs ambient CARGO_TARGET_DIR from child builds.

Co-Authored-By: Claude Fable 5 <[email protected]>
@mikolalysenko Mikola Lysenko (mikolalysenko) mentioned this pull request Jun 11, 2026
feat: vendor/scan/apply overhaul — lockfile-driven discovery, verified auto-fetch, mismatch-tolerant apply #114
Closed
@mikolalysenko

Mikola Lysenko (mikolalysenko) commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator Author

Folded into #114, which now targets main and carries this branch's commits (the stack #112 → #113 → #114 was combined into a single PR).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mikolalysenko Mikola Lysenko (mikolalysenko) closed this Jun 11, 2026
This was referenced Jun 11, 2026
fix(vendor): make scan --vendor / vendor / --prune just work #112
Closed
feat: vendor/scan/apply overhaul — lockfile-driven discovery, verified auto-fetch, mismatch-tolerant apply #115
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@mikolalysenko
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.