GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — feat: vendor/scan/apply overhaul — lockfile-driven discovery, verified auto-fetch, mismatch-tolerant apply by mikolalysenko · Pull Request #115 · SocketDev/socket-patch

feat: vendor/scan/apply overhaul — lockfile-driven discovery, verified auto-fetch, mismatch-tolerant apply #115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Mikola Lysenko (mikolalysenko) merged 19 commits into main from feat/vendor-overhaul

Jun 12, 2026

Uh oh!

There was an error while loading. Please reload this page.

Merged

feat: vendor/scan/apply overhaul — lockfile-driven discovery, verified auto-fetch, mismatch-tolerant apply #115

Show file tree

Hide file tree

Changes from 1 commit

Commits

Show all changes

19 commits

Select commit Hold shift + click to select a range

42e95be

fix(purl): percent-decode purl components from the API

mikolalysenko Jun 11, 2026

c3c012f

feat(vendor): auto-force staging on content mismatch + correct alread…

mikolalysenko Jun 11, 2026

7363c65

feat(vendor): take over exact-version override pins (pnpm + yarn berry)

mikolalysenko Jun 11, 2026

adc5179

feat(scan): prune lifecycle for vendored packages

mikolalysenko Jun 11, 2026

7042cbc

test: e2e coverage for encoded scoped purls, mismatch annotation, pru…

mikolalysenko Jun 11, 2026

1a2dc4a

feat(vendor): lockfile inventory module for npm-family locks

mikolalysenko Jun 11, 2026

9c93b90

feat(vendor): registry_fetch — verified pristine-artifact fetching

mikolalysenko Jun 11, 2026

18822dc

feat(vendor,scan): auto-fetch missing packages + lockfile/ledger disc…

mikolalysenko Jun 11, 2026

430145a

feat(vendor): yarn berry checksum-verified fetch + ledger artifact st…

mikolalysenko Jun 11, 2026

16e7e54

feat(vendor): cargo + golang lockfile inventory and verified fetch

mikolalysenko Jun 11, 2026

b5e9f63

feat(vendor): composer + gem + pypi lockfile inventory and verified f…

mikolalysenko Jun 11, 2026

772f98d

feat(scan): all-ecosystem lockfile supplement + docs

mikolalysenko Jun 11, 2026

0426e0b

feat(apply): beforeHash mismatch warns and applies the full blob by d…

mikolalysenko Jun 11, 2026

64c59f1

polish(apply): decode percent-encoded purls in human output

mikolalysenko Jun 11, 2026

18b4cb1

feat(vendor): hold patch blobs in memory — vendoring writes no .socke…

mikolalysenko Jun 12, 2026

c92f6b2

feat(repair): rebuild missing/corrupt vendored artifacts + no-ledger …

mikolalysenko Jun 12, 2026

c05805e

fix(pnpm): bind vendor edits to name@VERSION — multi-version vendorin…

mikolalysenko Jun 12, 2026

6091280

test(docker): pin scan --sync to --strict where apply --force must st…

mikolalysenko Jun 12, 2026

cb1b0a6

test(docker): pin the content_mismatch_overwritten warning in the for…

mikolalysenko Jun 12, 2026

File filter

Filter by extension .lock (1) .rs (4) .toml (1) All 3 file types selected

Only manifest files

Viewed files

Clear filters

Conversations

Failed to load comments.

Uh oh!

There was an error while loading. Please reload this page.

Jump to

Jump to file

Failed to load files.

Uh oh!

There was an error while loading. Please reload this page.

Diff view

Unified

Split

Hide whitespace

Diff view

Unified

Split

Hide whitespace

Previous commit

Next commit

feat(vendor,scan): auto-fetch missing packages + lockfile/ledger disc…

…overy

vendor: a manifest patch whose package has no installed copy is now
satisfied automatically (no flag) instead of failing with
package_not_installed:
- already-vendored purls stage from their own committed artifact,
  sha256-verified against the vendor ledger (fresh-clone re-vendor and
  in-sync runs work offline, no registry traffic);
- otherwise the lockfile-resolved pristine artifact is fetched
  (lock-recorded URL else conventional registry URL), verified against
  the lock's integrity FAIL-CLOSED, and staged from a private tempdir —
  the project tree is never touched.
Reason codes: vendor_fetched_missing (skip-warning beside the Applied
event), vendor_fetch_failed (distinct Failed, suppresses the duplicate
not-installed skip), vendor_fetch_unverifiable (no lock integrity →
calm skip). --offline keeps the calm skip and names the lockfile as
the would-be source.

scan: discovery now supplements the installed-tree crawl with
(a) lockfile-only dependencies — warned '[NOT INSTALLED]' in the table
    + a stderr note, JSON lockfileOnlyPackages + packages[].notInstalled,
    counted as scanned so a wiped node_modules no longer prunes
    lockfile-listed entries, partitioned out of --apply BEFORE download
    (calm skipped/package_not_installed records, exit 0, no manifest
    writes) while --vendor passes them to the auto-fetch; and
(b) vendored-ledger entries — the committed artifact IS the dependency,
    so updates[] detection and scan --vendor keep working on a fresh
    clone before any install.

scan --json --vendor now vendors a completely fresh clone end-to-end
(e2e-proven, second run already_vendored).

Co-Authored-By: Claude Fable 5 <[email protected]>

Loading branch information

Uh oh!

There was an error while loading. Please reload this page.

mikolalysenko and claude committed Jun 11, 2026

commit 18822dc9888998c76a99a94c9d8f4f6911a9fa4f

There are no files selected for viewing

2 changes: 2 additions & 0 deletions 2 Cargo.lock

Show comments

View file

Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

3 changes: 3 additions & 0 deletions 3 crates/socket-patch-cli/Cargo.toml

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -59,6 +59,9 @@ setup-e2e = []

		[dev-dependencies]
		sha2 = { workspace = true }
		# scan_vendor_e2e builds pristine registry tarballs for the auto-fetch tests.
		tar = { workspace = true }
		flate2 = { workspace = true }
		hex = { workspace = true }
		wiremock = { workspace = true }
		portable-pty = { workspace = true }
Expand Down

245 changes: 243 additions & 2 deletions 245 crates/socket-patch-cli/src/commands/scan.rs

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -281,6 +281,123 @@ pub(crate) fn detect_prunable(
		.collect()
		}

		/// Lockfile-only packages: dependencies the project's lockfile resolves
		/// that have no crawled (installed) counterpart.
		#[derive(Default)]
		struct LockfileSupplement {
		packages: Vec<socket_patch_core::crawlers::types::CrawledPackage>,
		/// Literal crawler-form purls, for fast membership tests.
		purls: HashSet<String>,
		/// The lockfile the entries came from, for messages.
		source: &'static str,
		}

		/// Inventory the project's lockfile(s) and fabricate crawl entries for
		/// dependencies that are not installed. The fabricated `path` is the
		/// WOULD-BE install dir — every consumer degrades safely on a nonexistent
		/// path (hash verify → NotFound, apply → partitioned skip, vendor →
		/// auto-fetch). Global scans target the machine's global tree, not this
		/// project's lockfile, so they get no supplement.
		async fn lockfile_supplement(
		common: &GlobalArgs,
		crawled: &[socket_patch_core::crawlers::types::CrawledPackage],
		) -> LockfileSupplement {
		use socket_patch_core::patch::vendor::lock_inventory;
		use socket_patch_core::patch::vendor::npm_flavor::NpmLockFlavor;

		let mut out = LockfileSupplement::default();
		if common.global \|\| common.global_prefix.is_some() {
		return out;
		}
		let Some((flavor, entries)) = lock_inventory::inventory_npm_lock(&common.cwd).await else {
		return out;
		};
		out.source = match flavor {
		NpmLockFlavor::PackageLock => "package-lock.json",
		NpmLockFlavor::Pnpm => "pnpm-lock.yaml",
		NpmLockFlavor::YarnClassic \| NpmLockFlavor::YarnBerry => "yarn.lock",
		NpmLockFlavor::Bun => "bun.lock",
		};
		let crawled_purls: HashSet<&str> = crawled.iter().map(\|p\| p.purl.as_str()).collect();
		for entry in entries {
		if crawled_purls.contains(entry.purl.as_str()) {
		continue;
		}
		let (namespace, name) = match entry.name.split_once('/') {
		Some((scope, bare)) => (Some(scope.to_string()), bare.to_string()),
		None => (None, entry.name.clone()),
		};
		out.purls.insert(entry.purl.clone());
		out.packages.push(socket_patch_core::crawlers::types::CrawledPackage {
		name,
		version: entry.version.clone(),
		namespace,
		purl: entry.purl.clone(),
		path: common.cwd.join("node_modules").join(&entry.name),
		});
		}
		out
		}

		/// A displayable crawl entry fabricated from a purl (decoded form). The
		/// path is a placeholder consumers degrade safely on.
		fn crawled_from_purl(
		purl: &str,
		cwd: &std::path::Path,
		) -> Option<socket_patch_core::crawlers::types::CrawledPackage> {
		let decoded = normalize_purl(strip_purl_qualifiers(purl)).into_owned();
		let rest = decoded.strip_prefix("pkg:")?;
		let (_eco, rest) = rest.split_once('/')?;
		let at = rest.rfind('@').filter(\|&i\| i > 0)?;
		let (name_part, version) = (&rest[..at], &rest[at + 1..]);
		let (namespace, name) = match name_part.rsplit_once('/') {
		Some((ns, n)) => (Some(ns.to_string()), n.to_string()),
		None => (None, name_part.to_string()),
		};
		Some(socket_patch_core::crawlers::types::CrawledPackage {
		name,
		version: version.to_string(),
		namespace,
		purl: decoded.clone(),
		path: cwd.join("node_modules").join(name_part),
		})
		}

		/// Vendored-ledger packages with no crawled counterpart: on a fresh clone
		/// the committed artifact IS the dependency, so these stay discoverable
		/// (updates[] detection, the table, and `scan --vendor` re-vendor/in-sync
		/// runs all keep working before any install). They are NOT "lockfile-only"
		/// — nothing needs installing; the artifact satisfies the lock.
		async fn vendored_ledger_supplement(
		common: &GlobalArgs,
		crawled: &[socket_patch_core::crawlers::types::CrawledPackage],
		) -> Vec<socket_patch_core::crawlers::types::CrawledPackage> {
		if common.global \|\| common.global_prefix.is_some() {
		return Vec::new();
		}
		let Ok(state) = socket_patch_core::patch::vendor::load_state(&common.cwd).await else {
		return Vec::new();
		};
		let crawled_norm: HashSet<String> = crawled
		.iter()
		.map(\|p\| normalize_purl(&p.purl).into_owned())
		.collect();
		let mut seen: HashSet<String> = HashSet::new();
		let mut out = Vec::new();
		for entry in state.entries.values() {
		let base = strip_purl_qualifiers(&entry.base_purl);
		let norm = normalize_purl(base).into_owned();
		if crawled_norm.contains(&norm) \|\| !seen.insert(norm) {
		continue;
		}
		if let Some(pkg) = crawled_from_purl(base, &common.cwd) {
		out.push(pkg);
		}
		}
		out.sort_by(\|a, b\| a.purl.cmp(&b.purl));
		out
		}

		/// Vendor-mode pre-prompt check: uuids of selected patches whose installed
		/// files match NEITHER beforeHash nor afterHash — the patch was built
		/// against different bytes than the installed artifact. Vendoring still
Expand All		@@ -296,6 +413,7 @@ async fn preverify_vendor_baselines(
		org_slug: Option<&str>,
		selected: &[PatchSearchResult],
		crawled: &[socket_patch_core::crawlers::types::CrawledPackage],
		lockfile_only: &HashSet<String>,
		) -> HashSet<String> {
		use socket_patch_core::manifest::schema::PatchFileInfo;
		use socket_patch_core::patch::apply::{verify_file_patch, VerifyStatus};
Expand All		@@ -306,6 +424,11 @@ async fn preverify_vendor_baselines(
		// API purls come percent-encoded, crawler purls literal — purl_eq
		// bridges the two spellings.
		let base = strip_purl_qualifiers(&patch.purl);
		// Lockfile-only packages have no installed bytes to compare — the
		// vendor engine fetches them pristine (nothing to annotate).
		if lockfile_only.contains(normalize_purl(base).as_ref()) {
		continue;
		}
		let Some(pkg) = crawled.iter().find(\|c\| purl_eq(&c.purl, base)) else {
		continue;
		};
Expand Down Expand Up		@@ -951,6 +1074,39 @@ fn partition_vendored_selected(
		(kept, vendored_records)
		}

		/// Lockfile-only patches are skipped BEFORE download in apply mode: the
		/// package is not on disk to patch in place, and downloading its patch
		/// into the manifest would create a not-yet-appliable entry (and flip the
		/// apply path's exit code). `scan --vendor` is the route that handles them
		/// (the vendor engine auto-fetches lockfile-resolved packages). Matching
		/// bridges API purl encoding via `normalize_purl`. Same shape/mechanics as
		/// [`partition_vendored_selected`].
		fn partition_not_installed_selected(
		selected: Vec<PatchSearchResult>,
		lockfile_only: &HashSet<String>,
		) -> (Vec<PatchSearchResult>, Vec<serde_json::Value>) {
		if lockfile_only.is_empty() {
		return (selected, Vec::new());
		}
		let is_lockfile_only = \|p: &str\| {
		lockfile_only.contains(normalize_purl(strip_purl_qualifiers(p)).as_ref())
		};
		let (not_installed, kept): (Vec<_>, Vec<_>) = selected
		.into_iter()
		.partition(\|p\| is_lockfile_only(&p.purl));
		let mut records: Vec<serde_json::Value> = not_installed
		.iter()
		.map(\|p\| {
		serde_json::json!({
		"purl": p.purl, "uuid": p.uuid,
		"action": "skipped", "errorCode": "package_not_installed",
		})
		})
		.collect();
		records.sort_by(\|a, b\| a["purl"].as_str().cmp(&b["purl"].as_str()));
		(kept, records)
		}

		/// Fold the pre-download vendored skips into the apply report returned by
		/// `download_and_apply_patches`: they were "found" by discovery and
		/// skipped here, never downloaded. Also strips the inner `status` (scan
Expand Down Expand Up		@@ -1184,7 +1340,24 @@ pub async fn run(args: ScanArgs) -> i32 {
		}

		// Crawl packages
		let (all_crawled, eco_counts) = crawl_all_ecosystems(&crawler_options).await;
		let (mut all_crawled, mut eco_counts) = crawl_all_ecosystems(&crawler_options).await;

		// Lockfile supplement: dependencies the project's lockfile resolves
		// that have NO installed copy (fresh clone, partial install). They join
		// discovery — counts, API lookup, table, the prune "scanned" set — and
		// are flagged "not yet installed" everywhere a user could act on them.
		let lockfile_only = lockfile_supplement(&args.common, &all_crawled).await;
		if !lockfile_only.packages.is_empty() {
		*eco_counts.entry(Ecosystem::Npm).or_insert(0) += lockfile_only.packages.len();
		all_crawled.extend(lockfile_only.packages.iter().cloned());
		}
		let ledger_supplement = vendored_ledger_supplement(&args.common, &all_crawled).await;
		for pkg in &ledger_supplement {
		if let Some(eco) = Ecosystem::from_purl(&pkg.purl) {
		*eco_counts.entry(eco).or_insert(0) += 1;
		}
		}
		all_crawled.extend(ledger_supplement);

		// Every PURL the crawl found, captured BEFORE the `--ecosystems`
		// display/query filter is applied. Prune (below) must reference the
Expand All		@@ -1193,6 +1366,9 @@ pub async fn run(args: ScanArgs) -> i32 {
		// prune used the filtered set instead, `scan --ecosystems npm --prune`
		// would treat every cargo/go/pypi/gem manifest entry as "uninstalled"
		// and delete it (plus its blobs) — silent cross-ecosystem data loss.
		// Lockfile-only purls are deliberately included: a dependency the
		// lockfile still resolves must not be pruned just because node_modules
		// is wiped or partially installed.
		let installed_purls: HashSet<String> = all_crawled.iter().map(\|p\| p.purl.clone()).collect();

		// Vendor-ledger purl keys, loaded once and shared by the prune
Expand Down Expand Up		@@ -1250,6 +1426,7 @@ pub async fn run(args: ScanArgs) -> i32 {
		let mut result = serde_json::json!({
		"status": "success",
		"scannedPackages": 0,
		"lockfileOnlyPackages": 0,
		"packagesWithPatches": 0,
		"totalPatches": 0,
		"freePatches": 0,
Expand Down Expand Up		@@ -1310,6 +1487,13 @@ pub async fn run(args: ScanArgs) -> i32 {
		} else {
		eprintln!("Found {package_count} packages{eco_summary}");
		}
		if !lockfile_only.purls.is_empty() {
		eprintln!(
		"Note: {} package(s) from {} are not yet installed (lockfile-only).",
		lockfile_only.purls.len(),
		lockfile_only.source,
		);
		}
		}

		// Query API in batches
Expand Down Expand Up		@@ -1495,6 +1679,7 @@ pub async fn run(args: ScanArgs) -> i32 {
		let mut result = serde_json::json!({
		"status": "success",
		"scannedPackages": package_count,
		"lockfileOnlyPackages": lockfile_only.purls.len(),
		"packagesWithPatches": all_packages_with_patches.len(),
		"totalPatches": total_patches,
		"freePatches": free_patches,
Expand All		@@ -1507,6 +1692,19 @@ pub async fn run(args: ScanArgs) -> i32 {
		"newUuid": u.new_uuid,
		})).collect::<Vec<_>>(),
		});
		// Flag lockfile-only packages so JSON consumers can tell "patch
		// available but not installed" from the installed case. Additive
		// field; absent means installed.
		if let Some(packages) = result["packages"].as_array_mut() {
		for pkg in packages {
		let is_lockfile_only = pkg["purl"]
		.as_str()
		.is_some_and(\|p\| lockfile_only.purls.contains(p));
		if is_lockfile_only {
		pkg["notInstalled"] = serde_json::json!(true);
		}
		}
		}

		// `apply` and `prune` are computed once at the top of run()
		// (factoring in --sync, which implies both). They're independent
Expand Down Expand Up		@@ -1549,6 +1747,17 @@ pub async fn run(args: ScanArgs) -> i32 {
		// operator's signal to run `scan --vendor` (or `vendor`).
		let (selected, vendored_records) =
		partition_vendored_selected(selected, &vendored_purls);
		// Lockfile-only purls leave the apply selection here (calm
		// skip records, never an error); the union rides the same
		// bookkeeping as the vendored skips.
		let (selected, vendored_records) = {
		let (kept, not_installed) =
		partition_not_installed_selected(selected, &lockfile_only.purls);
		let mut all = vendored_records;
		all.extend(not_installed);
		all.sort_by(\|a, b\| a["purl"].as_str().cmp(&b["purl"].as_str()));
		(kept, all)
		};

		let mut apply_code = 0i32;
		if dry {
Expand Down Expand Up		@@ -1791,14 +2000,22 @@ pub async fn run(args: ScanArgs) -> i32 {
		} else {
		String::new()
		};
		// Lockfile-only packages can be patched by `scan --vendor`
		// (which fetches them pristine) but not applied in place.
		let not_installed_marker = if lockfile_only.purls.contains(pkg.purl.as_str()) {
		color(" [NOT INSTALLED]", "33", use_color)
		} else {
		String::new()
		};

		println!(
		"{:<40} {:>8} {:<16} {}{}",
		"{:<40} {:>8} {:<16} {}{}{}",
		display_purl,
		count_str,
		format_severity(severity, use_color),
		vuln_str,
		update_marker,
		not_installed_marker,
		);
		}

Expand Down Expand Up		@@ -1930,6 +2147,29 @@ pub async fn run(args: ScanArgs) -> i32 {
		}
		}

		// Lockfile-only purls leave the in-place apply selection (calm skip,
		// mirrors the JSON path). In `--vendor` mode they stay: the vendor
		// engine fetches lockfile-resolved packages pristine.
		let (selected, not_installed_selected): (Vec<_>, Vec<String>) = if args.vendor {
		(selected, Vec::new())
		} else {
		let (kept, skipped) = partition_not_installed_selected(selected, &lockfile_only.purls);
		let printed: Vec<String> = skipped
		.iter()
		.filter_map(\|r\| r["purl"].as_str().map(str::to_string))
		.collect();
		(kept, printed)
		};
		if !args.common.silent {
		for purl in &not_installed_selected {
		println!(
		" [skip] {} (not installed — run your package manager's install first, \
		or `scan --vendor` to vendor it from the lockfile)",
		normalize_purl(purl)
		);
		}
		}

		if selected.is_empty() && !args.vendor {
		if !args.common.silent {
		println!("No patches selected.");
Expand All		@@ -1946,6 +2186,7 @@ pub async fn run(args: ScanArgs) -> i32 {
		effective_org_slug,
		&selected,
		&filtered_crawled,
		&lockfile_only.purls,
		)
		.await
		} else {
Expand Down

Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.