GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — Fix Python crawler missing .egg-info installs (#447) by mikolalysenko · Pull Request #452 · SocketDev/socket-patch

Conversation

Mikola Lysenko (mikolalysenko) commented Oct 1, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Collaborator

LLM Description written by Claude Code:claude-opus-5-5

Fixes #447

Root cause

list_dist_info_packages_sync in crates/socket-patch-core/src/crawlers/python_crawler.rs keeps only *.dist-info entries. Legacy installs record their metadata as <name>-<ver>[-pyX.Y].egg-info instead. That covers pip < 23.1 installing an sdist without wheel, distutils, and Debian/Ubuntu apt packages, and the crawler never sees any of them. Every consumer reads that one listing: agent apply/rollback lookups, the hosted redirect_pypi_stale_install guard, the Pipenv vendored stale-install probe, scan -g, and VEX. So all of them miss these packages.

Fix

The site-packages listing (and its test-only equivalence oracle) now reads:

.egg-info directories: the PKG-INFO headers, falling back to the <name>-<version>[-pyX.Y].egg-info name the way the .dist-info reader falls back to its dir name. setuptools escapes - to _, so the first - ends the name and the second starts the interpreter tag.
bare .egg-info files (distutils, distro packages such as PyGObject-3.48.2.egg-info). A file only counts when its headers parse, the same guard as the stray *.dist-info file case.

The wrappers under npm/, pypi/ and gem/ only dispatch to the binary, so they need no change.

Behaviour change worth a reviewer's eye

Apt-installed Python packages on Debian/Ubuntu (/usr/lib/python3/dist-packages/*.egg-info) are now visible. That's what #447 asks for under -g. A Python project with no venv already falls back to the global interpreters, so those copies now take part there too, the same way a pip-installed system copy always has. Two CI tests only passed because Ubuntu's apt python3-six 1.16.0 (the fixtures' exact release) was invisible. Each one claims "nothing installed", and both now point VIRTUAL_ENV at an empty venv to make that true:

e2e_vex_build::hatch hosted pin-basis step: vex correctly refused to attest over the unpatched apt copy.
vendor_eject_fresh_checkout::pypi_eject_needs_no_virtualenv: the release-variant probe compared the apt copy against the fixture's bytes.

Neither test's assertions changed.

Tests (red → green)

Issue item	Test	Before fix	After
agent mode patches an egg-info install (dir + bare file)	`in_process_pypi_apply::pypi_scan_sync_patches_egg_info_install`	FAIL (file left unpatched)	pass
hosted stale-install warning fires	`scan::hosted::python::tests::egg_info_install_gets_the_stale_install_warning`	FAIL (no warning)	pass
discovery / `-g` / `--global-prefix` reports egg-info packages	`in_process_python_envs::pypi_egg_info_layout_handled` (flipped from the pinned "not discovered" contract)	FAIL	pass
reader edge cases (`-pyX.Y`, headerless dir, bare file, headerless stray file)	`python_crawler::tests::egg_info_entries_are_listed`, `test_parse_egg_info_dir_name`	n/a	pass
parallel scan == async oracle (its generator already plants egg-info dirs)	`python_crawler::tests::equivalence::randomized_site_packages_match_the_async_oracle`	—	pass

Real-world check: pip 23.0.1 in a fresh CPython 3.11 venv without wheel (pip install --no-binary six six==1.16.0) writes exactly six-1.16.0-py3.11.egg-info/ + six.py, the layout the new agent test plants (Windows: Lib\site-packages).

Local runs:

cargo clippy --workspace --all-features -- -D warnings: clean.
cargo test --workspace --all-features, batched with --no-fail-fast to fit the sandbox disk: 8324 passed, 10 failed. All 10 are write-failure/permission tests that fail identically on main here because the sandbox runs as root.
cargo fmt --check: main itself isn't fmt-clean and CI doesn't run it, so I didn't reformat unrelated files. The touched files are rustfmt-clean.

CI on 4cb6673: all 481 check runs green (475 success, 6 conditional skips). Earlier single-cell failures on older heads (Bun 1.1.39, Pipenv 2018.11.26 vendored, PDM 0.12.3 space-unicode agent) didn't recur, which fits the live-API transport flakes noted in the comments. Bugbot reviewed 4cb6673: no issues.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MQdCDEbnNPApU1yUukp19S

Note

Medium Risk
Changes shared Python install discovery used by scan, apply, hosted redirects, and VEX; apt/global egg-info packages become visible by design, with broad test coverage but real-environment layout variance.

Overview
Fixes #447 by teaching the Python site-packages scan to treat legacy .egg-info installs like .dist-info, so scan/apply, hosted stale-install checks, and global discovery no longer skip older pip sdist installs, distutils layouts, and distro packages (e.g. Debian python3-*).

Crawler: list_dist_info_packages now reads .egg-info directories via PKG-INFO (with a <name>-<version>[-pyX.Y].egg-info filename fallback) and bare .egg-info files when headers parse; the test oracle matches the blocking scan.

Tests: Discovery expectations flip from “egg-info ignored” to reported PURLs; new coverage for scan --sync patching egg-info layouts, hosted redirect_pypi_stale_install on egg-info, and reader edge cases. Two e2e/vendor tests set VIRTUAL_ENV to an empty venv so Ubuntu’s apt python3-six egg-info does not masquerade as “nothing installed.”

^{Reviewed by Cursor Bugbot for commit 4cb6673. Configure here.}

Generated by Claude Code

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions


          Start fix for #447

81bac2c

Assisted-by: Claude Code:claude-opus-5-5

Mikola Lysenko (mikolalysenko) mentioned this pull request

Oct 1, 2026

Python crawler ignores .egg-info installs, so packages pip ≤ 23.0 installed from sdists are never patched, never get a stale-install warning, and are invisible to scan -g #447

Closed

Claude (claude) added 2 commits

October 1, 2026 09:38


          Find Python packages installed as .egg-info

264870d

pip before 23.1 installs an sdist without the wheel package via
setup.py install, which records the install as
<name>-<version>-pyX.Y.egg-info rather than .dist-info. That is the
default state of a fresh venv on CPython 3.11 and older, and distro
packages (Debian's python3-*) and distutils ship bare .egg-info
files. The crawler only listed .dist-info, so these real, importable
installs were reported "not installed" in agent mode, never got the
hosted stale-install warning, and were missing from scan -g.

The site-packages listing now also reads .egg-info directories
(PKG-INFO, falling back to the directory name) and bare .egg-info
files that carry metadata headers. Every consumer (scan, apply,
rollback, the stale-install guards, -g) shares that listing.

Fixes #447

Assisted-by: Claude Code:claude-opus-5-5


          Make the Hatch hosted pin-basis e2e hermetic

2801f42

The step asserts vex attests from the lock pin when nothing is
installed, but it ran with no venv, so the Hatch project fell back to
the global interpreters. On Ubuntu runners those carry apt's
python3-six 1.16.0, the fixture's exact release, as an .egg-info
install. Now that the crawler sees .egg-info, vex correctly refused
to attest over that unpatched copy. Point VIRTUAL_ENV at an empty
venv so the step tests what it says: a package that is not installed.

Assisted-by: Claude Code:claude-opus-5-5

Mikola Lysenko (mikolalysenko) commented Oct 1, 2026

Copy link

Copy Markdown

Collaborator Author

[agent] native (ubuntu-latest, 1.1.39) failed on 2801f42 (Bun backtest: 41/44 cells, hosted text / space-unicode / workspace-get-search). I don't think this PR caused it:

The diff only touches the Python site-packages listing and a Hatch e2e. The Bun cells run npm/Bun code paths that this change doesn't reach.
The log shows Connection reset by peer from the live https://patches-api.socket.dev/patch/batch in the same leg, which points to a transport flake against that host. The same leg was green on other recent PRs on the same base.

No fix to port. I'll re-run the failed jobs once when the workflow run finishes; a second failure gets treated as real.

Generated by Claude Code

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.


          Keep the pypi eject test off the system six

35394a9

The test vendors a hosted six 1.16.0 from a fresh checkout with
nothing installed, but it ran with no venv, so the project fell back
to the global interpreters. On Ubuntu those carry apt's python3-six
1.16.0 as an .egg-info install. Now that the crawler sees it, the
variant probe compared it with the fixture's bytes and skipped it.
Point VIRTUAL_ENV at an empty venv so nothing is installed.

Assisted-by: Claude Code:claude-opus-5-5

Mikola Lysenko (mikolalysenko) commented Oct 1, 2026

Copy link

Copy Markdown

Collaborator Author

[agent] On 35394a9, the Pipenv compatibility leg matrix (ubuntu-latest, 2018.11.26 …) failed in exactly one cell: 2018.11.26 direct vendored, check rescanIdempotent. Every other check in that cell passed (lock-only apply, stale-install warning, lock rewrite, vex), and so did the other 31 cells.

I don't think this PR caused it:

I re-ran that cell locally with this branch's binary. Its project venv holds only pip-24.0.dist-info, setuptools-69.5.1.dist-info and urllib3-1.26.18.dist-info, with no .egg-info. Because a venv exists, the global-interpreter fallback never runs. So the crawl is identical to main's.
The step drives the live patches-api.socket.dev. In the same window, the Bun leg logged Connection reset by peer from that host. I couldn't run it end to end here because the sandbox can't reach the API.

No fix to port. I'm re-running the failed job once; a second failure gets treated as real.

Generated by Claude Code

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Mikola Lysenko (mikolalysenko) commented Oct 1, 2026

Copy link

Copy Markdown

Collaborator Author

[agent] On 35394a9, the PDM compatibility leg native (ubuntu-latest, 0.12.3) failed in exactly one cell: 0.12.3 space-unicode agent, check appliedExactlyOne. The other 31 cells passed, including every other agent shape. The same leg is green on #425 and #383.

I can't yet rule this PR in or out. Agent mode does read the listing this PR changes, but nothing in this change depends on the project path, and the space/unicode path is the only thing different about that cell. The cell's envelope is in the run artifact, and this sandbox can't download artifacts. The harness also drives the live patches-api.socket.dev, so I can't reproduce the cell locally.

I've re-run the failed job once as the diagnostic. If it fails again, I'll treat it as real and dig into PDM 0.12's install layout for that shape.

Generated by Claude Code

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.


          Use the Windows venv layout in the egg-info test

4cb6673

The new agent-mode egg-info test planted its site-packages at
lib/python3.11/site-packages, which is where a venv keeps them on
Unix. On Windows a venv uses Lib\site-packages, so the crawler never
found the package and the Windows test job failed.

Assisted-by: Claude Code:claude-opus-5-5

Mikola Lysenko (mikolalysenko) marked this pull request as ready for review

October 1, 2026 11:04

Mikola Lysenko (mikolalysenko) commented Oct 1, 2026

Copy link

Copy Markdown

Collaborator Author

BugBot review

Generated by Claude Code

cursor[bot] reacted with thumbs up emoji

All reactions

👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

cursor Bot reviewed

Oct 1, 2026

View reviewed changes

cursor Bot left a comment

Copy link

Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 4cb6673. Configure here.}

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label

Oct 1, 2026

Mikola Lysenko (mikolalysenko) commented Oct 1, 2026

Copy link

Copy Markdown

Collaborator Author

[burn-down agent] Ready for review on 4cb6673: mergeable, 0 commits behind main.

CI: 476/482 check runs green, 6 skipped by workflow conditions, 0 failing. The earlier Pipenv/PDM leg failures were on 35394a9, not this head.
Bugbot: reviewed 4cb6673, no issues. No open review threads.
For the reviewer: the crawler now sees apt-installed .egg-info packages under /usr/lib/python3/dist-packages. Two e2e tests now point VIRTUAL_ENV at an empty venv so their "nothing installed" premise holds. See the description's "Behaviour change" section.

Generated by Claude Code

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Tanmay Singla (Tanmay182003) approved these changes

Oct 1, 2026

View reviewed changes

Mikola Lysenko (mikolalysenko) merged commit 2035cbb into main

Oct 1, 2026

482 checks passed

Uh oh!

There was an error while loading. Please reload this page.

Mikola Lysenko (mikolalysenko) deleted the agent/fix-python-crawler-egg-info branch

October 1, 2026 16:50

This was referenced Oct 1, 2026

With the same release in user site-packages and a system site-packages dir, agent mode patches the shadowed system copy, leaves the copy Python imports unpatched, and VEX attests it (worse since #452: now hits apt-installed packages) #501

Closed

Poetry with virtualenvs.in-project = true but no ./.venv keeps using its out-of-tree env, which socket-patch never probes: agent patches the global interpreter and hosted VEX falsely attests #476

Closed

Benchmark tracking: socket-patch scan #580

Open

Reviewers

cursor[bot]

cursor[bot] left review comments

Tanmay Singla (Tanmay182003)

Tanmay182003 approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

Ready for review

Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Python crawler ignores .egg-info installs, so packages pip ≤ 23.0 installed from sdists are never patched, never get a stale-install warning, and are invisible to scan -g

Uh oh!

There was an error while loading. Please reload this page.

3 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

Mikola Lysenko (mikolalysenko) commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LLM Description written by Claude Code:claude-opus-5-5

Root cause

Fix

Behaviour change worth a reviewer's eye

Tests (red → green)

Uh oh!

Mikola Lysenko (mikolalysenko) commented Oct 1, 2026

Uh oh!

Mikola Lysenko (mikolalysenko) commented Oct 1, 2026

Uh oh!

Mikola Lysenko (mikolalysenko) commented Oct 1, 2026

Uh oh!

Mikola Lysenko (mikolalysenko) commented Oct 1, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Mikola Lysenko (mikolalysenko) commented Oct 1, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Mikola Lysenko (mikolalysenko) commented Oct 1, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.