GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Keep python crawl tests off the host's venv by mikolalysenko · Pull Request #2055 · SocketDev/socket-patch

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Nov 11, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Collaborator

Problem

CI is healthy this run: the merge queue is empty, every merge_group run since 11-09 passed, main's latest push run and today's nightly are green (the nightly's first green since 10-27), and nothing is queued. So this run finishes the last open item from #1919 that #2054 left for a follow-up: python crawler unit tests that pass only because CI never activates a venv.

Reproduced on origin/main (c09cb0e), cargo test -p socket-patch-core --lib crawlers::python_crawler:

env result
clean 102 passed
VIRTUAL_ENV=<a real venv> 5 failed: test_crawl_all_python, test_crawl_all_not_poisoned_by_body_headers, test_crawl_all_recovers_metadata_less_package, base_copy_behind_a_system_site_venv_is_crawled, venv_copy_hides_the_base_copy_of_the_same_package
CONDA_PREFIX=<a real venv> 1 failed: poetry_out_of_tree_virtualenvs_are_discovered_without_a_dot_venv

A contributor working from an activated venv or conda env gets red tests on a clean checkout.

Root cause

PythonCrawler::crawl_all and get_site_packages_paths build their env lookup from std::env::var, and an activated VIRTUAL_ENV beats the fixture's ./.venv. The Poetry test already clears VIRTUAL_ENV under #[serial], but not CONDA_PREFIX, which poetry_active_prefix treats the same way.

Fix

  • Split out private get_site_packages_paths_with(options, var) and crawl_all_with(options, var). They take the env lookup as a closure, as local_site_packages_with already does. The public functions pass the process env exactly as before, so behaviour doesn't change.
  • The five tests call the _with variants with a no_env lookup. I injected the env instead of an EnvGuard + #[serial] on purpose: mutating process env while other tests run in parallel is itself a flake source.
  • The Poetry test's existing guard also clears CONDA_PREFIX.

No test is removed and no assertion changed. Each test asserts the same thing on the same fixtures.

Proof

cargo test -p socket-patch-core --lib crawlers:: after the change:

env result
clean 798 passed
VIRTUAL_ENV=<venv> 798 passed
CONDA_PREFIX=<venv> 798 passed
VIRTUAL_ENV + CONDA_PREFIX + CONDA_DEFAULT_ENV=x 798 passed
  • crawlers::python_crawler under VIRTUAL_ENV+CONDA_PREFIX, 20 repeats: 0 failures.
  • cargo clippy --locked -p socket-patch-core --all-features -- -D warnings (CI's clippy) is clean. With --all-targets the set of lints per file is identical to main (all pre-existing, in test code).
  • rustfmt --check passes on the touched file.

The public crawl_all stays covered by randomized_site_packages_match_the_async_oracle and the CLI integration suites.

Together with #2054 this meets #1919's acceptance criteria for crawlers::, apart from the two npm_crawler tests #2054 lists, which need the yarn env threaded through NpmCrawler.

Refs #1919


Generated by Claude Code


Generated by Claude Code

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@claude
Keep python crawl tests off the host's venv
144d4ab
Five python_crawler unit tests went through crawl_all and
get_site_packages_paths, which read VIRTUAL_ENV and CONDA_PREFIX
from the process. In an activated venv the crawler takes that venv
instead of the fixture's .venv, so the tests fail on a clean
checkout. CI passes only because no job activates a venv.

Split out crawl_all_with and get_site_packages_paths_with, which
take the env lookup as a closure like local_site_packages_with
already does. The public functions pass the process env as before,
so behaviour doesn't change. The five tests pass an empty env.

poetry_out_of_tree_virtualenvs_are_discovered_without_a_dot_venv
already cleared VIRTUAL_ENV under #[serial]; it now clears
CONDA_PREFIX too, which Poetry treats the same way.

Refs #1919

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_011f9aePBmHCMsHKaD7js5Fu
@mikolalysenko Mikola Lysenko (mikolalysenko) added the ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) label Nov 11, 2026
@mikolalysenko

Mikola Lysenko (mikolalysenko) commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

cursor[bot] reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@cursor

cursor Bot commented Nov 11, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

PR Summary

Low Risk
Production behavior is unchanged; only test helpers and internal _with methods were added, with no assertion changes.

Overview
Python crawler tests were failing when the developer had an activated venv or conda env because get_site_packages_paths and crawl_all always read VIRTUAL_ENV / CONDA_PREFIX from the real process environment.

This PR introduces injectable _with variants (get_site_packages_paths_with, crawl_all_with) that take an env lookup closure (matching local_site_packages_with). The public APIs still use std::env::var unchanged. Affected unit tests call the _with APIs with a no_env stub so fixtures aren’t overridden by the host. The Poetry virtualenv discovery test’s env guard also clears CONDA_PREFIX.

Reviewed by Cursor Bugbot for commit 144d4ab. Configure here.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

cursor[bot]
cursor Bot reviewed Nov 11, 2026
View reviewed changes

@cursor cursor Bot left a comment

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 144d4ab. Configure here.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mikolalysenko Mikola Lysenko (mikolalysenko) mentioned this pull request Nov 11, 2026
Keep the ruby crawl test off the host's Bundler env #2056
Open
@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Nov 11, 2026
@mikolalysenko

Mikola Lysenko (mikolalysenko) commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Ready for review at 144d4ab.

  • CI: all checks finished with no failures (ci-ok, clippy, coverage, e2e shards all green; the rest skipped by path filters). No main-wide failures.
  • Bugbot: reviewed 144d4ab and found nothing. No open review threads.
  • Worth a reviewer's look: the private get_site_packages_paths_with / crawl_all_with seams (env lookup passed as a closure). The public entry points pass std::env::var exactly as before. Also the Poetry test's guard now clears CONDA_PREFIX.

Generated by Claude Code

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mikolalysenko Mikola Lysenko (mikolalysenko) mentioned this pull request Nov 11, 2026
Keep Go unit tests off the host's Go env #2057
Open
@mikolalysenko

Mikola Lysenko (mikolalysenko) commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

Final review brief

What it does. Python crawler unit tests no longer depend on the host's activated venv or conda env. crawl_all and get_site_packages_paths now hand off to private *_with variants that take the env lookup as a closure. The public functions still pass the real process env, so production behaviour is unchanged. Five tests use an empty env, and the Poetry test's env guard also clears CONDA_PREFIX.

Risk: low. It touches one file. It is a pure refactor of two methods plus test-only changes. No assertion was changed and no test was removed.

Look here

  • get_site_packages_paths_with: the public fn passes std::env::var and the body is unchanged
  • crawl_all → crawl_all_with: same pattern
  • no_env: the empty env used by the five tests
  • Poetry test guard: CONDA_PREFIX cleared alongside VIRTUAL_ENV

Verified

  • Read the full diff. The only process-env reads left on the local crawl path are in the public wrappers. The remaining std::env reads in the file are on global or Windows paths, which these tests don't hit.
  • cargo test -p socket-patch-core --lib crawlers::python_crawler with VIRTUAL_ENV and CONDA_PREFIX set to a real venv: 102 passed on this head. On origin/main with VIRTUAL_ENV set: 5 failed, the same five listed in the description. So the tests fail without the change.
  • CHANGELOG.md is untouched. CI: ci-ok and clippy are green on 144d4ab and no checks failed. Bugbot reviewed 144d4ab and found nothing. No open review threads.

Changes I made: none.
Open questions: none.

Auto-merge is armed, so approving sends it straight to the merge queue.


Generated by Claude Code

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mikolalysenko
Mikola Lysenko (mikolalysenko) enabled auto-merge November 11, 2026 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cursor cursor[bot] cursor[bot] left review comments

At least 1 approving review is required to merge this pull request.

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@mikolalysenko @claude
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.