GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — perf(deps): sweep raw CDN across marker probes before any anonymous Contents API call by ShaoLOWu · Pull Request #3348 · microsoft/apm Conversation
Contributor
SummaryImplements the maintainer-approved scope from #3283 (apm-scope:v1): stop spending anonymous Contents API calls during virtual-package validation and marketplace registration on public github.com sources. 1. Virtual-package validation: raw CDN first sweep
2. Marketplace add: reuse the discovery response
TestingNew unit tests (12):
Local run (Python 3.13, Windows): all new tests pass; the pre-existing failure set ( AI-Generated PR disclosureThis PR was authored with AI assistance (Claude-based coding agent working under human direction): issue analysis, fix design, implementation, and test authoring were agent-assisted; scope was user-approved against the maintainer's approved-scope comment on #3283 before implementation. All changes were validated locally as described above, and the human reviewed and confirmed the final diff before submission. Fixes #3283
Sorry, something went wrong. Uh oh!
There was an error while loading. Please reload this page. All reactions…ontents API call Virtual-package validation on public github.com repos spent anonymous Contents API calls per marker file before discovering the layout, and `apm marketplace add` re-fetched the catalog immediately after the discovery probe had just downloaded it. On the default 60 req/h anonymous budget a single Claude-plugin-layout install could burn 4+ calls (validation + registration combined). - validate_virtual_package_exists: probe every marker path through the raw CDN first (zero API calls) before falling back to the existing per-marker Contents API probes; fallbacks and non-public hosts are unchanged - download_raw_file: new raw_only pass-through for CDN-only probes - marketplace add: reuse the catalog response captured during path discovery instead of re-fetching it (sidecar cache semantics kept) - tests: 12 new unit tests covering sweep ordering, API-spend assertions, fallback retention, and catalog reuse Fixes microsoft#3283
ShaoLOWu
requested review from
Daniel Meppiel (danielmeppiel) and
Sergio Sisternes (sergio-sisternes-epam)
as code owners
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Changes recommendedIncorrect and incomplete test mocks cause assertion failures and unintended network requests. 3 open issuesWhat changed in this PRThis PR addresses #3283 by reducing redundant GitHub API requests during virtual-package validation and marketplace registration. Changes:
Review effort: Balanced
Sorry, something went wrong. Uh oh!
There was an error while loading. Please reload this page. All reactions
Contributor
Author
All reactions
Sorry, something went wrong. Uh oh!
There was an error while loading. Please reload this page.
|
|
@microsoft-github-policy-service agree |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
…de-network sweep tests Addresses Copilot review feedback on microsoft#3348: - Retarget the traceback tests (phase3/surface) and the five wave7 patch sites from _auto_detect_path to _auto_detect_path_with_payload, returning (path, payload) tuples so discovery stays mocked instead of executing the real fetch path. - test_non_public_host_skips_sweep: use an actual non-public-host dependency and simulate legacy per-marker misses on the download strategy seam (a bare MagicMock return was accepted as file content), stubbing the directory and ls-remote fallbacks at the validation module boundary so the host boundary is checked fully offline. - test_virtual_file_probe_keeps_legacy_semantics: mock the auth callback and return controlled API 404s (matching the full-miss test) so the single-file probe never attempts real GitHub requests. Sweep test suite runtime drops from ~57s (network-dependent) to ~3s. Verified: 6/6 sweep tests green; 332 tests across the touched marketplace/deps suites pass; tests/unit/deps failure set identical to the pre-change Windows baseline (7 pre-existing symlink/routing failures); ruff clean.
|
Thank you for implementing the scoped fix. Maintainer approval for the bounded scope is recorded in #3283, but neither the PR nor linked issue currently carries status/accepted. Under the triage rule, this PR is labelled status/deferred until the maintainer reconciles that status. This recommendation is advisory only, not merge approval. Generated by autopilot-pr-triage-worker. This comment is AI-generated and may contain errors. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
The host-boundary unit test forced uses_public_github_anonymous_first to False on a github.com dependency, so it never exercised a non-public host and its CDN mock could be reached through the legacy fast path. Use an acme.ghe.com dependency through the real auth resolver, assert the CDN is never contacted and every probe is legacy, and apply ruff format to the PR's new unit tests. Addresses Copilot inline r4348166827 on tests/unit/deps/test_virtual_package_cdn_first_sweep.py. Co-authored-by: Copilot <[email protected]>
The microsoft#3283 scope requires installed-CLI request-count and state evidence, not only unit mocks. Add a loopback HTTPS proxy fixture that terminates CONNECT for raw.githubusercontent.com and api.github.com with a private CA and answers from owned bare repositories, so the uv console script and the frozen binary are both counted hermetically. Lifecycle cases cover multi-plugin, repeat and --update installs with zero Contents API calls, truthful missing-marker failure with one bounded API pass, private authenticated recovery, marketplace add catalog reuse plus marketplace installs, and no registration on a failed discovery. Mutation-break: disabling the raw sweep or the discovery payload reuse fails these tests. Co-authored-by: Copilot <[email protected]>
Document that public github.com subdirectory validation checks markers through raw.githubusercontent.com before the Contents API, and that marketplace add registers the catalog it discovered without downloading it again. Required by the microsoft#3283 scope record (update directly related documentation and usage guidance). Co-authored-by: Copilot <[email protected]>
Changelog entry for the microsoft#3283 request-budget repair, crediting the contributor. Co-authored-by: Copilot <[email protected]>
Addresses panel doc-writer findings: remove the duplicate apm-guide paragraph and update the Install validation chain to list all seven markers with the raw-first sweep; describe the sweep in getting-started without the misleading every-marker and fixed-ref wording; shorten the changelog entry and the marketplace add note. Co-authored-by: Copilot <[email protected]>
Addresses panel python-architect nit: the comment above the probe source still named _auto_detect_path after the call site switched to _auto_detect_path_with_payload. apm-spec-waiver: request scheduling only -- raw-file marker probes run before Contents API probes and marketplace add reuses the catalog it discovered; admission results, markers, refs and lockfile contents are unchanged, so there is no OpenAPM-observable behaviour delta. Co-authored-by: Copilot <[email protected]>
APM Review Panel:
|
| Persona | B | R | N | Takeaway |
|---|---|---|---|---|
| Python Architect | 0 | 0 | 2 | raw_only seam is clean; discovery payload reuse is coordinate-safe (same url/ref/path). |
| Auth Expert | 0 | 0 | 0 | Anonymous-first, token-free CDN, host gating, private recovery and throttle propagation preserved. |
| Performance Expert | 0 | 0 | 1 | Zero Contents API for public markers; no duplicate catalog fetch; full-miss raw repeat is bounded noise. |
| Test Coverage Expert | 0 | 0 | 0 | Done-when surfaces covered: 5 installed-CLI lifecycle tests with real request counts plus existing suites. |
| Doc Writer | 0 | 2 | 1 | Docs accurate after folds; raw-first behavior now stated once in the validation chain. |
| Scenario | Raw | Contents API |
|---|---|---|
Two nested Claude plugins at v1.0.0 (first, repeat, --update) |
10 each | 0 |
marketplace add |
0 | 3 discovery probes, no second fetch (was 4) |
install alpha@mkt beta@mkt |
10 | 0 |
| Missing marker | 14 | 8 (one per marker + directory), exit 1, manifest unchanged |
Private repo with GITHUB_APM_PAT |
14 | bounded, at most 2 per marker, one authenticated hit |
Verified against both the uv console script and a locally built frozen binary.
acme.ghe.com dependency through the real resolver and asserts the CDN is never contacted -- resolved in 0818391.--update, missing marker, private recovery, marketplace add reuse plus marketplace installs, and no registration after failed discovery -- resolved in f75184a.marketplace reference, apm-guide) -- resolved in e7a840a._auto_detect_path_with_payload; commit carries the apm-spec-waiver line for the Mode B gate (request scheduling only, no OpenAPM-observable delta) -- resolved in 68de490.src/apm_cli/commands/marketplace/__init__.py:451 -- LEGIT: stale _auto_detect_path patches would hit the real fetch path (resolved in 3bfc1f4 by the author).tests/unit/deps/test_virtual_package_cdn_first_sweep.py -- LEGIT: boundary test forced the resolver flag on a github.com dependency (resolved in 0818391).tests/unit/deps/test_virtual_package_cdn_first_sweep.py:192 -- LEGIT: single-file probe left the Contents API unmocked (resolved in 3bfc1f4 by the author).test_public_nested_plugins_validate_without_contents_api -- deleted the raw-only marker sweep; test FAILED as expected; guard restored.test_marketplace_add_reuses_discovered_catalog_and_installs_without_contents_api -- deleted discovery_payload reuse; test FAILED as expected; guard restored.TestRawCdnFirstSweep::test_non_public_host_skips_sweep -- deleted the host gate in _raw_cdn_first_applies; test FAILED as expected; guard restored.ruff check and ruff format --check on src/ tests/ silent; pylint R0801 10.00/10; lint-auth-signals.sh and lint-architecture-boundaries.sh clean. Canonical-owner gate: two existing owners touched (transport selection, single-file layout) as an owner extension; exact-head functional tests passed (242 and 24) and the semantic verifier passed.
All checks green on 68de490: 28 pass, 3 skipping, 0 failing, including Lint, Build and Test shards, Lifecycle Smoke shards (which run the new lifecycle file), Windows gates, Spec conformance and the required-checks aggregator (after 1 CI fix iteration: Spec conformance Mode B waiver).
| PR | head SHA | CEO stance | iters | folds | defers | Copilot rounds | CI | mergeable | mergeStateStatus | notes |
|---|---|---|---|---|---|---|---|---|---|---|
| #3348 | 68de490 | ship_now | 1 | 6 | 1 | 2 | green | MERGEABLE | BLOCKED | pending required review |
Labels were not changed. The approved apm-scope:v1 record on #3283 sets the scope for this PR.
flowchart TD
A["validate_virtual_package_exists"] --> C{"public github.com?"}
C -- "yes" --> D["sweep markers raw_only=True"]
D -- "raw hit" --> G["valid, zero Contents API"]
D -- "all miss" --> J["unchanged per-marker loop: raw then Contents API with auth fallback"]
C -- "no: GHE / ADO / other" --> J
J -- "hit" --> G
J -- "miss" --> N["directory Contents API, then ls-remote + ls-tree"]
N -- "hit" --> G
N -- "miss" --> P["invalid, diagnostic"]
MA["marketplace add"] --> MB["_auto_detect_path_with_payload"]
MB -- "path + catalog" --> MF["fetch_marketplace discovery_payload=catalog"]
MF --> MJ["parse + write cache, no second fetch"]
MB -- "none" --> ME["exit 1, nothing registered"]
Generated by autopilot-pr-merge-worker. This comment is AI-generated and may contain errors.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
Co-authored-by: Copilot App <[email protected]>
Keep the PR changelog bullet under [Unreleased] after the 0.35.1 release cut. Co-authored-by: Copilot App <[email protected]>
2ffc07c
into
microsoft:main
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
[BUG] Validating a public in-repo plugin spends one anonymous API call per marker the CDN misses (four for a .claude-plugin plugin)
There was an error while loading. Please reload this page.