GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Optimize Python repository-version membership queries by kdelee · Pull Request #1458 · pulp/pulp_python

Conversation

@kdelee

@kdelee kdelee commented Nov 5, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

What

Replace Python repository-version content lookups based on pk__in=repository_version.content with a direct version_memberships join driven by the package-name predicate.

This affects:

  • PyPI Simple package detail responses: /pypi/<distribution>/<package>/;
  • PyPI package metadata responses: /pypi/<distribution>/<package>/json;
  • published Python distribution JSON responses.

Why

The existing query expands the repository version's content_ids array before applying the package filter. GABI tests against production data showed that a direct membership join avoids array expansion and avoids the poor plan selected for a correlated EXISTS candidate.

Performance Insights ranked the affected SQL shapes among the highest database-load statements over the seven-day production window: the PyPI Simple detail query ranked #4 (B4233AAA3E27C4F1CD1366E8153D8FE3864735D6) and the Python metadata query ranked #5 (51F6729E84EE2E32271BAACF399A8A5D2A885FA3).

These are good optimization candidates because they do not require index-level changes. This is a targeted, lower-risk fix than the broader interval-query experiment attempted in pulp-service#1643, specifically patch 0081 (Jan's interval-query experiment).

Full metadata query results

Each repository's largest package. Warm execution times in milliseconds.

Repository Current ms Correlated EXISTS ms IN subquery ms Direct join ms (this PR)
main 314.8 402.5 397.2 76.3
python-validated-landing 265.1 13.7 14.0 25.9
cpu-torch2.13-el9.8-test 57.4 10.3 10.3 14.3
cpu-torch2.13-el9.8 56.7 10.3 10.7 10.9
cuda12.9-torch2.13-el9.8-test 43.6 10.0 9.9 10.4
cuda13.0-torch2.13-el9.8-test 43.5 10.1 9.9 10.5
cuda12.9-torch2.13-el9.8 43.5 10.1 10.0 8.6
cuda13.0-torch2.13-el9.8 43.9 10.1 10.0 8.2
cpu-torch2.11-el9.8-test 42.8 10.3 10.8 12.5
cpu-torch2.13-el9.8-sdists-test 38.8 10.2 10.2 10.2

The four SQL shapes tested are:

  1. Current: pk__in=repository_version.content, backed by unnest(content_ids).
  2. Correlated EXISTS: package filter plus per-row core_repositorycontent membership check.
  3. IN subquery: package filter plus content_ptr_id__in=RepositoryContent subquery.
  4. Direct join: package filter plus version_memberships repository/version predicates; this PR's shape.

The direct join preserves historical membership with version_added and version_removed bounds.

PyPI Simple detail query results

The same repositories and largest package, testing the narrower SQL shape used by /pypi/<distribution>/<package>/:

Repository Current ms Direct join ms (this PR)
main 333.3 75.3
python-validated-landing 276.5 23.1
cpu-torch2.13-el9.8-test 59.9 13.6
cpu-torch2.13-el9.8 59.6 10.2
cuda12.9-torch2.13-el9.8-test 45.3 9.7
cuda13.0-torch2.13-el9.8-test 45.8 9.2
cuda12.9-torch2.13-el9.8 45.1 7.3
cuda13.0-torch2.13-el9.8 45.3 7.5
cpu-torch2.11-el9.8-test 44.2 9.9
cpu-torch2.13-el9.8-sdists-test 39.9 9.5

The direct join preserves historical membership with version_added and version_removed bounds.

Validation

  • Generated SQL from Django ORM in an OCI Env with pulpcore and pulp_python.
  • Compared current, correlated-EXISTS, IN-subquery, and direct-join SQL with GABI using EXPLAIN (ANALYZE, BUFFERS, TIMING OFF).
  • Tested both Python SQL shapes across the ten largest current Python repository versions.
  • Tested representative largest packages: boto3, uv-build, and posthog.
  • git diff --check passed.
  • Python bytecode compilation passed.
  • Repository pre-commit checks passed.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@kdelee
Optimize Python repository content membership queries
7c2a039
Use indexed repository-content membership instead of expanding repository-version content IDs for PyPI endpoints.

Assisted-by: pi coding agent
@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

PyPIMixin.get_content now checks package membership through repository-version content relationships using an Exists subquery.

Changes: PyPI content query

Layer / File(s) Summary
Repository membership query
pulp_python/app/pypi/views.py
get_content uses an Exists subquery to match package primary keys to repository-version content IDs.

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Refactor

Possibly related PRs

  • pulp/pulp_python#1451: Both PRs change how package queries check repository-version content membership.

Suggested reviewers: gerrod3

Merge Risk: Minimal · up to 7c2a0

The change replaces package membership lookups with a relationship query. No concrete merge-blocking issue was identified; normal validation should confirm historical repository-version behavior.

Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: optimizing Python repository-version membership queries.
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The pull request description clearly explains what changed, why the change is needed, affected endpoints, performance results, SQL behavior, and validation performed. The repository checklist is not r…
Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 1 files.

Full details: Description check

Explanation

The pull request description clearly explains what changed, why the change is needed, affected endpoints, performance results, SQL behavior, and validation performed. The repository checklist is not reproduced or explicitly marked, and changelog or documentation status is not stated, but the description is otherwise complete and relevant.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

  • X
  • Mastodon
  • Reddit
  • LinkedIn
Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a6096b4f-5f30-404a-9c52-cd92bdcf041d
Commits

Reviewing files that changed from the base of the PR and between 6e6e0d9 and 7c2a039.

Files selected for processing: 1

  • pulp_python/app/pypi/views.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions github-actions Bot added no-changelog no-issue labels Nov 5, 2026
@kdelee
Optimize published Python metadata queries
44854c8
Reuse correlated repository membership for published JSON responses and narrow metadata projections.

Assisted-by: pi coding agent
@github-actions github-actions Bot added the multi-commit Add to bypass single commit lint check label Nov 5, 2026
@kdelee
kdelee marked this pull request as draft November 5, 2026 18:12
@kdelee
Use direct repository membership joins
237f575
Drive Python package queries from package-name predicates before applying repository-version membership.

Assisted-by: pi coding agent

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

multi-commit Add to bypass single commit lint check no-changelog no-issue

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@kdelee
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.