GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Count cached metadata against max_size and evict it LRU by pinguinfuss · Pull Request #507 · git-pkgs/proxy

Conversation

@pinguinfuss

@pinguinfuss pinguinfuss commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

Fixes #461.

max_size now counts cached metadata as well as artifacts, and eviction drops whatever was served least recently.

  1. metadata_cache gets last_accessed_at (migration 014, backfilled from updated_at), and every cached read records a hit, batched like artifact hits.
  2. Eviction merges the oldest artifacts and metadata rows. An artifact that was never served counts from its fetch time. Both orders are indexed.
  3. Arch writes a database blob again if eviction removed it while a refresh was re-creating the row.
  4. /stats is unchanged. New metric: proxy_metadata_evicted_total.

Tested live with max_size 5MB on SQLite and Postgres: floods of sumdb hash tiles and large npm packuments always went back to 5 MB after the next eviction pass. On main the same setup stayed over the limit.

Not covered: storage can exceed the limit until the next pass (once a minute), and a metadata flood now evicts artifacts too.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
pinguinfuss added 2 commits October 18, 2026 07:27
@pinguinfuss
max_size now counts cached metadata as well as artifacts, and evictio…
b87c656 
…n drops whichever was used least recently, so metadata no longer piles up for good (git-pkgs#461). Metadata rows record when they were last served, an artifact not yet served counts from its download, and Arch stores a database blob again if eviction removed it under a refreshed row.
@pinguinfuss
Shorter comments on the metadata last-access migration.
552f049
andrew
andrew requested changes Oct 18, 2026
View reviewed changes

@andrew andrew left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address durable metadata cleanup, move the data backfill out of the schema migration, and surface the new eviction metric in analytics.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread internal/server/eviction.go

// Metadata goes now too: readers hold the whole document in memory,
// and a row refetched meanwhile refetches again on a missing blob.
if err := store.Delete(ctx, path); err != nil {

@andrew andrew Oct 18, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new metadata path deletes the metadata_cache row before deleting the blob, and queues cleanup only after a storage error. A process crash between the row deletion and this call leaves a blob with neither a cache row nor a pending-delete record, so it disappears from max_size accounting and cannot be reclaimed by the queue. Delete the metadata row and enqueue its storage path in one database transaction, then let cleanup remove the blob. Please cover interruption and retry behavior.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread internal/database/schema.go Outdated
return fmt.Errorf("adding column last_accessed_at to metadata_cache: %w", err)
}
}
if err := backfillMetadataLastAccessed(db); err != nil {

@andrew andrew Oct 18, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please keep this data backfill outside the schema migration. backfillMetadataLastAccessed loads every existing row into memory, then performs one UPDATE per row in a single transaction before startup can finish. Move it into bounded, resumable work and keep eviction safe while last_accessed_at is still NULL. Cover an upgrade with existing rows and an interrupted backfill.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread internal/metrics/metrics.go

MetadataEvicted = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "proxy_metadata_evicted_total",

@andrew andrew Oct 18, 2026

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new proxy_metadata_evicted_total metric is absent from /ui/analytics and metricSurface, contrary to CONTRIBUTING.md. TestEveryMetricIsSurfaced also does not instantiate this counter, so the normal test order hides the omission. Reproduced with go test ./internal/server -run 'TestEveryMetricIsSurfaced|TestEvictLRU_CountsAndEvictsMetadata' -count=2. Add the analytics display, the metricSurface entry, and a RecordMetadataEvicted call in the coverage test setup.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
pinguinfuss and others added 4 commits October 19, 2026 19:39
@pinguinfuss
Show evicted metadata on the analytics page next to evicted artifacts…
e6c76b0 
…, and cover the new counter in the metric surface test.
@pinguinfuss
Evicting metadata now drops the row and queues its path in one transa…
b94533d 
…ction, so a crash before the blob is deleted leaves it to reclaim instead of orphaning it.
@pinguinfuss
The last-access backfill now runs in the background in batches of 100…
b87c751 
…0 instead of inside the migration, resuming after a restart, and until a row is backfilled eviction orders it by its write time.
@andrew
Merge main into issue-461-metadata-eviction
eca9d9e
@andrew
andrew merged commit d56d436 into git-pkgs:main Oct 20, 2026
6 checks passed

Uh oh!

There was an error while loading. Please reload this page.
@andrew andrew mentioned this pull request Oct 20, 2026
Separate artifact and metadata cache metrics on /metrics and the analytics page #463
Merged
@simonchrz simonchrz mentioned this pull request Oct 25, 2026
Extract portable npm metadata transformations #612
Merged
@pinguinfuss
pinguinfuss deleted the issue-461-metadata-eviction branch October 26, 2026 06:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrew andrew andrew requested changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Metadata cache never shrinks and isn't counted against max_size

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@pinguinfuss @andrew
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.