GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fix(knowledge): stop capping the unpaged knowledge-base list by waleedlatif1 · Pull Request #6771 · simstudioai/sim

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Aug 17, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Collaborator

Summary

  • Staging's integration test failed on the first run after fix(knowledge): list knowledge bases on the same authority that creates them #6770 deployed: GET /api/knowledge returned 500 with Knowledge base list exceeds the 10000 row limit. The same test had passed 28 consecutive runs over the prior two days against the same data — the delta was the code, not the volume.
  • fix(knowledge): list knowledge bases on the same authority that creates them #6770 routed the internal list through the workspace read, which carried a 10,000-row cap. Soft-delete cleanup reclaims archived knowledge bases only past a retention window, and not at all where none is configured, so a workspace that archives faster than that window crosses any fixed count on its own.
  • The cap could never have worked as a guard. Its throw was gated on limit === undefined, so it fired only for callers that had not asked for a page — precisely the callers with no cursor to retry with and no way to ask for less. A paged caller never reached it. It also read one row past the cap before throwing, so it refused to serve rows it had already materialized; most of the memory cost was already paid.
  • Removed. An unpaged read is unbounded, matching the sibling internal lists (listTables, workspace files, neither capped). Paged callers keep their page and their nextCursorKeys. This also clears the same latent 500 in three other unpaged callers — the archived list use case, the catalog read, and the VFS name lookup — rather than only the surface that happened to fail.

Two more of the same shape, found while auditing for others:

  • attachConnectorTypes threw a bare Error above its own row cap, on those same unpaged callers. Archiving a knowledge base archives its connectors, so the growth curve that broke staging could not reach it — but it is the identical construct, and the sibling latestJobsForTables has no equivalent.
  • The VFS path lookup read every knowledge base whose name merely contained the term and exact-matched in JS, so a single-row lookup scaled with the workspace. It now queries the exact name and reads two rows, mirroring findActiveTablesByExactName.

Regression test reproduces the incident: it fails with the cap restored, passes without.

Type of Change

  • Bug fix

Testing

Tested manually. Full apps/sim suite, type-check, lint, and all 29 audits pass.

Two things worth separate follow-up, neither in this PR:

  • The integration test's teardown leaves archived knowledge bases behind, which is how that workspace reached five figures.
  • sync-engine.ts accumulates every synced document's extracted text into one array with a page-count bound but no byte cap. That is where a bound would actually earn its keep.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

greptile-apps[bot] reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
@vercel

vercel Bot commented Aug 17, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Skipped Skipped Aug 17, 2026 6:11am

Request Review

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vercel
vercel Bot temporarily deployed to Preview August 17, 2026 05:47 Inactive
@cursor

cursor Bot commented Aug 17, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

PR Summary

Medium Risk
Unpaged list endpoints can return very large result sets and heavier connector projections, trading prior hard failures for memory/latency risk on huge workspaces.

Overview
Fixes staging 500s on GET /api/knowledge when a workspace had more than 10,000 knowledge bases (often archived rows with slow cleanup). Unpaged workspace, legacy, and merged list reads no longer enforce MAX_KNOWLEDGE_BASES_PER_WORKSPACE / legacy / connector caps; paging with limit still uses limit + 1 for nextCursorKeys.

Copilot VFS rename/delete no longer loads the whole workspace with a substring search and filters in JS. They call new findActiveKnowledgeBasesByExactName (exact name, limit 2), aligned with findActiveTablesByExactName.

Removed unused constants from constants.ts. Tests cover unbounded unpaged reads, large archived sets, paging, and exact-name lookup.

Reviewed by Cursor Bugbot for commit bc7d114. Configure here.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@greptile-apps

greptile-apps Bot commented Aug 17, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR removes fixed caps from unpaged knowledge-base and connector projections while preserving keyset limits for explicitly paged reads. It also replaces VFS name resolution through a workspace-wide search with a bounded exact-name query.

  • Makes unpaged workspace and legacy knowledge-base reads unbounded.
  • Keeps paged workspace reads at limit + 1 for cursor generation.
  • Adds an exact-name, two-row VFS lookup.
  • Updates regression and paging tests.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename Overview
apps/sim/lib/knowledge/service.ts Removes unpaged row and connector projection caps, preserves explicit paging, and adds a bounded exact-name lookup.
apps/sim/lib/knowledge/application/knowledge-vfs.ts Switches VFS knowledge-base resolution from workspace list filtering to the exact-name service query.
apps/sim/lib/knowledge/service.test.ts Replaces cap-failure coverage with unbounded-read, cursor paging, exact-name lookup, and oversized archived-list regression coverage.
apps/sim/lib/knowledge/constants.ts Removes constants for the deleted knowledge-base and connector projection limits.

Reviews (2): Last reviewed commit: "fix(knowledge): stop capping the unpaged..." | Re-trigger Greptile

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

greptile-apps[bot]
greptile-apps Bot reviewed Aug 17, 2026
View reviewed changes
Comment thread apps/sim/lib/knowledge/service.ts Resolved

Uh oh!

There was an error while loading. Please reload this page.

@waleedlatif1
fix(knowledge): stop capping the unpaged knowledge-base list
bc7d114
#6770 routed GET /api/knowledge through the workspace read, which carried a
10,000-row cap. Staging crossed it, and the first list request after the deploy
returned 500 with "Knowledge base list exceeds the 10000 row limit" — against
data that had served fine for days.

The cap could never have worked. Its throw was guarded by `limit === undefined`,
so it fired only for callers that had NOT asked for a page: exactly the callers
with no cursor to retry with and no way to ask for less. A paged caller never
reached it. It also read one row PAST the cap before throwing, so it refused to
serve rows it had already materialized — most of the memory was already spent.
Soft-delete cleanup reclaims archived rows only past a retention window, and not
at all where none is configured, so a workspace that archives faster than that
window crosses any fixed count on its own.

Remove it. An unpaged read is unbounded, matching the sibling internal lists
(`listTables`, workspace files), and paged callers keep their page. That fixes
the same latent 500 in the archived list, the catalog read, and the VFS name
lookup, which are all unpaged too, rather than only the surface that failed.

Two more of the same shape found while auditing for others:

- `attachConnectorTypes` threw a bare Error above its own cap, on those same
  unpaged callers. Archiving a knowledge base archives its connectors, so the
  growth curve that broke staging could not reach it — but it is the identical
  construct, and the sibling `latestJobsForTables` has no equivalent.
- The VFS path lookup read every knowledge base whose name merely CONTAINED the
  term and then exact-matched in JS, so a single-row lookup scaled with the
  workspace. It now queries the exact name and reads two rows, mirroring
  `findActiveTablesByExactName`.
@waleedlatif1
waleedlatif1 force-pushed the fix/kb-archived-list-cap branch from 0647e0b to bc7d114 Compare August 17, 2026 06:11
@vercel
vercel Bot temporarily deployed to Preview August 17, 2026 06:11 Inactive
@waleedlatif1

waleedlatif1 commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

@greptile

greptile-apps[bot] reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@waleedlatif1

waleedlatif1 commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

@cursor review

cursor[bot] reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

cursor[bot]
cursor Bot reviewed Aug 17, 2026
View reviewed changes

@cursor cursor Bot left a comment

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit bc7d114. Configure here.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@waleedlatif1
waleedlatif1 merged commit fd828f8 into staging Aug 17, 2026
31 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@waleedlatif1
waleedlatif1 deleted the fix/kb-archived-list-cap branch August 17, 2026 06:17
@waleedlatif1 waleedlatif1 mentioned this pull request Aug 17, 2026
v0.8.3: sim cli, new tools, credential groups, splunk, finder-like files refactor, security hardening #6765
Merged

This branch was previously deployed

1 inactive deployment
Preview — bc7d1141 Deployed Aug 17, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@greptile-apps greptile-apps[bot] greptile-apps[bot] left review comments
@cursor cursor[bot] cursor[bot] left review comments

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@waleedlatif1
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.