Problem
Tool responses are bounded in two places today:
- Per-tool row limits (
TOOL_RESULT_LIMITS, Fixes #1542: Parameterize relationship query limits, add truncation flags & CLI support #1594): these cut only the one list each handler names (ranked_results, unused, ...) and set truncated / result_limit.
- The final text cap (
_apply_response_token_limit in server.py, refined by the open fix(mcp): validate tool arguments and return consistent bounded errors #1836): when the serialized JSON is over MAX_TOOL_RESPONSE_TOKENS / MAX_PROMPT_CHARS, it becomes {"truncated": true, "preview": "<first N chars of the JSON text>", "notice": ...}.
Neither of these bounds the other lists in a response, such as sibling lists in the envelope or lists nested inside rows. Responses can also carry the same rows more than once. When such a response hits the text cap, the cut lands wherever the byte budget runs out. The client gets a string holding the first N characters of a JSON document. It does not parse, and the actual answer list may be missing from it entirely.
Repro on upstream/main 1b595a6, no code changes, Kùzu and LadybugDB (identical output)
Synthetic repo: 30 files pkg/mod_<i>.py, each holding one ~60-line function handle_request_<i>. After indexing:
resp = analysis_handlers.find_code(CodeFinder(mgr), query="handle_request", repo_path=repo)
text = encode_response(resp)
| measurement |
value |
lists in results |
content_matches 20, ranked_results 15 (+ 3 empty *_by_name) |
| row objects serialized / distinct |
35 / 20 (ranked_results re-emits rows already in content_matches) |
| response size |
155,177 chars ≈ 38.8k tokens (138,154 chars of it are source bodies) |
truncated flag on the response |
absent (the 20-row cap on ranked_results did not fire; the content_matches list was never in scope) |
That is over Claude Code's 25k-token MCP output limit for a 30-file repo. It is the same mechanism as the 83k-token response in #412.
With the text cap enabled (MAX_TOOL_RESPONSE_TOKENS=5000):
- the output is 20,579 chars with keys
notice, preview, truncated;
json.loads(preview) fails with Unterminated string, and the preview ends mid-line: " value_23 = payload.get('field_23', 23) # normalise fiel";"ranked_results", the ranked answer the tool exists to return, is not in the preview at all. Only the duplicated per-source lists that come before it in key order made it in.
#1836 makes this cap budget-exact, which is a real improvement: the wrapper always fits and is always valid JSON. The preview inside it is still a prefix of the serialized text, so the problem above stays the same.
Impact
Measured in a downstream fork on large real repositories before it bounded every list:
- a
module_deps response of 1.29 MB (one envelope list of 4,539 entries, emitted twice);
- a single row whose
definition_paths list had 611 entries.
A row cap does not catch either, and a byte cap turns both into an unparseable prefix. In the fork, find_all-style defaults of 500 rows came to about 220 KB (about 55k tokens), also over the client's limit.
Proposed fix
Make structured bounds the primary mechanism and keep the text cap as the last-resort guard:
- Walk each tool's response once, centrally in
handle_tool_call (e.g. a bound_tool_response(tool_name, result) step before encoding):
- cap the tool's row lists at its row limit (
TOOL_RESULT_LIMITS, with a default);
- cap every other list, envelope or inside a row, at a small envelope limit (e.g. 20, configurable).
- Each cut list keeps its first entries and gains a sibling
<key>_total with the full length, or reuses an existing total key such as count / total_jobs. A response then stays valid, complete-shaped JSON, and the client knows exactly what was cut.
- Don't serialize the same rows twice in one response (e.g.
find_code's ranked_results alongside the source lists it was built from).
- A registry with one entry per tool, plus a test that sends a pathological payload for every registered tool through the real dispatch, so a new tool can't skip the bound.
The text cap from #1836 then only fires for payloads that are huge even after structured bounding, such as a single giant source string.
A downstream fork implemented this approach (a per-tool bounds registry applied in handle_tool_call, with *_total keys and a registry test). After it, the module_deps response above went from 1.29 MB to 809 B, and definition_paths from 611 entries to 20. It's interleaved with fork-specific handlers and response shapes, so it can't be submitted as a clean patch. I'm opening this as an issue so the design can be agreed alongside #1836. I'm happy to send a PR built on upstream's current handlers once the direction is settled.
Related