GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Handle very long lines in `git rev-list --objects` output by simonkundrik · Pull Request #158 · github/git-sizer

Conversation

@simonkundrik

@simonkundrik simonkundrik commented Jul 6, 2026

Copy link
Copy Markdown

Problem

git-sizer aborts with:

error: error scanning repository: copy-oids: bufio.Scanner: token too long

on repositories that contain an object with a very long path name (the reporter of #157 saw a git rev-list --objects line of ~12 MiB). The copy-oids stage in git/obj_iter.go reads git rev-list --objects output via pipe.LinewiseFunction, whose underlying bufio.Scanner rejects any line longer than the default 64 KiB (bufio.MaxScanTokenSize).

Fix

Replace pipe.LinewiseFunction with pipe.ScannerFunction, supplying a bufio.Scanner that uses the same LF splitting (pipe.ScanLFTerminatedLines) and the same per-line logic, but whose buffer starts at the usual 64 KiB and is allowed to grow up to a new maxRevListLineLength (64 MiB). Because bufio.Scanner only grows its buffer as needed, normal usage still uses minimal memory, while pathologically long lines can now be processed — matching the "normally use fewer resources but deal with larger lines if need be" behaviour suggested in the issue.

The copy-oids per-line logic itself is unchanged.

Tests

Added TestObjectIterLongPath (git/obj_iter_test.go), which builds a tree whose single entry has a 100 KiB file name (created via git hash-object --literally to bypass git's path-length fsck check), then walks it with ObjectIter.

  • Before the fix: the test fails with copy-oids: bufio.Scanner: token too long.
  • After the fix: the test passes and the walk yields the commit, tree, and blob.

go test ./..., go vet ./..., and gofmt are all clean.

Note on the 64 MiB cap

bufio.Scanner requires an upper bound on token size, so this raises the limit rather than removing it entirely. 64 MiB is far beyond any realistic path name (and ~5× the largest line reported in #157) while still bounding per-line memory use. If you'd prefer a fully streaming approach instead — reading just the leading OID and discarding the rest of each line, so memory stays O(hash length) regardless of path length — I'm happy to take the PR in that direction.

Closes #157

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@simonkundrik
Handle very long lines in git rev-list --objects output
40eb644
The `copy-oids` stage used `pipe.LinewiseFunction`, whose scanner
rejects any line longer than 64 KiB with "bufio.Scanner: token too
long". A single line of `git rev-list --objects` output can exceed
that limit when an object has a very long path name, which aborts the
whole object walk.

Use `pipe.ScannerFunction` with a scanner that uses the same LF
line-splitting but whose buffer starts at the usual 64 KiB and may
grow up to 64 MiB, so normal usage stays cheap while pathologically
long lines can still be processed.

Closes github#157
@simonkundrik
simonkundrik requested a review from a team as a code owner July 6, 2026 12:33
@simonkundrik

simonkundrik commented Jul 10, 2026

Copy link
Copy Markdown
Author

Hey! 👋 Just following up on this one. I think it's my first PR to the repo, so the CI checks look like they're waiting on a maintainer to approve the workflow run before they'll kick off. Whenever someone has a minute, would you mind giving that the go-ahead? More than happy to fix anything the tests flag once they run. Thanks a lot!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

git-sizer can fail on some very long lines

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@simonkundrik
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.