GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fallback: return available bytes on short feed-based reads by zhechen-cs · Pull Request #752 · msgpack/msgpack-python

Conversation

@zhechen-cs

@zhechen-cs zhechen-cs commented Oct 17, 2026

Copy link
Copy Markdown

With a feed-based Unpacker, read_bytes(5) raises OutOfData in the pure-Python fallback when only b"ab" is buffered. The C extension returns b"ab", and both implementations already return short reads for file-backed input.

Honor raise_outofdata=False in the feed path so raw reads consume and return the available bytes. Normal unpacking still rolls back and raises OutOfData for incomplete objects. The regression covers empty and short buffers, reads after a decoded object, stream offsets, and feeding the next object afterward.

Validation on macOS arm64, CPython 3.12:

  • Both new cases fail on the unmodified fallback and pass on the unmodified C extension.
  • After the fix: fallback full suite 144 passed, 4 skipped; C extension full suite 147 passed, 1 skipped.
  • Ruff lint and format checks, and git diff --check, pass.

The existing thread_unsafe marker warning also occurs on the base. Other platforms and PyPy have not been tested locally.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@zhechen-cs
fallback: return available bytes on short raw reads
f5fd6d3
@JackyHsu-X JackyHsu-X mentioned this pull request Oct 26, 2026
Reject negative lengths in Unpacker.read_bytes #760
Merged
methane pushed a commit that referenced this pull request Oct 27, 2026
@JackyHsu-X
Reject negative lengths in Unpacker.read_bytes (#760)
c14af4b
`Unpacker.read_bytes()` does not reject negative lengths. The C
extension raises an internal `SystemError` from
`PyBytes_FromStringAndSize`; the Python implementation accepts the count
and can return an unintended slice of buffered data.

Raise `ValueError` for a negative byte count before reading or updating
the stream position in either implementation. Four regression cases
cover feed-based and file-based unpackers with `-1` and a larger
negative count. They also check that the failed call leaves the position
unchanged and that zero-length reads and subsequent reads/unpacking
still work.

On Windows/Python 3.13, all four cases fail with each unchanged
implementation. With the fix, the complete C-extension suite has 164
passed and 1 skipped; the Python suite has 161 passed and 4 skipped.
Both retain three existing unknown-marker warnings for `thread_unsafe`
when the free-threaded test plugin is absent. Repository-wide Ruff lint
and format checks pass. Other OS/Python and free-threaded matrices were
not run locally.

Related: #742 validates negative sizes in the packer header APIs; this
change covers the unpacker's raw-byte reader. #752 concerns short
feed-based reads, which is a separate case.

AI assistance: OpenAI Codex investigated, generated and self-reviewed
the patch, and ran the local checks. No separate human review is
claimed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@zhechen-cs
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.