GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-160309: Fix StreamReader.read() silently dropping an incomplete final character by saiyam-jain-dev · Pull Request #160642 · python/cpython

Conversation

@saiyam-jain-dev

@saiyam-jain-dev saiyam-jain-dev commented Oct 25, 2026 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Summary

Fixes #160309

codecs.StreamReader.read() silently drops bytes when the stream ends with an incomplete multibyte sequence. The read loop exits on empty
ewdata while �ytebuffer still holds the trailing bytes, and those bytes are never decoded.

Bug demonstration (before fix)

`python

import codecs, io
reader = codecs.getreader('utf-8')(io.BytesIO(b'ab\xe2'))
reader.read()
'ab' # \xe2 is silently dropped — no error raised
`

Every other decoding path raises or applies the error handler:

`python

b'ab\xe2'.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 2: unexpected end of data
codecs.getreader('utf-8')(io.BytesIO(b'ab\xe2'), 'replace').read()
'ab' # should be 'ab\ufffd' — replacement character is also silently dropped
`

Fix

When the underlying stream is exhausted and �ytebuffer is non-empty, decode the leftover bytes immediately before breaking out of the loop. The codec's stateless decode function applies inal=True semantics, so:

  • errors='strict' → raises UnicodeDecodeError (correct)
  • errors='replace' → emits the replacement character \ufffd (correct)
  • errors='ignore' → drops the bytes (correct)

This matches the behaviour of �ytes.decode() and io.TextIOWrapper.

  • Issue: codecs.StreamReader.read() silently drops an incomplete character at the end of the stream #160309

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@python-cla-bot

python-cla-bot Bot commented Oct 25, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot added the awaiting review label Oct 25, 2026
@bedevere-app bedevere-app Bot mentioned this pull request Oct 25, 2026
codecs.StreamReader.read() silently drops an incomplete character at the end of the stream #160309
Open
@bedevere-app

bedevere-app Bot commented Oct 25, 2026

Copy link
Copy Markdown

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

BHUVANSH855
BHUVANSH855 reviewed Oct 25, 2026
View reviewed changes

@BHUVANSH855 BHUVANSH855 left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CLA check is failing, sign it we can't review without that.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@saiyam-jain-dev
saiyam-jain-dev force-pushed the gh-160309-codecs-streamreader-truncation branch from 29f7951 to db97211 Compare October 25, 2026 20:54
@bedevere-app

bedevere-app Bot commented Oct 25, 2026

Copy link
Copy Markdown

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@saiyam-jain-dev

saiyam-jain-dev commented Oct 25, 2026

Copy link
Copy Markdown
Author

The commit author email has been updated to match the signed CLA, and the CLA check is now passing (green ✅). I have also ensured inal=True semantics are applied during EOF decode and added regression test coverage in est_codecs.py.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@saiyam-jain-dev
pythongh-160309: Fix StreamReader.read() silently dropping incomplete…
e6fb522
… final character

When a byte stream ends with an incomplete multibyte sequence (e.g. a
truncated UTF-8 character), StreamReader.read() would silently discard
those bytes: the read loop exited on empty newdata while bytebuffer still
held the fragment, and bytebuffer was never decoded.

Fix by detecting leftover bytes in bytebuffer when the underlying stream
is exhausted and decoding them immediately.  The codec's stateless decode
function (e.g. utf_8_decode with final=True) then applies the error
handler: 'strict' raises UnicodeDecodeError, 'replace' emits the
replacement character, and 'ignore' drops the bytes - matching the
behaviour of bytes.decode() and io.TextIOWrapper.
@saiyam-jain-dev
saiyam-jain-dev force-pushed the gh-160309-codecs-streamreader-truncation branch from db97211 to e6fb522 Compare October 25, 2026 21:14
@bedevere-app

bedevere-app Bot commented Oct 25, 2026

Copy link
Copy Markdown

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@blurb-it
📜🤖 Added by blurb_it.
51b3fda
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@BHUVANSH855 BHUVANSH855 BHUVANSH855 left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

codecs.StreamReader.read() silently drops an incomplete character at the end of the stream

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@saiyam-jain-dev @BHUVANSH855
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.