GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — gh-156939: Detect buffer overflow in PyBytesWriter in debug mode by vstinner · Pull Request #156943 · python/cpython

Conversation

vstinner commented Sep 4, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Member

Reserve one byte in PyBytesWriter used as a canary byte: set it to a special value. PyBytesWriter_Finish() checks if the canary byte has been overriden. Add a test on the feature.

Update buffer overflow check in fcntl: allocate extra guard bytes in the writer and then truncate these bytes.

Issue: C API: Detect buffer overflow in PyBytesWriter #156939

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

bedevere-app Bot mentioned this pull request

Sep 4, 2026

C API: Detect buffer overflow in PyBytesWriter #156939

Closed

vstinner commented Sep 7, 2026

Copy link

Copy Markdown

Member Author

Oh, the test_xml_etree_c failure is a legit failure: unicode_encode_ucs1() calls with errors="xmlcharrefreplace" writes a NUL byte after the writer buffer. xmlcharrefreplace() calls sprintf(str, "&#%d;", PyUnicode_READ(kind, data, i)) which writes a final NUL byte.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner commented Sep 7, 2026

Copy link

Copy Markdown

Member Author

The test_struct error is also a legit bug: a Pascal string of 0 characters writes a NUL byte into an empty buffer, and so triggers a buffer overflow.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner commented Sep 7, 2026

Copy link

Copy Markdown

Member Author

On Ubuntu, 8 tests are failing:

8 tests failed:
    test_codeccallbacks test_fcntl test_minidom test_sax test_struct
    test_urllib test_xml_etree test_xml_etree_c

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner mentioned this pull request

Sep 7, 2026

gh-156939: Fix memory corruption check in fcntl #157103

Closed


          pythongh-156939: Detect buffer overflow in PyBytesWriter in debug mode

89a8ac9

Reserve one byte in PyBytesWriter used as a canary byte: set it to a
special value. PyBytesWriter_Finish() checks if the canary byte has
been overriden. Add a test on the feature.

Update buffer overflow check in fcntl: allocate extra guard bytes in
the writer and then truncate these bytes.

vstinner force-pushed the writer_canary branch from 42640e3 to 89a8ac9 Compare

September 9, 2026 19:38

vstinner mentioned this pull request

Sep 9, 2026

gh-156939: Fix xmlcharrefreplace() buffer overflow #157109

Merged


          Skip test_canary_byte on release build

95e3701

vstinner commented Sep 9, 2026

Copy link

Copy Markdown

Member Author

I fixed struct.pack() in PR gh-157071 and xmlcharrefreplace in PR gh-157109.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.


          Cleanup test_canary_byte()

0dbd2cc

Avoid calling PyBytesWriter_Grow().

vstinner added the topic-C-API label

Sep 9, 2026

vstinner marked this pull request as ready for review

September 9, 2026 20:42

bedevere-app Bot added the awaiting core review label

Sep 9, 2026

vstinner commented Sep 9, 2026

Copy link

Copy Markdown

Member Author

cc @serhiy-storchaka

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.


          Merge branch 'main' into writer_canary

1f1d878

vstinner mentioned this pull request

Sep 11, 2026

gh-157242: Fix PyBytesWriter_Resize() on MemoryError #157243

Merged

vstinner added 3 commits

September 12, 2026 19:21


          Merge branch 'main' into writer_canary

c81cbd7


          Rewrite canary byte test in Python

7994e4e

* Add test_get_data_canary()
* Update test_get_data()


          Change fatal error message

19fdcb4

vstinner enabled auto-merge (squash)

September 12, 2026 17:46

vstinner added 5 commits

September 12, 2026 22:16


          Merge branch 'main' into writer_canary


          pythongh-156939: Do no modify writer->size directly in UCS1 encoder

dc79819

unicode_encode_ucs1() now calls PyBytesWriter_Grow() to update the
PyBytesWriter size.


          Update get_data() tests for canary byte

e14b81c


          Skip canary byte tests on release build

e029931


          Add test_grow_error()

276c315

PyBytesWriter_Grow() no longer calls byteswriter_resize() if grow is
smaller than 0.

vstinner merged commit 12a1de1 into python:main

Sep 13, 2026

53 of 54 checks passed

Uh oh!

There was an error while loading. Please reload this page.

vstinner deleted the writer_canary branch

September 13, 2026 00:28

bedevere-app Bot removed the awaiting core review label

Sep 13, 2026

vstinner commented Sep 13, 2026

Copy link

Copy Markdown

Member Author

Android x86_64 fails with:

  ERROR: test_str (test.test_android.TestAndroidOutput.test_str) [stdout]
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 171, in test_str
      write("\ud800\udc00", [r"\ud800\udc00"])
      ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 148, in write
      self.assertEqual(write_len, stream.write(s))
                                  ~~~~~~~~~~~~^^^
    File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 76, in write
      self._write_chunk(chunk)
      ~~~~~~~~~~~~~~~~~^^^^^^^
    File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 83, in _write_chunk
      b = s.encode(self.encoding, self.errors)
  ValueError: size larger than allocated size

This error comes from PyBytesWriter_FinishWithSize(): the requested size is too big. I can reproduce it with just:

>>> "\ud800\udc00".encode('utf8', 'backslashreplace')
Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    "\ud800\udc00".encode('utf8', 'backslashreplace')
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: size larger than allocated size

Ah! I forgot to update the UTF-8 encoder.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner commented Sep 13, 2026

Copy link

Copy Markdown

Member Author

Ah! I forgot to update the UTF-8 encoder.

I wrote PR gh-157383 for that.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bedevere-bot commented Sep 13, 2026

Copy link

Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Android 3.x (tier-3) has failed when building commit 12a1de1.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1594/builds/5913) and take a look at the build logs.
Check if the failure is related to this commit (12a1de1) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1594/builds/5913

Failed tests:

test_android

Failed subtests:

test_str - test.test_android.TestAndroidOutput.test_str

Summary of the results of the build (if available):

Click to see traceback logs

Traceback (most recent call last):
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 171, in test_str
    write("\ud800\udc00", [r"\ud800\udc00"])
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 148, in write
    self.assertEqual(write_len, stream.write(s))
                                ~~~~~~~~~~~~^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 76, in write
    self._write_chunk(chunk)
    ~~~~~~~~~~~~~~~~~^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 83, in _write_chunk
    b = s.encode(self.encoding, self.errors)
ValueError: size larger than allocated size
 
======================================================================
ERROR: test_str (test.test_android.TestAndroidOutput.test_str) [stderr]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 171, in test_str
    write("\ud800\udc00", [r"\ud800\udc00"])
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 148, in write
    self.assertEqual(write_len, stream.write(s))
                                ~~~~~~~~~~~~^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 76, in write
    self._write_chunk(chunk)
    ~~~~~~~~~~~~~~~~~^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 83, in _write_chunk
    b = s.encode(self.encoding, self.errors)
ValueError: size larger than allocated size
 
----------------------------------------------------------------------
Ran 3 tests in 0.081s
 
FAILED (errors=2)


Traceback (most recent call last):
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 171, in test_str
    write("\ud800\udc00", [r"\ud800\udc00"])
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 148, in write
    self.assertEqual(write_len, stream.write(s))
                                ~~~~~~~~~~~~^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 76, in write
    self._write_chunk(chunk)
    ~~~~~~~~~~~~~~~~~^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 83, in _write_chunk
    b = s.encode(self.encoding, self.errors)
ValueError: size larger than allocated size
 
======================================================================
ERROR: test_str (test.test_android.TestAndroidOutput.test_str) [stderr]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 171, in test_str
    write("\ud800\udc00", [r"\ud800\udc00"])
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 148, in write
    self.assertEqual(write_len, stream.write(s))
                                ~~~~~~~~~~~~^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 76, in write
    self._write_chunk(chunk)
    ~~~~~~~~~~~~~~~~~^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 83, in _write_chunk
    b = s.encode(self.encoding, self.errors)
ValueError: size larger than allocated size
 
----------------------------------------------------------------------
Ran 3 tests in 0.134s
 
FAILED (errors=2)
bytearraymemoryviewmmrve\xc0\x80\xc0\x80�\xc0\x80\xfe\xff\xff\xffaHelloHello world   olé中文😀test test_android failed

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bedevere-bot commented Sep 13, 2026

Copy link

Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Android 3.x (tier-3) has failed when building commit 12a1de1.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1591/builds/5239) and take a look at the build logs.
Check if the failure is related to this commit (12a1de1) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1591/builds/5239

Failed tests:

test_android

Failed subtests:

test_str - test.test_android.TestAndroidOutput.test_str

Summary of the results of the build (if available):

Click to see traceback logs

Traceback (most recent call last):
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 171, in test_str
    write("\ud800\udc00", [r"\ud800\udc00"])
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 148, in write
    self.assertEqual(write_len, stream.write(s))
                                ~~~~~~~~~~~~^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 76, in write
    self._write_chunk(chunk)
    ~~~~~~~~~~~~~~~~~^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 83, in _write_chunk
    b = s.encode(self.encoding, self.errors)
ValueError: size larger than allocated size
 
======================================================================
ERROR: test_str (test.test_android.TestAndroidOutput.test_str) [stderr]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 171, in test_str
    write("\ud800\udc00", [r"\ud800\udc00"])
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 148, in write
    self.assertEqual(write_len, stream.write(s))
                                ~~~~~~~~~~~~^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 76, in write
    self._write_chunk(chunk)
    ~~~~~~~~~~~~~~~~~^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 83, in _write_chunk
    b = s.encode(self.encoding, self.errors)
ValueError: size larger than allocated size
 
----------------------------------------------------------------------
Ran 3 tests in 0.595s
 
FAILED (errors=2)
bytearraymemoryviewmmrve\xc0\x80\xc0\x80�\xc0\x80\xfe\xff\xff\xffaHelloHello world   olé中文😀test test_android failed


Traceback (most recent call last):
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 171, in test_str
    write("\ud800\udc00", [r"\ud800\udc00"])
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 148, in write
    self.assertEqual(write_len, stream.write(s))
                                ~~~~~~~~~~~~^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 76, in write
    self._write_chunk(chunk)
    ~~~~~~~~~~~~~~~~~^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 83, in _write_chunk
    b = s.encode(self.encoding, self.errors)
ValueError: size larger than allocated size
 
======================================================================
ERROR: test_str (test.test_android.TestAndroidOutput.test_str) [stderr]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 171, in test_str
    write("\ud800\udc00", [r"\ud800\udc00"])
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/test/test_android.py", line 148, in write
    self.assertEqual(write_len, stream.write(s))
                                ~~~~~~~~~~~~^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 76, in write
    self._write_chunk(chunk)
    ~~~~~~~~~~~~~~~~~^^^^^^^
  File "/data/user/0/org.python.testbed/files/python/lib/python3.16/_android_support.py", line 83, in _write_chunk
    b = s.encode(self.encoding, self.errors)
ValueError: size larger than allocated size
 
----------------------------------------------------------------------
Ran 3 tests in 0.228s
 
FAILED (errors=2)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner mentioned this pull request

Sep 13, 2026

gh-156939: Detect buffer overflow in bytearray #157382

Closed

johnslavik pushed a commit to johnslavik/cpython that referenced this pull request

Sep 14, 2026


          pythongh-156939: Detect buffer overflow in PyBytesWriter in debug mode (

39d7d43

python#156943)

Reserve one byte in PyBytesWriter used as a canary byte: set it to a
special value. PyBytesWriter_Finish() checks if the canary byte has
been overriden to detect a buffer overflow. Add tests for this feature.

Update buffer overflow check in fcntl: allocate extra guard bytes in
the writer and then truncate these bytes.

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

topic-C-API

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

vstinner commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Sep 7, 2026

Uh oh!

vstinner commented Sep 7, 2026

Uh oh!

vstinner commented Sep 7, 2026

Uh oh!

vstinner commented Sep 9, 2026

Uh oh!

vstinner commented Sep 9, 2026

Uh oh!

Uh oh!

vstinner commented Sep 13, 2026

Uh oh!

vstinner commented Sep 13, 2026

Uh oh!

bedevere-bot commented Sep 13, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Sep 13, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vstinner commented Sep 4, 2026 •

edited

Loading

Uh oh!

There was an error while loading. Please reload this page.