Description

AdamKorcz

opened on Feb 16, 2026

Issue body actions

Bug report

Bug description:

OSS-Fuzz has found a heap buffer overflow in _PyTokenizer_ensure_utf8. Link to OSS-Fuzz bug report.

The root cause is that valid_utf8() in Parser/tokenizer/helpers.c checks continuation bytes in reverse order thus reader s[expected] before s[1] on these lines:

cpython/Parser/tokenizer/helpers.c

Lines 497 to 499 in 8b7b5a9

	for (; expected; expected--)
	if (s[expected] < 0x80 || s[expected] >= 0xC0)
	return 0;

When a multi-byte UTF-8 sequence is truncated - such as a 3-byte lead \xEA followed immediately by a null terminator - the backward loop reads past the end of the valid data before encountering the null byte that would stop it.

This is not a security-critical issue.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-144872: fix heap buffer overflow _PyTokenizer_ensure_utf8 #144807
• [3.14] gh-144872: fix heap buffer overflow _PyTokenizer_ensure_utf8 (GH-144807) #145287
• [3.13] gh-144872: fix heap buffer overflow _PyTokenizer_ensure_utf8 (GH-144807) #145441

Activity

1. AdamKorcz

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Feb 16, 2026
2. picnixz

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on Feb 16, 2026
3. pablogsal

   added a commit that references this issue on Feb 26, 2026

   pythongh-144872: move NEWS entry to Core_and_Builtins

   56d99c2
4. StanFromIreland

   added

   topic-parser

   on Feb 26, 2026
5. pablogsal

   added a commit that references this issue on Feb 26, 2026

   gh-144872: fix heap buffer overflow `_PyTokenizer_ensure_utf8` (#144807)

   3fc945d
6. miss-islington

   added a commit that references this issue on Feb 26, 2026

   pythongh-144872: fix heap buffer overflow `_PyTokenizer_ensure_utf8` (p…

   0fa0a35
7. pablogsal

   added a commit that references this issue on Feb 26, 2026

   [3.14] gh-144872: fix heap buffer overflow `_PyTokenizer_ensure_utf8` (…

   0810124
8. brijkapadia

   added a commit that references this issue on Feb 28, 2026

   pythongh-144872: fix heap buffer overflow `_PyTokenizer_ensure_utf8` (p…

   18910cc
9. StanFromIreland

   added a commit that references this issue on Mar 2, 2026

   [3.13] pythongh-144872: fix heap buffer overflow `_PyTokenizer_ensure…

   7665cee
10. pablogsal

    added a commit that references this issue on Mar 2, 2026

    [3.13] gh-144872: fix heap buffer overflow `_PyTokenizer_ensure_utf8` (…

    5577bd1
11. pablogsal commented on Mar 2, 2026

    

    pablogsal

    on Mar 2, 2026

    Member

    More actions

    Thanks a lot for the report @AdamKorcz !
12. pablogsal

    closed this as completedon Mar 2, 2026
13. ljfp

    added a commit that references this issue on Apr 25, 2026

    pythongh-144872: fix heap buffer overflow `_PyTokenizer_ensure_utf8` (p…

    8fc386a

Issue actions