GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-35214: Fix OOB memory access in unicode escape parser by gpshead · Pull Request #10506 · python/cpython

Conversation

@gpshead

@gpshead gpshead commented Nov 13, 2018 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Discovered using clang's MemorySanitizer when it ran
test_fstring's test_misformed_unicode_character_name.

An f-string ending in \N would access one byte beyond the end of
the string while looking for a potential }.

https://bugs.python.org/issue35214

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
gpshead added 2 commits November 13, 2018 00:30
@gpshead
bpo-35214: Fix OOB memory access in f-string parser.
782568a 
Discovered using clang's MemorySanitizer when it ran
test_fstring's test_misformed_unicode_character_name.

An f-string ending in `\N` would access one byte beyond the end of
the string while looking for a potential `}`.
@gpshead
Add a news entry.
1af8580
@gpshead gpshead added type-bug An unexpected behavior, bug, or error needs backport to 3.6 labels Nov 13, 2018
@gpshead gpshead self-assigned this Nov 13, 2018
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Nov 13, 2018
@bedevere-bot bedevere-bot added the awaiting merge label Nov 13, 2018
@gpshead

gpshead commented Nov 13, 2018

Copy link
Copy Markdown
Member Author

actually my news entry and description may not be correct. this may not be specific to f-strings but to any unicode string escape parsing?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@gpshead gpshead added the awaiting changes label Nov 13, 2018
@gpshead

gpshead commented Nov 13, 2018

Copy link
Copy Markdown
Member Author

Confirmed, this is in the unicode parser, not f-string related. that just happened to be the unittest that triggered it.

$ ./python -c 'u"\N"'
==17380==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x6a8d8f in _PyUnicode_DecodeUnicodeEscape /home/gps/test/b37/../cpython/Objects/unicodeobject.c:6045:17
    #1 0xbcd1c8 in decode_unicode_with_escapes /home/gps/test/b37/../cpython/Python/ast.c:4208:9
    #2 0xbcaf2c in parsestr /home/gps/test/b37/../cpython/Python/ast.c:5175:23
    #3 0xbc8be3 in parsestrplus /home/gps/test/b37/../cpython/Python/ast.c:5206:13
    #4 0xbc6516 in ast_for_atom /home/gps/test/b37/../cpython/Python/ast.c:2107:23
    #5 0xbc57af in ast_for_atom_expr /home/gps/test/b37/../cpython/Python/ast.c:2464:9
    #6 0xbc1796 in ast_for_power /home/gps/test/b37/../cpython/Python/ast.c:2501:9
    #7 0xbbec1d in ast_for_expr /home/gps/test/b37/../cpython/Python/ast.c:2689:20
    #8 0xbb99a4 in ast_for_testlist /home/gps/test/b37/../cpython/Python/ast.c:2878:16
    #9 0xbd606e in ast_for_expr_stmt /home/gps/test/b37/../cpython/Python/ast.c:2901:21
    #10 0xbb92cf in ast_for_stmt /home/gps/test/b37/../cpython/Python/ast.c:3983:24
    #11 0xbb820a in PyAST_FromNodeObject /home/gps/test/b37/../cpython/Python/ast.c:799:25
    #12 0x88d4cb in PyParser_ASTFromStringObject /home/gps/test/b37/../cpython/Python/pythonrun.c:1184:15
    #13 0x889f61 in PyRun_StringFlags /home/gps/test/b37/../cpython/Python/pythonrun.c:957:11
    #14 0x889c33 in PyRun_SimpleStringFlags /home/gps/test/b37/../cpython/Python/pythonrun.c:455:9
...
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@gpshead gpshead changed the title bpo-35214: Fix OOB memory access in f-string parser bpo-35214: Fix OOB memory access in unicode escape parser Nov 13, 2018
@gpshead
Correct the news entry.
db54123
@gpshead gpshead removed the awaiting changes label Nov 13, 2018
@gpshead

gpshead commented Nov 13, 2018

Copy link
Copy Markdown
Member Author

In terms of severity, this was only ever a single byte read. The code path it entered if that byte happened to be a } did check the bounds and would result in the desired error message (which is why this went unnoticed for so long). With most memory allocations it is unlikely that this byte would be on an unmapped page and crash the program, though I suspect it would be possible to trigger that by using a large enough string to generate such a specific allocation.

At most that is a crash of an interpreter from an errant memory read. Denial of service at best if someone is handing user data to the unicode escape parser.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@gpshead gpshead added the needs backport to 2.7 label Nov 13, 2018
@gpshead
gpshead merged commit 746b2d3 into python:master Nov 13, 2018
@miss-islington

miss-islington commented Nov 13, 2018

Copy link
Copy Markdown
Contributor

Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 2.7, 3.6, 3.7.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the awaiting merge label Nov 13, 2018
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Nov 13, 2018
@gpshead @miss-islington
bpo-35214: Fix OOB memory access in unicode escape parser (pythonGH-1…
266bb03 
…0506)

Discovered using clang's MemorySanitizer when it ran python3's
test_fstring test_misformed_unicode_character_name.

An msan build will fail by simply executing: ./python -c 'u"\N"'
(cherry picked from commit 746b2d3)

Co-authored-by: Gregory P. Smith <[email protected]>
@bedevere-bot bedevere-bot removed the needs backport to 3.7 label Nov 13, 2018
@bedevere-bot

bedevere-bot commented Nov 13, 2018

Copy link
Copy Markdown

GH-10522 is a backport of this pull request to the 3.7 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot

bedevere-bot commented Nov 13, 2018

Copy link
Copy Markdown

GH-10523 is a backport of this pull request to the 3.6 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the needs backport to 3.6 label Nov 13, 2018
@miss-islington

miss-islington commented Nov 13, 2018

Copy link
Copy Markdown
Contributor

Sorry, @gpshead, I could not cleanly backport this to 2.7 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 746b2d35ea47005054ed774fecaed64fab803d7d 2.7
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Nov 13, 2018
@gpshead @miss-islington
bpo-35214: Fix OOB memory access in unicode escape parser (pythonGH-1…
50127c1 
…0506)

Discovered using clang's MemorySanitizer when it ran python3's
test_fstring test_misformed_unicode_character_name.

An msan build will fail by simply executing: ./python -c 'u"\N"'
(cherry picked from commit 746b2d3)

Co-authored-by: Gregory P. Smith <[email protected]>
@gpshead
gpshead deleted the msan_fstring_oob branch November 13, 2018 21:18
miss-islington added a commit that referenced this pull request Nov 13, 2018
@miss-islington @gpshead
bpo-35214: Fix OOB memory access in unicode escape parser (GH-10506)
fdc485a 
Discovered using clang's MemorySanitizer when it ran python3's
test_fstring test_misformed_unicode_character_name.

An msan build will fail by simply executing: ./python -c 'u"\N"'
(cherry picked from commit 746b2d3)

Co-authored-by: Gregory P. Smith <[email protected]>
miss-islington added a commit that referenced this pull request Nov 14, 2018
@miss-islington
[3.7] bpo-35214: Fix OOB memory access in unicode escape parser (GH-1…
9fbcb14 
…0506) (GH-10522)

Discovered using clang's MemorySanitizer when it ran python3's
test_fstring test_misformed_unicode_character_name.

An msan build will fail by simply executing: ./python -c 'u"\N"'
(cherry picked from commit 746b2d3)


Co-authored-by: Gregory P. Smith <[email protected]>


https://bugs.python.org/issue35214
@bedevere-bot

bedevere-bot commented Nov 14, 2018

Copy link
Copy Markdown

GH-10538 is a backport of this pull request to the 2.7 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the needs backport to 2.7 label Nov 14, 2018
gpshead added a commit that referenced this pull request Nov 14, 2018
@gpshead
[2.7] bpo-35214: Fix OOB memory access in unicode escape parser (GH-1…
b6f4472 
…0506) (GH-10538)

Discovered using clang's MemorySanitizer.

A msan build will fail by simply executing: ./python -c 'u"\N"'
(cherry picked from commit 746b2d3)

Co-authored-by: Gregory P. Smith <[email protected]> [Google LLC]
@gpshead gpshead mentioned this pull request Aug 22, 2023
Get the test suite passing with clang Memory Sanitizer enabled #79395
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@gpshead gpshead

Labels

type-bug An unexpected behavior, bug, or error

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@gpshead @miss-islington @bedevere-bot @the-knights-who-say-ni
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.