Description

JohenanLi

opened on Apr 25, 2023

Issue body actions

Your environment

• CPython versions tested on: 3.12.0 alpha 7
• Operating system and architecture: ubuntu20.04.1,x86_64
• Compiler flags: clang with ASAN and UBSAN instrument

Bug description

The AddressSanitizer (ASAN) tool has detected multiple heap-use-after-free errors and a segmentation fault (SEGV) in the Python interpreter. The heap-use-after-free errors occurred in the ascii_decode and unicode_decode_utf8 functions in the unicodeobject.c file, and the SEGV occurred in the tok_backup function in the tokenizer.c file. Additionally, a memory leak was detected in the pystate.c file.

Steps to reproduce

1. Compile Python with ASAN enabled: ./configure && make
2. Run Python with ASAN enabled: ./python < poc_file
3. The heap-use-after-free errors and SEGV should be detected and logged by ASAN.

Expected behavior

No heap-use-after-free errors or SEGV should occur.

Actual behavior

ASAN detected multiple heap-use-after-free errors and a SEGV, as well as a memory leak.

Relevant logs and/or screenshots

The ASAN summary output is as follows:

AddressSanitizer: heap-use-after-free /src/cpython/Objects/unicodeobject.c:4474:28 in ascii_decode
AddressSanitizer: heap-use-after-free /src/cpython/Objects/unicodeobject.c:4506:28 in ascii_decode
AddressSanitizer: heap-use-after-free /src/cpython/Objects/unicodeobject.c:4483:32 in ascii_decode
AddressSanitizer: SEGV /src/cpython/Parser/tokenizer.c:1234:33 in tok_backup
AddressSanitizer: heap-use-after-free /src/cpython/Objects/unicodeobject.c:4526:37 in unicode_decode_utf8
AddressSanitizer: 3824 byte(s) leaked in 4 allocation(s).
AddressSanitizer: heap-use-after-free /src/cpython/Python/pystate.c:229:23 in bind_tstate
The full ASAN log can be found in the asan.log file.

asan.log
python_bug_poc.zip

Linked PRs

• gh-103824: fix use-after-free error in Parser/tokenizer.c #103993

Activity

1. JohenanLi

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Apr 25, 2023
2. sunmy2019 commented on Apr 25, 2023

   

   sunmy2019

   on Apr 25, 2023

   Member

   More actions

   CC: @lysnikolaou
3. arhadthedev

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   and removed

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Apr 25, 2023
4. sobolevn commented on Apr 25, 2023

   

   sobolevn

   on Apr 25, 2023

   Member

   More actions

   Sorry, I might be missing something, but your archive does not have poc_file.
   

   There are several bugN files. Do you mean to actually run them?
5. JohenanLi commented on Apr 25, 2023

   

   JohenanLi

   on Apr 25, 2023

   Author

   More actions

   Sorry, I might be missing something, but your archive does not have poc_file.

   There are several bugN files. Do you mean to actually run them?

   yes，run them. python < bugN.
6. chgnrdv commented on Apr 29, 2023

   

   chgnrdv

   on Apr 29, 2023

   Contributor

   More actions

   I can reproduce use-after-free errors detected in unicodeobject.c with bug_2, bug_6, bug_7 and bug_9 files. I don't get a segfault in tok_backup func with bug_7 on my machine, bug I get the same use-after-free.
   I'll submit a PR with possible fix for these errors, if nobody minds.

   Unfortunately, I don't get use-after-free error with bug_4 on my machine. I can't reproduce use-after-free in bind_tstate with bug_15 as well.
7. JohenanLi commented on May 1, 2023

   

   JohenanLi

   on May 1, 2023

   Author

   More actions

   I can reproduce use-after-free errors detected in unicodeobject.c with bug_2, bug_6, bug_7 and bug_9 files. I don't get a segfault in tok_backup func with bug_7 on my machine, bug I get the same use-after-free. I'll submit a PR with possible fix for these errors, if nobody minds.

   Unfortunately, I don't get use-after-free error with bug_4 on my machine. I can't reproduce use-after-free in bind_tstate with bug_15 as well.

   Thanks a lot.
8. pablogsal

   added a commit that references this issue on May 1, 2023

   gh-103824: fix use-after-free error in Parser/tokenizer.c (#103993)

   d5a9707
9. carljm

   added a commit that references this issue on May 1, 2023

   Merge branch 'main' into inlinecomp2

   1274e2b
10. lysnikolaou commented on May 3, 2023

    

    lysnikolaou

    on May 3, 2023

    Member

    More actions

    Resolved in #103993.
11. lysnikolaou

    closed this as completedon May 3, 2023
12. ajakk commented on Jun 9, 2023

    

    ajakk

    on Jun 9, 2023

    More actions

    Did this ever affect other release lines?
13. schribl commented on Jul 10, 2023

    

    schribl

    on Jul 10, 2023

    Contributor

    More actions

    Did this ever affect other release lines?

    We would also be interested in the answer to this question, if possible. Thanks a lot!
14. lysnikolaou commented on Jul 13, 2023

    

    lysnikolaou

    on Jul 13, 2023

    Member

    More actions

    I just did another check on 3.11 and everything appears to be okay. As far as I can see, this only ever affected 3.12, since it was introduced in the implementation of PEP 701.

    Reacted by Thomas Bininda, Anant Pathak, Jeffrey R. Van Voorst and Gregory P. Smith

Issue actions