GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-109496: Detect Py_DECREF() after dealloc in debug mode by vstinner · Pull Request #109539 · python/cpython

Conversation

@vstinner

@vstinner vstinner commented Sep 18, 2023 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

On a Python built in debug mode, Py_DECREF() now calls _Py_NegativeRefcount() if the object is a dangling pointer to deallocated memory: memory filled with 0xDD "dead byte" by the debug hook on memory allocators. The fix is to check the reference count before checking for _Py_IsImmortal().

Add test_decref_freed_object() to test_capi.test_misc.

  • Issue: Calling Py_DECREF twice does not result in error with debug build #109496

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
ericsnowcurrently and eduardo-elizondo reacted with thumbs up emoji
All reactions
  • 👍 2 reactions
@vstinner vstinner added the needs backport to 3.12 only security fixes label Sep 18, 2023
@bedevere-app bedevere-app Bot added the awaiting review label Sep 18, 2023
@bedevere-app bedevere-app Bot mentioned this pull request Sep 18, 2023
Calling Py_DECREF twice does not result in error with debug build #109496
Closed
@vstinner

vstinner commented Sep 18, 2023

Copy link
Copy Markdown
Member Author

cc @eduardo-elizondo @ericsnowcurrently @serhiy-storchaka
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner

vstinner commented Sep 18, 2023

Copy link
Copy Markdown
Member Author

Fix regression introduced in Python 3.12 PEP 683 Immortal Objects: https://peps.python.org/pep-0683/
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
serhiy-storchaka
serhiy-storchaka approved these changes Sep 18, 2023
View reviewed changes
@bedevere-app bedevere-app Bot added awaiting core review and removed awaiting review labels Sep 18, 2023
@vstinner
pythongh-109496: Detect Py_DECREF() after dealloc in debug mode
8ea43e9 
On a Python built in debug mode, Py_DECREF() now calls
_Py_NegativeRefcount() if the object is a dangling pointer to
deallocated memory: memory filled with 0xDD "dead byte" by the debug
hook on memory allocators. The fix is to check the reference count
*before* checking for _Py_IsImmortal().

Add test_decref_freed_object() to test_capi.test_misc.
@vstinner
vstinner force-pushed the detect_double_decref branch from e96fc04 to 8ea43e9 Compare September 18, 2023 14:32
@vstinner

vstinner commented Sep 18, 2023

Copy link
Copy Markdown
Member Author

vstinner force-pushed the detect_double_decref branch from e96fc04 to 8ea43e9

Sorry, I forgot a local change to fix a typo in the NEWS entry.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner
vstinner enabled auto-merge (squash) September 18, 2023 14:38
sorcio
sorcio reviewed Sep 18, 2023
View reviewed changes
Comment thread Modules/_testcapimodule.c
Comment on lines +2048 to +2049
Py_DECREF(obj);
// obj is a now a dangling pointer

@sorcio sorcio Sep 18, 2023

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a chance this could be an actual pointer to deallocated memory, or memory written to by another thread? Would it be safer to just call Py_DECREF on a fake object which is explicitly memset to 0xDD bytes?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@vstinner vstinner Sep 18, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to use a functional test: not mock anything here.

This test is very fragile, it makes sure that a dangling pointer has a determinstic behavior. Problem: by definition, it has an undefined behavior :-) But in practice, almost always, the memory should remain around and should just be filled by 0xDD bytes.

If tomorrow the test starts to fail randomly, sure, we can rewrite it by "mocking" the first DECREF() and just pass memory filled by 0xDD byte pattern.

test_capi should not be run in a process with more than 1 thread. libregrtest works hard to make sure that previous tests don't leak tests. Also, it's strongly recommended to run tests with -jN: each test file is run in a fresh process, so with a known number of threads: just 1 :-)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@vstinner
vstinner merged commit 0bb0d88 into python:main Sep 18, 2023
@vstinner
vstinner deleted the detect_double_decref branch September 18, 2023 14:59
@miss-islington

miss-islington commented Sep 18, 2023

Copy link
Copy Markdown
Contributor

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the awaiting core review label Sep 18, 2023
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Sep 18, 2023
@vstinner @miss-islington
pythongh-109496: Detect Py_DECREF() after dealloc in debug mode (pyth…
a356eed 
…onGH-109539)

On a Python built in debug mode, Py_DECREF() now calls
_Py_NegativeRefcount() if the object is a dangling pointer to
deallocated memory: memory filled with 0xDD "dead byte" by the debug
hook on memory allocators. The fix is to check the reference count
*before* checking for _Py_IsImmortal().

Add test_decref_freed_object() to test_capi.test_misc.
(cherry picked from commit 0bb0d88)

Co-authored-by: Victor Stinner <[email protected]>
@bedevere-app

bedevere-app Bot commented Sep 18, 2023

Copy link
Copy Markdown

GH-109545 is a backport of this pull request to the 3.12 branch.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-app bedevere-app Bot removed the needs backport to 3.12 only security fixes label Sep 18, 2023
Yhg1s pushed a commit that referenced this pull request Sep 18, 2023
@miss-islington @vstinner
[3.12] gh-109496: Detect Py_DECREF() after dealloc in debug mode (GH-…
74a7f5d 
…109539) (#109545)

gh-109496: Detect Py_DECREF() after dealloc in debug mode (GH-109539)

On a Python built in debug mode, Py_DECREF() now calls
_Py_NegativeRefcount() if the object is a dangling pointer to
deallocated memory: memory filled with 0xDD "dead byte" by the debug
hook on memory allocators. The fix is to check the reference count
*before* checking for _Py_IsImmortal().

Add test_decref_freed_object() to test_capi.test_misc.
(cherry picked from commit 0bb0d88)

Co-authored-by: Victor Stinner <[email protected]>
@vstinner vstinner mentioned this pull request Sep 19, 2023
gh-109496: Skip test_capi.test_decref_freed_object() on ASAN #109573
Merged
csm10495 pushed a commit to csm10495/cpython that referenced this pull request Sep 28, 2023
@vstinner @csm10495
pythongh-109496: Detect Py_DECREF() after dealloc in debug mode (pyth…
6d4c96a 
…on#109539)

On a Python built in debug mode, Py_DECREF() now calls
_Py_NegativeRefcount() if the object is a dangling pointer to
deallocated memory: memory filled with 0xDD "dead byte" by the debug
hook on memory allocators. The fix is to check the reference count
*before* checking for _Py_IsImmortal().

Add test_decref_freed_object() to test_capi.test_misc.
@pablogsal

pablogsal commented Oct 2, 2023

Copy link
Copy Markdown
Member

@vstinner This PR broke the AMD64 Arch Linux Asan Debug 3.12 builder. I will make a fix
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@pablogsal

pablogsal commented Oct 2, 2023

Copy link
Copy Markdown
Member

Nevermind I can see this was fixed in 0a31ff0050e but the backport didn't land
vstinner reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@serhiy-storchaka serhiy-storchaka serhiy-storchaka approved these changes
+1 more reviewer
@sorcio sorcio sorcio left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@vstinner @miss-islington @pablogsal @sorcio @serhiy-storchaka
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.