GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-44184: Fix subtype_dealloc() for freed type by vstinner · Pull Request #26274 · python/cpython

Conversation

@vstinner

@vstinner vstinner commented May 20, 2021 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Fix a crash at Python exit when a deallocator function removes the
last strong reference to a heap type.

Don't read type memory after calling basedealloc() since
basedealloc() can deallocate the type and free its memory.

https://bugs.python.org/issue44184

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@vstinner
vstinner requested a review from markshannon as a code owner May 20, 2021 23:46
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label May 20, 2021
@bedevere-bot bedevere-bot added the awaiting core review label May 20, 2021
@vstinner

vstinner commented May 20, 2021

Copy link
Copy Markdown
Member Author

cc @pablogsal @erlend-aasland

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland
erlend-aasland approved these changes May 20, 2021
View reviewed changes
pablogsal
pablogsal previously requested changes May 21, 2021
View reviewed changes

@pablogsal pablogsal left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fix makes no sense to me, if what you say in the comment was actually the root cause, we would be seeing this under vallgrind or the address sanitizer and I could not reproduce it

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot

bedevere-bot commented May 21, 2021

Copy link
Copy Markdown

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot added awaiting changes and removed awaiting core review labels May 21, 2021
@pablogsal
pablogsal dismissed their stale review May 21, 2021 00:16

I was able to reproduce the crash under ASAN: https://bugs.python.org/msg394085

@pablogsal

pablogsal commented May 21, 2021 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This is enough to crash it:

import ast
import builtins
import sys
import os

tree = ast.parse("pass")
x = [tree]
x.append(x)

# Put the cycle somewhere to survive until the *last* GC collection
def callback(*args):
    pass
callback.a = x
os.register_at_fork(after_in_child=callback)

can we maybe add a test based on this?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented May 21, 2021

Copy link
Copy Markdown
Member Author

I would prefer to not rely directly on os.register_at_fork() to ensure that the object will survive until the last GC collection. I would prefer an abstraction function in the test.support module. I will try to add a test.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@erlend-aasland erlend-aasland mentioned this pull request May 21, 2021
bpo-42862: Use functools.lru_cache iso. _sqlite.Cache in sqlite3 module #24203
Merged
@erlend-aasland

erlend-aasland commented May 21, 2021

Copy link
Copy Markdown
Contributor

I can reproduce the bug on macOS using the address sanitiser. The patch also fixes the buildbot problems on #24203 (comment)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner added 3 commits May 21, 2021 18:03
@vstinner
bpo-44184: Fix subtype_dealloc() for freed type
5bcd788
Fix a crash at Python exit when a deallocator function removes the
last strong reference to a heap type.

Don't read type memory after calling basedealloc() since
basedealloc() can deallocate the type and free its memory.
@vstinner
Add an unit test
454e996
@vstinner
_PyMem_IsPtrFreed() argument is now const
b670383
@vstinner

vstinner commented May 21, 2021

Copy link
Copy Markdown
Member Author

@pablogsal: I added an unit test, would you mind to review the updated PR?

I was too lazy to add an abstraction. Since the test is in test_gc, IMO it's ok. First, I wanted to add the test to test_ast.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
Port the unit test to Windows
efd45ad
@vstinner

vstinner commented May 21, 2021

Copy link
Copy Markdown
Member Author

Ah, register_at_fork() is not available on Windows. I modified my test to use codecs.register() instead.

You can test manually that the unit test fails as expected with this change:

diff --git a/Objects/typeobject.c b/Objects/typeobject.c
index af99ab79d1..acbedc64e6 100644
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -5,6 +5,7 @@
 #include "pycore_compile.h"       // _Py_Mangle()
 #include "pycore_initconfig.h"
 #include "pycore_moduleobject.h"  // _PyModule_GetDef()
+#include "pycore_pymem.h"  // _PyModule_GetDef()
 #include "pycore_object.h"
 #include "pycore_pyerrors.h"
 #include "pycore_pystate.h"       // _PyThreadState_GET()
@@ -1459,6 +1460,7 @@ subtype_dealloc(PyObject *self)
        our type from a HEAPTYPE to a non-HEAPTYPE, so be careful about
        reference counting. Only decref if the base type is not already a heap
        allocated type. Otherwise, basedealloc should have decref'd it already */
+    assert(!_PyMem_IsPtrFreed(type->tp_name));
     if (type_needs_decref) {
         Py_DECREF(type);
     }
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

erlend-aasland
erlend-aasland approved these changes May 21, 2021
View reviewed changes

@erlend-aasland erlend-aasland left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting core review and removed awaiting changes labels May 21, 2021
@vstinner
vstinner merged commit 615069e into python:main May 21, 2021
@bedevere-bot bedevere-bot removed the awaiting core review label May 21, 2021
@vstinner
vstinner deleted the subtype_dealloc branch May 21, 2021 17:19
@vstinner vstinner added the needs backport to 3.10 label May 21, 2021
@miss-islington

miss-islington commented May 21, 2021

Copy link
Copy Markdown
Contributor

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the needs backport to 3.10 label May 21, 2021
@bedevere-bot

bedevere-bot commented May 21, 2021

Copy link
Copy Markdown

GH-26290 is a backport of this pull request to the 3.10 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 21, 2021
@vstinner @miss-islington
bpo-44184: Fix subtype_dealloc() for freed type (pythonGH-26274)
4a549e4
Fix a crash at Python exit when a deallocator function removes the
last strong reference to a heap type.

Don't read type memory after calling basedealloc() since
basedealloc() can deallocate the type and free its memory.

_PyMem_IsPtrFreed() argument is now constant.
(cherry picked from commit 615069e)

Co-authored-by: Victor Stinner <[email protected]>
@vstinner

vstinner commented May 21, 2021

Copy link
Copy Markdown
Member Author

Thanks for reviews @pablogsal and @erlend-aasland!

erlend-aasland reacted with hooray emoji
All reactions
  • 🎉 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner added a commit that referenced this pull request May 21, 2021
@miss-islington @vstinner
bpo-44184: Fix subtype_dealloc() for freed type (GH-26274) (GH-26290)
50b0d14
Fix a crash at Python exit when a deallocator function removes the
last strong reference to a heap type.

Don't read type memory after calling basedealloc() since
basedealloc() can deallocate the type and free its memory.

_PyMem_IsPtrFreed() argument is now constant.
(cherry picked from commit 615069e)

Co-authored-by: Victor Stinner <[email protected]>

Co-authored-by: Victor Stinner <[email protected]>
@erlend-aasland erlend-aasland mentioned this pull request May 21, 2021
bpo-44032: Defer clearing last thread state until last GC has been run. #26285
Merged
Yhg1s added a commit to Yhg1s/cpython that referenced this pull request Jul 15, 2021
@Yhg1s
Apply pythonGH-26274 to the non-GC-type branch of subtype_dealloc, as…
f098930
… it is also

trying to use a type after it's potentially been freed.
@Yhg1s Yhg1s mentioned this pull request Jul 15, 2021
bpo-44184: Apply GH-26274 to the non-GC-type branch of subtype_dealloc. #27165
Merged
Yhg1s added a commit that referenced this pull request Jul 15, 2021
@Yhg1s
bpo-44184: Apply GH-26274 to the non-GC-type branch of subtype_dealloc (
074e765
GH-27165)

The non-GC-type branch of subtype_dealloc is using the type of an object after freeing in the same unsafe way as GH-26274 fixes. (I believe the old news entry covers this change well enough.)

https://bugs.python.org/issue44184
@miss-islington miss-islington mentioned this pull request Jul 15, 2021
[3.10] bpo-44184: Apply GH-26274 to the non-GC-type branch of subtype_dealloc (GH-27165) #27174
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 15, 2021
@Yhg1s @miss-islington
bpo-44184: Apply pythonGH-26274 to the non-GC-type branch of subtype_…
f2302d8
…dealloc (pythonGH-27165)

The non-GC-type branch of subtype_dealloc is using the type of an object after freeing in the same unsafe way as pythonGH-26274 fixes. (I believe the old news entry covers this change well enough.)

https://bugs.python.org/issue44184
(cherry picked from commit 074e765)

Co-authored-by: T. Wouters <[email protected]>
@miss-islington miss-islington mentioned this pull request Jul 15, 2021
[3.9] bpo-44184: Apply GH-26274 to the non-GC-type branch of subtype_dealloc (GH-27165) #27175
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 15, 2021
@Yhg1s @miss-islington
bpo-44184: Apply pythonGH-26274 to the non-GC-type branch of subtype_…
48ad320
…dealloc (pythonGH-27165)

The non-GC-type branch of subtype_dealloc is using the type of an object after freeing in the same unsafe way as pythonGH-26274 fixes. (I believe the old news entry covers this change well enough.)

https://bugs.python.org/issue44184
(cherry picked from commit 074e765)

Co-authored-by: T. Wouters <[email protected]>
@Yhg1s Yhg1s added the needs backport to 3.9 label Jul 15, 2021
@miss-islington

miss-islington commented Jul 15, 2021

Copy link
Copy Markdown
Contributor

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 15, 2021
@vstinner @miss-islington
bpo-44184: Fix subtype_dealloc() for freed type (pythonGH-26274)
04ff8e8
Fix a crash at Python exit when a deallocator function removes the
last strong reference to a heap type.

Don't read type memory after calling basedealloc() since
basedealloc() can deallocate the type and free its memory.

_PyMem_IsPtrFreed() argument is now constant.
(cherry picked from commit 615069e)

Co-authored-by: Victor Stinner <[email protected]>
@bedevere-bot bedevere-bot removed the needs backport to 3.9 label Jul 15, 2021
@bedevere-bot

bedevere-bot commented Jul 15, 2021

Copy link
Copy Markdown

GH-27176 is a backport of this pull request to the 3.9 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Yhg1s added a commit that referenced this pull request Jul 15, 2021
@miss-islington @Yhg1s
bpo-44184: Apply GH-26274 to the non-GC-type branch of subtype_dealloc (
6aa59c6
GH-27165) (GH-27174)

The non-GC-type branch of subtype_dealloc is using the type of an object after freeing in the same unsafe way as GH-26274 fixes. (I believe the old news entry covers this change well enough.)

https://bugs.python.org/issue44184
(cherry picked from commit 074e765)

Co-authored-by: T. Wouters <[email protected]>
Yhg1s added a commit that referenced this pull request Jul 15, 2021
@miss-islington @Yhg1s
bpo-44184: Apply GH-26274 to the non-GC-type branch of subtype_dealloc (
0b47049
GH-27165) (GH-27175)

The non-GC-type branch of subtype_dealloc is using the type of an object after freeing in the same unsafe way as GH-26274 fixes. (I believe the old news entry covers this change well enough.)

https://bugs.python.org/issue44184
(cherry picked from commit 074e765)

Co-authored-by: T. Wouters <[email protected]>
miss-islington added a commit that referenced this pull request Jul 15, 2021
@miss-islington @vstinner
bpo-44184: Fix subtype_dealloc() for freed type (GH-26274)
298ee65
Fix a crash at Python exit when a deallocator function removes the
last strong reference to a heap type.

Don't read type memory after calling basedealloc() since
basedealloc() can deallocate the type and free its memory.

_PyMem_IsPtrFreed() argument is now constant.
(cherry picked from commit 615069e)

Co-authored-by: Victor Stinner <[email protected]>
@pablogsal pablogsal mentioned this pull request Jul 18, 2021
[3.10] Correct the order of regen-abidump #27228
Closed
@encukou

encukou commented Oct 21, 2021 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

For posterity: bpo-42961 reports the same issue and has a nice reproducer.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@erlend-aasland erlend-aasland erlend-aasland approved these changes
@pablogsal pablogsal pablogsal left review comments
@markshannon markshannon Awaiting requested review from markshannon markshannon is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

8 participants

@vstinner @bedevere-bot @pablogsal @erlend-aasland @miss-islington @encukou @Yhg1s @the-knights-who-say-ni
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.