GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-94673: Always Finalize Static Builtin Types by ericsnowcurrently · Pull Request #95153 · python/cpython

Conversation

@ericsnowcurrently

@ericsnowcurrently ericsnowcurrently commented Jul 22, 2022 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Static builtin types are finalized by calling _PyStaticType_Dealloc(). Currently we skip finalizing such a type if it still has subtypes (i.e. its tp_subclasses hasn't been cleared yet). The problem is that static builtin types hold several heap objects, which leak if we skip the type's finalization. This PR addresses that.

For context, there's an old comment that says the following:

// If a type still has subtypes, it cannot be deallocated.
// A subtype can inherit attributes and methods of its parent type,
// and a type must no longer be used once it's deallocated.

However, it isn't clear that is actually still true. Clearing tp_dict should mean it isn't a problem.

Furthermore, the only subtypes that might still be around come from extension modules that didn't clean them up when unloaded (i.e. extensions that do not implement multi-phase initialization, AKA PEP 489). Those objects are already leaking, so this PR doesn't change anything in that regard. Instead, this change means more objects gets cleaned up that before.

  • Issue: [subinterpreters] Static types incorrectly share some objects between interpreters #94673

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
ericsnowcurrently added 5 commits July 22, 2022 17:06
@ericsnowcurrently
Add a note about skipping on subclasses.
cc0f56c
@ericsnowcurrently
Throw away the leaking subclasses.
34c5d3b
@ericsnowcurrently
Factor out subclass_from_ref().
8d64d72
@ericsnowcurrently
Handle MemoryError in remove_subclass().
fe8cce9
@ericsnowcurrently
Call _PyTypes_FiniTypes() before _PyTypes_Fini().
74d5af3
@ericsnowcurrently ericsnowcurrently added the skip news label Jul 22, 2022
@ericsnowcurrently
ericsnowcurrently requested a review from vstinner July 22, 2022 23:24
@ericsnowcurrently
ericsnowcurrently requested a review from markshannon as a code owner July 22, 2022 23:24
@bedevere-bot bedevere-bot added the awaiting core review label Jul 22, 2022
@ericsnowcurrently
ericsnowcurrently removed the request for review from markshannon July 22, 2022 23:24
@ericsnowcurrently ericsnowcurrently mentioned this pull request Jul 22, 2022
[subinterpreters] Static types incorrectly share some objects between interpreters #94673
Closed
@ericsnowcurrently

ericsnowcurrently commented Jul 22, 2022

Copy link
Copy Markdown
Member Author

@vstinner, you introduced the original skipping logic that I'm basically removing. Are there any problems with the approach I'm taking?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ericsnowcurrently

ericsnowcurrently commented Jul 22, 2022

Copy link
Copy Markdown
Member Author

This was split out of #94995.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ericsnowcurrently ericsnowcurrently added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 22, 2022
@bedevere-bot

bedevere-bot commented Jul 22, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit 74d5af3 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 22, 2022
@kumaraditya303

kumaraditya303 commented Jul 23, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

you introduced the original skipping logic that I'm basically removing. Are there any problems with the approach I'm taking?

Hmm, this would affected extension modules which store PyTypeObject in static global if the interpreter is initialized multiple times and the extension does not initializes it's static type via PyType_Ready when it is (re)imported. Such extensions modules are already unsafe in multiple initialization scenario so there should be no impact.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
kumaraditya303
kumaraditya303 reviewed Jul 23, 2022
View reviewed changes
Comment thread Objects/typeobject.c Resolved

Uh oh!

There was an error while loading. Please reload this page.
kumaraditya303
kumaraditya303 approved these changes Jul 23, 2022
View reviewed changes

@kumaraditya303 kumaraditya303 left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@ericsnowcurrently @kumaraditya303
Update Objects/typeobject.c
d91a0d6 
Co-authored-by: Kumar Aditya <[email protected]>
@ericsnowcurrently
ericsnowcurrently merged commit 2d26449 into python:main Jul 25, 2022
@bedevere-bot bedevere-bot removed the awaiting core review label Jul 25, 2022
@ericsnowcurrently
ericsnowcurrently deleted the always-finalize-static-builtin-standalone branch July 25, 2022 20:23
@vstinner

vstinner commented Aug 3, 2022

Copy link
Copy Markdown
Member

However, it isn't clear that is actually still true. Clearing tp_dict should mean it isn't a problem.

I skipped type deallocation if they have sub-types to prevent crashes. I don't recall the details.

I got issues with subclasses: https://bugs.python.org/issue46417#msg411122 I fixed this issue with: 8ee07dd

It's a complicated issue: https://bugs.python.org/issue46417#msg411304

See also #84258
ericsnowcurrently reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner

vstinner commented Aug 3, 2022

Copy link
Copy Markdown
Member

Are there any problems with the approach I'm taking?

If Python is initialized multiple times (Py_Initialize + Py_Finalize called multiple times), a sub-type may try to get data from its parent static type which has been cleared. If the static type is not properly initialized, Python can crash. If the heap type is used during the finalization after the static type has been cleared, Python can crash.

Again, I don't recall the details. I don't know the crashes that I got.
ericsnowcurrently reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kumaraditya303 kumaraditya303 kumaraditya303 approved these changes
@vstinner vstinner Awaiting requested review from vstinner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@ericsnowcurrently @bedevere-bot @kumaraditya303 @vstinner
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.