GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-36356: Fix memory leak in _asynciomodule.c during "setup.py build_ext" by btharper · Pull Request #16598 · python/cpython

Conversation

@btharper

@btharper btharper commented Oct 5, 2019 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

The module seems to be imported twice during the build process which causes a leak. Since these objects are intended to be global for the life of the import, retain them during the second import.

The double import seems to be an artifact of the build system, and not indicative of normal use.

Multiple PRs for BPO-36356 were largely grouped under a single NEWS entry, should this continue to be skipped, or should a new note be added for the 3.9 branch?

https://bugs.python.org/issue36356

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@btharper
Fix memory leak during setup.py build_ext
3f1669e
@btharper
btharper requested review from 1st1 and asvetlov as code owners October 5, 2019 17:44
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Oct 5, 2019
@bedevere-bot bedevere-bot added the awaiting review label Oct 5, 2019
@1st1

1st1 commented Oct 5, 2019

Copy link
Copy Markdown
Member

I think you should add a static _asyncio_module_inited and check that once, as there are many other places that could be initialized twice. I also wonder how exactly can it be imported twice...

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@btharper

btharper commented Oct 6, 2019

Copy link
Copy Markdown
Contributor Author

Turns out I was incorrect about the double import. It's importing itself as part of the import process, once as "_asyncio" that then imports itself as "asyncio" gdb trace (with edited paths) below. The spec is passed in from setup.py line 529, but the _module naming convention doesn't seem out of place compared to others (_sha*, _ctypes, _decimal, etc).

#0 module_init () at cpython/Modules/_asynciomodule.c:3251
#1 0x00007ffff2dca8a7 in PyInit__asyncio () at cpython/Modules/_asynciomodule.c:3352
#2 0x000055555593e5ff in _PyImport_LoadDynamicModuleWithSpec (
spec=spec@entry=<ModuleSpec(name='_asyncio', loader=<ExtensionFileLoader(name='_asyncio', path='cpython/build/lib.linux-x86_64-3.9-pydebug/_asyncio.cpython-39d-x86_64-linux-gnu.so') at remote 0x607000ea0320>, origin='cpython/build/lib.linux-x86_64-3.9-pydebug/_asyncio.cpython-39d-x86_64-linux-gnu.so', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached=None) at remote 0x607000ea0390>, fp=fp@entry=0x0) at ./Python/importdl.c:164
[... trimmed for brevity ...]
#214 0x000055555593d20c in PyImport_ImportModule (name=name@entry=0x7ffff2dcbe80 "asyncio") at Python/import.c:1482
#215 0x00007ffff2dc3c6f in module_init () at cpython/Modules/_asynciomodule.c:3246
#216 0x00007ffff2dca8a7 in PyInit__asyncio () at cpython/Modules/_asynciomodule.c:3352
#217 0x000055555593e5ff in _PyImport_LoadDynamicModuleWithSpec (
spec=spec@entry=<ModuleSpec(name='_asyncio', loader=<ExtensionFileLoader(name='_asyncio', path='build/lib.linux-x86_64-3.9-pydebug/_asyncio.cpython-39d-x86_64-linux-gnu.so') at remote 0x607000cfc9b0>, origin='build/lib.linux-x86_64-3.9-pydebug/_asyncio.cpython-39d-x86_64-linux-gnu.so', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached=None) at remote 0x607000cfca20>, fp=fp@entry=0x0) at ./Python/importdl.c:164

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@btharper
Ensure asyncio initialization only happens once
4a342f9
@btharper

btharper commented Oct 6, 2019

Copy link
Copy Markdown
Contributor Author

Updated with a static variable to check initialization. left initialization for the inner call and return early for the outside call, although I don't think the difference would be significant.

If everything looks good I can squash the previous commit since the first one is entirely reverted by the second.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

1st1
1st1 reviewed Oct 7, 2019
View reviewed changes
Comment thread Modules/_asynciomodule.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@1st1

1st1 commented Oct 7, 2019

Copy link
Copy Markdown
Member

LGTM, but please fix the nit I highlighted.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@btharper Yury Selivanov
Fix brackets on else clause
604b615
Co-Authored-By: Yury Selivanov <[email protected]>
@btharper

btharper commented Oct 7, 2019

Copy link
Copy Markdown
Contributor Author

Good catch, I've fixed the one introduced by this changeset, but grep shows 6 more instances. Would it be worth fixing them along with these changes, or leaving it as it is?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Oct 7, 2019

Copy link
Copy Markdown
Member

Good catch, I've fixed the one introduced by this changeset, but grep shows 6 more instances. Would it be worth fixing them along with these changes, or leaving it as it is?

@btharper: Which instances? In which file?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@btharper

btharper commented Oct 7, 2019 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor Author

Good catch, I've fixed the one introduced by this changeset, but grep shows 6 more instances. Would it be worth fixing them along with these changes, or leaving it as it is?

@btharper: Which instances? In which file?

Sorry for the lack of specificity. 6 more instances of the closing bracket being on the same line as an else/else if in the Modules/_asynciomodule.c file. I wasn't sure if it would be useful to clean up the rest of the file or just correct what was modified in this PR, though I expect not changing unrelated lines is the safest answer. Anything else for bpo-36356 would be under another PR.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@1st1

1st1 commented Oct 7, 2019 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Sorry for the lack of specificity. 6 more instances of the closing bracket being on the same line as an else/else if in the Modules/_asynciomodule.c file. I wasn't sure if it would be useful to clean up the rest of the file or just correct what was modified in this PR, though I expect not changing unrelated lines is the safest answer. Anything else for bpo-36356 would be under another PR.

We can fix them in a separate PR.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

1st1
1st1 approved these changes Oct 7, 2019
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting review labels Oct 7, 2019
@1st1

1st1 commented Oct 7, 2019

Copy link
Copy Markdown
Member

Could you please generate a NEWS entry with blurb?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner
vstinner approved these changes Oct 7, 2019
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

I'm fine with merging it with new NEWS entry. It's uncommon to load a C extension twice.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@1st1 1st1 added needs backport to 3.8 skip news labels Oct 7, 2019
@1st1
1st1 merged commit 321def8 into python:master Oct 7, 2019
@miss-islington

miss-islington commented Oct 7, 2019

Copy link
Copy Markdown
Contributor

Thanks @btharper for the PR, and @1st1 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the awaiting merge label Oct 7, 2019
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Oct 7, 2019
@btharper @miss-islington
bpo-36356: Fix memory leak in _asynciomodule.c (pythonGH-16598)
855c4f9
(cherry picked from commit 321def8)

Co-authored-by: Ben Harper <[email protected]>
@1st1

1st1 commented Oct 7, 2019

Copy link
Copy Markdown
Member

I'm fine with merging it with new NEWS entry. It's uncommon to load a C extension twice.

Alright!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the needs backport to 3.8 label Oct 7, 2019
@bedevere-bot

bedevere-bot commented Oct 7, 2019

Copy link
Copy Markdown

GH-16622 is a backport of this pull request to the 3.8 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

miss-islington added a commit that referenced this pull request Oct 7, 2019
@miss-islington @btharper
bpo-36356: Fix memory leak in _asynciomodule.c (GH-16598)
13915a3
(cherry picked from commit 321def8)

Co-authored-by: Ben Harper <[email protected]>
jacobneiltaylor pushed a commit to jacobneiltaylor/cpython that referenced this pull request Dec 5, 2019
@btharper Jake Taylor
bpo-36356: Fix memory leak in _asynciomodule.c (pythonGH-16598)
dda68d7
@jquesnelle jquesnelle mentioned this pull request Apr 15, 2020
bpo-40294: Fix use-after-free in _asynciomodule.c when module is loaded/unloaded multiple times #19542
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner approved these changes
@1st1 1st1 1st1 approved these changes
@asvetlov asvetlov Awaiting requested review from asvetlov

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@btharper @1st1 @vstinner @miss-islington @bedevere-bot @the-knights-who-say-ni
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.