GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-104510: Fix refleaks in _io base types by kumaraditya303 · Pull Request #104516 · python/cpython

Conversation

@kumaraditya303

@kumaraditya303 kumaraditya303 commented May 15, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor
  • Issue: Ref leaks introduced by _io isolation (gh-101948) #104510

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@kumaraditya303
visit instance dict
5ee2358
@bedevere-bot bedevere-bot added the awaiting core review label May 15, 2023
@bedevere-bot bedevere-bot mentioned this pull request May 15, 2023
Ref leaks introduced by _io isolation (gh-101948) #104510
Closed
@kumaraditya303 kumaraditya303 added skip news release-blocker 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section labels May 15, 2023
@bedevere-bot

bedevere-bot commented May 15, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @kumaraditya303 for commit 5ee2358 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label May 15, 2023
@kumaraditya303
kumaraditya303 requested a review from erlend-aasland May 15, 2023 23:44
sunmy2019
sunmy2019 approved these changes May 16, 2023
View reviewed changes

@sunmy2019 sunmy2019 left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Eclips4
Eclips4 reviewed May 16, 2023
View reviewed changes
Comment thread Modules/_io/iobase.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Eclips4
Eclips4 approved these changes May 16, 2023
View reviewed changes

@Eclips4 Eclips4 left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All ref leaks are gone!
Thanks for the fix.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@erlend-aasland

erlend-aasland commented May 16, 2023

Copy link
Copy Markdown
Contributor

The reason why this fix works is not obvious. Can you add a clarifying comment, alternatively expand on the C API docs for tp_traverse?

Note to self: we also need to document the GC peculiarities of heap-type-inherits-from-static-type (for example defdict in the _collections extension module).

Eclips4 reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@erlend-aasland erlend-aasland changed the title GH-104510: fix refleaks GH-104510: Fix refleaks in \_io base types May 16, 2023
@erlend-aasland erlend-aasland changed the title GH-104510: Fix refleaks in \_io base types GH-104510: Fix refleaks in _io base types May 16, 2023
@erlend-aasland erlend-aasland linked an issue May 16, 2023 that may be closed by this pull request
Ref leaks introduced by _io isolation (gh-101948) #104510
Closed
kumaraditya303 added 2 commits May 16, 2023 11:34
@kumaraditya303
fix by inheriting vsitor
1e710b4
@kumaraditya303
Add comment
78c12ff
erlend-aasland
erlend-aasland reviewed May 16, 2023
View reviewed changes
Comment thread Modules/_io/bufferedio.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@kumaraditya303 kumaraditya303 added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label May 16, 2023
@bedevere-bot

bedevere-bot commented May 16, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @kumaraditya303 for commit 78c12ff 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label May 16, 2023
erlend-aasland
erlend-aasland approved these changes May 16, 2023
View reviewed changes

@erlend-aasland erlend-aasland left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICS, this is the correct solution.

We still need to write a sentence or two in the docs about our recent GC adventures (_io, _collections).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels May 16, 2023
@sunmy2019

sunmy2019 commented May 16, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

The heap type still has GC but it now inherits traverse and clear slots from its base because Py_TPFLAGS_HAVE_GC isn't set. See https://docs.python.org/3.12/c-api/gcsupport.html

The doc's a bit unclear here.

When calling PyType_Ready() or some of the APIs that indirectly call it like PyType_FromSpecWithBases() or PyType_FromSpec() the interpreter will automatically populate the tp_flags, tp_traverse and tp_clear fields if the type inherits from a class that implements the garbage collector protocol and the child class does not include the Py_TPFLAGS_HAVE_GC flag.

From the code, we see,

cpython/Objects/typeobject.c

Lines 6670 to 6678 in 0bb61dd

if (!(type->tp_flags & Py_TPFLAGS_HAVE_GC) &&
(base->tp_flags & Py_TPFLAGS_HAVE_GC) &&
(!type->tp_traverse && !type->tp_clear)) {
type->tp_flags |= Py_TPFLAGS_HAVE_GC;
if (type->tp_traverse == NULL)
type->tp_traverse = base->tp_traverse;
if (type->tp_clear == NULL)
type->tp_clear = base->tp_clear;
}

To get the tp_traverse and tp_clear inherited, the child type must have them both NULL.

This is against the doc. Also, the comparisons in lines 6674 and 6676 are unnecessary.

It was a 22-yr old code 13d52f0. I guess the author (@gvanrossum) was trying to write

-      (!type->tp_traverse && !type->tp_clear)) { 
+      (!type->tp_traverse || !type->tp_clear)) {  

Don't know what to do. Maybe we should only update the doc.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@kumaraditya303
kumaraditya303 merged commit 442a3e6 into python:main May 16, 2023
@bedevere-bot bedevere-bot removed the awaiting merge label May 16, 2023
@kumaraditya303

kumaraditya303 commented May 16, 2023

Copy link
Copy Markdown
Contributor Author

Don't know what to do. Maybe we should only update the doc.

The best we can do is to fix docs, the behavior here is ancient.

erlend-aasland reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

carljm added a commit to carljm/cpython that referenced this pull request May 16, 2023
@carljm
Merge branch 'main' into supermon
a6dd626
* main:
  pythonGH-104510: Fix refleaks in `_io` base types (python#104516)
  pythongh-104539: Fix indentation error in logging.config.rst (python#104545)
  pythongh-104050: Don't star-import 'types' in Argument Clinic (python#104543)
  pythongh-104050: Add basic typing to CConverter in clinic.py (python#104538)
  pythongh-64595: Fix write file logic in Argument Clinic (python#104507)
  pythongh-104523: Inline minimal PGO rules (python#104524)
  pythongh-103861: Fix Zip64 extensions not being properly applied in some cases (python#103863)
  pythongh-69152: add method get_proxy_response_headers to HTTPConnection class (python#104248)
  pythongh-103763: Implement PEP 695 (python#103764)
  pythongh-104461: Run tkinter test_configure_screen on X11 only (pythonGH-104462)
  pythongh-104469: Convert _testcapi/watchers.c to use Argument Clinic (python#104503)
  pythongh-104482: Fix error handling bugs in ast.c (python#104483)
  pythongh-104341: Adjust tstate_must_exit() to Respect Interpreter Finalization (pythongh-104437)
  pythonGH-102613: Fix recursion error from `pathlib.Path.glob()` (pythonGH-104373)
@vstinner

vstinner commented May 17, 2023

Copy link
Copy Markdown
Member

Next time, it would be nice to elaborate the rationale for the fix in the commit, rather than just saying "fix" without any commit message ;-) It's really hard to handle the GC correctly with heap types. Better documentation is required.

The _io module is one of the few C extensions which use class inheritance with heap types implemented in C.

erlend-aasland reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented May 17, 2023

Copy link
Copy Markdown
Member

cc @pablogsal: GC issues are hard :-(

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

carljm added a commit to carljm/cpython that referenced this pull request May 17, 2023
@carljm
Merge branch 'main' into 709funconly
b1bb3a7
* main: (26 commits)
  pythonGH-101520: Move tracemalloc functionality into core, leaving interface in Modules. (python#104508)
  typing: Add more tests for TypeVar (python#104571)
  pythongh-104572: Improve error messages for invalid constructs in PEP 695 contexts (python#104573)
  typing: Use PEP 695 syntax in typing.py (python#104553)
  pythongh-102153: Start stripping C0 control and space chars in `urlsplit` (python#102508)
  pythongh-104469: Update README.txt for _testcapi (pythongh-104529)
  pythonGH-103092: isolate `_elementtree` (python#104561)
  pythongh-104050: Add typing to Argument Clinic converters (python#104547)
  pythonGH-103906: Remove immortal refcounting in the interpreter (pythonGH-103909)
  pythongh-87474: Fix file descriptor leaks in subprocess.Popen (python#96351)
  pythonGH-103092: isolate `pyexpat`  (python#104506)
  pythongh-75367: Fix data descriptor detection in inspect.getattr_static (python#104517)
  pythongh-104050: Add more annotations to `Tools/clinic.py` (python#104544)
  pythongh-104555: Fix isinstance() and issubclass() for runtime-checkable protocols that use PEP 695 (python#104556)
  pythongh-103865: add monitoring support to LOAD_SUPER_ATTR (python#103866)
  CODEOWNERS: Assign new PEP 695 files to myself (python#104551)
  pythonGH-104510: Fix refleaks in `_io` base types (python#104516)
  pythongh-104539: Fix indentation error in logging.config.rst (python#104545)
  pythongh-104050: Don't star-import 'types' in Argument Clinic (python#104543)
  pythongh-104050: Add basic typing to CConverter in clinic.py (python#104538)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sunmy2019 sunmy2019 sunmy2019 approved these changes
@erlend-aasland erlend-aasland erlend-aasland approved these changes
@Eclips4 Eclips4 Eclips4 approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

release-blocker skip news

Projects

Release and Deferred blockers 🚫
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Ref leaks introduced by _io isolation (gh-101948)

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@kumaraditya303 @bedevere-bot @erlend-aasland @sunmy2019 @vstinner @Eclips4
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.