GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-1635741: Fix refleaks of encodings module by removing the encodings._aliases by shihai1991 · Pull Request #21896 · python/cpython

Conversation

@shihai1991

@shihai1991 shihai1991 commented Aug 16, 2020 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Fix refleaks of encodings._aliases by using encodings.aliases directly in encodings.search_function.

Co-authored-by: Victor Stinner [email protected]

https://bugs.python.org/issue1635741

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@shihai1991
remove encodings._aliases to search_function
c933b1a
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Aug 16, 2020
@bedevere-bot bedevere-bot added the awaiting review label Aug 16, 2020
@shihai1991 shihai1991 changed the title WIP: Move encodings._aliases to search_function bpo-1635741: Move encodings._aliases to search_function Aug 16, 2020
@shihai1991

shihai1991 commented Aug 16, 2020

Copy link
Copy Markdown
Member Author

I use the test case of https://bugs.python.org/issue1635741#msg355187 to test the refleaks in debug mode.

Before this PR:
sys.gettotalrefcount: 14288
sys.gettotalrefcount: 18042
sys.gettotalrefcount: 21796
sys.gettotalrefcount: 25550
sys.gettotalrefcount: 29304
sys.gettotalrefcount: 33058
sys.gettotalrefcount: 36812
sys.gettotalrefcount: 40566
sys.gettotalrefcount: 44320
sys.gettotalrefcount: 48074

After this PR:
sys.gettotalrefcount: 13641
sys.gettotalrefcount: 16748
sys.gettotalrefcount: 19855
sys.gettotalrefcount: 22962
sys.gettotalrefcount: 26069
sys.gettotalrefcount: 29176
sys.gettotalrefcount: 32283
sys.gettotalrefcount: 35390
sys.gettotalrefcount: 38497
sys.gettotalrefcount: 41604

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@shihai1991

shihai1991 commented Aug 16, 2020

Copy link
Copy Markdown
Member Author

@vstinner Hi, victor. Pls take a look if you have free time, thanks.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ghost
ghost reviewed Aug 16, 2020
View reviewed changes
Comment thread Lib/encodings/__init__.py Outdated

def search_function(encoding):

_aliases = aliases.aliases

@ghost ghost Aug 16, 2020 •
edited by ghost
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't understand the problem.
This statement should be placed at below so that it does not affect the performance of the cache.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@shihai1991 shihai1991 Aug 16, 2020 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't understand the problem.

Thanks for your comment. It will affect the encodings module's refcount in C level and reduce the refleaks.

This statement should be placed at below so that it does not affect the performance of the cache.

MAYBE removing this line and using aliases.aliases to replace _aliases is fine too :)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@shihai1991 shihai1991 Aug 16, 2020

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking this comment: #21896 (comment)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@ghost ghost Aug 16, 2020 •
edited by ghost
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking this comment: #21896 (comment)

The usage of aliases.aliasesis is very normal, maybe the root of the problem is not here.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
shihai1991 added 3 commits August 16, 2020 22:12
@shihai1991
deleting the _aliases in encodings module
4dd90d4
@shihai1991
update news
1dc9733
@shihai1991
Fix typo
9cc86b1
@shihai1991 shihai1991 changed the title bpo-1635741: Move encodings._aliases to search_function bpo-1635741: Fix refleaks of encodings module by removing the encodings._aliases Aug 16, 2020
vstinner
vstinner reviewed Aug 17, 2020
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see how using encodings._aliases in search_function() creates a "reference leak". A leak is when calling a function multiple times leaks memory. Here, there is no leak.

Maybe you're talking about a "reference cycle".

I guess that you're trying to clear variables at exit.

You should try to trigger an explicit GC collection after calling PyInterpreterState_Clear(). In finalize_interp_clear(), try to replace:

    /* Trigger a GC collection on subinterpreters*/
    if (!is_main_interp) {
        _PyGC_CollectNoFail();
    }

with:

    // Last explicit GC collection
    _PyGC_CollectNoFail();

(without this change)

Does it fix your issue?

PyInterpreterState_Clear() clears the reference to the search function: Py_CLEAR(interp->codec_search_path).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@shihai1991

shihai1991 commented Aug 17, 2020 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

Maybe you're talking about a "reference cycle".

Thanks, victor. "reference cycle" would be more exact. And I will try your idea in my interpreter.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@shihai1991

shihai1991 commented Aug 17, 2020

Copy link
Copy Markdown
Member Author
    /* Trigger a GC collection on subinterpreters*/
    if (!is_main_interp) {
        _PyGC_CollectNoFail();
    }

with:

    // Last explicit GC collection
    _PyGC_CollectNoFail();

Oh, amazing result:

sys.gettotalrefcount: 10537
sys.gettotalrefcount: 10540
sys.gettotalrefcount: 10543
sys.gettotalrefcount: 10546
sys.gettotalrefcount: 10549
sys.gettotalrefcount: 10552
sys.gettotalrefcount: 10555
sys.gettotalrefcount: 10558
sys.gettotalrefcount: 10561
sys.gettotalrefcount: 10564

the pr in: #21902

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@shihai1991

shihai1991 commented Aug 17, 2020 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

Pablo created this PR(don't calling explict collection in main interpreter): #17457
I am not sure I missed some info or not~

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Aug 17, 2020

Copy link
Copy Markdown
Member

Since #17457 is merged, is this PR still relevant/useless? If not, please close it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@shihai1991

shihai1991 commented Aug 17, 2020

Copy link
Copy Markdown
Member Author

Since #17457 is merged, is this PR still relevant/useless? If not, please close it.

#17457 is worked, thanks.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@shihai1991 shihai1991 closed this Aug 17, 2020
@kylotan kylotan mannequin mentioned this pull request Sep 19, 2022
Py_Finalize() doesn't clear all Python objects at exit #44470
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner left review comments
+1 more reviewer
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@shihai1991 @vstinner @the-knights-who-say-ni @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.