GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — GH-104787: use managed dict in `_asyncio` by kumaraditya303 · Pull Request #104795 · python/cpython

Conversation

@kumaraditya303

@kumaraditya303 kumaraditya303 commented May 23, 2023 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor
  • Issue: Optimize and reduce memory usage for asyncio.Task and others #104787

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@kumaraditya303
use managed dict in _asyncio
d1cee85
@kumaraditya303 kumaraditya303 added performance Performance or resource usage topic-asyncio labels May 23, 2023
@bedevere-bot bedevere-bot mentioned this pull request May 23, 2023
Optimize and reduce memory usage for asyncio.Task and others #104787
Closed
@kumaraditya303 kumaraditya303 added skip news 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section labels May 23, 2023
@bedevere-bot

bedevere-bot commented May 23, 2023

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @kumaraditya303 for commit d1cee85 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label May 23, 2023
PixelCode01
PixelCode01 approved these changes May 23, 2023
View reviewed changes
@bedevere-bot bedevere-bot added the awaiting core review label May 23, 2023
@kumaraditya303
kumaraditya303 marked this pull request as ready for review May 23, 2023 10:33
@kumaraditya303
kumaraditya303 requested review from 1st1, asvetlov, gvanrossum and willingc as code owners May 23, 2023 10:33
@itamaro

itamaro commented May 23, 2023

Copy link
Copy Markdown
Contributor

is this a runtime optimization? memory optimization? what's the impact?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gvanrossum

gvanrossum commented May 23, 2023

Copy link
Copy Markdown
Member

From reading the 3.12 docs this seems to be primarily meant as a "better" way to support __dict__ , letting the standard object infrastructure manage the dict instead of managing it "by hand". (The dict is still present, so it doesn't really save memory, but there's a potential for future optimizations once a type adopts this.)

@kumaraditya303 Shouldn't we do this for weakrefs too? There's a corresponding Py_TPFLAGS_MANAGED_WEAKREF flag.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@kumaraditya303

kumaraditya303 commented May 23, 2023

Copy link
Copy Markdown
Contributor Author

From reading the 3.12 docs this seems to be primarily meant as a "better" way to support dict , letting the standard object infrastructure manage the dict instead of managing it "by hand". (The dict is still present, so it doesn't really save memory, but there's a potential for future optimizations once a type adopts this.)

See #95707, it does save memory as the dict isn't really allocated, the values pointer is used directly so it saves memory.

Shouldn't we do this for weakrefs too? There's a corresponding Py_TPFLAGS_MANAGED_WEAKREF flag.

Haven't thought about using that yet, will check it in another PR for now.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gvanrossum gvanrossum mentioned this pull request May 23, 2023
Support the use of the managed pre-header in builtin classes. #95707
Open
2 of 3 tasks
@gvanrossum

gvanrossum commented May 23, 2023

Copy link
Copy Markdown
Member

I see. I've forgotten how this works, I've added a ping asking for more documentation.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@kumaraditya303

kumaraditya303 commented May 26, 2023

Copy link
Copy Markdown
Contributor Author

I see. I've forgotten how this works, I've added a ping asking for more documentation.

Ok, I'll merge this now.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@kumaraditya303
kumaraditya303 merged commit 72c3d2e into python:main May 26, 2023
@bedevere-bot bedevere-bot removed the awaiting core review label May 26, 2023
@kumaraditya303
kumaraditya303 deleted the managed-dict branch May 26, 2023 05:00
@bedevere-bot

bedevere-bot commented May 26, 2023

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.x has failed when building commit 72c3d2e.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/345/builds/4934) and take a look at the build logs.
  4. Check if the failure is related to this commit (72c3d2e) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/345/builds/4934

Failed tests:

  • test__xxsubinterpreters

Summary of the results of the build (if available):

==

Click to see traceback logs
remote: Enumerating objects: 7, done.        
remote: Counting objects:  14% (1/7)        
remote: Counting objects:  28% (2/7)        
remote: Counting objects:  42% (3/7)        
remote: Counting objects:  57% (4/7)        
remote: Counting objects:  71% (5/7)        
remote: Counting objects:  85% (6/7)        
remote: Counting objects: 100% (7/7)        
remote: Counting objects: 100% (7/7), done.        
remote: Compressing objects:  25% (1/4)        
remote: Compressing objects:  50% (2/4)        
remote: Compressing objects:  75% (3/4)        
remote: Compressing objects: 100% (4/4)        
remote: Compressing objects: 100% (4/4), done.        
remote: Total 4 (delta 3), reused 0 (delta 0), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to '72c3d2e105f120f6f2bce410699b34dac4e948fd'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 72c3d2e105 GH-104787: use managed dict in `_asyncio` (#104795)
Switched to and reset branch 'main'

configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

Fatal Python error: drop_gil: PyMUTEX_LOCK(gil->switch_mutex) failed
Python runtime state: initialized

Thread 0x00007fa8c3e13280 (most recent call first):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test__xxsubinterpreters.py", line 80 in clean_up_interpreters
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test__xxsubinterpreters.py", line 88 in tearDown
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 594 in _callTearDown
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 637 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 690 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/runner.py", line 240 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 1115 in _run_suite
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 1241 in run_unittest
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 294 in _test_module
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 330 in _runtest_inner2
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 373 in _runtest_inner
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 248 in _runtest
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 278 in runtest
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 365 in rerun_failed_tests
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 783 in _main
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 738 in main
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 802 in main
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/__main__.py", line 2 in <module>
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/runpy.py", line 88 in _run_code
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/runpy.py", line 198 in _run_module_as_main
Fatal Python error: Aborted

Thread 0x00007fa8c3e13280 (most recent call first):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test__xxsubinterpreters.py", line 80 in clean_up_interpreters
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test__xxsubinterpreters.py", line 88 in tearDown
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 594 in _callTearDown
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 637 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/case.py", line 690 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/unittest/runner.py", line 240 in run
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 1115 in _run_suite
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 1241 in run_unittest
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 294 in _test_module
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 330 in _runtest_inner2
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 373 in _runtest_inner
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 248 in _runtest
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 278 in runtest
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 365 in rerun_failed_tests
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 783 in _main
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 738 in main
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 802 in main
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/__main__.py", line 2 in <module>
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/runpy.py", line 88 in _run_code
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: _testcapi, _xxsubinterpreters (total: 2)
Aborted
make: *** [Makefile:1998: buildbottest] Error 134

Cannot open file '/root/buildarea/3.x.angelico-debian-amd64/build/test-results.xml' for upload
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@1st1 1st1 Awaiting requested review from 1st1 1st1 is a code owner
@asvetlov asvetlov Awaiting requested review from asvetlov asvetlov is a code owner
@gvanrossum gvanrossum Awaiting requested review from gvanrossum
@willingc willingc Awaiting requested review from willingc willingc is a code owner
1 more reviewer
@PixelCode01 PixelCode01 PixelCode01 approved these changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

performance Performance or resource usage skip news topic-asyncio

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

5 participants

@kumaraditya303 @bedevere-bot @itamaro @gvanrossum @PixelCode01
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.