GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-90815: Add mimalloc memory allocator by tiran · Pull Request #31164 · python/cpython

Conversation

@tiran

@tiran tiran commented Feb 6, 2022 •
edited by ericsnowcurrently
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

https://bugs.python.org/issue46657

gh-90815

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bratao, vovavili, erlend-aasland, dgiger42, rdenadai, Rongronggg9, vasily-v-ryabov, shihai1991, and rbtz-openai reacted with heart emoji
All reactions
  • ❤️ 9 reactions
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Feb 6, 2022
@bedevere-bot bedevere-bot added the awaiting core review label Feb 6, 2022
@tiran tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 6, 2022
@bedevere-bot

bedevere-bot commented Feb 6, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @tiran for commit d4d545e5d54cced48793a1e67d36553588d4c560 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 6, 2022
@tiran
tiran force-pushed the bpo-46657-mimalloc branch from d4d545e to d6f5f01 Compare February 8, 2022 12:56
@tiran tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 8, 2022
@bedevere-bot

bedevere-bot commented Feb 8, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @tiran for commit 973b6178c0b46c2cd839b06cbe543ea039962442 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 8, 2022
@tiran tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 9, 2022
@bedevere-bot

bedevere-bot commented Feb 9, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @tiran for commit 9b8d0dac0c8ca184f0d7a93ee4b880edcacf64fd 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 9, 2022
@tiran tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 9, 2022
@bedevere-bot

bedevere-bot commented Feb 9, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @tiran for commit a9c29e66902900022bd74357647d0cc8c1538292 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 9, 2022
@tiran tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 9, 2022
@bedevere-bot

bedevere-bot commented Feb 9, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @tiran for commit e7231f186360d756df7d1320e39433fb46077e99 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 9, 2022
@tiran tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 9, 2022
@bedevere-bot

bedevere-bot commented Feb 9, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @tiran for commit 2472ecbf4e4723186145d3476dcbee7ef95cb8a4 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 9, 2022
@tiran tiran mentioned this pull request Feb 10, 2022
UBSAN: mi_slice_t index out of bounce microsoft/mimalloc#543
Closed
@tiran tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 10, 2022
@bedevere-bot

bedevere-bot commented Feb 10, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @tiran for commit 10498c3826865ac62811be3a61db019b01f23fc8 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 10, 2022
@neonene

neonene commented Feb 11, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

JFYI on Windows,

  • test_fstring.test_many_expressions reserves 150MB memory which seems never released. The following case holds 500MB:
build_fstr = lambda n: "f'" + ('{1}' * n) + "'"
exec(build_fstr(350)*1000)
  • MSVC fails to build x86 PGO, without generating python311*.pgc profiles if test_embed.EmbeddingTests.test_bpo20891 is used for an exercise. The same goes for x64 PGO when trained with only the test. (Fixed or improved by gh-94790)
ericsnowcurrently reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tiran
tiran force-pushed the bpo-46657-mimalloc branch 3 times, most recently from 210b5db to 60d6011 Compare March 10, 2022 12:10
@tiran
tiran force-pushed the bpo-46657-mimalloc branch from 01a3321 to 42c681a Compare April 6, 2022 12:38
tiran added 7 commits August 30, 2022 23:38
@tiran
Fix refleak MallocInfoType
51deac9
@tiran
Move mimalloc headers into internal header directory
95d904a
@tiran
Update to mimalloc 2.0.6
82642cc
@tiran
Hook up mi_collect for debugging
716a5f3
@tiran
Use heap type for malloc info struct
cc0d5e5
@tiran
Regen global symbols
5d4c193
@tiran
Fix memory leak
65af44c
@tiran
tiran force-pushed the bpo-46657-mimalloc branch from f56577e to 65af44c Compare August 30, 2022 21:40
@tiran

tiran commented Aug 30, 2022

Copy link
Copy Markdown
Member Author

I have rebased the PR again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tiran tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 30, 2022
@bedevere-bot

bedevere-bot commented Aug 30, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @tiran for commit 65af44c 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 30, 2022
@ericsnowcurrently

ericsnowcurrently commented Aug 30, 2022

Copy link
Copy Markdown
Member

Thanks!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bobfang1992

bobfang1992 commented Apr 15, 2023

Copy link
Copy Markdown

Hi, just want to check if this is still under development?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@zooba

zooba commented Apr 25, 2023

Copy link
Copy Markdown
Member

tiran is currently taking a break, so it's not being actively developed. I think we're still interested in using mimalloc though? Probably @ericsnowcurrently is the best person to sync with.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@erlend-aasland

erlend-aasland commented Apr 25, 2023

Copy link
Copy Markdown
Contributor

Probably @ericsnowcurrently is the best person to sync with.

... or @colesbury.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently

ericsnowcurrently commented Apr 25, 2023

Copy link
Copy Markdown
Member

FWIW, I don't have much relationship with this effort currently. At one point I thought I needed it for per-interpreter GIL, but realized later I didn't. That said, I would like to see this happen as I think it opens up several valuable opportunities to us. Furthermore, I did work with @daanx, the creator of mimalloc, to address the remaining issues, so this PR should be mostly ready, aside from conflicts and pulling in the latest mimalloc.

Ultimately, this change will probably need a PEP. (I also don't know what our options are regarding linking mimalloc in rather than vendoring it.) Other than that, I don't have much insight to offer, sadly.

daanx reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@daanx

daanx commented Apr 25, 2023

Copy link
Copy Markdown

Hi, @daanx here from mimalloc -- let me know if I can help with this PR. Just wanted to add that it would be good to merge the latest version of mimalloc as recently I made many improvements in particular for Valgrind and ASAN (and ETW) tools.

vasily-v-ryabov and thesamesam reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Aeron Aeron mentioned this pull request Sep 6, 2023
Consider to drop mimalloc emmett-framework/granian#121
Closed
@colesbury colesbury mentioned this pull request Sep 26, 2023
PEP 703 -- Making the Global Interpreter Lock Optional in CPython #108219
Closed
@ericsnowcurrently ericsnowcurrently mentioned this pull request Sep 27, 2023
gh-90815: Add mimalloc memory allocator #109914
Merged
DinoV added a commit to DinoV/cpython that referenced this pull request Oct 2, 2023
@DinoV
Use macro defs for exports as done in python#31164
4c22f38
DinoV added a commit to DinoV/cpython that referenced this pull request Oct 10, 2023
@DinoV
Use macro defs for exports as done in python#31164
1b2d106
DinoV added a commit to DinoV/cpython that referenced this pull request Oct 12, 2023
@DinoV
Use macro defs for exports as done in python#31164
8975bd1
vstinner added a commit that referenced this pull request Oct 30, 2023
@DinoV @colesbury
@tiran @vstinner
gh-90815: Add mimalloc memory allocator (#109914)
05f2f0a
* Add mimalloc v2.12

Modified src/alloc.c to remove include of alloc-override.c and not
compile new handler.

Did not include the following files:

 - include/mimalloc-new-delete.h
 - include/mimalloc-override.h
 - src/alloc-override-osx.c
 - src/alloc-override.c
 - src/static.c
 - src/region.c

mimalloc is thread safe and shares a single heap across all runtimes,
therefore finalization and getting global allocated blocks across all
runtimes is different.

* mimalloc: minimal changes for use in Python:

 - remove debug spam for freeing large allocations
 - use same bytes (0xDD) for freed allocations in CPython and mimalloc
   This is important for the test_capi debug memory tests

* Don't export mimalloc symbol in libpython.
* Enable mimalloc as Python allocator option.
* Add mimalloc MIT license.
* Log mimalloc in Lib/test/pythoninfo.py.
* Document new mimalloc support.
* Use macro defs for exports as done in:
  #31164

Co-authored-by: Sam Gross <[email protected]>
Co-authored-by: Christian Heimes <[email protected]>
Co-authored-by: Victor Stinner <[email protected]>
@erlend-aasland

erlend-aasland commented Oct 30, 2023

Copy link
Copy Markdown
Contributor

Superseded by #109914.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@erlend-aasland erlend-aasland closed this Oct 30, 2023
FullteaR pushed a commit to FullteaR/cpython that referenced this pull request Nov 3, 2023
@DinoV @colesbury
@tiran @vstinner @FullteaR
pythongh-90815: Add mimalloc memory allocator (python#109914)
37fa7b1
* Add mimalloc v2.12

Modified src/alloc.c to remove include of alloc-override.c and not
compile new handler.

Did not include the following files:

 - include/mimalloc-new-delete.h
 - include/mimalloc-override.h
 - src/alloc-override-osx.c
 - src/alloc-override.c
 - src/static.c
 - src/region.c

mimalloc is thread safe and shares a single heap across all runtimes,
therefore finalization and getting global allocated blocks across all
runtimes is different.

* mimalloc: minimal changes for use in Python:

 - remove debug spam for freeing large allocations
 - use same bytes (0xDD) for freed allocations in CPython and mimalloc
   This is important for the test_capi debug memory tests

* Don't export mimalloc symbol in libpython.
* Enable mimalloc as Python allocator option.
* Add mimalloc MIT license.
* Log mimalloc in Lib/test/pythoninfo.py.
* Document new mimalloc support.
* Use macro defs for exports as done in:
  python#31164

Co-authored-by: Sam Gross <[email protected]>
Co-authored-by: Christian Heimes <[email protected]>
Co-authored-by: Victor Stinner <[email protected]>
aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024
@DinoV @colesbury
@tiran @vstinner @aisk
pythongh-90815: Add mimalloc memory allocator (python#109914)
5f7475d
* Add mimalloc v2.12

Modified src/alloc.c to remove include of alloc-override.c and not
compile new handler.

Did not include the following files:

 - include/mimalloc-new-delete.h
 - include/mimalloc-override.h
 - src/alloc-override-osx.c
 - src/alloc-override.c
 - src/static.c
 - src/region.c

mimalloc is thread safe and shares a single heap across all runtimes,
therefore finalization and getting global allocated blocks across all
runtimes is different.

* mimalloc: minimal changes for use in Python:

 - remove debug spam for freeing large allocations
 - use same bytes (0xDD) for freed allocations in CPython and mimalloc
   This is important for the test_capi debug memory tests

* Don't export mimalloc symbol in libpython.
* Enable mimalloc as Python allocator option.
* Add mimalloc MIT license.
* Log mimalloc in Lib/test/pythoninfo.py.
* Document new mimalloc support.
* Use macro defs for exports as done in:
  python#31164

Co-authored-by: Sam Gross <[email protected]>
Co-authored-by: Christian Heimes <[email protected]>
Co-authored-by: Victor Stinner <[email protected]>
Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
@DinoV @colesbury
@tiran @vstinner @Glyphack
pythongh-90815: Add mimalloc memory allocator (python#109914)
554c420
* Add mimalloc v2.12

Modified src/alloc.c to remove include of alloc-override.c and not
compile new handler.

Did not include the following files:

 - include/mimalloc-new-delete.h
 - include/mimalloc-override.h
 - src/alloc-override-osx.c
 - src/alloc-override.c
 - src/static.c
 - src/region.c

mimalloc is thread safe and shares a single heap across all runtimes,
therefore finalization and getting global allocated blocks across all
runtimes is different.

* mimalloc: minimal changes for use in Python:

 - remove debug spam for freeing large allocations
 - use same bytes (0xDD) for freed allocations in CPython and mimalloc
   This is important for the test_capi debug memory tests

* Don't export mimalloc symbol in libpython.
* Enable mimalloc as Python allocator option.
* Add mimalloc MIT license.
* Log mimalloc in Lib/test/pythoninfo.py.
* Document new mimalloc support.
* Use macro defs for exports as done in:
  python#31164

Co-authored-by: Sam Gross <[email protected]>
Co-authored-by: Christian Heimes <[email protected]>
Co-authored-by: Victor Stinner <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently will be requested when the pull request is marked ready for review ericsnowcurrently is a code owner
@ZeroIntensity ZeroIntensity Awaiting requested review from ZeroIntensity ZeroIntensity will be requested when the pull request is marked ready for review ZeroIntensity is a code owner
@FFY00 FFY00 Awaiting requested review from FFY00 FFY00 will be requested when the pull request is marked ready for review FFY00 is a code owner
@erlend-aasland erlend-aasland Awaiting requested review from erlend-aasland erlend-aasland will be requested when the pull request is marked ready for review erlend-aasland is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner will be requested when the pull request is marked ready for review AA-Turner is a code owner
@emmatyping emmatyping Awaiting requested review from emmatyping emmatyping will be requested when the pull request is marked ready for review emmatyping is a code owner
@itamaro itamaro Awaiting requested review from itamaro itamaro will be requested when the pull request is marked ready for review itamaro is a code owner
@corona10 corona10 Awaiting requested review from corona10 corona10 will be requested when the pull request is marked ready for review corona10 is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting core review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

10 participants

@tiran @bedevere-bot @neonene @ericsnowcurrently @zooba @bobfang1992 @erlend-aasland @daanx @the-knights-who-say-ni @ezio-melotti
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.