GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-42825: Enable /OPT:REF by Austin-Lamb · Pull Request #24098 · python/cpython

Conversation

@Austin-Lamb

@Austin-Lamb Austin-Lamb commented Jan 4, 2021 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

This PR enables the /OPT:REF linker optimization on Windows, allowing the linker to discard uncalled code. This is analogous to --gc-sections for ld on linux, if you're familiar with that. Basically the linker can determine that a function is never called and not include it in the final binary.

Note that I had to explicitly keep /OPT:ICF disabled because enabling Identical COMDAT Folding (ICF) causes test failures. This is because Python310.dll depends on some identical things not folding together (such as wrap_binary_func and wrap_binary_func_l) as their addresses are compared to determine equality in some important places. There are even helpful tests that verify COMDAT folding is disabled which helped me catch this 😊.

This could be backported to previous versions if desired, it should be simple and safe to do to whichever versions are deployed at scale in the wild, but I'll leave that to the community and maintainers to decide, I didn't want to go through the backport process.

Note: I am not sure how to build for PGO locally, but I think it's good to have this on for Configuration == PGInstrument and/or PGUpdate, so I used a Condition of "!= Debug". This should be verified in the real build pipeline that does PGO, or let me know how I can do it locally.

In terms of the size savings, here's a few example binaries and then the total across all of them in the build output folder - all measurements were done for amd64 Release binaries. It's over a 10% savings in the size of everything, and some binaries are considerably more than that.

Binary Size before Size after Delta (%)
py.exe 1,024,000 735,744 -288,256 (-28.2%)
python.exe 93,696 91,136 -2,560 (-2.7%)
python_uwp.exe 225,792 140,288 -85,504 (-37.9%)
python310.dll 5,268,992 4,753,408 -515,584 (-9.8%)
sqlite3.dll 1,505,280 1,376,256 -129,024 (-8.6%)
Total of .exe/.dll/.pyd 21,632,408 19,278,744 -2,353,664 (-10.9%)

https://bugs.python.org/issue42825

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

asklar and methane reacted with thumbs up emoji
All reactions
  • 👍 2 reactions
@Austin-Lamb
bpo-42825: Enable /OPT:REF, but explicitly disable /OPT:ICF since it …
bcbdad1
…causes test failures as Python310.dll depends on some identical things not folding together (such as wrap_binary_func and wrap_binary_func_l).
@Austin-Lamb
Austin-Lamb requested a review from a team as a code owner January 4, 2021 17:59
@the-knights-who-say-ni the-knights-who-say-ni added the CLA not signed label Jan 4, 2021
@bedevere-bot bedevere-bot added the awaiting review label Jan 4, 2021
@ammaraskar
ammaraskar requested a review from zooba January 6, 2021 06:28
@the-knights-who-say-ni the-knights-who-say-ni added CLA signed and removed CLA not signed labels Jan 7, 2021
@python python deleted a comment from the-knights-who-say-ni Jan 8, 2021
@Austin-Lamb

Austin-Lamb commented Jan 19, 2021

Copy link
Copy Markdown
Contributor Author

@zooba - anything I should do to help this get reviewed?

Thanks!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions

github-actions Bot commented Feb 19, 2021

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Feb 19, 2021
@zooba zooba added the skip news label Feb 19, 2021
@zooba
zooba merged commit b4af629 into python:master Feb 19, 2021
@bedevere-bot bedevere-bot removed the awaiting review label Feb 19, 2021
adorilson pushed a commit to adorilson/cpython that referenced this pull request Mar 13, 2021
@Austin-Lamb @adorilson
bpo-42825: Enable /OPT:REF (pythonGH-24098)
176b0e8
We explicitly disable /OPT:ICF as some manual optimisations depend on some functions still having distinct pointers (such as wrap_binary_func and wrap_binary_func_l).
geofft added a commit to geofft/cpython that referenced this pull request May 25, 2025
@geofft
Use only safe identical code folding with BOLT
c8f458d
"Identical code folding" (ICF) is the feature of an optimizer to find that two
functions have the same code and that they can therefore be deduplicated
in the binary. While this is usually safe, it can cause observable
behavior differences if the program relies on the fact that the two
functions have different addresses.

CPython relies on this in (at least) Objects/typeobject.c, which defines
two functions wrap_binaryfunc() and wrap_binaryfunc_l() with the same
implementation, and stores their addresses in the slotdefs array. If
these two functions have the same address, update_one_slot() in that
file will fill in slots it shouldn't, causing, for instances,
classes defined in Python that inherit from some built-in types to
misbehave.

As of LLVM 20 (llvm/llvm-project#116275), BOLT has a "safe ICF" mode,
where it looks to see if there are any uses of a function symbol outside
function calls (e.g., relocations in data sections) and skips ICF on
such functions. The intent is that this avoids observable behavior
differences but still saves storage as much as possible.

This version is about two months old at the time of writing. To support
older LLVM versions, we have to turn off ICF entirely.

This problem was previously noticed for Windows/MSVC in python#53093 (and
again in python#24098), where the default behavior of PGO is to enable ICF
(which they expand to "identical COMDAT folding") and we had to turn it
off.
geofft added a commit to geofft/cpython that referenced this pull request May 25, 2025
@geofft
Use only safe identical code folding with BOLT
91fc5ae
"Identical code folding" (ICF) is the feature of an optimizer to find that two
functions have the same code and that they can therefore be deduplicated
in the binary. While this is usually safe, it can cause observable
behavior differences if the program relies on the fact that the two
functions have different addresses.

CPython relies on this in (at least) Objects/typeobject.c, which defines
two functions wrap_binaryfunc() and wrap_binaryfunc_l() with the same
implementation, and stores their addresses in the slotdefs array. If
these two functions have the same address, update_one_slot() in that
file will fill in slots it shouldn't, causing, for instances,
classes defined in Python that inherit from some built-in types to
misbehave.

As of LLVM 20 (llvm/llvm-project#116275), BOLT has a "safe ICF" mode,
where it looks to see if there are any uses of a function symbol outside
function calls (e.g., relocations in data sections) and skips ICF on
such functions. The intent is that this avoids observable behavior
differences but still saves storage as much as possible.

This version is about two months old at the time of writing. To support
older LLVM versions, we have to turn off ICF entirely.

This problem was previously noticed for Windows/MSVC in python#53093 (and
again in python#24098), where the default behavior of PGO is to enable ICF
(which they expand to "identical COMDAT folding") and we had to turn it
off.
@geofft geofft mentioned this pull request May 25, 2025
gh-101525: Use only safe identical code folding with BOLT #134642
Open
@icanhasmath icanhasmath mentioned this pull request Oct 29, 2026
3.9.25.3: link Windows builds with /OPT:REF,NOICF (bpo-42825) ActiveState/cpython#116
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zooba zooba Awaiting requested review from zooba

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news stale Stale PR or inactive for long period of time.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@Austin-Lamb @zooba @the-knights-who-say-ni @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.