GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-46576: bpo-46524: Disable compiler optimization within test_peg_generator. by gpshead · Pull Request #31015 · python/cpython

Conversation

@gpshead

@gpshead gpshead commented Jan 29, 2022 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This speed up test_peg_generator by always disabling compiler
optimizations by using -O0 by default when the test is building its
own C extensions.

A build not using --with-pydebug in order to speed up test execution
winds up with this test taking a very long time as it would do
repeated compilation of parser C code using the same optimization
flags as CPython was built with.

This speeds the test up 6-8x on gps-raspbian.

Includes the win32 changes from #31017 by Kumar Aditya

https://bugs.python.org/issue46576

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@gpshead
bpo-46576: Disable compiler optimization within test_peg_generator.
efbb649 
This speed up test_peg_generator by always disabling compiler
optimizations by using -O0 by default when the test is building its
own C extensions.

A build not using --with-pydebug in order to speed up test execution
winds up with this test taking a very long time as it would do
repeated compilation of parser C code using the same optimization
flags as CPython was built with.

This speeds the test up 6-8x on gps-raspbian.
@gpshead gpshead added tests Tests in the Lib/test dir needs backport to 3.9 labels Jan 29, 2022
@gpshead
gpshead requested a review from pablogsal January 29, 2022 20:40
@gpshead
gpshead requested a review from lysnikolaou as a code owner January 29, 2022 20:40
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Jan 29, 2022
@bedevere-bot bedevere-bot added the awaiting core review label Jan 29, 2022
@gpshead gpshead assigned pablogsal Jan 29, 2022
pablogsal
pablogsal approved these changes Jan 29, 2022
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels Jan 29, 2022
@sweeneyde

sweeneyde commented Jan 30, 2022

Copy link
Copy Markdown
Member

Windows tests show:

cl : Command line warning D9025 : overriding '/DNDEBUG' with '/UNDEBUG'
cl : Command line warning D9002 : ignoring unknown option '-O0'

Is there a corresponding option for msvc?
gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@kumaraditya303

kumaraditya303 commented Jan 30, 2022

Copy link
Copy Markdown
Contributor

MSVC has /Od to speed up compilation, See https://docs.microsoft.com/en-us/cpp/build/reference/od-disable-debug?view=msvc-170
gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
gpshead
gpshead commented Jan 30, 2022
View reviewed changes
Comment thread Tools/peg_generator/pegen/build.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@gpshead gpshead assigned gpshead and unassigned gpshead and pablogsal Jan 30, 2022
gpshead added 2 commits January 30, 2022 00:40
@gpshead
incorporate python#31017's win32 conditional and flags
36044f3
@gpshead
reword news
42b989c
@gpshead gpshead changed the title bpo-46576: Disable compiler optimization within test_peg_generator. bpo-46576: bpo-46524: Disable compiler optimization within test_peg_generator. Jan 30, 2022
tiran
tiran reviewed Jan 30, 2022
View reviewed changes
Comment thread Tools/peg_generator/pegen/build.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@gpshead @tiran
Add -fno-lto
c4457db 
Co-authored-by: Christian Heimes <[email protected]>
@gpshead gpshead added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jan 31, 2022
@bedevere-bot

bedevere-bot commented Jan 31, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @gpshead for commit c4457db 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jan 31, 2022
@pablogsal pablogsal added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 1, 2022
@bedevere-bot

bedevere-bot commented Feb 1, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit c4457db 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 1, 2022
@gpshead

gpshead commented Feb 1, 2022

Copy link
Copy Markdown
Member Author

I believe I need to rebase my PR branch before buildbottest is useful.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@gpshead
Merge branch 'python:main' into faster_test_pegen
5b99059
@gpshead gpshead added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 1, 2022
@bedevere-bot

bedevere-bot commented Feb 1, 2022

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @gpshead for commit 5b99059 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 1, 2022
@gpshead
gpshead merged commit 164a017 into python:main Feb 2, 2022
@miss-islington

miss-islington commented Feb 2, 2022

Copy link
Copy Markdown
Contributor

Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 3.9, 3.10.
🐍🍒⛏🤖
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the awaiting merge label Feb 2, 2022
@bedevere-bot

bedevere-bot commented Feb 2, 2022

Copy link
Copy Markdown

@gpshead: Please replace # with GH- in the commit message next time. Thanks!
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@miss-islington

miss-islington commented Feb 2, 2022

Copy link
Copy Markdown
Contributor

Sorry, @gpshead, I could not cleanly backport this to 3.10 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 164a017e13ee96bd1ea1ae79f5ac9e25fe83994e 3.10
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@miss-islington

miss-islington commented Feb 2, 2022

Copy link
Copy Markdown
Contributor

Sorry @gpshead, I had trouble checking out the 3.9 backport branch.
Please backport using cherry_picker on command line.
cherry_picker 164a017e13ee96bd1ea1ae79f5ac9e25fe83994e 3.9
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@gpshead
gpshead deleted the faster_test_pegen branch February 2, 2022 20:15
@gpshead gpshead mentioned this pull request Feb 2, 2022
bpo-46524: speed up compilation of peg_generator on windows #31017
Closed
gpshead added a commit to gpshead/cpython that referenced this pull request Feb 2, 2022
@gpshead
[3.10] bpo-46576: bpo-46524: Disable compiler optimization within tes…
4b0efc6 
…t_peg_generator. (pythonGH-31015)

Disable compiler optimization within test_peg_generator.

This speed up test_peg_generator by always disabling compiler
optimizations by using -O0 or equivalent when the test is building its
own C extensions.

A build not using --with-pydebug in order to speed up test execution
winds up with this test taking a very long time as it would do
repeated compilation of parser C code using the same optimization
flags as CPython was built with.

This speeds the test up 6-8x on gps-raspbian.

Also incorporate's pythonGH-31017's win32 conditional and flags.

Co-authored-by: Kumar Aditya kumaraditya303.
(cherry picked from commit 164a017)

Co-authored-by: Gregory P. Smith <[email protected]>
@gpshead gpshead mentioned this pull request Feb 2, 2022
[3.10] bpo-46576: bpo-46524: Disable compiler optimization within test_peg_generator #31089
Merged
@vstinner

vstinner commented Feb 2, 2022

Copy link
Copy Markdown
Member

In the Windows (x64) job, test_peg_generator only took 3 min 40 sec: it's way better than 8 minutes, yeah!
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
gpshead added a commit that referenced this pull request Feb 3, 2022
@gpshead
[3.10] bpo-46576: bpo-46524: Disable compiler optimization within tes…
f5ebec4 
…t_peg_generator. (GH-31015) (GH-31089)

Disable compiler optimization within test_peg_generator.

This speed up test_peg_generator by always disabling compiler
optimizations by using -O0 or equivalent when the test is building its
own C extensions.

A build not using --with-pydebug in order to speed up test execution
winds up with this test taking a very long time as it would do
repeated compilation of parser C code using the same optimization
flags as CPython was built with.

This speeds the test up 6-8x on gps-raspbian.

Also incorporate's GH-31017's win32 conditional and flags.

Co-authored-by: Kumar Aditya kumaraditya303.
(cherry picked from commit 164a017)

Co-authored-by: Gregory P. Smith <[email protected]>
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Feb 3, 2022
@gpshead @miss-islington
[3.10] bpo-46576: bpo-46524: Disable compiler optimization within tes…
1f04b87 
…t_peg_generator. (pythonGH-31015) (pythonGH-31089)

Disable compiler optimization within test_peg_generator.

This speed up test_peg_generator by always disabling compiler
optimizations by using -O0 or equivalent when the test is building its
own C extensions.

A build not using --with-pydebug in order to speed up test execution
winds up with this test taking a very long time as it would do
repeated compilation of parser C code using the same optimization
flags as CPython was built with.

This speeds the test up 6-8x on gps-raspbian.

Also incorporate's pythonGH-31017's win32 conditional and flags.

Co-authored-by: Kumar Aditya kumaraditya303.
(cherry picked from commit 164a017)

Co-authored-by: Gregory P. Smith <[email protected]>
(cherry picked from commit f5ebec4)

Co-authored-by: Gregory P. Smith <[email protected]>
@gpshead gpshead removed needs backport to 3.9 labels Feb 3, 2022
miss-islington added a commit that referenced this pull request Feb 3, 2022
@miss-islington
[3.9] [3.10] bpo-46576: bpo-46524: Disable compiler optimization with…
e825860 
…in test_peg_generator. (GH-31015) (GH-31089) (GH-31093)

Disable compiler optimization within test_peg_generator.

This speed up test_peg_generator by always disabling compiler
optimizations by using -O0 or equivalent when the test is building its
own C extensions.

A build not using --with-pydebug in order to speed up test execution
winds up with this test taking a very long time as it would do
repeated compilation of parser C code using the same optimization
flags as CPython was built with.

This speeds the test up 6-8x on gps-raspbian.

Also incorporate's GH-31017's win32 conditional and flags.

Co-authored-by: Kumar Aditya kumaraditya303.
(cherry picked from commit 164a017)

Co-authored-by: Gregory P. Smith <[email protected]>
(cherry picked from commit f5ebec4)


Co-authored-by: Gregory P. Smith <[email protected]>

Automerge-Triggered-By: GH:gpshead
@gpshead

gpshead commented Feb 3, 2022

Copy link
Copy Markdown
Member Author

In the Windows (x64) job, test_peg_generator only took 3 min 40 sec: it's way better than 8 minutes, yeah!

yeah. though i also saw it take both 16 minutes or 5 minutes in similar CI runs on other PRs prior to this... so I can't make any judgement call. it certainly should not be worse. I hope it trends better.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
hello-adam pushed a commit to hello-adam/cpython that referenced this pull request Jun 2, 2022
@miss-islington @hello-adam
[3.9] [3.10] bpo-46576: bpo-46524: Disable compiler optimization with…
930cd57 
…in test_peg_generator. (pythonGH-31015) (pythonGH-31089) (pythonGH-31093)

Disable compiler optimization within test_peg_generator.

This speed up test_peg_generator by always disabling compiler
optimizations by using -O0 or equivalent when the test is building its
own C extensions.

A build not using --with-pydebug in order to speed up test execution
winds up with this test taking a very long time as it would do
repeated compilation of parser C code using the same optimization
flags as CPython was built with.

This speeds the test up 6-8x on gps-raspbian.

Also incorporate's pythonGH-31017's win32 conditional and flags.

Co-authored-by: Kumar Aditya kumaraditya303.
(cherry picked from commit 164a017)

Co-authored-by: Gregory P. Smith <[email protected]>
(cherry picked from commit f5ebec4)


Co-authored-by: Gregory P. Smith <[email protected]>

Automerge-Triggered-By: GH:gpshead
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kumaraditya303 kumaraditya303 kumaraditya303 left review comments
@pablogsal pablogsal pablogsal approved these changes
@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner
+1 more reviewer
@tiran tiran tiran left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@gpshead gpshead

Labels

tests Tests in the Lib/test dir

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

9 participants

@gpshead @sweeneyde @kumaraditya303 @bedevere-bot @miss-islington @vstinner @tiran @pablogsal @the-knights-who-say-ni
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.