GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Ensure ``make -j`` uses a reasonable argument by AA-Turner · Pull Request #1541 · python/devguide

Conversation

@AA-Turner

@AA-Turner AA-Turner commented Apr 18, 2025 •
edited by github-actions Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

📚 Documentation preview 📚: https://cpython-devguide--1541.org.readthedocs.build/

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@AA-Turner
Ensure make -j uses a reasonable argument
4be7737
@AA-Turner
AA-Turner requested a review from Yhg1s April 18, 2025 12:18
@AA-Turner AA-Turner mentioned this pull request Apr 18, 2025
Don't suggest to start build with infinite number of threads #1087
Closed
hugovk
hugovk reviewed Apr 18, 2025
View reviewed changes
Comment thread developer-workflow/extension-modules.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread getting-started/setup-building.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

AA-Turner and others added 3 commits April 18, 2025 13:23
@AA-Turner @hugovk
Simplify
c2784fa
Co-authored-by: Hugo van Kemenade <[email protected]>
@AA-Turner @hugovk
Expand
1b10013
Co-authored-by: Hugo van Kemenade <[email protected]>
@Yhg1s @AA-Turner
Do not suggest make -j for building CPython. make -j means _no_ l…
9ce0c3c
…imit on

the number of concurrent jobs, not "a sensible limit considering the
available memory and CPU cores". With LTO it's easy to run out of memory
when too many jobs run at the same time.
@AA-Turner

AA-Turner commented Apr 18, 2025

Copy link
Copy Markdown
Member Author

I've cherry-picked @Yhg1s's changes from #1543 into this branch, the question is if we want to suggest a static number (-j4) or use the number of CPU cores (-j $(nproc)).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@hugovk

hugovk commented Apr 18, 2025

Copy link
Copy Markdown
Member

Let's use $(nproc) so we can copy and paste and get something more useful. Many modern machines have much more than 4 cores; I have an 8 and a 10.

AA-Turner and chris-eibl reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

picnixz
picnixz reviewed Apr 18, 2025
View reviewed changes
Comment thread getting-started/setup-building.rst

Once ``configure`` is done, you can then compile CPython with::

$ make -s -j2

@picnixz picnixz Apr 18, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that nproc is part of GNU coreutils and may not necessarily be available on all Unix-like systems (namely, macOS). So Mac users should first do brew install coreutils or use sysctl instead to get the number of physical or logical cores (I don't remember which one nproc returns)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@AA-Turner AA-Turner Apr 18, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've merged, as Hugo uses macOS and didn't complain, but we can open a follow-up if need be.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@hugovk hugovk Apr 18, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out. I do have coreutils installed via Homebrew, which is why it worked for me. Let's not require or assume others do.

Shall we use a hardcoded value under the macOS tabs? Something like 8 feels like a good default?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@gpshead gpshead Apr 18, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TL;DR - this is good enough.

I feel like people who understand how to run make already understand what $(nproc) can be substituted with manually if they don't have the command. if we wanted to be ironic we'd suggest $(python3 -c 'import os; print(os.cpu_count())') to the person who is building python and may not yet have a built python3 (but realistically do anyways because all Linux distros offer it and macOS xcode comes with one). 😛

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hugovk reacted with laugh emoji
All reactions
  • 😄 1 reaction

@zware zware Apr 18, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like people who understand how to run make already understand what $(nproc) can be substituted with

That seems like a rash assumption to me; I would expect anyone with that level of comfort with make to not need to be walked through "run ./configure && make". There's also the point that make -j $(nproc) in the absense of a working nproc will leave the user with make -j , which is what we were trying to avoid with this change.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

hugovk reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@picnixz picnixz Apr 18, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The assumption would be correct if we were in the extension-modules.rst file, but this file is for the "getting started". I don't think we should assume that the reader is extra familiar with make and even knows about nproc and co (this section would be read by newcomers I think).

By the way, nproc being part of coreutils and not necessarily available on macOS caused some issues in other projects: https://www.drupal.org/project/drupal/issues/3407360. So it's not necessarily a theoretical use case.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@AA-Turner
AA-Turner merged commit f1b7e25 into python:main Apr 18, 2025
@AA-Turner
AA-Turner deleted the make-jobs branch April 18, 2025 19:40
This was referenced Apr 18, 2025
gh-1542: Do not suggest make -j #1543
Closed
Bug: devguide gives incorrect advice for make -j #1542
Closed
@hugovk hugovk mentioned this pull request Apr 29, 2025
Use make -j8 for macOS #1545
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead gpshead left review comments
@hugovk hugovk hugovk left review comments
@zware zware zware left review comments
@picnixz picnixz picnixz left review comments
@Yhg1s Yhg1s Awaiting requested review from Yhg1s

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@AA-Turner @hugovk @gpshead @zware @picnixz @Yhg1s
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.