GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Don't suggest to start build with infinite number of threads by Delgan · Pull Request #1087 · python/devguide

Conversation

@Delgan

@Delgan Delgan commented Apr 29, 2023 •
edited by github-actions Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Hi.

Just a small enhancement proposal, as I don't think calling make with unbounded number of threads is desirable.

Using make -j will start "infinite" number of jobs according to make docs, which slows down the build and consumes too many resources.

Suggesting make -j4 seems to be a sane default, advanced users will know how to adjust it.


📚 Documentation preview 📚: https://cpython-devguide--1087.org.readthedocs.build/

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@Delgan
Don't suggest to start build with infinite number of threads
e68aadb
@arhadthedev arhadthedev added the type-feature Additions; New content or section needed label Apr 29, 2023
@hugovk

hugovk commented Apr 29, 2023

Copy link
Copy Markdown
Member

Thanks for the suggestion.

What machine are you using? Does the build take longer with make -j than make -j4, or otherwise slow the machine down? "consumes too many resources" - in what way?

On macOS with 8 CPUs, make -j uses them all and I don't notice any other slowdown. Having said that, it takes about the same time to run as make -j4, around 32s.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Delgan

Delgan commented Apr 29, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Author

Hi @hugovk.

It's not so much related to my machine in particular. It's just that once all the CPU cores are used for compiling, adding more jobs generally do not improve the performance and rather tend to make it worst because of the overhead of context switches 1. It's not very much noticeable in practice, though 23.

However, what is growing continuously is the RAM consumption. Each started job will consume memory, which causes a noticeable increase in overall RAM usage (finally "wasted" since an infinite number of jobs cannot run in parallel). This is not really desirable, and this is frequently related to out-of-memory problems for a beginner who is not familiar with make building process 4.

Also I suggested -j4 but maybe -j2 is preferable as it matches the example given when clicking "more detailed instructions" on the page.

Footnotes

  1. Disadvantages to high make job values ↩

  2. How to speed up compilation time in linux ↩

  3. GNU make: should the number of jobs equal the number of CPU cores in a system? ↩

  4. Is make -j (with no argument) dangerous? ↩

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@terryjreedy

terryjreedy commented Apr 29, 2023

Copy link
Copy Markdown
Member

For the test suite, -j starts a number of parallel tests appropriate to the CPU. Since, you say, make -j does not do that, why not recommend -j# where # is appropriate for the machine?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Delgan

Delgan commented Apr 29, 2023

Copy link
Copy Markdown
Author

@terryjreedy On the same page, tests are suggested to be run with ./python -m test -j3, details about the -j argument are only provided on the linked page. I suppose it's best to show a simple and functional example by default, and only go into detail on the appropriate page. It's easier to copy and paste. But in the end, it's up to what you prefer.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@willingc willingc added the needs: decision Needs consensus decision from the core team label Oct 10, 2023
@picnixz picnixz mentioned this pull request Jul 18, 2024
Explain how to add an extension module #1350
Merged
@python-cla-bot

python-cla-bot Bot commented Apr 18, 2025

Copy link
Copy Markdown

The following commit authors need to sign the Contributor License Agreement:

  • [email protected]

CLA signed

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@AA-Turner

AA-Turner commented Apr 18, 2025 •
edited by zware
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Superseeded by either #1541 or #1543 or both.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@AA-Turner AA-Turner closed this Apr 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

needs: decision Needs consensus decision from the core team type-feature Additions; New content or section needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@Delgan @hugovk @terryjreedy @AA-Turner @willingc @arhadthedev
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.