GeezerVison

I updated my PR:

• Rename usable parameter to affinity.
• Add Windows implementation using GetProcessAffinityMask().
• Add _Py_popcount64().

I don't think that my Windows implementation works with more than 64 CPUs :-(

https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-getprocessaffinitymask#return-value

It should be fixed before this PR can be merged.

Manual test on Windows, affinity.py:

import os
with open("output", "w") as fp:
    try:
        print(os.cpu_count(affinity=True), file=fp)
    except Exception as exc:
        print(f"ERROR: {exc!r}", file=fp)

Without CPU affinity (all CPUs):

vstinner@WIN C:\victor\python\main>PCbuild\amd64\python_d.exe affinity.py                    
vstinner@WIN C:\victor\python\main>type output
2

With CPU affinity (limit to 1 CPU):

vstinner@WIN C:\victor\python\main>start /affinity 01 PCbuild\amd64\python_d.exe affinity.py
vstinner@WIN C:\victor\python\main>type output
1

cc @taleinat

I mark the PR as a draft to remind myself that either I should drop Windows support for now, or I should fix Windows support, before this PR can be considered ready to be reviewed (and then merged).

As I wrote previous, my current implementation is limited to 64 CPUs on Windows which looks wrong. A Windows machine can have more than 64 CPUs: #67226 But I'm not sure if a process can be assigned to more than 64 CPUs. Well, I have to investigate :-)

I updated my PR. It's now ready for review.

Changes:

• Remove Windows implementation.
• Test that os.cpu_count(affinity=True) <= os.cpu_count(affinity=False).
• I reverted my minor test change when cpu_count() returns None.
• The doc no longer announces that os.cpu_count(affinity=True) raises an exception on error, since it falls back on os.cpu_count(affinity=False) code path is os.sched_getaffinity() is not available. It may be rephrased later when the feature will be implemented for Windows.

I mark the PR as a draft to remind myself that either I should drop Windows support for now, or I should fix Windows support, before this PR can be considered ready to be reviewed (and then merged).

I chose the easy way: remove the Windows implementation for now.

Honestly, IMO cpu_count(affinity=True) is what most users expect by default: decide how many workers (threads or CPUs) should be spawn to maximize efficiency without killing performances. If a server has 100 CPUs but Python is limited by the admin to 2 CPUS, spawning 100 worker processes is likely to kill latency and may cause many timeout issues.

Problem: changing the default behavior is wrong for different reasons. For example, a program can simply query cpu_count() to display how many logical CPUs a machine has. It doesn't matter if this program is limited to 1 CPUs or has access to all 100 CPUs of the server. It should always display 100.

PYTHONCPUCOUNT env var and -X cpu_count cmdline option sounds complementary with my change:
#109595 (comment)

If we can use PYTHONCPUCOUNT=affinity / -X cpu_count=affinity, it would fit into the first use case (spawn worker processes). It will break the second use case which will have to modify their code to use cpu_count(affinity=False). But it's different: someone sets the env var knowing consequences.

@gpshead @corona10: So, are you ok with this change? Do you think that we can continue this approach later to maybe add cgroups parameter (if it makes sense and if it is needed)? This approach fits with issue gh-109595 design, no?

So, are you ok with this change?

No objection, it will need to people who get cpu_count with affinity option :)

Same, lets do this. I'm even okay with an envvar + -X option to override the return value after our discussions (followon PR i'd assume). And if Windows only supports a return value of up to 64 for the affinity feature (that came up in one of these discussions or PRs iirc?), just document the caveat while we get guidance from windows experts on how to get a better answer there.

I marked again my PR as a draft after a discussion with @corona10. I'm sorry about this back and forth.

It seems like there is a misunderstanding about "system CPU count" and "process CPU count". Depending on the use case, you may pick one or the other. The problem is that if you consider -X cpu_count=4 option: which value should be overriden? System CPU count or process CPU count? It becomes very blurry.

So.

I created PR gh-109907 which adds a new os.process_cpu_count(). The os.cpu_count() stays unchanged: it sticks to its documentation, it returns the number of logical CPUs of the system.

• process_cpu_count(): currently gets sched_getaffinity(), but later it may read cgroups and be affected by -X cpu_count=value option.
• cpu_count(): unchanged, number of system logical CPUs.

I understood that @corona10 is unhappy about this proposition because he wants to overriden the CPU count of Python applications which currently use os.cpu_count(). I understand this use case, but there are other ways like injecting a sitecustomize scripts to get a new PYTHONCPUCOUNT variable which would simply replace the whole function:

Try sitecustomize.py:

import os
ncpu = os.environ.get('PYTHONCPUCOUNT', None)
if ncpu:
    try:
        ncpu = int(ncpu)
    except ValueError:
        print(f"WARNING: invalid PYTHONCPUCOUNT value: {ncpu!r}")
    else:
        def cpu_count():
            return ncpu
        cpu_count.__doc__ = os.cpu_count.__doc__
        os.cpu_count = cpu_count

Example:

vstinner@mona$ PYTHONPATH=$PWD ./python -c "import os; print(os.cpu_count())"
12
vstinner@mona$ PYTHONCPUCOUNT=4096 PYTHONPATH=$PWD ./python -c "import os; print(os.cpu_count())"
4096
vstinner@mona$ PYTHONCPUCOUNT=xxx PYTHONPATH=$PWD ./python -c "import os; print(os.cpu_count())"
WARNING: invalid PYTHONCPUCOUNT value: 'xxx'
12

Then put your sitecustomize.py somewhere in one of sys.path directories. It also works with a magic .pth file I suppose.

Then put your sitecustomize.py somewhere in one of sys.path directories. It also works with a magic .pth file I suppose.

No, it doesn't solve the issue what I reported, How you can inject the customer's sys.path that is already written Docker image file? User will just pull the docker image from the already stored docker container image store. From the K8S admin there is no way to control CPU count except cmdline or environment variable.

The problem is that if you consider -X cpu_count=4 option: which value should be overridden? System CPU count or process CPU count? It becomes very blurry.

When a -X or equivalent environment variable is encountered, override all return values from os.cpu_count(). That potential feature can be documented as the user explicitly asking our API to ignore whatever the OS APIs might have said.

Realize that no matter what, these new features don't solve anyones existing problems because they're having those problems on Python 3.12 and earlier which will never see these features. They already need to concoct workarounds (which are still going to work in 3.13+).

Adding an affinity= parameter or adding a new os.process_cpu_count() API is useful as a feature regardless of any possible future plans for ways to override the return values here.

Closed in favor of adding os.process_cpu_count() instead: PR gh-109907.