GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-109649: Add affinity parameter to os.cpu_count() by vstinner · Pull Request #109652 · python/cpython

gh-109649: Add affinity parameter to os.cpu_count() #109652

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vstinner wants to merge 1 commit into python:main from vstinner:cpu_count_usable

Uh oh!

There was an error while loading. Please reload this page.
Closed

gh-109649: Add affinity parameter to os.cpu_count() #109652

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
2762144
gh-109649: Add affinity parameter to os.cpu_count()
vstinner Sep 21, 2023
File filter
Filter by extension
Clear filters
Conversations
Failed to load comments.
Loading

Uh oh!

There was an error while loading. Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading

Uh oh!

There was an error while loading. Please reload this page.
Diff view
Diff view

There are no files selected for viewing

17 changes: 12 additions & 5 deletions 17 Doc/library/os.rst
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5153,14 +5153,21 @@ Miscellaneous System Information
.. availability:: Unix.


.. function:: cpu_count()
.. function:: cpu_count(*, affinity=False)

Return the number of CPUs in the system. Returns ``None`` if undetermined.
Return the number of logical CPUs in the system. Returns ``None`` if
Comment thread

@corona10 corona10 Sep 21, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have to update the documentation that it will return the number of logical CPUs for the process if the usable is True.
Because the current os.cpu_count returns the available CPU counts from the system not the process.
It's different layer.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@vstinner vstinner Sep 21, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc says:

If affinity is true, return the number of logical CPUs the current process can use.

It's not clear enough? Do you want to propose a different phrasing?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@gpshead gpshead Sep 21, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is reasonable.

I'd add "this may be less than the number of logical cpus returned by affinity=False due to OS or container limitations imposed upon the process" to make it more clear why people should want to use the affinity=True argument.

PS thanks for making it keyword only!

I do wish this API never used the term "cpu"... everything these days is really a "logical_core" and what that even means depends a lot on underlying infrastructure and platform that Python may not be able to introspect. Way too late for that though. :)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@vstinner vstinner Sep 21, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My PR adds "logical CPU" to the doc. In previous bug reports, I saw some confusion between physical CPU core, CPU packages, CPU threads, and logical CPUs.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
undetermined.

This number is not equivalent to the number of CPUs the current process can
use. The number of usable CPUs can be obtained with
``len(os.sched_getaffinity(0))``
If *affinity* is true, return the number of logical CPUs the current process
can use. If the :func:`sched_getaffinity` function is available,
return ``len(os.sched_getaffinity(0))``. Otherwise, return
``cpu_count(affinity=False)``.

Linux control groups, *cgroups*, are not taken in account to get the number
Comment thread

@gpshead gpshead Sep 21, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like people wanting such an API may also want cgroups to also be considered when the real question being answered is really "How parallel am I usefully allowed to be?".

Does that need to be separated out into its own cgroups_cpuset=True flag so that people could query one or the other or both? The use cases I have in mind are all around the above question where I'd always want the combination aka min(logical_cpus, affinity_cores, cgroups_cpuset_cores).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@vstinner vstinner Sep 21, 2023

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My PR just fix the documentation to avoid any misunderstading.

I feel like people wanting such an API may also want cgroups to also be considered

It is discussed in PR #80235. So far, nobody proposes any PR to implement this.

Maybe this PR is a baby step forward :-)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@corona10 corona10 Sep 21, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even if we decide to support cgroup in the future, I would like to propose not to use the flag name that can represent the implementation detail. If some platform suggests new things do we have to add a new flag for them?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
gpshead reacted with thumbs up emoji
All reactions
  • 👍 1 reaction
of logical CPUs.

.. versionchanged:: 3.13
Add *affinity* parameter.

.. versionadded:: 3.4

Expand Down
7 changes: 7 additions & 0 deletions 7 Doc/whatsnew/3.13.rst
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,13 @@ opcode
documented or exposed through ``dis``, and were not intended to be
used externally.

os
--

* Add *affinity* parameter to :func:`os.cpu_count` to get the number of CPUs
the current process can use.
(Contributed by Victor Stinner in :gh:`109649`.)

pathlib
-------

Expand Down
1 change: 1 addition & 0 deletions 1 Include/internal/pycore_global_objects_fini_generated.h
Show comments
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
1 change: 1 addition & 0 deletions 1 Include/internal/pycore_global_strings.h
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -273,6 +273,7 @@ struct _Py_global_strings {
STRUCT_FOR_ID(aclose)
STRUCT_FOR_ID(add)
STRUCT_FOR_ID(add_done_callback)
STRUCT_FOR_ID(affinity)
STRUCT_FOR_ID(after_in_child)
STRUCT_FOR_ID(after_in_parent)
STRUCT_FOR_ID(aggregate_class)
Expand Down
1 change: 1 addition & 0 deletions 1 Include/internal/pycore_runtime_init_generated.h
Show comments
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
3 changes: 3 additions & 0 deletions 3 Include/internal/pycore_unicodeobject_generated.h
Show comments
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
36 changes: 32 additions & 4 deletions 36 Lib/test/test_os.py
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3996,14 +3996,42 @@ def test_oserror_filename(self):
self.fail(f"No exception thrown by {func}")

class CPUCountTests(unittest.TestCase):
def check_cpu_count(self, cpus):
if cpus is None:
self.skipTest("Could not determine the number of CPUs")

self.assertIsInstance(cpus, int)
self.assertGreater(cpus, 0)

def test_cpu_count(self):
cpus = os.cpu_count()
if cpus is not None:
self.assertIsInstance(cpus, int)
self.assertGreater(cpus, 0)
else:
self.check_cpu_count(cpus)

def test_cpu_count_affinity(self):
Comment thread
vstinner marked this conversation as resolved.
Resolved

Uh oh!

There was an error while loading. Please reload this page.
cpus = os.cpu_count(affinity=True)
self.assertLessEqual(cpus, os.cpu_count())
self.check_cpu_count(cpus)

@unittest.skipUnless(hasattr(os, 'sched_setaffinity'),
"don't have sched affinity support")
def test_cpu_count_affinity_setaffinity(self):
ncpu = os.cpu_count()
if ncpu is None:
self.skipTest("Could not determine the number of CPUs")

# Disable one CPU
mask = os.sched_getaffinity(0)
if len(mask) <= 1:
self.skipTest(f"sched_getaffinity() returns less than "
f"2 CPUs: {sorted(mask)}")
self.addCleanup(os.sched_setaffinity, 0, list(mask))
mask.pop()
os.sched_setaffinity(0, mask)

# test cpu_count(affinity=True)
affinity = os.cpu_count(affinity=True)
self.assertEqual(affinity, ncpu - 1)


# FD inheritance check is only useful for systems with process support.
@support.requires_subprocess()
Expand Down
1 change: 1 addition & 0 deletions 1 Lib/test/test_posix.py
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1205,6 +1205,7 @@ def test_sched_getaffinity(self):
@requires_sched_affinity
def test_sched_setaffinity(self):
mask = posix.sched_getaffinity(0)
self.addCleanup(posix.sched_setaffinity, 0, list(mask))
if len(mask) > 1:
# Empty masks are forbidden
mask.pop()
Expand Down
2 changes: 2 additions & 0 deletions 2 Misc/NEWS.d/next/Library/2023-09-21-16-21-19.gh-issue-109649.YYCjAF.rst
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Add *affinity* parameter to :func:`os.cpu_count` to get the number of CPUs the
current process can use. Patch by Victor Stinner.
61 changes: 52 additions & 9 deletions 61 Modules/clinic/posixmodule.c.h
Show comments
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
111 changes: 79 additions & 32 deletions 111 Modules/posixmodule.c
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8138,39 +8138,45 @@ static PyObject *
os_sched_getaffinity_impl(PyObject *module, pid_t pid)
/*[clinic end generated code: output=f726f2c193c17a4f input=983ce7cb4a565980]*/
{
int cpu, ncpus, count;
int ncpus = NCPUS_START;
size_t setsize;
cpu_set_t *mask = NULL;
PyObject *res = NULL;
cpu_set_t *mask;

ncpus = NCPUS_START;
while (1) {
setsize = CPU_ALLOC_SIZE(ncpus);
mask = CPU_ALLOC(ncpus);
if (mask == NULL)
if (mask == NULL) {
return PyErr_NoMemory();
if (sched_getaffinity(pid, setsize, mask) == 0)
}
if (sched_getaffinity(pid, setsize, mask) == 0) {
break;
}
CPU_FREE(mask);
if (errno != EINVAL)
if (errno != EINVAL) {
return posix_error();
}
if (ncpus > INT_MAX / 2) {
PyErr_SetString(PyExc_OverflowError, "could not allocate "
"a large enough CPU set");
PyErr_SetString(PyExc_OverflowError,
"could not allocate a large enough CPU set");
return NULL;
}
ncpus = ncpus * 2;
ncpus *= 2;
}

res = PySet_New(NULL);
if (res == NULL)
PyObject *res = PySet_New(NULL);
if (res == NULL) {
goto error;
for (cpu = 0, count = CPU_COUNT_S(setsize, mask); count; cpu++) {
}

int cpu = 0;
int count = CPU_COUNT_S(setsize, mask);
for (; count; cpu++) {
if (CPU_ISSET_S(cpu, setsize, mask)) {
PyObject *cpu_num = PyLong_FromLong(cpu);
--count;
if (cpu_num == NULL)
if (cpu_num == NULL) {
goto error;
}
if (PySet_Add(res, cpu_num)) {
Py_DECREF(cpu_num);
goto error;
Expand All @@ -8182,12 +8188,12 @@ os_sched_getaffinity_impl(PyObject *module, pid_t pid)
return res;

error:
if (mask)
if (mask) {
CPU_FREE(mask);
}
Py_XDECREF(res);
return NULL;
}

#endif /* HAVE_SCHED_SETAFFINITY */

#endif /* HAVE_SCHED_H */
Expand Down Expand Up @@ -14338,44 +14344,85 @@ os_get_terminal_size_impl(PyObject *module, int fd)
/*[clinic input]
os.cpu_count

*
affinity: bool = False

Return the number of CPUs in the system; return None if indeterminable.

This number is not equivalent to the number of CPUs the current process can
use. The number of usable CPUs can be obtained with
``len(os.sched_getaffinity(0))``
If 'affinity' is true, return the number of CPUs the current process can use.
[clinic start generated code]*/

static PyObject *
os_cpu_count_impl(PyObject *module)
/*[clinic end generated code: output=5fc29463c3936a9c input=e7c8f4ba6dbbadd3]*/
{
int ncpu = 0;
os_cpu_count_impl(PyObject *module, int affinity)
/*[clinic end generated code: output=0cd2ead51703a781 input=8b61eda84766f638]*/
{
int ncpu;
if (affinity) {
#if defined(HAVE_SCHED_H) && defined(HAVE_SCHED_SETAFFINITY)
int ncpus = NCPUS_START;
cpu_set_t *mask;
size_t setsize;

while (1) {
setsize = CPU_ALLOC_SIZE(ncpus);
mask = CPU_ALLOC(ncpus);
if (mask == NULL) {
return PyErr_NoMemory();
}
if (sched_getaffinity(0, setsize, mask) == 0) {
break;
}
CPU_FREE(mask);
if (errno != EINVAL) {
return posix_error();
}
if (ncpus > INT_MAX / 2) {
PyErr_SetString(PyExc_OverflowError,
"could not allocate a large enough CPU set");
return NULL;
}
ncpus *= 2;
}

ncpu = CPU_COUNT_S(setsize, mask);
CPU_FREE(mask);
return PyLong_FromLong(ncpu);
#endif
}

#ifdef MS_WINDOWS
#ifdef MS_WINDOWS_DESKTOP
# ifdef MS_WINDOWS_DESKTOP
ncpu = GetActiveProcessorCount(ALL_PROCESSOR_GROUPS);
#endif
# else
ncpu = 0;
# endif

#elif defined(__hpux)
ncpu = mpctl(MPC_GETNUMSPUS, NULL, NULL);

#elif defined(HAVE_SYSCONF) && defined(_SC_NPROCESSORS_ONLN)
ncpu = sysconf(_SC_NPROCESSORS_ONLN);

#elif defined(__VXWORKS__)
ncpu = _Py_popcount32(vxCpuEnabledGet());

#elif defined(__DragonFly__) || \
defined(__OpenBSD__) || \
defined(__FreeBSD__) || \
defined(__NetBSD__) || \
defined(__APPLE__)
int mib[2];
ncpu = 0;
size_t len = sizeof(ncpu);
mib[0] = CTL_HW;
mib[1] = HW_NCPU;
if (sysctl(mib, 2, &ncpu, &len, NULL, 0) != 0)
int mib[2] = {CTL_HW, HW_NCPU};
if (sysctl(mib, 2, &ncpu, &len, NULL, 0) != 0) {
ncpu = 0;
}
#endif
if (ncpu >= 1)
return PyLong_FromLong(ncpu);
else

if (ncpu < 1) {
Py_RETURN_NONE;
}
return PyLong_FromLong(ncpu);
}


Expand Down
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.