GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — [3.14] gh-139653: Add PyUnstable_ThreadState_SetStackProtection() (GH-139668) by encukou · Pull Request #141661 · python/cpython

Conversation

@encukou

@encukou encukou commented Nov 17, 2025 •
edited by github-actions Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This backports recent changes to the stack protection mechanism, and the unstable API to control it:

  • gh-139231: Fix estimation of available stack size for recursion limit on macOS #139232
  • GH-139291: Fix C stack limits by factoring out finding hardware stack limits #139294
  • gh-139653: Add PyUnstable_ThreadState_SetStackProtection() #139668
  • gh-139653: Remove assertions in _Py_InitializeRecursionLimits() #141551
  • Issue: Python 3.14 stack overflow detection is incompatible with C++ Boost make_fcontext() coroutines #139653

📚 Documentation preview 📚: https://cpython-previews--141661.org.readthedocs.build/

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
rokm and others added 5 commits November 17, 2025 14:16
@rokm @encukou
pythongh-139231: Fix estimation of available stack size for recursion…
7752629
… limit on macOS (pythonGH-139232)

Use `pthread_get_stackaddr_np()` and `pthread_get_stacksize_np()` to determine the stack address and size.
@markshannon @encukou
pythonGH-139291: Fix C stack limits by factoring out finding hardware…
0300367
… stack limits (pythonGH-139294)
@encukou
pythongh-139653: Add PyUnstable_ThreadState_SetStackProtection() (pyt…
2cca51f
…honGH-139668)

Add PyUnstable_ThreadState_SetStackProtection() and
PyUnstable_ThreadState_ResetStackProtection() functions
to set the stack base address and stack size of a Python
thread state.

Co-authored-by: Petr Viktorin <[email protected]>
@vstinner @encukou
pythongh-139653: Remove assertions in _Py_InitializeRecursionLimits() (…
5d2c92c
…pythonGH-141551)

These checks were invalid and failed randomly on FreeBSD
and Alpine Linux.
@encukou
Add What's New & compatibility warnings
fa01bf8
@encukou
encukou requested a review from vstinner November 17, 2025 13:32
@encukou
encukou requested review from ericsnowcurrently and markshannon as code owners November 17, 2025 13:32
@bedevere-app bedevere-app Bot added the awaiting core review label Nov 17, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Nov 17, 2025
Python 3.14 stack overflow detection is incompatible with C++ Boost make_fcontext() coroutines #139653
Open
@encukou encukou changed the title gh-139653: Add PyUnstable_ThreadState_SetStackProtection() (GH-139668) [3.14] gh-139653: Add PyUnstable_ThreadState_SetStackProtection() (GH-139668) Nov 17, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Nov 17, 2025
gh-139653: Add PyUnstable_ThreadState_SetStackProtection() #139668
Merged
encukou added 2 commits November 17, 2025 14:35
@encukou
Put new fields at the end of _PyThreadStateImpl
24858cf
@encukou
Regenerate the ABI dump
ecbf10e
@encukou
encukou requested a review from hugovk November 17, 2025 13:50
@encukou

encukou commented Nov 17, 2025

Copy link
Copy Markdown
Member Author

@hugovk, this will change the internal ABI.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Nov 17, 2025

Copy link
Copy Markdown
Member

I would prefer to wait to see how the discussion https://discuss.python.org/t/python-3-14-0-is-incompatible-with-stack-switching-systems-what-do-we-do/104880 goes and see if this API solves the issue, or if we need another solution.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Nov 17, 2025

Copy link
Copy Markdown
Member

If this API cannot be used by most projects and a better fix can be designed, I would even suggest to remove this API from the main branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@encukou encukou added the release-blocker label Nov 19, 2025
@github-project-automation github-project-automation Bot added this to Release and Deferred blockers 🚫 Nov 19, 2025
@github-project-automation github-project-automation Bot moved this to Todo in Release and Deferred blockers 🚫 Nov 19, 2025
@vstinner

vstinner commented Nov 24, 2025

Copy link
Copy Markdown
Member

#141404 has been merged causing conflicts.

Can you please update your PR to solve conflicts?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner vstinner mentioned this pull request Nov 24, 2025
[3.14] GH-139653: Only raise an exception (or fatal error) when the stack pointer is about to overflow the stack. (GH-141711) #141892
Closed
vstinner
vstinner reviewed Nov 24, 2025
View reviewed changes
Comment thread Doc/c-api/init.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Nov 24, 2025

Copy link
Copy Markdown
Member

I would prefer to wait to see how the discussion https://discuss.python.org/t/python-3-14-0-is-incompatible-with-stack-switching-systems-what-do-we-do/104880 goes and see if this API solves the issue, or if we need another solution.

If this API cannot be used by most projects and a better fix can be designed, I would even suggest to remove this API from the main branch.

#141711 has been merged: it should fix #139653 for most impacted projects. But according to @markshannon, PyUnstable_ThreadState_SetStackProtection() remains useful for these projects. So let's backport this function to the 3.14 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

encukou added 2 commits November 25, 2025 13:36
@encukou
Merge in the 3.14 branch
c3b1186
@encukou
… will need explicit confiuration
7580763
vstinner
vstinner approved these changes Nov 25, 2025
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Nov 25, 2025
@encukou
encukou merged commit 32a38a2 into python:3.14 Nov 25, 2025
57 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting merge label Nov 25, 2025
@github-project-automation github-project-automation Bot moved this from Todo to Done in Release and Deferred blockers 🚫 Nov 25, 2025
@encukou
encukou deleted the backport-b99db92-3.14 branch November 25, 2025 13:21
@hroncok hroncok mentioned this pull request Nov 25, 2025
[3.14] GH-139653: Only raise an exception (or fatal error) when the stack pointer is about to overflow the stack. (GH-141711) #141944
Merged
@pseudo-rnd-thoughts pseudo-rnd-thoughts mentioned this pull request Jul 15, 2026
[Core] Fix Python 3.14 async-actor memory leak by re-anchoring stack protection to fiber stacks ray-project/ray#64772
Merged
MengjinYan added a commit to ray-project/ray that referenced this pull request Aug 3, 2026
@pseudo-rnd-thoughts Mark Towers @MengjinYan
[Core] Fix Python 3.14 async-actor memory leak by re-anchoring stack …
35591ba
…protection to fiber stacks (#64772)

# Description

On Python 3.14 + Linux, every async-actor task permanently leaks ~518
KiB of live malloc (the per-task `asyncio.Task`,
`concurrent.futures.Future`, Cython coroutine + scopes, and two msgpack
`Packer`s with 256 KiB internal buffers).
Closes #63290

### Root cause

**1. CPython 3.14 changed how it avoids stack overflow when freeing
objects.** Freeing one object can recursively free many others (a dict
frees its values, which free their contents, …), and each level is a
nested C call. To keep that from overflowing the C stack, CPython has
long had a safety mechanism (the "trashcan"): when it decides it's too
deep, it doesn't free the object right away. Instead it parks the object
on a per-thread *delete-later* list and drains the list once there's
stack headroom again. Up to 3.13, "too deep" was a simple recursion
counter. In 3.14 it's decided by comparing the actual machine **stack
pointer** against the stack bounds CPython recorded for the thread when
it attached (from pthreads, on Linux).

**2. Ray async actors don't run task code on the thread's normal
stack.** Each task executes on a small 256 KiB boost fiber stack
allocated elsewhere in memory. The problem is that CPython still thinks
the thread runs on its original pthread stack.

So while a task runs on a fiber, every "am I near the stack limit?"
check compares the fiber's stack pointer against the *pthread* stack's
bounds. On Linux, fiber stacks happen to be allocated at lower addresses
than the pthread stack, so CPython concludes the stack is hopelessly
overflowed and parks **every** object freed during the task (including
return-value serialization and end-of-task cleanup) on the delete-later
list.

That list is only ever drained by a later free on the same thread state
at a healthy stack margin, which never happens here as the Ray thread
only runs on fibers and Ray creates a fresh Python thread state per task
and destroys it at task end. This means that CPython destroys a thread
state **without draining its delete-later list** and the parked objects
are orphaned permanently. That's the leak.

Why the confusing symptoms:

- `boost::make_fcontext` in the issue's flamegraphs just marks *where*
the leaked allocations were made (on a fiber stack); the fiber stacks
themselves are freed correctly.
- macOS is unaffected only by luck: fiber stacks there land at *higher*
addresses than the pthread stack, so the check passes.
- 3.13 and earlier are unaffected because their trashcan uses the
counter, not the stack pointer.

### Fix

CPython 3.14.2 added an official API for exactly this situation:
`PyUnstable_ThreadState_SetStackProtection` (python/cpython#141661) lets
an embedder tell CPython "this thread is currently executing on *this*
stack." We call it with the fiber's stack bounds:

- at async-actor task entry in `task_execution_handler`, and
- whenever a fiber resumes after `YieldCurrentFiber` (concurrent fibers
share the thread state, so each must re-register its own stack).

With the bounds correct, the near-limit check returns to normal
behavior: objects are freed immediately, and the rare genuinely-deep
free is parked and then properly drained.

Implementation notes: the symbol is looked up via `dlsym`, so `_raylet`
still imports on 3.14.0/3.14.1 (fix skipped there; those releases have a
more severe, since-fixed stack-check bug anyway, python/cpython#141944).
No-op below 3.14 (preprocessor-gated) and on Windows. Stack bounds are
derived from the current stack pointer minus a conservative allowance
for stack already used, so the protection errs toward triggering
slightly early rather than missing an overflow. Side benefit: fibers
gain real C-stack overflow protection (RecursionError) on 3.14, which
they currently lack entirely (`boost::fibers::fixedsize_stack` has no
guard pages). Also makes `FiberState::kStackSize` public so the
anchoring uses the real fiber stack size.

## Related issue number

Closes #63290. Supersedes #63284 (same diagnosis direction, but
hand-rolled `_PyThreadStateImpl` offsets, a deliberate
`gilstate_counter` leak that freezes non-main threads, and a crash
premise that CPython 3.14.2 already fixed upstream).

## Checks

- Verified with a locally built cp314 Linux (aarch64, python:3.14.6
docker) wheel:
- refcount probe: **+4.00 refs/task → 0.00/task** (100 tasks)
- `__del__` deferral probe: dealloc during return serialization on the
fiber **deferred → immediate**
- live-malloc probe (`mallinfo2`, 300 tasks/shape): **~518 KiB/task → ~3
KiB/task** across async call → dict/bytes, async generator, sync
generator on async actor
- reporter-shaped streaming workload (400 tasks, 10 concurrent
sessions): live-malloc delta **0.2 MB total**, fiber-sized mapped
regions 0 → 0
- async-actor smoke: correctness (echo, state, async generators,
recursion), concurrency (20 overlapping 0.5 s sleeps in 0.51 s)
- throughput A/B (500 sequential echo tasks, 3 runs fixed / 2 runs
baseline, same container image): fixed 5624–6076 tasks/s vs unpatched
4551–4825 tasks/s meaning no regression (the unpatched build is slower
while leaking)
- baseline (unpatched) wheel from the same tree reproduces the bug:
+4.00 refs/task, fiber dealloc deferred=True

note: fable did a majority of the heavy lifting in this investigation
with prompting on what to check next and validate the solution

---------

Signed-off-by: Mark Towers <[email protected]>
Signed-off-by: myan <[email protected]>
Co-authored-by: Mark Towers <[email protected]>
Co-authored-by: myan <[email protected]>
Co-authored-by: Mengjin Yan <[email protected]>
@elliot-barn elliot-barn mentioned this pull request Aug 3, 2026
[Core] Fix Python 3.14 async-actor memory leak by re-anchoring stack … ray-project/ray#65177
Merged
elliot-barn added a commit to ray-project/ray that referenced this pull request Aug 4, 2026
@elliot-barn @pseudo-rnd-thoughts
Mark Towers @MengjinYan
[Core] Fix Python 3.14 async-actor memory leak by re-anchoring stack … (
8778646
#65177)

…protection to fiber stacks (#64772)

# Description

On Python 3.14 + Linux, every async-actor task permanently leaks ~518
KiB of live malloc (the per-task `asyncio.Task`,
`concurrent.futures.Future`, Cython coroutine + scopes, and two msgpack
`Packer`s with 256 KiB internal buffers).
Closes #63290

### Root cause

**1. CPython 3.14 changed how it avoids stack overflow when freeing
objects.** Freeing one object can recursively free many others (a dict
frees its values, which free their contents, …), and each level is a
nested C call. To keep that from overflowing the C stack, CPython has
long had a safety mechanism (the "trashcan"): when it decides it's too
deep, it doesn't free the object right away. Instead it parks the object
on a per-thread *delete-later* list and drains the list once there's
stack headroom again. Up to 3.13, "too deep" was a simple recursion
counter. In 3.14 it's decided by comparing the actual machine **stack
pointer** against the stack bounds CPython recorded for the thread when
it attached (from pthreads, on Linux).

**2. Ray async actors don't run task code on the thread's normal
stack.** Each task executes on a small 256 KiB boost fiber stack
allocated elsewhere in memory. The problem is that CPython still thinks
the thread runs on its original pthread stack.

So while a task runs on a fiber, every "am I near the stack limit?"
check compares the fiber's stack pointer against the *pthread* stack's
bounds. On Linux, fiber stacks happen to be allocated at lower addresses
than the pthread stack, so CPython concludes the stack is hopelessly
overflowed and parks **every** object freed during the task (including
return-value serialization and end-of-task cleanup) on the delete-later
list.

That list is only ever drained by a later free on the same thread state
at a healthy stack margin, which never happens here as the Ray thread
only runs on fibers and Ray creates a fresh Python thread state per task
and destroys it at task end. This means that CPython destroys a thread
state **without draining its delete-later list** and the parked objects
are orphaned permanently. That's the leak.

Why the confusing symptoms:

- `boost::make_fcontext` in the issue's flamegraphs just marks *where*
the leaked allocations were made (on a fiber stack); the fiber stacks
themselves are freed correctly.
- macOS is unaffected only by luck: fiber stacks there land at *higher*
addresses than the pthread stack, so the check passes.
- 3.13 and earlier are unaffected because their trashcan uses the
counter, not the stack pointer.

### Fix

CPython 3.14.2 added an official API for exactly this situation:
`PyUnstable_ThreadState_SetStackProtection` (python/cpython#141661) lets
an embedder tell CPython "this thread is currently executing on *this*
stack." We call it with the fiber's stack bounds:

- at async-actor task entry in `task_execution_handler`, and
- whenever a fiber resumes after `YieldCurrentFiber` (concurrent fibers
share the thread state, so each must re-register its own stack).

With the bounds correct, the near-limit check returns to normal
behavior: objects are freed immediately, and the rare genuinely-deep
free is parked and then properly drained.

Implementation notes: the symbol is looked up via `dlsym`, so `_raylet`
still imports on 3.14.0/3.14.1 (fix skipped there; those releases have a
more severe, since-fixed stack-check bug anyway, python/cpython#141944).
No-op below 3.14 (preprocessor-gated) and on Windows. Stack bounds are
derived from the current stack pointer minus a conservative allowance
for stack already used, so the protection errs toward triggering
slightly early rather than missing an overflow. Side benefit: fibers
gain real C-stack overflow protection (RecursionError) on 3.14, which
they currently lack entirely (`boost::fibers::fixedsize_stack` has no
guard pages). Also makes `FiberState::kStackSize` public so the
anchoring uses the real fiber stack size.

## Related issue number

Closes #63290. Supersedes #63284 (same diagnosis direction, but
hand-rolled `_PyThreadStateImpl` offsets, a deliberate
`gilstate_counter` leak that freezes non-main threads, and a crash
premise that CPython 3.14.2 already fixed upstream).

## Checks

- Verified with a locally built cp314 Linux (aarch64, python:3.14.6
docker) wheel:
- refcount probe: **+4.00 refs/task → 0.00/task** (100 tasks)
- `__del__` deferral probe: dealloc during return serialization on the
fiber **deferred → immediate**
- live-malloc probe (`mallinfo2`, 300 tasks/shape): **~518 KiB/task → ~3
KiB/task** across async call → dict/bytes, async generator, sync
generator on async actor
- reporter-shaped streaming workload (400 tasks, 10 concurrent
sessions): live-malloc delta **0.2 MB total**, fiber-sized mapped
regions 0 → 0
- async-actor smoke: correctness (echo, state, async generators,
recursion), concurrency (20 overlapping 0.5 s sleeps in 0.51 s)
- throughput A/B (500 sequential echo tasks, 3 runs fixed / 2 runs
baseline, same container image): fixed 5624–6076 tasks/s vs unpatched
4551–4825 tasks/s meaning no regression (the unpatched build is slower
while leaking)
- baseline (unpatched) wheel from the same tree reproduces the bug:
+4.00 refs/task, fiber dealloc deferred=True

note: fable did a majority of the heavy lifting in this investigation
with prompting on what to check next and validate the solution

---------






(cherry picked from commit 35591ba)

Signed-off-by: Mark Towers <[email protected]>
Signed-off-by: myan <[email protected]>
Signed-off-by: elliot-barn <[email protected]>
Co-authored-by: Mark Towers <[email protected]>
Co-authored-by: Mark Towers <[email protected]>
Co-authored-by: myan <[email protected]>
Co-authored-by: Mengjin Yan <[email protected]>
Artimislyy pushed a commit to Artimislyy/ray that referenced this pull request Aug 11, 2026
@pseudo-rnd-thoughts Mark Towers
@MengjinYan @artimislyy-netizen
[Core] Fix Python 3.14 async-actor memory leak by re-anchoring stack …
27a040d
…protection to fiber stacks (ray-project#64772)

# Description

On Python 3.14 + Linux, every async-actor task permanently leaks ~518
KiB of live malloc (the per-task `asyncio.Task`,
`concurrent.futures.Future`, Cython coroutine + scopes, and two msgpack
`Packer`s with 256 KiB internal buffers).
Closes ray-project#63290

### Root cause

**1. CPython 3.14 changed how it avoids stack overflow when freeing
objects.** Freeing one object can recursively free many others (a dict
frees its values, which free their contents, …), and each level is a
nested C call. To keep that from overflowing the C stack, CPython has
long had a safety mechanism (the "trashcan"): when it decides it's too
deep, it doesn't free the object right away. Instead it parks the object
on a per-thread *delete-later* list and drains the list once there's
stack headroom again. Up to 3.13, "too deep" was a simple recursion
counter. In 3.14 it's decided by comparing the actual machine **stack
pointer** against the stack bounds CPython recorded for the thread when
it attached (from pthreads, on Linux).

**2. Ray async actors don't run task code on the thread's normal
stack.** Each task executes on a small 256 KiB boost fiber stack
allocated elsewhere in memory. The problem is that CPython still thinks
the thread runs on its original pthread stack.

So while a task runs on a fiber, every "am I near the stack limit?"
check compares the fiber's stack pointer against the *pthread* stack's
bounds. On Linux, fiber stacks happen to be allocated at lower addresses
than the pthread stack, so CPython concludes the stack is hopelessly
overflowed and parks **every** object freed during the task (including
return-value serialization and end-of-task cleanup) on the delete-later
list.

That list is only ever drained by a later free on the same thread state
at a healthy stack margin, which never happens here as the Ray thread
only runs on fibers and Ray creates a fresh Python thread state per task
and destroys it at task end. This means that CPython destroys a thread
state **without draining its delete-later list** and the parked objects
are orphaned permanently. That's the leak.

Why the confusing symptoms:

- `boost::make_fcontext` in the issue's flamegraphs just marks *where*
the leaked allocations were made (on a fiber stack); the fiber stacks
themselves are freed correctly.
- macOS is unaffected only by luck: fiber stacks there land at *higher*
addresses than the pthread stack, so the check passes.
- 3.13 and earlier are unaffected because their trashcan uses the
counter, not the stack pointer.

### Fix

CPython 3.14.2 added an official API for exactly this situation:
`PyUnstable_ThreadState_SetStackProtection` (python/cpython#141661) lets
an embedder tell CPython "this thread is currently executing on *this*
stack." We call it with the fiber's stack bounds:

- at async-actor task entry in `task_execution_handler`, and
- whenever a fiber resumes after `YieldCurrentFiber` (concurrent fibers
share the thread state, so each must re-register its own stack).

With the bounds correct, the near-limit check returns to normal
behavior: objects are freed immediately, and the rare genuinely-deep
free is parked and then properly drained.

Implementation notes: the symbol is looked up via `dlsym`, so `_raylet`
still imports on 3.14.0/3.14.1 (fix skipped there; those releases have a
more severe, since-fixed stack-check bug anyway, python/cpython#141944).
No-op below 3.14 (preprocessor-gated) and on Windows. Stack bounds are
derived from the current stack pointer minus a conservative allowance
for stack already used, so the protection errs toward triggering
slightly early rather than missing an overflow. Side benefit: fibers
gain real C-stack overflow protection (RecursionError) on 3.14, which
they currently lack entirely (`boost::fibers::fixedsize_stack` has no
guard pages). Also makes `FiberState::kStackSize` public so the
anchoring uses the real fiber stack size.

## Related issue number

Closes ray-project#63290. Supersedes ray-project#63284 (same diagnosis direction, but
hand-rolled `_PyThreadStateImpl` offsets, a deliberate
`gilstate_counter` leak that freezes non-main threads, and a crash
premise that CPython 3.14.2 already fixed upstream).

## Checks

- Verified with a locally built cp314 Linux (aarch64, python:3.14.6
docker) wheel:
- refcount probe: **+4.00 refs/task → 0.00/task** (100 tasks)
- `__del__` deferral probe: dealloc during return serialization on the
fiber **deferred → immediate**
- live-malloc probe (`mallinfo2`, 300 tasks/shape): **~518 KiB/task → ~3
KiB/task** across async call → dict/bytes, async generator, sync
generator on async actor
- reporter-shaped streaming workload (400 tasks, 10 concurrent
sessions): live-malloc delta **0.2 MB total**, fiber-sized mapped
regions 0 → 0
- async-actor smoke: correctness (echo, state, async generators,
recursion), concurrency (20 overlapping 0.5 s sleeps in 0.51 s)
- throughput A/B (500 sequential echo tasks, 3 runs fixed / 2 runs
baseline, same container image): fixed 5624–6076 tasks/s vs unpatched
4551–4825 tasks/s meaning no regression (the unpatched build is slower
while leaking)
- baseline (unpatched) wheel from the same tree reproduces the bug:
+4.00 refs/task, fiber dealloc deferred=True

note: fable did a majority of the heavy lifting in this investigation
with prompting on what to check next and validate the solution

---------

Signed-off-by: Mark Towers <[email protected]>
Signed-off-by: myan <[email protected]>
Co-authored-by: Mark Towers <[email protected]>
Co-authored-by: myan <[email protected]>
Co-authored-by: Mengjin Yan <[email protected]>
400Ping pushed a commit to 400Ping/ray that referenced this pull request Aug 18, 2026
@pseudo-rnd-thoughts Mark Towers
@MengjinYan @400Ping
[Core] Fix Python 3.14 async-actor memory leak by re-anchoring stack …
7a18304
…protection to fiber stacks (ray-project#64772)

# Description

On Python 3.14 + Linux, every async-actor task permanently leaks ~518
KiB of live malloc (the per-task `asyncio.Task`,
`concurrent.futures.Future`, Cython coroutine + scopes, and two msgpack
`Packer`s with 256 KiB internal buffers).
Closes ray-project#63290

### Root cause

**1. CPython 3.14 changed how it avoids stack overflow when freeing
objects.** Freeing one object can recursively free many others (a dict
frees its values, which free their contents, …), and each level is a
nested C call. To keep that from overflowing the C stack, CPython has
long had a safety mechanism (the "trashcan"): when it decides it's too
deep, it doesn't free the object right away. Instead it parks the object
on a per-thread *delete-later* list and drains the list once there's
stack headroom again. Up to 3.13, "too deep" was a simple recursion
counter. In 3.14 it's decided by comparing the actual machine **stack
pointer** against the stack bounds CPython recorded for the thread when
it attached (from pthreads, on Linux).

**2. Ray async actors don't run task code on the thread's normal
stack.** Each task executes on a small 256 KiB boost fiber stack
allocated elsewhere in memory. The problem is that CPython still thinks
the thread runs on its original pthread stack.

So while a task runs on a fiber, every "am I near the stack limit?"
check compares the fiber's stack pointer against the *pthread* stack's
bounds. On Linux, fiber stacks happen to be allocated at lower addresses
than the pthread stack, so CPython concludes the stack is hopelessly
overflowed and parks **every** object freed during the task (including
return-value serialization and end-of-task cleanup) on the delete-later
list.

That list is only ever drained by a later free on the same thread state
at a healthy stack margin, which never happens here as the Ray thread
only runs on fibers and Ray creates a fresh Python thread state per task
and destroys it at task end. This means that CPython destroys a thread
state **without draining its delete-later list** and the parked objects
are orphaned permanently. That's the leak.

Why the confusing symptoms:

- `boost::make_fcontext` in the issue's flamegraphs just marks *where*
the leaked allocations were made (on a fiber stack); the fiber stacks
themselves are freed correctly.
- macOS is unaffected only by luck: fiber stacks there land at *higher*
addresses than the pthread stack, so the check passes.
- 3.13 and earlier are unaffected because their trashcan uses the
counter, not the stack pointer.

### Fix

CPython 3.14.2 added an official API for exactly this situation:
`PyUnstable_ThreadState_SetStackProtection` (python/cpython#141661) lets
an embedder tell CPython "this thread is currently executing on *this*
stack." We call it with the fiber's stack bounds:

- at async-actor task entry in `task_execution_handler`, and
- whenever a fiber resumes after `YieldCurrentFiber` (concurrent fibers
share the thread state, so each must re-register its own stack).

With the bounds correct, the near-limit check returns to normal
behavior: objects are freed immediately, and the rare genuinely-deep
free is parked and then properly drained.

Implementation notes: the symbol is looked up via `dlsym`, so `_raylet`
still imports on 3.14.0/3.14.1 (fix skipped there; those releases have a
more severe, since-fixed stack-check bug anyway, python/cpython#141944).
No-op below 3.14 (preprocessor-gated) and on Windows. Stack bounds are
derived from the current stack pointer minus a conservative allowance
for stack already used, so the protection errs toward triggering
slightly early rather than missing an overflow. Side benefit: fibers
gain real C-stack overflow protection (RecursionError) on 3.14, which
they currently lack entirely (`boost::fibers::fixedsize_stack` has no
guard pages). Also makes `FiberState::kStackSize` public so the
anchoring uses the real fiber stack size.

## Related issue number

Closes ray-project#63290. Supersedes ray-project#63284 (same diagnosis direction, but
hand-rolled `_PyThreadStateImpl` offsets, a deliberate
`gilstate_counter` leak that freezes non-main threads, and a crash
premise that CPython 3.14.2 already fixed upstream).

## Checks

- Verified with a locally built cp314 Linux (aarch64, python:3.14.6
docker) wheel:
- refcount probe: **+4.00 refs/task → 0.00/task** (100 tasks)
- `__del__` deferral probe: dealloc during return serialization on the
fiber **deferred → immediate**
- live-malloc probe (`mallinfo2`, 300 tasks/shape): **~518 KiB/task → ~3
KiB/task** across async call → dict/bytes, async generator, sync
generator on async actor
- reporter-shaped streaming workload (400 tasks, 10 concurrent
sessions): live-malloc delta **0.2 MB total**, fiber-sized mapped
regions 0 → 0
- async-actor smoke: correctness (echo, state, async generators,
recursion), concurrency (20 overlapping 0.5 s sleeps in 0.51 s)
- throughput A/B (500 sequential echo tasks, 3 runs fixed / 2 runs
baseline, same container image): fixed 5624–6076 tasks/s vs unpatched
4551–4825 tasks/s meaning no regression (the unpatched build is slower
while leaking)
- baseline (unpatched) wheel from the same tree reproduces the bug:
+4.00 refs/task, fiber dealloc deferred=True

note: fable did a majority of the heavy lifting in this investigation
with prompting on what to check next and validate the solution

---------

Signed-off-by: Mark Towers <[email protected]>
Signed-off-by: myan <[email protected]>
Co-authored-by: Mark Towers <[email protected]>
Co-authored-by: myan <[email protected]>
Co-authored-by: Mengjin Yan <[email protected]>
Signed-off-by: 400Ping <[email protected]>
nadongjun pushed a commit to nadongjun/ray that referenced this pull request Oct 6, 2026
@pseudo-rnd-thoughts Mark Towers @MengjinYan
[Core] Fix Python 3.14 async-actor memory leak by re-anchoring stack …
5654983
…protection to fiber stacks (ray-project#64772)

# Description

On Python 3.14 + Linux, every async-actor task permanently leaks ~518
KiB of live malloc (the per-task `asyncio.Task`,
`concurrent.futures.Future`, Cython coroutine + scopes, and two msgpack
`Packer`s with 256 KiB internal buffers).
Closes ray-project#63290

### Root cause

**1. CPython 3.14 changed how it avoids stack overflow when freeing
objects.** Freeing one object can recursively free many others (a dict
frees its values, which free their contents, …), and each level is a
nested C call. To keep that from overflowing the C stack, CPython has
long had a safety mechanism (the "trashcan"): when it decides it's too
deep, it doesn't free the object right away. Instead it parks the object
on a per-thread *delete-later* list and drains the list once there's
stack headroom again. Up to 3.13, "too deep" was a simple recursion
counter. In 3.14 it's decided by comparing the actual machine **stack
pointer** against the stack bounds CPython recorded for the thread when
it attached (from pthreads, on Linux).

**2. Ray async actors don't run task code on the thread's normal
stack.** Each task executes on a small 256 KiB boost fiber stack
allocated elsewhere in memory. The problem is that CPython still thinks
the thread runs on its original pthread stack.

So while a task runs on a fiber, every "am I near the stack limit?"
check compares the fiber's stack pointer against the *pthread* stack's
bounds. On Linux, fiber stacks happen to be allocated at lower addresses
than the pthread stack, so CPython concludes the stack is hopelessly
overflowed and parks **every** object freed during the task (including
return-value serialization and end-of-task cleanup) on the delete-later
list.

That list is only ever drained by a later free on the same thread state
at a healthy stack margin, which never happens here as the Ray thread
only runs on fibers and Ray creates a fresh Python thread state per task
and destroys it at task end. This means that CPython destroys a thread
state **without draining its delete-later list** and the parked objects
are orphaned permanently. That's the leak.

Why the confusing symptoms:

- `boost::make_fcontext` in the issue's flamegraphs just marks *where*
the leaked allocations were made (on a fiber stack); the fiber stacks
themselves are freed correctly.
- macOS is unaffected only by luck: fiber stacks there land at *higher*
addresses than the pthread stack, so the check passes.
- 3.13 and earlier are unaffected because their trashcan uses the
counter, not the stack pointer.

### Fix

CPython 3.14.2 added an official API for exactly this situation:
`PyUnstable_ThreadState_SetStackProtection` (python/cpython#141661) lets
an embedder tell CPython "this thread is currently executing on *this*
stack." We call it with the fiber's stack bounds:

- at async-actor task entry in `task_execution_handler`, and
- whenever a fiber resumes after `YieldCurrentFiber` (concurrent fibers
share the thread state, so each must re-register its own stack).

With the bounds correct, the near-limit check returns to normal
behavior: objects are freed immediately, and the rare genuinely-deep
free is parked and then properly drained.

Implementation notes: the symbol is looked up via `dlsym`, so `_raylet`
still imports on 3.14.0/3.14.1 (fix skipped there; those releases have a
more severe, since-fixed stack-check bug anyway, python/cpython#141944).
No-op below 3.14 (preprocessor-gated) and on Windows. Stack bounds are
derived from the current stack pointer minus a conservative allowance
for stack already used, so the protection errs toward triggering
slightly early rather than missing an overflow. Side benefit: fibers
gain real C-stack overflow protection (RecursionError) on 3.14, which
they currently lack entirely (`boost::fibers::fixedsize_stack` has no
guard pages). Also makes `FiberState::kStackSize` public so the
anchoring uses the real fiber stack size.

## Related issue number

Closes ray-project#63290. Supersedes ray-project#63284 (same diagnosis direction, but
hand-rolled `_PyThreadStateImpl` offsets, a deliberate
`gilstate_counter` leak that freezes non-main threads, and a crash
premise that CPython 3.14.2 already fixed upstream).

## Checks

- Verified with a locally built cp314 Linux (aarch64, python:3.14.6
docker) wheel:
- refcount probe: **+4.00 refs/task → 0.00/task** (100 tasks)
- `__del__` deferral probe: dealloc during return serialization on the
fiber **deferred → immediate**
- live-malloc probe (`mallinfo2`, 300 tasks/shape): **~518 KiB/task → ~3
KiB/task** across async call → dict/bytes, async generator, sync
generator on async actor
- reporter-shaped streaming workload (400 tasks, 10 concurrent
sessions): live-malloc delta **0.2 MB total**, fiber-sized mapped
regions 0 → 0
- async-actor smoke: correctness (echo, state, async generators,
recursion), concurrency (20 overlapping 0.5 s sleeps in 0.51 s)
- throughput A/B (500 sequential echo tasks, 3 runs fixed / 2 runs
baseline, same container image): fixed 5624–6076 tasks/s vs unpatched
4551–4825 tasks/s meaning no regression (the unpatched build is slower
while leaking)
- baseline (unpatched) wheel from the same tree reproduces the bug:
+4.00 refs/task, fiber dealloc deferred=True

note: fable did a majority of the heavy lifting in this investigation
with prompting on what to check next and validate the solution

---------

Signed-off-by: Mark Towers <[email protected]>
Signed-off-by: myan <[email protected]>
Co-authored-by: Mark Towers <[email protected]>
Co-authored-by: myan <[email protected]>
Co-authored-by: Mengjin Yan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner approved these changes
@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner
@markshannon markshannon Awaiting requested review from markshannon markshannon is a code owner
@hugovk hugovk Awaiting requested review from hugovk

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

release-blocker

Projects

Release and Deferred blockers 🚫
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@encukou @vstinner @rokm @markshannon
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.