GeezerVison

Copy link

Copy Markdown

Member

Fidget-Spinner marked this pull request as ready for review April 30, 2024 21:09

Fidget-Spinner requested review from a team, brandtbucher, brettcannon, ericsnowcurrently, gvanrossum, kumaraditya303, markshannon, methane, ncoghlan and warsaw as code owners April 30, 2024 21:09

Fidget-Spinner requested a review from colesbury April 30, 2024 21:10

colesbury reviewed May 1, 2024

View reviewed changes

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Fidget-Spinner requested a review from markshannon June 26, 2024 14:32

Fidget-Spinner dismissed markshannon’s stale review June 26, 2024 16:53

colesbury approved these changes Jun 26, 2024

View reviewed changes

Fidget-Spinner deleted the stackref_all branch June 26, 2024 19:10

…` arguments in configuration command after #118450 (#121083)



Signed-off-by: Manjusaka <[email protected]>
Co-authored-by: Ken Jin <[email protected]>

…18450)

This PR sets up tagged pointers for CPython.

The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly.

Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out.

This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it.

The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs.

Please read Include/internal/pycore_stackref.h for more information!

---------

Co-authored-by: Mark Shannon <[email protected]>

…ystats` arguments in configuration command after python#118450 (python#121083)



Signed-off-by: Manjusaka <[email protected]>
Co-authored-by: Ken Jin <[email protected]>

…18450)

This PR sets up tagged pointers for CPython.

The general idea is to create a separate struct _PyStackRef for everything on the evaluation stack to store the bits. This forces the C compiler to warn us if we try to cast things or pull things out of the struct directly.

Only for free threading: We tag the low bit if something is deferred - that means we skip incref and decref operations on it. This behavior may change in the future if Mark's plans to defer all objects in the interpreter loop pans out.

This implies a strict stack reference discipline is required. ALL incref and decref operations on stackrefs must use the stackref variants. It is unsafe to untag something then do normal incref/decref ops on it.

The new incref and decref variants are called dup and close. They mimic a "handle" API operating on these stackrefs.

Please read Include/internal/pycore_stackref.h for more information!

---------

Co-authored-by: Mark Shannon <[email protected]>

…ystats` arguments in configuration command after python#118450 (python#121083)



Signed-off-by: Manjusaka <[email protected]>
Co-authored-by: Ken Jin <[email protected]>

## Description

This PR adds support for Python 3.14 in the profiler by updating it to
handle CPython internal changes.

### Key CPython changes addressed 

**`_PyInterpreterFrame` Structure Changes**
1. Moved from `Include/internal/pycore_frame.h` to
`Include/internal/pycore_interpframe_structs.h`
2. `PyObject *f_executable` and `PyObject *f_funcobj` changed to
`_PyStackRef` type. Profilers like us now need to clear the LSB of these
fields to get the `PyObject*`. See
python/cpython#123923 for details
3. `int stacktop` field removed, replaced with `_PyStackRef
*stackpointer` pointer. See
python/cpython#121923 (GH-120024) for details
4. `PyObject *localsplus[1]` changed to `_PyStackRef localsplus[1]`. See
python/cpython#118450 (gh-117139) for details

**`FutureObj`/`TaskObj` Changes**
1. Added fields: `awaited_by`, `is_task`, `awaited_by_is_set` in
`FutureObj_HEAD` macro
2. Added `struct llist_node_task_node` field for linked-list storage

**Asyncio Task Storage Changes**

Prior to Python 3.14, 
- All tasks are stored in `_scheduled_tasks` WeakSet
([exported](https://github.com/python/cpython/blob/e96367da1fdc1e1cf17ca523e93a127b1961b443/Modules/_asynciomodule.c#L3738)
from C extension)
- Eager tasks are stored in `_eager_tasks` set
([exported](https://github.com/python/cpython/blob/e96367da1fdc1e1cf17ca523e93a127b1961b443/Modules/_asynciomodule.c#L3742)
from C extension)

From Python 3.14,
- Native `asyncio.Tasks` are stored in a linked-list (`struct
llist_node`) per thread and per interpreter
-
[Per-thread](https://github.com/python/cpython/blob/0114178911f8713bfcb935ff5542fe61b4a5d551/Include/internal/pycore_tstate.h#L46):
`tstate->asyncio_tasks_head` (in `_PyThreadStateImpl`)
-
[Per-interpreter](https://github.com/python/cpython/blob/0114178911f8713bfcb935ff5542fe61b4a5d551/Include/internal/pycore_interp_structs.h#L897):
`interp->asyncio_tasks_head` (for lingering tasks)
- Each `TaskObj` has a `task_node` field with `next` and `prev` pointers
- Third-party tasks: Still stored in Python-level `_scheduled_tasks`
WeakSet (now Python-only, not exported from C extension)
- Eager tasks:  Still stored in Python-level `_eager_tasks` set


### Implementation Summary

- **Frame reading** (`frame.h`, `frame.cc`): Updated header includes to
use `pycore_interpframe_structs.h` for Python 3.14+. Implemented tagged
pointer handling: clear LSB of `f_executable` to recover `PyObject*`
(per gh-123923). Replaced `stacktop` field access with `stackpointer`
pointer arithmetic for stack depth calculation. Updated `PyGen_yf()` to
use `_PyStackRef` and `stackpointer[-1]` instead of
`localsplus[stacktop-1]`. Added handling for
`FRAME_OWNED_BY_INTERPRETER` frame type (introduced in 3.14).

- **Task structures** (`cpython/tasks.h`): Added Python 3.14+
`FutureObj_HEAD` macro with new fields: `awaited_by`, `is_task`,
`awaited_by_is_set`. Added `struct llist_node task_node` field to
`TaskObj` for linked-list storage. Updated `PyGen_yf()` implementation
to handle `_PyStackRef` and `stackpointer` instead of `stacktop`.

- **Asyncio discovery** (`tasks.h`, `threads.h`): Implemented
`get_tasks_from_linked_list()` to safely iterate over circular
linked-lists with iteration limits (`MAX_ITERATIONS = 2 << 15`). Added
`get_tasks_from_thread_linked_list()` to read tasks from
`_PyThreadStateImpl.asyncio_tasks_head` (per-thread active tasks). Added
`get_tasks_from_interpreter_linked_list()` to read lingering tasks from
`PyInterpreterState.asyncio_tasks_head` (per-interpreter). Updated
`get_all_tasks()` to handle both linked-list (native `asyncio.Task`
instances) and WeakSet (third-party tasks).

- **Python integration** (`_asyncio.py`): Added compatibility handling
for `BaseDefaultEventLoopPolicy` → `_BaseDefaultEventLoopPolicy` rename
in 3.14. Updated `_scheduled_tasks` access to handle Python-only WeakSet
(no longer exported from C extension in 3.14+).

## Testing

All existing tests pass except for
tests/profiling/collector/test_memalloc.py which needed some edits.

## Risks

<!-- Note any risks associated with this change, or "None" if no risks
-->

## Additional Notes

---------

Co-authored-by: Brett Langdon <[email protected]>

Problem 1:

    golang/_golang.cpp: In function ‘void XPyFrame_WhiteoutFastLocal(PyFrameObject*, PyObject*)’:
    golang/_golang.cpp:1572:43: error: invalid use of incomplete type ‘_PyInterpreterFrame’ {aka ‘struct _PyInterpreterFrame’}
     1572 |                                 f->f_frame->localsplus;
          |                                           ^~
    In file included from /home/kirr/local/py3.14/include/python3.14/pyframe.h:19,
                     from /home/kirr/local/py3.14/include/python3.14/Python.h:113,
                     from golang/_golang.cpp:150:
    /home/kirr/local/py3.14/include/python3.14/cpython/pyframe.h:25:8: note: forward declaration of ‘_PyInterpreterFrame’ {aka ‘struct _PyInterpreterFrame’}
       25 | struct _PyInterpreterFrame;
          |        ^~~~~~~~~~~~~~~~~~~

-> include internal/pycore_interpframe.h like we already do with e.g. internal/pycore_frame.h .

Problem 2:

    golang/_golang.cpp: In function ‘void XPyFrame_WhiteoutFastLocal(PyFrameObject*, PyObject*)’:
    golang/_golang.cpp:1575:45: error: cannot convert ‘_PyStackRef*’ to ‘PyObject**’ {aka ‘_object**’} in initialization
     1575 |                                 f->f_frame->localsplus;
          |                                 ~~~~~~~~~~~~^~~~~~~~~~
          |                                             |
          |                                             _PyStackRef*

It turns out _PyInterpreterFrame.localsplus was changed from PyObject to _PyStackRef:

    python/cpython@22b0de2755ee2
    python/cpython#118450
    python/cpython#117139

-> Adjust the code to work with localsplus as _PyStackRef on py3.14+ .

/reviewed-by @jerome
/reviewed-on https://lab.nexedi.com/nexedi/pygolang/-/merge_requests/39