GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — C-API for signalling monitoring events · Issue #111997 · python/cpython

C-API for signalling monitoring events #111997

New issue

Copy link

New issue

Copy link

Closed

C-API for signalling monitoring events#111997

Copy link

Assignees

Labels

3.13only security fixesonly security fixesdeferred-blockertopic-C-APItype-featureA feature request or enhancementA feature request or enhancement

Description

scoder

opened

on Nov 12, 2023

Issue body actions

Feature or enhancement

Proposal:

Language implementations for the CPython runtime (Cython, JIT compilers, regular expression engines, template engines, etc.) need a way to signal PEP-669 monitoring events to the registered listeners.

We need a way to create events and inject them into the monitoring system. Since events have more than one signature, we might end up needing more than one C-API function for this.
We need a way to map 3D source code positions (file name, line, character) to 1D integer offsets. Code objects help, but branches might cross source file boundaries, so that's more tricky. For many use cases, a mapping between (line, character) positions and an integer offset would probably suffice, although templating languages usually provide some kind of include commands, as does Cython. There should be some help in the C-API for building up such a mapping.

The reason why I think we need CPython's help for mapping indices is that both sides, listeners and even producers, need to agree on the same mapping. Sadly, the events don't include line/character positions directly but only a single integer. So event producers need a way to produce a number that event listeners like coverage analysers can map back to a source code position.

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/61?u=scoder

Linked PRs

Reacted by Iago Chaves and Antoine Pitrou

Reacted by Dan D'Avella

Activity

scoder
added
type-featureA feature request or enhancementA feature request or enhancement
on Nov 12, 2023
Eclips4
added
topic-C-API
on Nov 12, 2023
scoder commented on Nov 12, 2023
scoder
on Nov 12, 2023
ContributorAuthor
More actions
@encukou @markshannon @vstinner
gvanrossum
assigned
markshannon
on Nov 12, 2023
vstinner commented on Nov 12, 2023
vstinner
on Nov 12, 2023
Member
More actions
Do you want to propose an API for that?
drdavella commented on Nov 13, 2023
drdavella
on Nov 13, 2023
More actions
PEP-669 seems to discuss a mechanism to modify the bytestream for the purpose of implementing monitoring. Would access to this mechanism also be provided as part of this API? I haven't seen too much mention of this feature elsewhere.

Source: https://peps.python.org/pep-0669/#rationale

The quickening mechanism provided by PEP 659 provides a way to dynamically modify executing Python bytecode. These modifications have little cost beyond the parts of the code that are modified and a relatively low cost to those parts that are modified. We can leverage this to provide an efficient mechanism for monitoring that was not possible in 3.10 or earlier.

IMO it would be very useful for an API of this nature to expose low-level interpreter state to monitoring tools. For example, if we are able to see events like INSTRUCTION, it would also be useful to have access to the operands involved.

scoder commented on Nov 21, 2023

scoder

on Nov 21, 2023

ContributorAuthor

More actions

Do you want to propose an API for that?

In order to map source/line/column information to an offset, we need a way to build a co_linetable for a code object: https://github.com/python/cpython/blob/main/Objects/locations.md
That's probably the most complex thing to do here. IIUC, CPython implements this in C here:

cpython/Python/assemble.c

Lines 190 to 444 in d67f947

	/* Code location emitting code. See locations.md for a description of the format. */

	#define MSB 0x80

	static void
	write_location_byte(struct assembler* a, int val)
	{
	PyBytes_AS_STRING(a->a_linetable)[a->a_location_off] = val&255;
	a->a_location_off++;
	}


	static uint8_t *
	location_pointer(struct assembler* a)
	{
	return (uint8_t *)PyBytes_AS_STRING(a->a_linetable) +
	a->a_location_off;
	}

	static void
	write_location_first_byte(struct assembler* a, int code, int length)
	{
	a->a_location_off += write_location_entry_start(
	location_pointer(a), code, length);
	}

	static void
	write_location_varint(struct assembler* a, unsigned int val)
	{
	uint8_t *ptr = location_pointer(a);
	a->a_location_off += write_varint(ptr, val);
	}


	static void
	write_location_signed_varint(struct assembler* a, int val)
	{
	uint8_t *ptr = location_pointer(a);
	a->a_location_off += write_signed_varint(ptr, val);
	}

	static void
	write_location_info_short_form(struct assembler* a, int length, int column, int end_column)
	{
	assert(length > 0 && length <= 8);
	int column_low_bits = column & 7;
	int column_group = column >> 3;
	assert(column < 80);
	assert(end_column >= column);
	assert(end_column - column < 16);
	write_location_first_byte(a, PY_CODE_LOCATION_INFO_SHORT0 + column_group, length);
	write_location_byte(a, (column_low_bits << 4) \| (end_column - column));
	}

	static void
	write_location_info_oneline_form(struct assembler* a, int length, int line_delta, int column, int end_column)
	{
	assert(length > 0 && length <= 8);
	assert(line_delta >= 0 && line_delta < 3);
	assert(column < 128);
	assert(end_column < 128);
	write_location_first_byte(a, PY_CODE_LOCATION_INFO_ONE_LINE0 + line_delta, length);
	write_location_byte(a, column);
	write_location_byte(a, end_column);
	}

	static void
	write_location_info_long_form(struct assembler* a, location loc, int length)
	{
	assert(length > 0 && length <= 8);
	write_location_first_byte(a, PY_CODE_LOCATION_INFO_LONG, length);
	write_location_signed_varint(a, loc.lineno - a->a_lineno);
	assert(loc.end_lineno >= loc.lineno);
	write_location_varint(a, loc.end_lineno - loc.lineno);
	write_location_varint(a, loc.col_offset + 1);
	write_location_varint(a, loc.end_col_offset + 1);
	}

	static void
	write_location_info_none(struct assembler* a, int length)
	{
	write_location_first_byte(a, PY_CODE_LOCATION_INFO_NONE, length);
	}

	static void
	write_location_info_no_column(struct assembler* a, int length, int line_delta)
	{
	write_location_first_byte(a, PY_CODE_LOCATION_INFO_NO_COLUMNS, length);
	write_location_signed_varint(a, line_delta);
	}

	#define THEORETICAL_MAX_ENTRY_SIZE 25 /* 1 + 6 + 6 + 6 + 6 */


	static int
	write_location_info_entry(struct assembler* a, location loc, int isize)
	{
	Py_ssize_t len = PyBytes_GET_SIZE(a->a_linetable);
	if (a->a_location_off + THEORETICAL_MAX_ENTRY_SIZE >= len) {
	assert(len > THEORETICAL_MAX_ENTRY_SIZE);
	RETURN_IF_ERROR(_PyBytes_Resize(&a->a_linetable, len*2));
	}
	if (loc.lineno < 0) {
	write_location_info_none(a, isize);
	return SUCCESS;
	}
	int line_delta = loc.lineno - a->a_lineno;
	int column = loc.col_offset;
	int end_column = loc.end_col_offset;
	assert(column >= -1);
	assert(end_column >= -1);
	if (column < 0 \|\| end_column < 0) {
	if (loc.end_lineno == loc.lineno \|\| loc.end_lineno == -1) {
	write_location_info_no_column(a, isize, line_delta);
	a->a_lineno = loc.lineno;
	return SUCCESS;
	}
	}
	else if (loc.end_lineno == loc.lineno) {
	if (line_delta == 0 && column < 80 && end_column - column < 16 && end_column >= column) {
	write_location_info_short_form(a, isize, column, end_column);
	return SUCCESS;
	}
	if (line_delta >= 0 && line_delta < 3 && column < 128 && end_column < 128) {
	write_location_info_oneline_form(a, isize, line_delta, column, end_column);
	a->a_lineno = loc.lineno;
	return SUCCESS;
	}
	}
	write_location_info_long_form(a, loc, isize);
	a->a_lineno = loc.lineno;
	return SUCCESS;
	}

	static int
	assemble_emit_location(struct assembler* a, location loc, int isize)
	{
	if (isize == 0) {
	return SUCCESS;
	}
	while (isize > 8) {
	RETURN_IF_ERROR(write_location_info_entry(a, loc, 8));
	isize -= 8;
	}
	return write_location_info_entry(a, loc, isize);
	}

	static int
	assemble_location_info(struct assembler a, instr_sequence instrs,
	int firstlineno)
	{
	a->a_lineno = firstlineno;
	location loc = NO_LOCATION;
	int size = 0;
	for (int i = 0; i < instrs->s_used; i++) {
	instruction *instr = &instrs->s_instrs[i];
	if (!same_location(loc, instr->i_loc)) {
	RETURN_IF_ERROR(assemble_emit_location(a, loc, size));
	loc = instr->i_loc;
	size = 0;
	}
	size += instr_size(instr);
	}
	RETURN_IF_ERROR(assemble_emit_location(a, loc, size));
	return SUCCESS;
	}

	static void
	write_instr(_Py_CODEUNIT codestr, instruction instr, int ilen)
	{
	int opcode = instr->i_opcode;
	assert(!IS_PSEUDO_INSTR(opcode));
	int oparg = instr->i_oparg;
	assert(OPCODE_HAS_ARG(opcode) \|\| oparg == 0);
	int caches = _PyOpcode_Caches[opcode];
	switch (ilen - caches) {
	case 4:
	codestr->op.code = EXTENDED_ARG;
	codestr->op.arg = (oparg >> 24) & 0xFF;
	codestr++;
	/* fall through */
	case 3:
	codestr->op.code = EXTENDED_ARG;
	codestr->op.arg = (oparg >> 16) & 0xFF;
	codestr++;
	/* fall through */
	case 2:
	codestr->op.code = EXTENDED_ARG;
	codestr->op.arg = (oparg >> 8) & 0xFF;
	codestr++;
	/* fall through */
	case 1:
	codestr->op.code = opcode;
	codestr->op.arg = oparg & 0xFF;
	codestr++;
	break;
	default:
	Py_UNREACHABLE();
	}
	while (caches--) {
	codestr->op.code = CACHE;
	codestr->op.arg = 0;
	codestr++;
	}
	}

	/* assemble_emit_instr()
	Extend the bytecode with a new instruction.
	Update lnotab if necessary.
	*/

	static int
	assemble_emit_instr(struct assembler a, instruction instr)
	{
	Py_ssize_t len = PyBytes_GET_SIZE(a->a_bytecode);
	_Py_CODEUNIT *code;

	int size = instr_size(instr);
	if (a->a_offset + size >= len / (int)sizeof(_Py_CODEUNIT)) {
	if (len > PY_SSIZE_T_MAX / 2) {
	return ERROR;
	}
	RETURN_IF_ERROR(_PyBytes_Resize(&a->a_bytecode, len * 2));
	}
	code = (_Py_CODEUNIT *)PyBytes_AS_STRING(a->a_bytecode) + a->a_offset;
	a->a_offset += size;
	write_instr(code, instr, size);
	return SUCCESS;
	}

	static int
	assemble_emit(struct assembler a, instr_sequence instrs,
	int first_lineno, PyObject *const_cache)
	{
	RETURN_IF_ERROR(assemble_init(a, first_lineno));

	for (int i = 0; i < instrs->s_used; i++) {
	instruction *instr = &instrs->s_instrs[i];
	RETURN_IF_ERROR(assemble_emit_instr(a, instr));
	}

	RETURN_IF_ERROR(assemble_location_info(a, instrs, a->a_lineno));

	RETURN_IF_ERROR(assemble_exception_table(a, instrs));

	RETURN_IF_ERROR(_PyBytes_Resize(&a->a_except_table, a->a_except_table_off));
	RETURN_IF_ERROR(_PyCompile_ConstCacheMergeOne(const_cache, &a->a_except_table));

	RETURN_IF_ERROR(_PyBytes_Resize(&a->a_linetable, a->a_location_off));
	RETURN_IF_ERROR(_PyCompile_ConstCacheMergeOne(const_cache, &a->a_linetable));

	RETURN_IF_ERROR(_PyBytes_Resize(&a->a_bytecode, a->a_offset * sizeof(_Py_CODEUNIT)));
	RETURN_IF_ERROR(_PyCompile_ConstCacheMergeOne(const_cache, &a->a_bytecode));
	return SUCCESS;
	}

That's purely internal code. I don't think we need C-API support for this, but a Python implementation would be nice. Or a static method on code objects to build the byte string from a Python list of positions. Given the complexity of the format and the risk of future changes, I think it's in CPython's responsibility to generate this format from something user friendly.

compile.h defines the location like this:

/* source location information */
typedef struct {
    int lineno;
    int end_lineno;
    int col_offset;
    int end_col_offset;
} _PyCompilerSrcLocation;

This looks like a function could help that accepts a sequence of tuples of these four values and maps it to a bytes object with the corresponding line table in it.

We'd probably still have to reimplement it in Cython in order to generate the line table string in "older" Python versions. But we could at least rely on it from CPython 3.13 on.

Reacted by Antoine Pitrou and Mahmoud Rusty Abdelkader

scoder commented on Nov 22, 2023
scoder
on Nov 22, 2023
ContributorAuthor
More actions
@markshannon, could you please comment whether you consider this the right approach? To me, a line table seems required to generate source level monitoring events.
scoder commented on Dec 14, 2023
scoder
on Dec 14, 2023
ContributorAuthor
More actions
@markshannon any comments? Given that Python 3.12 broke an entire C-API feature, I consider it a blocker for 3.13 to fix it.
pitrou
added
release-blocker

3.13only security fixesonly security fixes
on Dec 20, 2023
ezio-melotti
added this to Release and Deferred blockers 🚫on Dec 20, 2023
github-project-automation
moved this to Todo in Release and Deferred blockers 🚫on Dec 20, 2023
markshannon commented on Dec 20, 2023
markshannon
on Dec 20, 2023
Member
More actions
What C-API was broken?
AFAICT there was no API to call the tstate->c_tracefunc and tstate->c_profilefunc functions.
How were you calling those functions, and why does it no longer work?
markshannon commented on Jan 22, 2024
markshannon
on Jan 22, 2024
Member
More actions
C extensions can already register callbacks and set events using the Python API, so I assume we are talking about adding an API for C extensions to fire events.

There will be two parts to such an API:

Specifying the mapping from "code objects" and offsets to full locations.

Firing events from C extension code.

Handling locations

Apart from LINE events, all event callback functions include code: CodeType, offset: int. Let's not force C extensions to build a full code objects, so we should relax code to be a "code like" object. We define exactly what "code like" means later.

We also need a specification for the table mapping offsets to locations. The existing table format is quite fiddly, and can only be searched linearly, so designing a new table format might be a good idea.

Firing events from C extension code

Any potential API has to be more complex than for sys.settrace as we need to handle disabling of event locations and multiple tools.

We will need 16 bits of data for each event location, to track active tools and which tool/location pairs are disabled.

We will need to initialize this data, and to specify which event location maps to which event.

With that in mind, and using the PY_START event as an example, an API could look something like this:

typedef struct _PyMonitoringState { uint8_t active; uint8_t opaque; } PyMonitoringState; typedef struct _PyCodeLikeObject PyCodeLikeObject; /* Returns a version, which should be stored and passed to the next call of _PyMonitioringEnterScope. Arguments: previous_version: Points to a per code-like object value. The value must be set to 0 before first call per code-like object. state_array: The array of all PyMonitoringStates for this code-like object. Should be initialized to all zeroes before first call. event_types: Array of the event types describing the event type for each state in state array. length: The length of the state and event arrays, which must be the same. */ void _PyMonitoringEnterScope(uint64_t *previous_version, PyMonitoringState *state_array, uint8_t event_types, uint32_t length); int _PyMonitoring_FirePyStartEvent(PyCodeLikeObject *codelike, uint32_t offset, PyMonitoringState *state);

If performance is going to be an issue, then the above could be implemented as inline functions.
We would lose ABI compatibility, but would still retain API compatibility across CPython versions.

Example usage.

Suppose we have a code-like object that that should fire two events when called, PY_START then PY_RETURN.

We would need to describe the events:

static const uint8_t EVENTS = { PY_START, PY_RETURN };

And the code-like object would need an array for the state and a version number.

struct MyCodeLike { PyObject_HEAD PyMonitoringState monitoring_state[2]; uint64_t version; /* Other fields */ };

The C code for the function:

void init_code(MyCodeLike *code) { code->monitoring_state = {0, 0}; code->version = 0; } PyObject *call_code(MyCodeLike *code) { _PyMonitoringEnterScope(&code->version, code->monitoring_state, EVENTS, 2); ... _PyMonitoring_FirePyStartEvent(code, 0, &code->monitoring_state[0]); ... _PyMonitoring_FirePyReturnEvent(code, 1, &code->monitoring_state[1]); ... }
21 remaining items
markshannon commented on May 21, 2024
markshannon
on May 21, 2024
Member
More actions
We can remove the assert, nothing will crash.
However, there might be an issue with the frame passed to the sys.settrace callback function, as it will be the frame of the closest Python function, not the Cython function.

How did Cython work with profiling prior to 3.12? Did you create a fake frame?

scoder commented on May 25, 2024

scoder

on May 25, 2024

ContributorAuthor

More actions

How did Cython work with profiling prior to 3.12? Did you create a fake frame?

Yes. We could keep doing that, but a) frames aren't "officially" exposed from the C-API and b) as I understand it, the goal of creating the monitoring C-API was to allow non-Python non-frame code to interact with the tracing infrastructure, so requiring frames seems counter-productive.

I now found that coverage.py was actually updated to work directly with sys.monitoring, so I tried that and ran into some more issues:

  File "/home/stefan/source/Python/cython/cython-git/venv/py314/lib/python3.14/site-packages/coverage/sysmon.py", line 337, in sysmon_py_start
    sys_monitoring.set_local_events(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.myid,
        ^^^^^^^^^^
    ...<7 lines>...
        # | events.JUMP
        ^^^^^^^^^^^^^^^
    )
    ^
SystemError: cannot instrument shim code object 'func1'

The exception originates here and seems to trigger on generally all code objects that do not have byte code:

cpython/Python/instrumentation.c

Lines 1971 to 1974 in de19694

	if (code->_co_firsttraceable >= Py_SIZE(code)) {
	PyErr_Format(PyExc_SystemError, "cannot instrument shim code object '%U'", code->co_name);
	return -1;
	}

set_local_events() also has this code, requiring the "code-like object" to be an actual code object:

cpython/Python/instrumentation.c

Lines 2253 to 2259 in de19694

	if (!PyCode_Check(code)) {
	PyErr_Format(
	PyExc_TypeError,
	"code must be a code object"
	);
	return NULL;
	}

I'm afraid that we might still run into a lot of similar assumptions about what is traceable code.

scoder commented on May 25, 2024
scoder
on May 25, 2024
ContributorAuthor
More actions
Changing the code from
if (code->_co_firsttraceable >= Py_SIZE(code)) {
to
if (code->_co_firsttraceable && code->_co_firsttraceable >= Py_SIZE(code)) {
lets me get past the check and continue.
miss-islington
added a commit that references this issue on May 26, 2024
pythongh-111997: Fix argument count for LINE event and clarify type o…
b68915f
iritkatriel
added a commit that references this issue on May 26, 2024
gh-111997: Fix argument count for LINE event and clarify type of argu…
70b07aa
scoder
added a commit that references this issue on May 26, 2024
[3.13] gh-111997: Fix argument count for LINE event and clarify type …
825a5ae
iritkatriel commented on Jun 3, 2024
iritkatriel
on Jun 3, 2024
Member
More actions
Should we make sys_trace_line_func and other functions in legacy_tracing.c create a fake frame if frame passed in is NULL?
iritkatriel commented on Jun 13, 2024
iritkatriel
on Jun 13, 2024
Member
More actions
What are the open issues here currently?
estyxx
added a commit that references this issue on Jul 17, 2024
pythongh-111997: Fix argument count for LINE event and clarify type o…
4b9f326
iritkatriel commented on Sep 7, 2024
iritkatriel
on Sep 7, 2024
Member
More actions
I'm closing this as complete. Let's create new issues for any followup work.
iritkatriel
closed this as completed on Sep 7, 2024
github-project-automation
moved this from In Progress to Done in Release and Deferred blockers 🚫on Sep 7, 2024
iritkatriel
added 2 commits that reference this issue on Sep 7, 2024
pythongh-111997: update what's new in 3.13
875e056
gh-111997: update what's new in 3.13 re C API for monitoring events (#…
77c68b4
miss-islington
added a commit that references this issue on Sep 12, 2024
pythongh-111997: update what's new in 3.13 re C API for monitoring ev…
5c68e6b

Metadata

Assignees

Labels

3.13only security fixesonly security fixesdeferred-blockertopic-C-APItype-featureA feature request or enhancementA feature request or enhancement

Projects

Release and Deferred blockers 🚫
Status
Done
Show more project fields

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

C-API for signalling monitoring events #111997

Description

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Linked PRs

Activity

scoder commented on Nov 12, 2023

vstinner commented on Nov 12, 2023

drdavella commented on Nov 13, 2023

scoder commented on Nov 21, 2023

scoder commented on Nov 22, 2023

scoder commented on Dec 14, 2023

markshannon commented on Dec 20, 2023

markshannon commented on Jan 22, 2024

Handling locations

Firing events from C extension code

Example usage.

21 remaining items

markshannon commented on May 21, 2024

scoder commented on May 25, 2024

scoder commented on May 25, 2024

iritkatriel commented on Jun 3, 2024

iritkatriel commented on Jun 13, 2024

iritkatriel commented on Sep 7, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions