GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — gh-98894: Fix dtrace tests in shared builds by stratakis · Pull Request #153372 · python/cpython

Conversation

stratakis commented Jul 8, 2026 •

edited by bedevere-app Bot

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

When Python is configured with --with-dtrace and --enable-shared, its USDT probes are in libpython, not the executable. The SystemTap and BPFTrace backends in test_dtrace were searching the executable, causing the tests to fail or time out. This PR makes both backends target the object containing the probes. Static builds are unaffected.

On Fedora, this can be reproduced by running test_dtrace for BPFTrace as sudo and for SystemTap either as sudo or as a user in the stapusr and stapdev groups.

Also added a commit to truncate the output of the tools as it can be quite verbose when it errors out (you can inspect by running the same tests without sudo).

Issue: Some DTrace probes are broken in 3.11 #98894

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

bedevere-app Bot added the awaiting review label

Jul 8, 2026

bedevere-app Bot mentioned this pull request

Jul 8, 2026

Some DTrace probes are broken in 3.11 #98894

Open

stratakis commented Jul 8, 2026

Copy link

Copy Markdown

Contributor Author

cc @vstinner

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

stratakis mentioned this pull request

Jul 8, 2026

Add dtrace worker tag support python/buildmaster-config#754

Merged

stratakis added 2 commits

July 10, 2026 00:05


          pythongh-98894: Fix dtrace tests in shared builds

90f8699

Generate SystemTap probe definitions targeting libpython for shared
builds and use centralized USDT probe object discovery for readelf
and BPFTrace.


          Truncate external tracer error output

9d87605

stratakis force-pushed the shared_probes branch from 37adf0f to 9d87605 Compare

July 9, 2026 22:13

stratakis commented Jul 9, 2026

Copy link

Copy Markdown

Contributor Author

Rebased to fix conflicts

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner commented Jul 13, 2026

Copy link

Copy Markdown

Member

Python configured with: ./configure --with-pydebug --with-dtrace --enable-shared.

Without the change, I get:

# LANG= LD_LIBRARY_PATH=$PWD ./python -m test -v test_dtrace 
== CPython 3.16.0a0 (heads/main:ed716551e13, Jul 13 2026, 07:59:05) [GCC 16.1.1 20260515 (Red Hat 16.1.1-2)]
== Linux-7.0.12-201.fc44.x86_64-x86_64-with-glibc2.43 little-endian
== Python build: debug shared dtrace
== cwd: /home/vstinner/python/main/build/test_python_worker_3188349æ
== CPU count: 10 (process) / 12 (system)
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 3520453529
0:00:00 load avg: 2.02 mem: 22.5 MiB Run 1 test sequentially in a single process
0:00:00 load avg: 2.02 mem: 22.5 MiB [1/1] test_dtrace
setUpClass (test.test_dtrace.BPFTraceNormalTests) ... skipped 'bpftrace timed out during usability check'
setUpClass (test.test_dtrace.BPFTraceOptimizedTests) ... skipped 'bpftrace timed out during usability check'
test_filter_probe_rows_ignores_warnings (test.test_dtrace.BPFTraceOutputTests.test_filter_probe_rows_ignores_warnings) ... ok
readelf version: 2.46
test_check_probes (test.test_dtrace.CheckDtraceProbes.test_check_probes) ... ok
test_missing_probes (test.test_dtrace.CheckDtraceProbes.test_missing_probes) ... expected failure
skipped 'dtrace(1) failed: /usr/sbin/dtrace invalid option -q\nUsage /usr/sbin/dtrace [--help] [-h | -G] [-C [-I<Path>]] -s File.d [-o <File>]'
setUpClass (test.test_dtrace.DTraceOptimizedTests) ... skipped 'dtrace(1) failed: /usr/sbin/dtrace invalid option -q\nUsage /usr/sbin/dtrace [--help] [-h | -G] [-C [-I<Path>]] -s File.d [-o <File>]'
setUpClass (test.test_dtrace.SystemTapNormalTests) ... skipped 'stap(1) failed: parse error: expected literal string or number\n        saw: operator \'*\' at /usr/share/systemtap/tapset/libvirt_probes-64.stp:204:3\n     source:   *cert = user_string($arg4);\n               ^\n\n1 parse error.\nstap_69e06176e3befa287a4f6741352860fa_990_src.c:12:10: fatal error: runtime_defines.h: No such file or directory\n   12 | #include "runtime_defines.h"\n      |          ^~~~~~~~~~~~~~~~~~~\ncompilation terminated.\nmake[2]: *** [/usr/src/kernels/7.0.12-201.fc44.x86_64/scripts/Makefile.build:289: stap_69e06176e3befa287a4f6741352860fa_990_src.o] Error 1\nmake[2]: *** Waiting for unfinished jobs....\nstap_symbols.c:3:10: fatal error: sym.h: No such file or directory\n    3 | #include <sym.h>\n      |          ^~~~~~~\ncompilation terminated.\nmake[2]: *** [/usr/src/kernels/7.0.12-201.fc44.x86_64/scripts/Makefile.build:289: stap_symbols.o] Error 1\nmake[1]: *** [/usr/src/kernels/7.0.12-201.fc44.x86_64/Makefile:2140: .] Error 2\nmake: *** [Makefile:248: __sub-make] Error 2\nWARNING: kbuild exited with status: 2\nPass 4: compilation failed.  [man error::pass4]\nKernel version 7.0.12 is outside tested range 3.10 ... 7.0'
setUpClass (test.test_dtrace.SystemTapOptimizedTests) ... skipped 'stap(1) failed: parse error: expected literal string or number\n        saw: operator \'*\' at /usr/share/systemtap/tapset/libvirt_probes-64.stp:204:3\n     source:   *cert = user_string($arg4);\n               ^\n\n1 parse error.\nstap_69e06176e3befa287a4f6741352860fa_990_src.c:12:10: fatal error: runtime_defines.h: No such file or directory\n   12 | #include "runtime_defines.h"\n      |          ^~~~~~~~~~~~~~~~~~~\ncompilation terminated.\nmake[2]: *** [/usr/src/kernels/7.0.12-201.fc44.x86_64/scripts/Makefile.build:289: stap_69e06176e3befa287a4f6741352860fa_990_src.o] Error 1\nmake[2]: *** Waiting for unfinished jobs....\nstap_symbols.c:3:10: fatal error: sym.h: No such file or directory\n    3 | #include <sym.h>\n      |          ^~~~~~~\ncompilation terminated.\nmake[2]: *** [/usr/src/kernels/7.0.12-201.fc44.x86_64/scripts/Makefile.build:289: stap_symbols.o] Error 1\nmake[1]: *** [/usr/src/kernels/7.0.12-201.fc44.x86_64/Makefile:2140: .] Error 2\nmake: *** [Makefile:248: __sub-make] Error 2\nWARNING: kbuild exited with status: 2\nPass 4: compilation failed.  [man error::pass4]\nKernel version 7.0.12 is outside tested range 3.10 ... 7.0'

----------------------------------------------------------------------
Ran 3 tests in 24.814s

OK (skipped=6, expected failures=1)
0:00:24 load avg: 1.75 mem: 22.9 MiB [1/1] test_dtrace passed

== Tests result: SUCCESS ==

1 test OK.

Total duration: 24.9 sec
Total tests: run=3 skipped=6
Total test files: run=1/1
Result: SUCCESS

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

stratakis commented Jul 13, 2026

Copy link

Copy Markdown

Contributor Author

The systemtap tests require the same kernel-devel package as the running kernel, if it's not there the test will be skipped.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner commented Aug 6, 2026

Copy link

Copy Markdown

Member

I tested again the change. Python built with ./configure --with-pydebug --with-dtrace --enable-shared and test_dtrace runs as root.

Without the change:

setUpClass (test.test_dtrace.BPFTraceNormalTests) ... skipped 'bpftrace timed out during usability check'
setUpClass (test.test_dtrace.BPFTraceOptimizedTests) ... skipped 'bpftrace timed out during usability check'
test_filter_probe_rows_ignores_warnings (test.test_dtrace.BPFTraceOutputTests.test_filter_probe_rows_ignores_warnings) ... ok
readelf version: 2.46
test_check_probes (test.test_dtrace.CheckDtraceProbes.test_check_probes) ... ok
test_missing_probes (test.test_dtrace.CheckDtraceProbes.test_missing_probes) ... expected failure
skipped 'dtrace(1) failed: /usr/sbin/dtrace invalid option -q\nUsage /usr/sbin/dtrace [--help] [-h | -G] [-C [-I<Path>]] -s File.d [-o <File>]'
setUpClass (test.test_dtrace.DTraceOptimizedTests) ... skipped 'dtrace(1) failed: /usr/sbin/dtrace invalid option -q\nUsage /usr/sbin/dtrace [--help] [-h | -G] [-C [-I<Path>]] -s File.d [-o <File>]'
setUpClass (test.test_dtrace.SystemTapNormalTests) ... skipped 'stap(1) failed: parse error: expected literal string or number\n        saw: operator \'*\' at /usr/share/systemtap/tapset/libvirt_probes-64.stp:204:3\n     source:   *cert = user_string($arg4);\n               ^\n\n1 parse error.\nstap_ad00f0008fef1242c8a51840ffeb6b1e_983_src.c:12:10: fatal error: runtime_defines.h: No such file or directory\n   12 | #include "runtime_defines.h"\n      |          ^~~~~~~~~~~~~~~~~~~\ncompilation terminated.\nmake[2]: *** [/usr/src/kernels/7.1.4-204.fc44.x86_64/scripts/Makefile.build:289: stap_ad00f0008fef1242c8a51840ffeb6b1e_983_src.o] Error 1\nmake[2]: *** Waiting for unfinished jobs....\nstap_symbols.c:3:10: fatal error: sym.h: No such file or directory\n    3 | #include <sym.h>\n      |          ^~~~~~~\ncompilation terminated.\nmake[2]: *** [/usr/src/kernels/7.1.4-204.fc44.x86_64/scripts/Makefile.build:289: stap_symbols.o] Error 1\nmake[1]: *** [/usr/src/kernels/7.1.4-204.fc44.x86_64/Makefile:2192: .] Error 2\nmake: *** [Makefile:248: __sub-make] Error 2\nWARNING: kbuild exited with status: 2\nPass 4: compilation failed.  [man error::pass4]\nKernel version 7.1.4 is outside tested range 3.10 ... 7.0'
setUpClass (test.test_dtrace.SystemTapOptimizedTests) ... skipped 'stap(1) failed: parse error: expected literal string or number\n        saw: operator \'*\' at /usr/share/systemtap/tapset/libvirt_probes-64.stp:204:3\n     source:   *cert = user_string($arg4);\n               ^\n\n1 parse error.\nstap_ad00f0008fef1242c8a51840ffeb6b1e_983_src.c:12:10: fatal error: runtime_defines.h: No such file or directory\n   12 | #include "runtime_defines.h"\n      |          ^~~~~~~~~~~~~~~~~~~\ncompilation terminated.\nmake[2]: *** [/usr/src/kernels/7.1.4-204.fc44.x86_64/scripts/Makefile.build:289: stap_ad00f0008fef1242c8a51840ffeb6b1e_983_src.o] Error 1\nmake[2]: *** Waiting for unfinished jobs....\nstap_symbols.c:3:10: fatal error: sym.h: No such file or directory\n    3 | #include <sym.h>\n      |          ^~~~~~~\ncompilation terminated.\nmake[2]: *** [/usr/src/kernels/7.1.4-204.fc44.x86_64/scripts/Makefile.build:289: stap_symbols.o] Error 1\nmake[1]: *** [/usr/src/kernels/7.1.4-204.fc44.x86_64/Makefile:2192: .] Error 2\nmake: *** [Makefile:248: __sub-make] Error 2\nWARNING: kbuild exited with status: 2\nPass 4: compilation failed.  [man error::pass4]\nKernel version 7.1.4 is outside tested range 3.10 ... 7.0'

----------------------------------------------------------------------
Ran 3 tests in 24.678s

OK (skipped=6, expected failures=1)
...
Total tests: run=3 skipped=6
Total test files: run=1/1

With the change:

test_function_entry_return (test.test_dtrace.BPFTraceNormalTests.test_function_entry_return) ... ok
test_gc (test.test_dtrace.BPFTraceNormalTests.test_gc) ... ok
test_verify_call_opcodes (test.test_dtrace.BPFTraceNormalTests.test_verify_call_opcodes)
Ensure our call stack test hits all function call opcodes ... ok
test_function_entry_return (test.test_dtrace.BPFTraceOptimizedTests.test_function_entry_return) ... ok
test_gc (test.test_dtrace.BPFTraceOptimizedTests.test_gc) ... ok
test_verify_call_opcodes (test.test_dtrace.BPFTraceOptimizedTests.test_verify_call_opcodes)
Ensure our call stack test hits all function call opcodes ... ok
test_filter_probe_rows_ignores_warnings (test.test_dtrace.BPFTraceOutputTests.test_filter_probe_rows_ignores_warnings) ... ok
readelf version: 2.46
test_check_probes (test.test_dtrace.CheckDtraceProbes.test_check_probes) ... ok
test_missing_probes (test.test_dtrace.CheckDtraceProbes.test_missing_probes) ... expected failure
skipped 'dtrace(1) failed: /usr/sbin/dtrace invalid option -q; Usage /usr/sbin/dtrace [--help] [-h | -G] [-C [-I<Path>]] -s File.d [-o <File>]'
setUpClass (test.test_dtrace.DTraceOptimizedTests) ... skipped 'dtrace(1) failed: /usr/sbin/dtrace invalid option -q; Usage /usr/sbin/dtrace [--help] [-h | -G] [-C [-I<Path>]] -s File.d [-o <File>]'
setUpClass (test.test_dtrace.SystemTapNormalTests) ... skipped "stap(1) failed: parse error: expected literal string or number; saw: operator '*' at /usr/share/systemtap/tapset/libvirt_probes-64.stp:204:3; source:   *cert = user_string($arg4);; ... (18 lines omitted)"
setUpClass (test.test_dtrace.SystemTapOptimizedTests) ... skipped "stap(1) failed: parse error: expected literal string or number; saw: operator '*' at /usr/share/systemtap/tapset/libvirt_probes-64.stp:204:3; source:   *cert = user_string($arg4);; ... (18 lines omitted)"

----------------------------------------------------------------------
Ran 9 tests in 7.410s

OK (skipped=4, expected failures=1)
...
Total tests: run=9 skipped=4
Total test files: run=1/1

So we get: "Total tests: run=3 skipped=6" => "Total tests: run=9 skipped=4". 6 more tests are run thanks to the change.

With the change, BPFTraceNormalTests and BPFTraceOptimizedTests tests are run.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner reviewed

Aug 6, 2026

View reviewed changes

Comment thread

Lib/test/test_dtrace.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

Lib/test/test_dtrace.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

Lib/test/test_dtrace.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

Lib/test/test_dtrace.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

Lib/test/test_dtrace.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

stratakis added 2 commits

August 7, 2026 15:14


          Revert "Truncate external tracer error output"

bf2da78

This reverts commit 9d87605.


          Address review

b07f88b

vstinner added the skip news label

Aug 7, 2026

vstinner enabled auto-merge (squash)

August 7, 2026 14:12

vstinner merged commit 298bef3 into python:main

Aug 7, 2026

48 of 49 checks passed

Uh oh!

There was an error while loading. Please reload this page.

bedevere-app Bot removed the awaiting review label

Aug 7, 2026

stratakis commented Aug 11, 2026

Copy link

Copy Markdown

Contributor Author

Should be backported on 3.15 as well.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

mbeijen pushed a commit to mbeijen/cpython that referenced this pull request

Aug 14, 2026


          pythongh-98894: Fix dtrace tests in shared builds (python#153372)

61230f6

Generate SystemTap probe definitions targeting libpython for shared
builds and use centralized USDT probe object discovery for readelf
and BPFTrace.

vstinner added the needs backport to 3.15 bugs and security fixes label

Aug 14, 2026

miss-islington-app Bot commented Aug 14, 2026

Copy link

Copy Markdown

Thanks @stratakis for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bedevere-app Bot commented Aug 14, 2026

Copy link

Copy Markdown

GH-155803 is a backport of this pull request to the 3.15 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bedevere-app Bot removed the needs backport to 3.15 bugs and security fixes label

Aug 14, 2026

hugovk pushed a commit that referenced this pull request

Aug 19, 2026


          [3.15] gh-98894: Fix dtrace tests in shared builds (GH-153372) (#155803)

775b2c3

Co-authored-by: stratakis <[email protected]>

vstinner added a commit that referenced this pull request

Oct 16, 2026


          [3.14] gh-156701: Backport test_dtrace from main (#159596)

566f815

* gh-98894: Fix DTrace test_check_probes for shared builds (#151122)

When building with --enable-shared, the SystemTap/DTrace notes
live in libpython. Add detection logic to be used by readelf.

Force the C locale on readelf output.

(cherry picked from commit 3a08e93)

* gh-151496: Use process groups in test_dtrace (#151512)

Create a new process group to run bpftrace commands, so it's possible
to kill also child processes on timeout.

(cherry picked from commit a064b32)

* gh-151496: Use process groups in TraceBackend in test_dtrace (#152039)

Run the generic DTrace/SystemTap commands in a new process group and
terminate the whole group on timeout.

This prevents a forked tracer child from keeping stdout/stderr pipes
open after the direct tracer process is killed.

(cherry picked from commit 1785f4b)

* gh-98894: Check readelf failures in test_dtrace (#152345)

Report readelf command failures directly instead of later failing with
missing-probe assertions.

(cherry picked from commit d60a677)

* gh-98894: Check tracer exit status in test_dtrace (#152893)

Fail functional dtrace and SystemTap cases directly when the tracer exits
non-zero.

(cherry picked from commit 311e7e7)

* gh-98894: Quote test_dtrace tracer subcommands (#152901)

Use shlex.join() when building tracer -c command strings
for avoiding issues with spaces.

(cherry picked from commit a159c68)

* gh-98894: Fix dtrace tests in shared builds (#153372)

Generate SystemTap probe definitions targeting libpython for shared
builds and use centralized USDT probe object discovery for readelf
and BPFTrace.

(cherry picked from commit 298bef3)

* gh-156703: Set LD_LIBRARY_PATH in SystemTap trace commands (GH-156424)

Prefix commands passed to stap -c with LD_LIBRARY_PATH via env, when
probes reside in libpython. This allows the interpreter to find
libpython when staprun does not preserve the loader path.

(cherry picked from commit faf4e52)

* gh-156722, test_dtrace: Terminate tracing processes gracefully on timeout (#156753)

Send SIGTERM to the tracing process group before falling back to SIGKILL.

(cherry picked from commit 55ecc0a)

* gh-156701: Increase the SystemTap usability timeout (#156720)

SystemTap may compile a kernel module during its usability check
(if no cached module is available). We need to allow a larger timeout
for that case.

(cherry picked from commit e1be613)

---------

Co-authored-by: stratakis <[email protected]>
Co-authored-by: stratakis <[email protected]>

Reviewers

vstinner

vstinner left review comments

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

stratakis commented Jul 8, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stratakis commented Jul 8, 2026

Uh oh!

stratakis commented Jul 9, 2026

Uh oh!

vstinner commented Jul 13, 2026

Uh oh!

stratakis commented Jul 13, 2026

Uh oh!

vstinner commented Aug 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stratakis commented Aug 11, 2026

Uh oh!

miss-islington-app Bot commented Aug 14, 2026

Uh oh!

bedevere-app Bot commented Aug 14, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stratakis commented Jul 8, 2026 •

edited by bedevere-app Bot

Loading

Uh oh!

There was an error while loading. Please reload this page.