GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-151496: Use process groups in test_dtrace by vstinner · Pull Request #151512 · python/cpython

Conversation

@vstinner

@vstinner vstinner commented Jun 15, 2026 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Create a new process group to run bpftrace commands, so it's possible to kill also child processes on timeout.

  • Issue: test_dtrace can hang indefinitely when bpftrace usability check times out #151496

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@vstinner
pythongh-151496: Use process groups in test_dtrace
665bb1d
Create a new process group to run bpftrace commands, so it's possible
to kill also child processes on timeout.
@vstinner vstinner added the needs backport to 3.15 bugs and security fixes label Jun 15, 2026
@bedevere-app bedevere-app Bot added the awaiting core review label Jun 15, 2026
@bedevere-app bedevere-app Bot mentioned this pull request Jun 15, 2026
test_dtrace can hang indefinitely when bpftrace usability check times out #151496
Closed
@bedevere-app bedevere-app Bot added the tests Tests in the Lib/test dir label Jun 15, 2026
@vstinner
Simplify kill_process_group()
81ebe07
@vstinner

vstinner commented Jun 15, 2026

Copy link
Copy Markdown
Member Author

cc @pablogsal @stratakis

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner vstinner added the skip news label Jun 15, 2026
stratakis
stratakis reviewed Jun 16, 2026
View reviewed changes
Comment thread Lib/test/test_dtrace.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

stratakis
stratakis reviewed Jun 16, 2026
View reviewed changes
Comment thread Lib/test/test_dtrace.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
Address stratakis' review
b5bf287
@vstinner

vstinner commented Jun 16, 2026

Copy link
Copy Markdown
Member Author

@stratakis: I addressed your review, thanks. Please review the updated PR.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Jun 16, 2026

Copy link
Copy Markdown
Member Author

Tests / Sanitizers / TSan (free-threading) (pull_request): Failing after 22m

Hum. test_socket failed with a strange error. I just re-ran the job.

ERROR: testSourceAddress (test.test_socket.NetworkConnectionAttributesTest.testSourceAddress)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython/Lib/test/test_socket.py", line 442, in raise_queued_exception
    raise self.queue.get()
  File "/home/runner/work/cpython/cpython/Lib/test/test_socket.py", line 479, in clientRun
    test_func()
    ~~~~~~~~~^^
  File "/home/runner/work/cpython/cpython/Lib/test/test_socket.py", line 6048, in _testSourceAddress
    self.assertEqual(self.cli.getsockname()[1], self.source_port)
                     ~~~~~~~~~~~~~~~~~~~~^^
OSError: [Errno 9] Bad file descriptor
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@stratakis

stratakis commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Nitpick: I'd change the helper name to something other as kill has a very specific meaning and the helper also closes the pipes, it doesn't only kill the processes. Maybe cleanup_process_group() or something?

For testing the assert_usable() path (10s timeout):

Fake bpftrace script:
(sleep 300) & sleep 300

And adding it to the $PATH when running the tests:
PATH="fake bpftrace path:$PATH" ./python -m test test_dtrace -m 'BPFTraceNormalTests' -v

This hangs and also leaves leaked children:
$ pgrep -af 'sleep 300'
194272 sleep 300

For testing the run_case() path (60s timeout):

Fake bpftrace script:

case "$*" in
  *'probe: success'*)
  echo 'probe: success'
  exit 0
  ;;
esac
(sleep 300) & sleep 300

PATH="fake bpftrace path:$PATH" ./python -m unittest -v test.test_dtrace.BPFTraceNormalTests.test_gc

After 60 secs it should failed and children are leaked:
$ pgrep -af 'sleep 300'
250104 sleep 300
250105 sleep 300

With the PR the assert_usable() path doesn't hang anymore and doesn't leak and the run_case() path still times outs but doesn't leak.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Jun 17, 2026

Copy link
Copy Markdown
Member Author

Tests / Sanitizers / TSan (free-threading) (pull_request): Cancelled after 60m

Oh, test_abc of ./python -m test --tsan-parallel --parallel-threads=4 -j4 -W seems to be stuck, and the job was cancelled after 60 minutes :-(

Run ./python -m test --tsan-parallel --parallel-threads=4 -j4 -W
Using random seed: 2657327358
0:00:00 load avg: 2.63 mem: 274.1 MiB Run 2 tests in parallel using 2 worker processes
0:00:13 load avg: 2.96 mem: 588.5 MiB [1/2] test_hashlib passed
0:00:43 load avg: 1.79 mem: 588.6 MiB running (1): test_abc (42.9 sec)
0:01:13 load avg: 1.08 mem: 589.1 MiB running (1): test_abc (1 min 13 sec)
0:01:43 load avg: 0.66 mem: 590.1 MiB running (1): test_abc (1 min 43 sec)
...
0:36:00 load avg: 0.07 mem: 531.8 MiB running (1): test_abc (35 min 59 sec)
0:36:30 load avg: 0.04 mem: 531.8 MiB running (1): test_abc (36 min 29 sec)
Error: The operation was canceled.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Jun 17, 2026

Copy link
Copy Markdown
Member Author

With the PR the assert_usable() path doesn't hang anymore and doesn't leak and the run_case() path still times outs but doesn't leak.

Oh thanks for testing manually! It's good that you managed to confirm that my fix works as expected :-)

Nitpick: I'd change the helper name to something other as kill has a very specific meaning and the helper also closes the pipes, it doesn't only kill the processes. Maybe cleanup_process_group() or something?

Calling proc.communicate() # Clean up in kill_process_group() is just there to call waitpid(pid) to avoid zombie process. Since the process and its child processes are killed, it should no longer have to wait for pipes to be closed, they should be already closed.

I prefer to keep "kill" in the function name, since it's the main action of the function.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
vstinner merged commit a064b32 into python:main Jun 17, 2026
165 of 171 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
vstinner deleted the test_dtrace_pg branch June 17, 2026 08:58
@miss-islington-app

miss-islington-app Bot commented Jun 17, 2026

Copy link
Copy Markdown

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the awaiting core review label Jun 17, 2026
@bedevere-app

bedevere-app Bot commented Jun 17, 2026

Copy link
Copy Markdown

GH-151589 is a backport of this pull request to the 3.15 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-app bedevere-app Bot removed the needs backport to 3.15 bugs and security fixes label Jun 17, 2026
vstinner added a commit that referenced this pull request Jun 17, 2026
@miss-islington @vstinner
[3.15] gh-151496: Use process groups in test_dtrace (GH-151512) (#151589
d7e7d85
)

gh-151496: Use process groups in test_dtrace (GH-151512)

Create a new process group to run bpftrace commands, so it's possible
to kill also child processes on timeout.
(cherry picked from commit a064b32)

Co-authored-by: Victor Stinner <[email protected]>
philthompson10 pushed a commit to philthompson10/cpython that referenced this pull request Jun 17, 2026
@vstinner @philthompson10
pythongh-151496: Use process groups in test_dtrace (python#151512)
f33d8f5
Create a new process group to run bpftrace commands, so it's possible
to kill also child processes on timeout.
@stratakis stratakis mentioned this pull request Jun 24, 2026
gh-151496: Use process groups in TraceBackend in test_dtrace #152039
Merged
vstinner added a commit that referenced this pull request Oct 16, 2026
@vstinner @stratakis
[3.14] gh-156701: Backport test_dtrace from main (#159596)
566f815
* gh-98894: Fix DTrace test_check_probes for shared builds (#151122)

When building with --enable-shared, the SystemTap/DTrace notes
live in libpython. Add detection logic to be used by readelf.

Force the C locale on readelf output.

(cherry picked from commit 3a08e93)

* gh-151496: Use process groups in test_dtrace (#151512)

Create a new process group to run bpftrace commands, so it's possible
to kill also child processes on timeout.

(cherry picked from commit a064b32)

* gh-151496: Use process groups in TraceBackend in test_dtrace (#152039)

Run the generic DTrace/SystemTap commands in a new process group and
terminate the whole group on timeout.

This prevents a forked tracer child from keeping stdout/stderr pipes
open after the direct tracer process is killed.

(cherry picked from commit 1785f4b)

* gh-98894: Check readelf failures in test_dtrace (#152345)

Report readelf command failures directly instead of later failing with
missing-probe assertions.

(cherry picked from commit d60a677)

* gh-98894: Check tracer exit status in test_dtrace (#152893)

Fail functional dtrace and SystemTap cases directly when the tracer exits
non-zero.

(cherry picked from commit 311e7e7)

* gh-98894: Quote test_dtrace tracer subcommands (#152901)

Use shlex.join() when building tracer -c command strings
for avoiding issues with spaces.

(cherry picked from commit a159c68)

* gh-98894: Fix dtrace tests in shared builds (#153372)

Generate SystemTap probe definitions targeting libpython for shared
builds and use centralized USDT probe object discovery for readelf
and BPFTrace.

(cherry picked from commit 298bef3)

* gh-156703: Set LD_LIBRARY_PATH in SystemTap trace commands (GH-156424)

Prefix commands passed to stap -c with LD_LIBRARY_PATH via env, when
probes reside in libpython. This allows the interpreter to find
libpython when staprun does not preserve the loader path.

(cherry picked from commit faf4e52)

* gh-156722, test_dtrace: Terminate tracing processes gracefully on timeout (#156753)

Send SIGTERM to the tracing process group before falling back to SIGKILL.

(cherry picked from commit 55ecc0a)

* gh-156701: Increase the SystemTap usability timeout (#156720)

SystemTap may compile a kernel module during its usability check
(if no cached module is available). We need to allow a larger timeout
for that case.

(cherry picked from commit e1be613)

---------

Co-authored-by: stratakis <[email protected]>
Co-authored-by: stratakis <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@stratakis stratakis stratakis left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news tests Tests in the Lib/test dir

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@vstinner @stratakis
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.