GeezerVison is showing github.com — Missing DTrace probes · Issue #104280 · python/cpython
Missing DTrace probes #104280Labels 3.13only security fixesonly security fixes3.14bugs and security fixesbugs and security fixes3.15bugs and security fixesbugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error Description
Bug report
On Linux, you can list available DTrace probes by running /bcc/tools/tplist.py tool;
$ ~/bcc/tools/tplist.py -l cpython/python
cpython/python python:function__entry
cpython/python python:function__return
cpython/python python:line
cpython/python python:import__find__load__done
cpython/python python:import__find__load__start
cpython/python python:audit
cpython/python python:gc__start
cpython/python python:gc__done The function__entry, function__return and line probes appear to be missing after the GH-103082: Implementation of PEP 669: Low Impact Monitoring for CPython update.
$ ~/bcc/tools/tplist.py -l cpython/python
cpython/python python:import__find__load__done
cpython/python python:import__find__load__start
cpython/python python:audit
cpython/python python:gc__start
cpython/python python:gc__done Your environment
- CPython versions tested on: Python 3.12.0a7+
- Operating system and architecture: Arch Linux x86_64
Linked PRs
Reacted by Aryeh Hillman and Mark Smith Activitysunmy2019 commented on May 8, 2023 gaogaotiantian commented on May 9, 2023 I think DTrace was intentionally removed because it was not being used. I forgot where I read that, but Mark will have a better idea of it. brandtbucher commented on May 10, 2023 I think a bigger issue is that I'm pretty sure it isn't tested on any CI jobs or buildbots, so stuff can break (like #98894) without us noticing.
If it's not tested, it effectively isn't supported. Not sure whether the right path forward is for some motivated person to fix and maintain it, for us to remove the broken stuff and keep the working stuff (with a big disclaimer in the docs), or to just rip it out entirely.
Anecdotally, it's always been a bit uncomfortable to touch this stuff when modifying the interpreter, since I'm not very sure how it works or how to test it.
@ambv might have an idea of what the right path forward is here. furkanonder commented on May 25, 2023 ContributorAuthor More actions
I think DTrace was intentionally removed because it was not being used. I forgot where I read that, but Mark will have a better idea of it.
Dtrace support was actually being used. DTrace markers, also known as "probes", are not only used by DTtrace. For example, these probes can be used with bpftrace.
There are tools written using these probes. I will even give a talk about these probes at EuroPython 2023 :) Reacted by Daniel Golding and Mark Smith kchoudhu commented on Jul 10, 2023 Just chiming in here: we use these probes in anger in production, and their absence is a blocker preventing us from upgrading to 3.11.
If tests are needed to make sure dtrace support is accounted for in the release process, I am happy to contribute time to write tests and set them up. furkanonder commented on Jul 29, 2023 ContributorAuthor More actions @ambv @brandtbucher I wrote a test case to check for the presence of probes. Could you review the PR? encukou commented on Aug 8, 2023 The newly added tests fail in Fedora:
0:07:40 load avg: 5.34 Re-running test_dtrace in verbose mode (matching: test_check_probes, test_check_probes, test_check_probes, test_check_probes, test_check_probes)
readelf version: 2.40
test_check_probes (test.test_dtrace.CheckDtraceProbes.test_check_probes) ...
test_check_probes (test.test_dtrace.CheckDtraceProbes.test_check_probes) (probe_name='Name: import__find__load__done') ... FAIL
test_check_probes (test.test_dtrace.CheckDtraceProbes.test_check_probes) (probe_name='Name: import__find__load__start') ... FAIL
test_check_probes (test.test_dtrace.CheckDtraceProbes.test_check_probes) (probe_name='Name: audit') ... FAIL
test_check_probes (test.test_dtrace.CheckDtraceProbes.test_check_probes) (probe_name='Name: gc__start') ... FAIL
test_check_probes (test.test_dtrace.CheckDtraceProbes.test_check_probes) (probe_name='Name: gc__done') ... FAIL
======================================================================
FAIL: test_check_probes (test.test_dtrace.CheckDtraceProbes.test_check_probes) (probe_name='Name: import__find__load__done')
----------------------------------------------------------------------
Traceback (most recent call last):
File "/builddir/build/BUILD/Python-3.12.0rc1/Lib/test/test_dtrace.py", line 241, in test_check_probes
self.assertIn(probe_name, readelf_output)
AssertionError: 'Name: import__find__load__done' not found in '\nDisplaying notes found in: .note.gnu.property\n Owner Data size \tDescription\n GNU 0x00000010\tNT_GNU_PROPERTY_TYPE_0\n Properties: AArch64 feature: BTI, PAC\n\nDisplaying notes found in: .note.gnu.build-id\n Owner Data size \tDescription\n GNU 0x00000014\tNT_GNU_BUILD_ID (unique build ID bitstring)\n Build ID: 3b49ac28c70a95104b93815144881aadefb7bad1\n\nDisplaying notes found in: .note.ABI-tag\n Owner Data size \tDescription\n GNU 0x00000010\tNT_GNU_ABI_TAG (ABI version tag)\n OS: Linux, ABI: 3.7.0\n\nDisplaying notes found in: .note.package\n Owner Data size \tDescription\n FDO 0x00000084\tFDO_PACKAGING_METADATA\n Packaging Metadata: {"type":"rpm","name":"python3.12","version":"3.12.0~rc1-1.fc39","architecture":"aarch64","osCpe":"cpe:/o:fedoraproject:fedora:39"}\n\nDisplaying notes found in: .gnu.build.attributes\n Owner Data size \tDescription\n GA$<version>3a1 0x00000010\tOPEN\n Applies to region from 0x100c0 to 0x100f4\n GA$<version>3a1 0x00000010\tOPEN\n Applies to region from 0x100f4 to 0x10108\n GA$<version>3a1 0x00000010\tOPEN\n Applies to region from 0x10000 to 0x10010\n GA$<version>3a1 0x00000010\tOPEN\n Applies to region from 0x10204 to 0x10210\n GA$<version>3a1 0x00000010\tOPEN\n Applies to region from 0x10110 to 0x101d8\n GA$<version>3a1 0x00000010\tOPEN\n Applies to region from 0x10204 to 0x10204\n GA$<version>3a1 0x00000010\tOPEN\n Applies to region from 0x10204 to 0x10204\n GA$<version>3a1 0x00000010\tOPEN\n Applies to region from 0x10010 to 0x1001c\n GA$<version>3a1 0x00000010\tOPEN\n Applies to region from 0x10210 to 0x1021c\n'
Full log: https://kojipkgs.fedoraproject.org//work/tasks/6927/104486927/build.log
Here's a readelf output printed out:
Displaying notes found in: .note.gnu.property
Owner Data size Description
GNU 0x00000010 NT_GNU_PROPERTY_TYPE_0
Properties: AArch64 feature: BTI, PAC
Displaying notes found in: .note.gnu.build-id
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: 3b49ac28c70a95104b93815144881aadefb7bad1
Displaying notes found in: .note.ABI-tag
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 3.7.0
Displaying notes found in: .note.package
Owner Data size Description
FDO 0x00000084 FDO_PACKAGING_METADATA
Packaging Metadata: {"type":"rpm","name":"python3.12","version":"3.12.0~rc1-1.fc39","architecture":"aarch64","osCpe":"cpe:/o:fedoraproject:fedora:39"}
Displaying notes found in: .gnu.build.attributes
Owner Data size Description
GA$<version>3a1 0x00000010 OPEN
Applies to region from 0x100c0 to 0x100f4
GA$<version>3a1 0x00000010 OPEN
Applies to region from 0x100f4 to 0x10108
GA$<version>3a1 0x00000010 OPEN
Applies to region from 0x10000 to 0x10010
GA$<version>3a1 0x00000010 OPEN
Applies to region from 0x10204 to 0x10210
GA$<version>3a1 0x00000010 OPEN
Applies to region from 0x10110 to 0x101d8
GA$<version>3a1 0x00000010 OPEN
Applies to region from 0x10204 to 0x10204
GA$<version>3a1 0x00000010 OPEN
Applies to region from 0x10204 to 0x10204
GA$<version>3a1 0x00000010 OPEN
Applies to region from 0x10010 to 0x1001c
GA$<version>3a1 0x00000010 OPEN
Applies to region from 0x10210 to 0x1021c
I'm unfortunately unfamiliar with DTrace. Is this expected? How can I debug this? ambv commented on Aug 8, 2023 Are you building your Python --with-dtrace? If not, it's a surprise that sysconfig.get_config_var('WITH_DTRACE') returns 1 in your case...
When I build --with-dtrace locally, I get 1 but when I don't, that config var returns 0. In this case the test is just skipped:
setUpClass (test.test_dtrace.CheckDtraceProbes) ... skipped 'CPython must be configured with the --with-dtrace option.'
ambv commented on Aug 8, 2023 BTW, this says "dtrace" but those probes are also SystemTap / eBPF. markshannon commented on Oct 5, 2023 dtrace is supposed to have zero (or very close to it) overhead when not in use.
The CPython implementation failed in this. If it had, perhaps PEP 669 would not have been needed.
I think it is best to push users towards using sys.monitoring now that 3.12 has been released.
I realize that's not the ideal solution for existing users of dtrace, but sys.monitoring doesn't need a special build and is tested on CI. markshannon commented on Oct 5, 2023 It also looks like dtrace function entry robes are broken in 3.11, as they are only invoked if sys.settrace tracing is also turned on.
The probe point is present, but the normal code path skips it. kchoudhu commented on Oct 5, 2023 With dtrace support effectively removed in 3.12, a few thoughts.
My initial read of sys.monitoring is that while it is great for bespoke, python-specific monitoring and tooling, it doesn't quite work in a world that is increasingly moving towards one-size fits all distributed tracing via ebpf (i.e. dtrace).
Is there room in Python for both visions of tracing? If not, why? If yes, what kind of resourcing would be needed to make sure that dtrace isn't removed again from the build? Zheaoli commented on Oct 9, 2023
dtrace is supposed to have zero (or very close to it) overhead when not in use. The CPython implementation failed in this. If it had, perhaps PEP 669 would not have been needed. I think it is best to push users towards using sys.monitoring now that 3.12 has been released.
I realize that's not the ideal solution for existing users of dtrace, but sys.monitoring doesn't need a special build and is tested on CI.
This is a little bit weird, the DTrace and the PEP 669 are different features.
the DTrace API is for people who want to trace the Python process without restarting the process and injecting some code for the PEP 669
We can solve the process tracing problem by using #96123 in some circumstances. But there are still some issues.
- the perf API seems unstable
- we can only get the address after the process has been started. it will cause us to miss some information we need.
If we decide to remove DTrace Support officially, we may need to update the documentation https://docs.python.org/3/howto/instrumentation.html avioligo commented on Jan 9, 2024 When building python 3.11.7 from source, I ran into this error:
python configure: error: dtrace command not found on $PATH
Fixed using:
sudo apt-get install systemtap-sdt-dev
By installing dtrace (well almost) toolkit - the binary is not really used for tracing but only to list probes if I am correct. Bill-Sommerfeld commented on Feb 9, 2024 I took a look at this yesterday in 3.12 as part of my current effort to package 3.12 for OpenIndiana (illumos) and I got the function entry/exit probes restored and working well enough to pass the test_dtrace.py test.
I'll share when I get a chance - I'm doing this with original dtrace on illumos (the open-source continuation of opensolaris) but it would likely also work for those who want to use these probes via systemtap.
I'll also look at the line probes when I get a chance. Bill-Sommerfeld commented on Feb 9, 2024 https://github.com/Bill-Sommerfeld/cpython/tree/dtrace-3.12 has patches that restore the function-entry and function-return probes for me on illumos. The tests now pass for me, except for test.test_dtrace.DTraceNormalTests.test_line and
test.test_dtrace.DTraceOptimizedTests.test_line which depend on the line probe which I haven't restored yet.
I'd appreciate it if others could give these a whirl on other platforms.
MetadataMetadata3.13only security fixesonly security fixes3.14bugs and security fixesbugs and security fixes3.15bugs and security fixesbugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DevelopmentNo branches or pull requestsIssue actions
|