GeezerVison is showing github.com — test_perf_profiler fails on Linux (both AArch64 and x86) · Issue #124613 · python/cpython
test_perf_profiler fails on Linux (both AArch64 and x86) #124613DescriptionBug report
Bug description:
Our CI picked up this failure recently. We are using Ubuntu 24.04 and it is failing both on AArch64 and x86.
$ ./python -m test test_perf_profiler -v
== CPython 3.14.0a0 (heads/main-dirty:7a839d6013, Sep 26 2024, 14:53:59) [GCC 13.2.0]
== Linux-6.8.0-44-generic-aarch64-with-glibc2.39 little-endian
== Python build: release
== cwd: /home/user/work/ce-sw/repos/cpython/build/build/test_python_worker_1359145æ
== CPU count: 8
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests
Using random seed: 1215453149
0:00:00 load avg: 0.44 Run 1 test sequentially in a single process
0:00:00 load avg: 0.44 [1/1] test_perf_profiler
test_pre_fork_compile (test.test_perf_profiler.TestPerfProfiler.test_pre_fork_compile) ... skipped 'Unwinding is unreliable with frame pointers'
test_python_calls_appear_in_the_stack_if_perf_activated (test.test_perf_profiler.TestPerfProfiler.test_python_calls_appear_in_the_stack_if_perf_activated) ... skipped 'Unwinding is unreliable with frame pointers'
test_python_calls_do_not_appear_in_the_stack_if_perf_deactivated (test.test_perf_profiler.TestPerfProfiler.test_python_calls_do_not_appear_in_the_stack_if_perf_deactivated) ... skipped 'Unwinding is unreliable with frame pointers'
test_python_calls_appear_in_the_stack_if_perf_activated (test.test_perf_profiler.TestPerfProfilerWithDwarf.test_python_calls_appear_in_the_stack_if_perf_activated) ... perf record -g --call-graph=dwarf,65528 -F99 -k1 -o /tmp/tmp52ljdy43/perf_output.perf -- /home/user/work/ce-sw/repos/cpython/build/python -Xperf_jit /tmp/tmp52ljdy43/perftest.py
FAIL
test_python_calls_do_not_appear_in_the_stack_if_perf_deactivated (test.test_perf_profiler.TestPerfProfilerWithDwarf.test_python_calls_do_not_appear_in_the_stack_if_perf_deactivated) ... perf record -g --call-graph=dwarf,65528 -F99 -k1 -o /tmp/tmpj41x5k9d/perf_output.perf -- /home/user/work/ce-sw/repos/cpython/build/python /tmp/tmpj41x5k9d/perftest.py
ok
test_sys_api (test.test_perf_profiler.TestPerfTrampoline.test_sys_api) ... ok
test_sys_api_get_status (test.test_perf_profiler.TestPerfTrampoline.test_sys_api_get_status) ... ok
test_sys_api_with_existing_trampoline (test.test_perf_profiler.TestPerfTrampoline.test_sys_api_with_existing_trampoline) ... ok
test_sys_api_with_invalid_trampoline (test.test_perf_profiler.TestPerfTrampoline.test_sys_api_with_invalid_trampoline) ... ok
test_trampoline_works (test.test_perf_profiler.TestPerfTrampoline.test_trampoline_works) ... ok
test_trampoline_works_with_forks (test.test_perf_profiler.TestPerfTrampoline.test_trampoline_works_with_forks) ... ok
======================================================================
FAIL: test_python_calls_appear_in_the_stack_if_perf_activated (test.test_perf_profiler.TestPerfProfilerWithDwarf.test_python_calls_appear_in_the_stack_if_perf_activated)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/user/work/ce-sw/repos/cpython/Lib/test/test_perf_profiler.py", line 357, in test_python_calls_appear_in_the_stack_if_perf_activated
self.assertIn(f"py::bar:{script}", stdout)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'py::bar:/tmp/tmp52ljdy43/perftest.py' not found in 'python 1359175 378278.191585: 10101010 task-clock:ppp: \n\t b5794db789fc _PyInterpreterState_GET+0x8 (inlined)\n\t b5794db789fc get_state+0x8 (inlined)\n\t b5794db789fc _PyObject_Malloc+0x8 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794db2c383 _PyLong_FromMedium+0xff (inlined)\n\t b5794db2c383 PyLong_FromLong+0xff (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da8e087 _PyEval_EvalFrameDefault+0xa4f7 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.201687: 10101010 task-clock:ppp: \n\t b5794db250f0 x_add+0x20 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da8800f _PyEval_EvalFrameDefault+0x447f (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.211789: 10101010 task-clock:ppp: \n\t b5794db789fc _PyInterpreterState_GET+0x8 (inlined)\n\t b5794db789fc get_state+0x8 (inlined)\n\t b5794db789fc _PyObject_Malloc+0x8 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794db25143 _PyLong_New+0x73 (inlined)\n\t b5794db25143 x_add+0x73 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da8800f _PyEval_EvalFrameDefault+0x447f (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.221886: 10101010 task-clock:ppp: \n\t b5794db60eac _PyInterpreterState_GET+0xc (inlined)\n\t b5794db60eac get_state+0xc (inlined)\n\t b5794db60eac _PyObject_Free+0xc (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794db60eac _PyObject_Free+0xc (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da88257 _PyEval_EvalFrameDefault+0x46c7 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.231989: 10101010 task-clock:ppp: \n\t b5794db25218 long_normalize+0x148 (inlined)\n\t b5794db25218 x_add+0x148 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da8800f _PyEval_EvalFrameDefault+0x447f (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.242093: 10101010 task-clock:ppp: \n\t b5794da88034 _PyEval_EvalFrameDefault+0x44a4 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.252194: 10101010 task-clock:ppp: \n\t b5794da88020 _PyEval_EvalFrameDefault+0x4490 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.262291: 10101010 task-clock:ppp: \n\t b5794da88250 _PyEval_EvalFrameDefault+0x46c0 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.272398: 10101010 task-clock:ppp: \n\t b5794da8822c _PyEval_EvalFrameDefault+0x469c (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.282497: 10101010 task-clock:ppp: \n\t b5794db78ac4 pymalloc_pool_extend+0xd0 (inlined)\n\t b5794db78ac4 pymalloc_alloc+0xd0 (inlined)\n\t b5794db78ac4 _PyObject_Malloc+0xd0 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794db2c383 _PyLong_FromMedium+0xff (inlined)\n\t b5794db2c383 PyLong_FromLong+0xff (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da8e087 _PyEval_EvalFrameDefault+0xa4f7 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.292598: 10101010 task-clock:ppp: \n\t b5794db250f0 x_add+0x20 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da8800f _PyEval_EvalFrameDefault+0x447f (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.302700: 10101010 task-clock:ppp: \n\t b5794db60eac _PyInterpreterState_GET+0xc (inlined)\n\t b5794db60eac get_state+0xc (inlined)\n\t b5794db60eac _PyObject_Free+0xc (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794db60eac _PyObject_Free+0xc (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da88257 _PyEval_EvalFrameDefault+0x46c7 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.312798: 10101010 task-clock:ppp: \n\t b5794db250f0 x_add+0x20 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da8800f _PyEval_EvalFrameDefault+0x447f (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.322900: 10101010 task-clock:ppp: \n\t b5794db25124 _PyLong_New+0x54 (inlined)\n\t b5794db25124 x_add+0x54 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da8800f _PyEval_EvalFrameDefault+0x447f (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.333001: 10101010 task-clock:ppp: \n\t b5794da8a200 _PyEval_EvalFrameDefault+0x6670 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.343099: 10101010 task-clock:ppp: \n\t b5794da89784 Py_INCREF+0x5bf4 (inlined)\n\t b5794da89784 _Py_NewRef+0x5bf4 (inlined)\n\t b5794da89784 _PyEval_EvalFrameDefault+0x5bf4 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.353200: 10101010 task-clock:ppp: \n\t b5794db25214 long_normalize+0x144 (inlined)\n\t b5794db25214 x_add+0x144 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da8800f _PyEval_EvalFrameDefault+0x447f (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.363305: 10101010 task-clock:ppp: \n\t b5794da89784 Py_INCREF+0x5bf4 (inlined)\n\t b5794da89784 _Py_NewRef+0x5bf4 (inlined)\n\t b5794da89784 _PyEval_EvalFrameDefault+0x5bf4 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.373421: 10101010 task-clock:ppp: \n\t b5794db2c3d4 PyLong_FromLong+0x150 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da8e087 _PyEval_EvalFrameDefault+0xa4f7 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.383504: 10101010 task-clock:ppp: \n\t b5794db251b8 x_add+0xe8 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da8800f _PyEval_EvalFrameDefault+0x447f (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.393612: 10101010 task-clock:ppp: \n\t b5794da897b0 _PyEval_EvalFrameDefault+0x5c20 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.403708: 10101010 task-clock:ppp: \n\t b5794db60eac _PyInterpreterState_GET+0xc (inlined)\n\t b5794db60eac get_state+0xc (inlined)\n\t b5794db60eac _PyObject_Free+0xc (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794db60eac _PyObject_Free+0xc (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da88257 _PyEval_EvalFrameDefault+0x46c7 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.413813: 10101010 task-clock:ppp: \n\t b5794db77c14 PyObject_Malloc+0x14 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794db25143 _PyLong_New+0x73 (inlined)\n\t b5794db25143 x_add+0x73 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da8800f _PyEval_EvalFrameDefault+0x447f (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.423910: 10101010 task-clock:ppp: \n\t b5794db60f28 _PyObject_Free+0x88 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da88257 _PyEval_EvalFrameDefault+0x46c7 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.434011: 10101010 task-clock:ppp: \n\t b5794db251dc x_add+0x10c (/home/user/work/ce-sw/repos/cpython/build/python)\n\t b5794da8800f _PyEval_EvalFrameDefault+0x447f (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\npython 1359175 378278.444112: 10101010 task-clock:ppp: \n\t b5794da8e080 _PyEval_EvalFrameDefault+0xa4f0 (/home/user/work/ce-sw/repos/cpython/build/python)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\t e917a72f05cb py::foo:/tmp/tmp52ljdy43/perftest.py+0xb (/tmp/perf-1359175.map)\n\n'
----------------------------------------------------------------------
Ran 11 tests in 2.981s
FAILED (failures=1, skipped=3)
test test_perf_profiler failed
test_perf_profiler failed (1 failure)
== Tests result: FAILURE ==
1 test failed:
test_perf_profiler
Total duration: 5.2 sec
Total tests: run=11 failures=1 skipped=3
Total test files: run=1/1 failed=1
Result: FAILURE
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Linked PRs
Activitymdboom commented on Sep 26, 2024 Cc: @pablogsal (as someone who knows a lot about Linux perf) pablogsal commented on Sep 26, 2024 The integration has been broken since #123472 (comment) with the stable buildbots failing. Seems that something on that PR broke the buildbot or something changed at that time. vstinner commented on Sep 27, 2024 vstinner commented on Sep 28, 2024 diegorusso commented on Sep 30, 2024 ContributorAuthor More actions This is still failing on our x86-64 machine
[2024-09-30T12:05:29.521Z] ======================================================================
[2024-09-30T12:05:29.521Z] FAIL: test_python_calls_appear_in_the_stack_if_perf_activated (test.test_perf_profiler.TestPerfProfilerWithDwarf.test_python_calls_appear_in_the_stack_if_perf_activated)
[2024-09-30T12:05:29.522Z] ----------------------------------------------------------------------
[2024-09-30T12:05:29.522Z] Traceback (most recent call last):
[2024-09-30T12:05:29.522Z] File "/home/user/ci-scripts/cpython/Lib/test/test_perf_profiler.py", line 355, in test_python_calls_appear_in_the_stack_if_perf_activated
[2024-09-30T12:05:29.522Z] self.assertIn(f"py::foo:{script}", stdout)
[2024-09-30T12:05:29.522Z] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-09-30T12:05:29.522Z] AssertionError: 'py::foo:/tmp/test_python_a_xa7rb8/tmpwz5q1j12/perftest.py' not found in 'perf-exec 20962 10544768.681288: 1 cycles:P: \n\tffffffff9f80eea4 [unknown] ([unknown])\n\tffffffff9f80f83b [unknown] ([unknown])\n\tffffffff9faa9718 [unknown] ([unknown])\n\tffffffff9faaf531 [unknown] ([unknown])\n\tffffffff9fba88fa [unknown] ([unknown])\n\tffffffff9fc20990 [unknown] ([unknown])\n\tffffffff9fba6881 [unknown] ([unknown])\n\tffffffff9fba6a61 [unknown] ([unknown])\n\tffffffff9fba75a1 [unknown] ([unknown])\n\tffffffff9fba769e [unknown] ([unknown])\n\tffffffff9fba79e8 [unknown] ([unknown])\n\tffffffff9fba7d07 [unknown] ([unknown])\n\tffffffff9f8054f0 [unknown] ([unknown])\n\tffffffffa05c2ce6 [unknown] ([unknown])\n\tffffffffa06000df [unknown] ([unknown])\n\nperf-exec 20962 10544768.681312: 1 cycles:P: \n\tffffffff9f80eea4 [unknown] ([unknown])\n\tffffffff9f80f83b [unknown] ([unknown])\n\tffffffff9faa9718 [unknown] ([unknown])\n\tffffffff9faaf531 [unknown] ([unknown])\n\tffffffff9fba88fa [unknown] ([unknown])\n\tffffffff9fc20990 [unknown] ([unknown])\n\tffffffff9fba6881 [unknown] ([unknown])\n\tffffffff9fba6a61 [unknown] ([unknown])\n\tffffffff9fba75a1 [unknown] ([unknown])\n\tffffffff9fba769e [unknown] ([unknown])\n\tffffffff9fba79e8 [unknown] ([unknown])\n\tffffffff9fba7d07 [unknown] ([unknown])\n\tffffffff9f8054f0 [unknown] ([unknown])\n\tffffffffa05c2ce6 [unknown] ([unknown])\n\tffffffffa06000df [unknown] ([unknown])\n\nperf-exec 20962 10544768.681318: 1 cycles:P: \n\tffffffff9f80eea4 [unknown] ([unknown])\n\tffffffff9f80f83b [unknown] ([unknown])\n\tffffffff9faa9718 [unknown] ([unknown])\n\tffffffff9faaf531 [unknown] ([unknown])\n\tffffffff9fba88fa [unknown] ([unknown])\n\tffffffff9fc20990 [unknown] ([unknown])\n\tffffffff9fba6881 [unknown] ([unknown])\n\tffffffff9fba6a61 [unknown] ([unknown])\n\tffffffff9fba75a1 [unknown] ([unknown])\n\tffffffff9fba769e [unknown] ([unknown])\n\tffffffff9fba79e8 [unknown] ([unknown])\n\tffffffff9fba7d07 [unknown] ([unknown])\n\tffffffff9f8054f0 [unknown] ([unknown])\n\tffffffffa05c2ce6 [unknown] ([unknown])\n\tffffffffa06000df [unknown] ([unknown])\n\nperf-exec 20962 10544768.681324: 51 cycles:P: \n\tffffffff9f80eea4 [unknown] ([unknown])\n\tffffffff9f80f83b [unknown] ([unknown])\n\tffffffff9faa9718 [unknown] ([unknown])\n\tffffffff9faaf531 [unknown] ([unknown])\n\tffffffff9fba88fa [unknown] ([unknown])\n\tffffffff9fc20990 [unknown] ([unknown])\n\tffffffff9fba6881 [unknown] ([unknown])\n\tffffffff9fba6a61 [unknown] ([unknown])\n\tffffffff9fba75a1 [unknown] ([unknown])\n\tffffffff9fba769e [unknown] ([unknown])\n\tffffffff9fba79e8 [unknown] ([unknown])\n\tffffffff9fba7d07 [unknown] ([unknown])\n\tffffffff9f8054f0 [unknown] ([unknown])\n\tffffffffa05c2ce6 [unknown] ([unknown])\n\tffffffffa06000df [unknown] ([unknown])\n\nperf-exec 20962 10544768.681330: 9627 cycles:P: \n\tffffffff9f80eea4 [unknown] ([unknown])\n\tffffffff9f80f83b [unknown] ([unknown])\n\tffffffff9faa9718 [unknown] ([unknown])\n\tffffffff9faaf531 [unknown] ([unknown])\n\tffffffff9fba88fa [unknown] ([unknown])\n\tffffffff9fc20990 [unknown] ([unknown])\n\tffffffff9fba6881 [unknown] ([unknown])\n\tffffffff9fba6a61 [unknown] ([unknown])\n\tffffffff9fba75a1 [unknown] ([unknown])\n\tffffffff9fba769e [unknown] ([unknown])\n\tffffffff9fba79e8 [unknown] ([unknown])\n\tffffffff9fba7d07 [unknown] ([unknown])\n\tffffffff9f8054f0 [unknown] ([unknown])\n\tffffffffa05c2ce6 [unknown] ([unknown])\n\tffffffffa06000df [unknown] ([unknown])\n\npython 20962 10544768.682237: 2175145 cycles:P: \n\t 7f0f4da5560a do_lookup_x+0xda (/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)\n\t 7f0f4da561fb _dl_lookup_symbol_x+0x12b (/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)\n\t 7f0f4da5aae2 resolve_map+0x8a2 (inlined)\n\t 7f0f4da5aae2 elf_machine_rela+0x8a2 (inlined)\n\t 7f0f4da5aae2 elf_dynamic_do_Rela+0x8a2 (inlined)\n\t 7f0f4da5aae2 _dl_relocate_object+0x8a2 (/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)\n\t 7f0f4da6d788 dl_main+0x1c98 (/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)\n\t 7f0f4da69f45 _dl_sysdep_start+0x85 (/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)\n\t 7f0f4da6b75d _dl_start_final+0x58d (inlined)\n\t 7f0f4da6b75d _dl_start+0x58d (/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)\n\t 7f0f4da6a547 _start+0x7 (/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)\n\npython 20962 10544768.859556: 425076627 cycles:P: \n\t 55dbd22a5029 arena_map_is_used+0x79 (inlined)\n\t 55dbd22a5029 address_in_range+0x79 (inlined)\n\t 55dbd22a5029 pymalloc_free+0x79 (inlined)\n\t 55dbd22a5029 _PyObject_Free+0x79 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a5029 _PyObject_Free+0x79 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\npython 20962 10544768.870755: 26855347 cycles:P: \n\t 55dbd2267a46 long_dealloc+0x56 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d18828a [unknown] ([unknown])\n\npython 20962 10544768.881594: 25984233 cycles:P: \n\t 55dbd22bc261 pymalloc_alloc+0x41 (inlined)\n\t 55dbd22bc261 _PyObject_Malloc+0x41 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd226864e _PyLong_New+0x5e (inlined)\n\t 55dbd226864e x_add+0x5e (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544768.892192: 25400576 cycles:P: \n\t 7f0f4d188279 [unknown] ([unknown])\n\npython 20962 10544768.902620: 25009362 cycles:P: \n\t 55dbd2268663 Py_SET_TYPE+0x73 (inlined)\n\t 55dbd2268663 _PyObject_Init+0x73 (inlined)\n\t 55dbd2268663 _PyLong_New+0x73 (inlined)\n\t 55dbd2268663 x_add+0x73 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544768.912977: 24822660 cycles:P: \n\t 55dbd22a5025 arena_map_is_used+0x75 (inlined)\n\t 55dbd22a5025 address_in_range+0x75 (inlined)\n\t 55dbd22a5025 pymalloc_free+0x75 (inlined)\n\t 55dbd22a5025 _PyObject_Free+0x75 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a5025 _PyObject_Free+0x75 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\npython 20962 10544768.923245: 24622936 cycles:P: \n\t 55dbd22a4fd1 _PyInterpreterState_GET+0x21 (inlined)\n\t 55dbd22a4fd1 get_state+0x21 (inlined)\n\t 55dbd22a4fd1 _PyObject_Free+0x21 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a4fd1 _PyObject_Free+0x21 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\npython 20962 10544768.933474: 24527621 cycles:P: \n\t 55dbd22a4fd9 get_state+0x29 (inlined)\n\t 55dbd22a4fd9 _PyObject_Free+0x29 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a4fd9 _PyObject_Free+0x29 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d18828a [unknown] ([unknown])\n\npython 20962 10544768.943661: 24426125 cycles:P: \n\t 55dbd229df92 new_reference+0x12 (inlined)\n\t 55dbd229df92 _Py_NewReference+0x12 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd2268686 _PyObject_Init+0x96 (inlined)\n\t 55dbd2268686 _PyLong_New+0x96 (inlined)\n\t 55dbd2268686 x_add+0x96 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544768.953830: 24377304 cycles:P: \n\t 7f0f4d188208 [unknown] ([unknown])\n\npython 20962 10544768.963972: 24323472 cycles:P: \n\t 55dbd2271b34 Py_INCREF+0x154 (inlined)\n\t 55dbd2271b34 _PyObject_Init+0x154 (inlined)\n\t 55dbd2271b34 _PyLong_FromMedium+0x154 (inlined)\n\t 55dbd2271b34 PyLong_FromLong+0x154 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d18815f [unknown] ([unknown])\n\npython 20962 10544768.974108: 24298036 cycles:P: \n\t 55dbd22bc261 pymalloc_alloc+0x41 (inlined)\n\t 55dbd22bc261 _PyObject_Malloc+0x41 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd226864e _PyLong_New+0x5e (inlined)\n\t 55dbd226864e x_add+0x5e (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544768.984236: 24273562 cycles:P: \n\t 55dbd22a5007 arena_map_get+0x57 (inlined)\n\t 55dbd22a5007 arena_map_is_used+0x57 (inlined)\n\t 55dbd22a5007 address_in_range+0x57 (inlined)\n\t 55dbd22a5007 pymalloc_free+0x57 (inlined)\n\t 55dbd22a5007 _PyObject_Free+0x57 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a5007 _PyObject_Free+0x57 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\npython 20962 10544768.994349: 24251966 cycles:P: \n\t 55dbd2267a18 long_dealloc+0x28 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\npython 20962 10544769.004465: 24245115 cycles:P: \n\t 55dbd22bc277 pymalloc_alloc+0x57 (inlined)\n\t 55dbd22bc277 _PyObject_Malloc+0x57 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd226864e _PyLong_New+0x5e (inlined)\n\t 55dbd226864e x_add+0x5e (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.014568: 24236552 cycles:P: \n\t 55dbd22686da x_add+0xea (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.024676: 24231731 cycles:P: \n\t 55dbd227746e _PyLong_Add+0x3e (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.034780: 24229045 cycles:P: \n\t 55dbd22bc270 pymalloc_alloc+0x50 (inlined)\n\t 55dbd22bc270 _PyObject_Malloc+0x50 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd2271b09 _PyLong_FromMedium+0x129 (inlined)\n\t 55dbd2271b09 PyLong_FromLong+0x129 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d18815f [unknown] ([unknown])\n\npython 20962 10544769.044887: 24225959 cycles:P: \n\t 7f0f4d188208 [unknown] ([unknown])\n\npython 20962 10544769.054987: 24221906 cycles:P: \n\t 55dbd2268663 Py_SET_TYPE+0x73 (inlined)\n\t 55dbd2268663 _PyObject_Init+0x73 (inlined)\n\t 55dbd2268663 _PyLong_New+0x73 (inlined)\n\t 55dbd2268663 x_add+0x73 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.065095: 24221703 cycles:P: \n\t 55dbd22a4ff4 arena_map_get+0x44 (inlined)\n\t 55dbd22a4ff4 arena_map_is_used+0x44 (inlined)\n\t 55dbd22a4ff4 address_in_range+0x44 (inlined)\n\t 55dbd22a4ff4 pymalloc_free+0x44 (inlined)\n\t 55dbd22a4ff4 _PyObject_Free+0x44 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a4ff4 _PyObject_Free+0x44 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\npython 20962 10544769.075194: 24219126 cycles:P: \n\t 7f0f4d188208 [unknown] ([unknown])\n\npython 20962 10544769.085298: 24218678 cycles:P: \n\t 55dbd22bc261 pymalloc_alloc+0x41 (inlined)\n\t 55dbd22bc261 _PyObject_Malloc+0x41 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd2271b09 _PyLong_FromMedium+0x129 (inlined)\n\t 55dbd2271b09 PyLong_FromLong+0x129 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d18815f [unknown] ([unknown])\n\npython 20962 10544769.095396: 24217814 cycles:P: \n\t 55dbd22a4fd1 _PyInterpreterState_GET+0x21 (inlined)\n\t 55dbd22a4fd1 get_state+0x21 (inlined)\n\t 55dbd22a4fd1 _PyObject_Free+0x21 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a4fd1 _PyObject_Free+0x21 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\npython 20962 10544769.105497: 24217334 cycles:P: \n\t 55dbd22bc234 _PyInterpreterState_GET+0x14 (inlined)\n\t 55dbd22bc234 get_state+0x14 (inlined)\n\t 55dbd22bc234 _PyObject_Malloc+0x14 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd226864e _PyLong_New+0x5e (inlined)\n\t 55dbd226864e x_add+0x5e (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.115596: 24219156 cycles:P: \n\t 55dbd2268663 Py_SET_TYPE+0x73 (inlined)\n\t 55dbd2268663 _PyObject_Init+0x73 (inlined)\n\t 55dbd2268663 _PyLong_New+0x73 (inlined)\n\t 55dbd2268663 x_add+0x73 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.125701: 24218620 cycles:P: \n\t 55dbd22a5029 arena_map_is_used+0x79 (inlined)\n\t 55dbd22a5029 address_in_range+0x79 (inlined)\n\t 55dbd22a5029 pymalloc_free+0x79 (inlined)\n\t 55dbd22a5029 _PyObject_Free+0x79 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a5029 _PyObject_Free+0x79 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\npython 20962 10544769.135798: 24217127 cycles:P: \n\t 55dbd22bc26d pymalloc_alloc+0x4d (inlined)\n\t 55dbd22bc26d _PyObject_Malloc+0x4d (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd226864e _PyLong_New+0x5e (inlined)\n\t 55dbd226864e x_add+0x5e (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.145904: 24217819 cycles:P: \n\t 55dbd22bc261 pymalloc_alloc+0x41 (inlined)\n\t 55dbd22bc261 _PyObject_Malloc+0x41 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd2271b09 _PyLong_FromMedium+0x129 (inlined)\n\t 55dbd2271b09 PyLong_FromLong+0x129 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d18815f [unknown] ([unknown])\n\npython 20962 10544769.156002: 24217028 cycles:P: \n\tffffffff9f84f3f3 [unknown] ([unknown])\n\tffffffff9f97d85a [unknown] ([unknown])\n\tffffffff9f887161 [unknown] ([unknown])\n\tffffffffa05c634e [unknown] ([unknown])\n\tffffffffa0600ecb [unknown] ([unknown])\n\t 55dbd22bc2d0 pymalloc_pool_extend+0xb0 (inlined)\n\t 55dbd22bc2d0 pymalloc_alloc+0xb0 (inlined)\n\t 55dbd22bc2d0 _PyObject_Malloc+0xb0 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd2271b09 _PyLong_FromMedium+0x129 (inlined)\n\t 55dbd2271b09 PyLong_FromLong+0x129 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d18815f [unknown] ([unknown])\n\npython 20962 10544769.166107: 24216861 cycles:P: \n\t 55dbd22bc261 pymalloc_alloc+0x41 (inlined)\n\t 55dbd22bc261 _PyObject_Malloc+0x41 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd226864e _PyLong_New+0x5e (inlined)\n\t 55dbd226864e x_add+0x5e (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.176207: 24216156 cycles:P: \n\t 55dbd22bc261 pymalloc_alloc+0x41 (inlined)\n\t 55dbd22bc261 _PyObject_Malloc+0x41 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd226864e _PyLong_New+0x5e (inlined)\n\t 55dbd226864e x_add+0x5e (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.186303: 24215716 cycles:P: \n\t 55dbd22a5007 arena_map_get+0x57 (inlined)\n\t 55dbd22a5007 arena_map_is_used+0x57 (inlined)\n\t 55dbd22a5007 address_in_range+0x57 (inlined)\n\t 55dbd22a5007 pymalloc_free+0x57 (inlined)\n\t 55dbd22a5007 _PyObject_Free+0x57 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a5007 _PyObject_Free+0x57 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\n'
pablogsal commented on Sep 30, 2024
This is still failing on our x86 machine
[2024-09-30T12:05:29.521Z] ======================================================================
[2024-09-30T12:05:29.521Z] FAIL: test_python_calls_appear_in_the_stack_if_perf_activated (test.test_perf_profiler.TestPerfProfilerWithDwarf.test_python_calls_appear_in_the_stack_if_perf_activated)
[2024-09-30T12:05:29.522Z] ----------------------------------------------------------------------
[2024-09-30T12:05:29.522Z] Traceback (most recent call last):
[2024-09-30T12:05:29.522Z] File "/home/user/ci-scripts/cpython/Lib/test/test_perf_profiler.py", line 355, in test_python_calls_appear_in_the_stack_if_perf_activated
[2024-09-30T12:05:29.522Z] self.assertIn(f"py::foo:{script}", stdout)
[2024-09-30T12:05:29.522Z] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-09-30T12:05:29.522Z] AssertionError: 'py::foo:/tmp/test_python_a_xa7rb8/tmpwz5q1j12/perftest.py' not found in 'perf-exec 20962 10544768.681288: 1 cycles:P: \n\tffffffff9f80eea4 [unknown] ([unknown])\n\tffffffff9f80f83b [unknown] ([unknown])\n\tffffffff9faa9718 [unknown] ([unknown])\n\tffffffff9faaf531 [unknown] ([unknown])\n\tffffffff9fba88fa [unknown] ([unknown])\n\tffffffff9fc20990 [unknown] ([unknown])\n\tffffffff9fba6881 [unknown] ([unknown])\n\tffffffff9fba6a61 [unknown] ([unknown])\n\tffffffff9fba75a1 [unknown] ([unknown])\n\tffffffff9fba769e [unknown] ([unknown])\n\tffffffff9fba79e8 [unknown] ([unknown])\n\tffffffff9fba7d07 [unknown] ([unknown])\n\tffffffff9f8054f0 [unknown] ([unknown])\n\tffffffffa05c2ce6 [unknown] ([unknown])\n\tffffffffa06000df [unknown] ([unknown])\n\nperf-exec 20962 10544768.681312: 1 cycles:P: \n\tffffffff9f80eea4 [unknown] ([unknown])\n\tffffffff9f80f83b [unknown] ([unknown])\n\tffffffff9faa9718 [unknown] ([unknown])\n\tffffffff9faaf531 [unknown] ([unknown])\n\tffffffff9fba88fa [unknown] ([unknown])\n\tffffffff9fc20990 [unknown] ([unknown])\n\tffffffff9fba6881 [unknown] ([unknown])\n\tffffffff9fba6a61 [unknown] ([unknown])\n\tffffffff9fba75a1 [unknown] ([unknown])\n\tffffffff9fba769e [unknown] ([unknown])\n\tffffffff9fba79e8 [unknown] ([unknown])\n\tffffffff9fba7d07 [unknown] ([unknown])\n\tffffffff9f8054f0 [unknown] ([unknown])\n\tffffffffa05c2ce6 [unknown] ([unknown])\n\tffffffffa06000df [unknown] ([unknown])\n\nperf-exec 20962 10544768.681318: 1 cycles:P: \n\tffffffff9f80eea4 [unknown] ([unknown])\n\tffffffff9f80f83b [unknown] ([unknown])\n\tffffffff9faa9718 [unknown] ([unknown])\n\tffffffff9faaf531 [unknown] ([unknown])\n\tffffffff9fba88fa [unknown] ([unknown])\n\tffffffff9fc20990 [unknown] ([unknown])\n\tffffffff9fba6881 [unknown] ([unknown])\n\tffffffff9fba6a61 [unknown] ([unknown])\n\tffffffff9fba75a1 [unknown] ([unknown])\n\tffffffff9fba769e [unknown] ([unknown])\n\tffffffff9fba79e8 [unknown] ([unknown])\n\tffffffff9fba7d07 [unknown] ([unknown])\n\tffffffff9f8054f0 [unknown] ([unknown])\n\tffffffffa05c2ce6 [unknown] ([unknown])\n\tffffffffa06000df [unknown] ([unknown])\n\nperf-exec 20962 10544768.681324: 51 cycles:P: \n\tffffffff9f80eea4 [unknown] ([unknown])\n\tffffffff9f80f83b [unknown] ([unknown])\n\tffffffff9faa9718 [unknown] ([unknown])\n\tffffffff9faaf531 [unknown] ([unknown])\n\tffffffff9fba88fa [unknown] ([unknown])\n\tffffffff9fc20990 [unknown] ([unknown])\n\tffffffff9fba6881 [unknown] ([unknown])\n\tffffffff9fba6a61 [unknown] ([unknown])\n\tffffffff9fba75a1 [unknown] ([unknown])\n\tffffffff9fba769e [unknown] ([unknown])\n\tffffffff9fba79e8 [unknown] ([unknown])\n\tffffffff9fba7d07 [unknown] ([unknown])\n\tffffffff9f8054f0 [unknown] ([unknown])\n\tffffffffa05c2ce6 [unknown] ([unknown])\n\tffffffffa06000df [unknown] ([unknown])\n\nperf-exec 20962 10544768.681330: 9627 cycles:P: \n\tffffffff9f80eea4 [unknown] ([unknown])\n\tffffffff9f80f83b [unknown] ([unknown])\n\tffffffff9faa9718 [unknown] ([unknown])\n\tffffffff9faaf531 [unknown] ([unknown])\n\tffffffff9fba88fa [unknown] ([unknown])\n\tffffffff9fc20990 [unknown] ([unknown])\n\tffffffff9fba6881 [unknown] ([unknown])\n\tffffffff9fba6a61 [unknown] ([unknown])\n\tffffffff9fba75a1 [unknown] ([unknown])\n\tffffffff9fba769e [unknown] ([unknown])\n\tffffffff9fba79e8 [unknown] ([unknown])\n\tffffffff9fba7d07 [unknown] ([unknown])\n\tffffffff9f8054f0 [unknown] ([unknown])\n\tffffffffa05c2ce6 [unknown] ([unknown])\n\tffffffffa06000df [unknown] ([unknown])\n\npython 20962 10544768.682237: 2175145 cycles:P: \n\t 7f0f4da5560a do_lookup_x+0xda (/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)\n\t 7f0f4da561fb _dl_lookup_symbol_x+0x12b (/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)\n\t 7f0f4da5aae2 resolve_map+0x8a2 (inlined)\n\t 7f0f4da5aae2 elf_machine_rela+0x8a2 (inlined)\n\t 7f0f4da5aae2 elf_dynamic_do_Rela+0x8a2 (inlined)\n\t 7f0f4da5aae2 _dl_relocate_object+0x8a2 (/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)\n\t 7f0f4da6d788 dl_main+0x1c98 (/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)\n\t 7f0f4da69f45 _dl_sysdep_start+0x85 (/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)\n\t 7f0f4da6b75d _dl_start_final+0x58d (inlined)\n\t 7f0f4da6b75d _dl_start+0x58d (/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)\n\t 7f0f4da6a547 _start+0x7 (/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)\n\npython 20962 10544768.859556: 425076627 cycles:P: \n\t 55dbd22a5029 arena_map_is_used+0x79 (inlined)\n\t 55dbd22a5029 address_in_range+0x79 (inlined)\n\t 55dbd22a5029 pymalloc_free+0x79 (inlined)\n\t 55dbd22a5029 _PyObject_Free+0x79 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a5029 _PyObject_Free+0x79 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\npython 20962 10544768.870755: 26855347 cycles:P: \n\t 55dbd2267a46 long_dealloc+0x56 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d18828a [unknown] ([unknown])\n\npython 20962 10544768.881594: 25984233 cycles:P: \n\t 55dbd22bc261 pymalloc_alloc+0x41 (inlined)\n\t 55dbd22bc261 _PyObject_Malloc+0x41 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd226864e _PyLong_New+0x5e (inlined)\n\t 55dbd226864e x_add+0x5e (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544768.892192: 25400576 cycles:P: \n\t 7f0f4d188279 [unknown] ([unknown])\n\npython 20962 10544768.902620: 25009362 cycles:P: \n\t 55dbd2268663 Py_SET_TYPE+0x73 (inlined)\n\t 55dbd2268663 _PyObject_Init+0x73 (inlined)\n\t 55dbd2268663 _PyLong_New+0x73 (inlined)\n\t 55dbd2268663 x_add+0x73 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544768.912977: 24822660 cycles:P: \n\t 55dbd22a5025 arena_map_is_used+0x75 (inlined)\n\t 55dbd22a5025 address_in_range+0x75 (inlined)\n\t 55dbd22a5025 pymalloc_free+0x75 (inlined)\n\t 55dbd22a5025 _PyObject_Free+0x75 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a5025 _PyObject_Free+0x75 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\npython 20962 10544768.923245: 24622936 cycles:P: \n\t 55dbd22a4fd1 _PyInterpreterState_GET+0x21 (inlined)\n\t 55dbd22a4fd1 get_state+0x21 (inlined)\n\t 55dbd22a4fd1 _PyObject_Free+0x21 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a4fd1 _PyObject_Free+0x21 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\npython 20962 10544768.933474: 24527621 cycles:P: \n\t 55dbd22a4fd9 get_state+0x29 (inlined)\n\t 55dbd22a4fd9 _PyObject_Free+0x29 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a4fd9 _PyObject_Free+0x29 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d18828a [unknown] ([unknown])\n\npython 20962 10544768.943661: 24426125 cycles:P: \n\t 55dbd229df92 new_reference+0x12 (inlined)\n\t 55dbd229df92 _Py_NewReference+0x12 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd2268686 _PyObject_Init+0x96 (inlined)\n\t 55dbd2268686 _PyLong_New+0x96 (inlined)\n\t 55dbd2268686 x_add+0x96 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544768.953830: 24377304 cycles:P: \n\t 7f0f4d188208 [unknown] ([unknown])\n\npython 20962 10544768.963972: 24323472 cycles:P: \n\t 55dbd2271b34 Py_INCREF+0x154 (inlined)\n\t 55dbd2271b34 _PyObject_Init+0x154 (inlined)\n\t 55dbd2271b34 _PyLong_FromMedium+0x154 (inlined)\n\t 55dbd2271b34 PyLong_FromLong+0x154 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d18815f [unknown] ([unknown])\n\npython 20962 10544768.974108: 24298036 cycles:P: \n\t 55dbd22bc261 pymalloc_alloc+0x41 (inlined)\n\t 55dbd22bc261 _PyObject_Malloc+0x41 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd226864e _PyLong_New+0x5e (inlined)\n\t 55dbd226864e x_add+0x5e (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544768.984236: 24273562 cycles:P: \n\t 55dbd22a5007 arena_map_get+0x57 (inlined)\n\t 55dbd22a5007 arena_map_is_used+0x57 (inlined)\n\t 55dbd22a5007 address_in_range+0x57 (inlined)\n\t 55dbd22a5007 pymalloc_free+0x57 (inlined)\n\t 55dbd22a5007 _PyObject_Free+0x57 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a5007 _PyObject_Free+0x57 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\npython 20962 10544768.994349: 24251966 cycles:P: \n\t 55dbd2267a18 long_dealloc+0x28 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\npython 20962 10544769.004465: 24245115 cycles:P: \n\t 55dbd22bc277 pymalloc_alloc+0x57 (inlined)\n\t 55dbd22bc277 _PyObject_Malloc+0x57 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd226864e _PyLong_New+0x5e (inlined)\n\t 55dbd226864e x_add+0x5e (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.014568: 24236552 cycles:P: \n\t 55dbd22686da x_add+0xea (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.024676: 24231731 cycles:P: \n\t 55dbd227746e _PyLong_Add+0x3e (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.034780: 24229045 cycles:P: \n\t 55dbd22bc270 pymalloc_alloc+0x50 (inlined)\n\t 55dbd22bc270 _PyObject_Malloc+0x50 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd2271b09 _PyLong_FromMedium+0x129 (inlined)\n\t 55dbd2271b09 PyLong_FromLong+0x129 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d18815f [unknown] ([unknown])\n\npython 20962 10544769.044887: 24225959 cycles:P: \n\t 7f0f4d188208 [unknown] ([unknown])\n\npython 20962 10544769.054987: 24221906 cycles:P: \n\t 55dbd2268663 Py_SET_TYPE+0x73 (inlined)\n\t 55dbd2268663 _PyObject_Init+0x73 (inlined)\n\t 55dbd2268663 _PyLong_New+0x73 (inlined)\n\t 55dbd2268663 x_add+0x73 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.065095: 24221703 cycles:P: \n\t 55dbd22a4ff4 arena_map_get+0x44 (inlined)\n\t 55dbd22a4ff4 arena_map_is_used+0x44 (inlined)\n\t 55dbd22a4ff4 address_in_range+0x44 (inlined)\n\t 55dbd22a4ff4 pymalloc_free+0x44 (inlined)\n\t 55dbd22a4ff4 _PyObject_Free+0x44 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a4ff4 _PyObject_Free+0x44 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\npython 20962 10544769.075194: 24219126 cycles:P: \n\t 7f0f4d188208 [unknown] ([unknown])\n\npython 20962 10544769.085298: 24218678 cycles:P: \n\t 55dbd22bc261 pymalloc_alloc+0x41 (inlined)\n\t 55dbd22bc261 _PyObject_Malloc+0x41 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd2271b09 _PyLong_FromMedium+0x129 (inlined)\n\t 55dbd2271b09 PyLong_FromLong+0x129 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d18815f [unknown] ([unknown])\n\npython 20962 10544769.095396: 24217814 cycles:P: \n\t 55dbd22a4fd1 _PyInterpreterState_GET+0x21 (inlined)\n\t 55dbd22a4fd1 get_state+0x21 (inlined)\n\t 55dbd22a4fd1 _PyObject_Free+0x21 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a4fd1 _PyObject_Free+0x21 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\npython 20962 10544769.105497: 24217334 cycles:P: \n\t 55dbd22bc234 _PyInterpreterState_GET+0x14 (inlined)\n\t 55dbd22bc234 get_state+0x14 (inlined)\n\t 55dbd22bc234 _PyObject_Malloc+0x14 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd226864e _PyLong_New+0x5e (inlined)\n\t 55dbd226864e x_add+0x5e (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.115596: 24219156 cycles:P: \n\t 55dbd2268663 Py_SET_TYPE+0x73 (inlined)\n\t 55dbd2268663 _PyObject_Init+0x73 (inlined)\n\t 55dbd2268663 _PyLong_New+0x73 (inlined)\n\t 55dbd2268663 x_add+0x73 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.125701: 24218620 cycles:P: \n\t 55dbd22a5029 arena_map_is_used+0x79 (inlined)\n\t 55dbd22a5029 address_in_range+0x79 (inlined)\n\t 55dbd22a5029 pymalloc_free+0x79 (inlined)\n\t 55dbd22a5029 _PyObject_Free+0x79 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a5029 _PyObject_Free+0x79 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\npython 20962 10544769.135798: 24217127 cycles:P: \n\t 55dbd22bc26d pymalloc_alloc+0x4d (inlined)\n\t 55dbd22bc26d _PyObject_Malloc+0x4d (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd226864e _PyLong_New+0x5e (inlined)\n\t 55dbd226864e x_add+0x5e (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.145904: 24217819 cycles:P: \n\t 55dbd22bc261 pymalloc_alloc+0x41 (inlined)\n\t 55dbd22bc261 _PyObject_Malloc+0x41 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd2271b09 _PyLong_FromMedium+0x129 (inlined)\n\t 55dbd2271b09 PyLong_FromLong+0x129 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d18815f [unknown] ([unknown])\n\npython 20962 10544769.156002: 24217028 cycles:P: \n\tffffffff9f84f3f3 [unknown] ([unknown])\n\tffffffff9f97d85a [unknown] ([unknown])\n\tffffffff9f887161 [unknown] ([unknown])\n\tffffffffa05c634e [unknown] ([unknown])\n\tffffffffa0600ecb [unknown] ([unknown])\n\t 55dbd22bc2d0 pymalloc_pool_extend+0xb0 (inlined)\n\t 55dbd22bc2d0 pymalloc_alloc+0xb0 (inlined)\n\t 55dbd22bc2d0 _PyObject_Malloc+0xb0 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd2271b09 _PyLong_FromMedium+0x129 (inlined)\n\t 55dbd2271b09 PyLong_FromLong+0x129 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d18815f [unknown] ([unknown])\n\npython 20962 10544769.166107: 24216861 cycles:P: \n\t 55dbd22bc261 pymalloc_alloc+0x41 (inlined)\n\t 55dbd22bc261 _PyObject_Malloc+0x41 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd226864e _PyLong_New+0x5e (inlined)\n\t 55dbd226864e x_add+0x5e (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.176207: 24216156 cycles:P: \n\t 55dbd22bc261 pymalloc_alloc+0x41 (inlined)\n\t 55dbd22bc261 _PyObject_Malloc+0x41 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd226864e _PyLong_New+0x5e (inlined)\n\t 55dbd226864e x_add+0x5e (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d188201 [unknown] ([unknown])\n\npython 20962 10544769.186303: 24215716 cycles:P: \n\t 55dbd22a5007 arena_map_get+0x57 (inlined)\n\t 55dbd22a5007 arena_map_is_used+0x57 (inlined)\n\t 55dbd22a5007 address_in_range+0x57 (inlined)\n\t 55dbd22a5007 pymalloc_free+0x57 (inlined)\n\t 55dbd22a5007 _PyObject_Free+0x57 (/home/user/ci-scripts/cpython-head/build/python)\n\t 55dbd22a5007 _PyObject_Free+0x57 (/home/user/ci-scripts/cpython-head/build/python)\n\t 7f0f4d1881ac [unknown] ([unknown])\n\n'
The buildbot is green and I cannot reproduce on my x86_64 machine. Are you sure you are using the latest main? If this was working before can you bisect? vstinner commented on Sep 30, 2024
This is still failing on our x86 machine
I reopen the issue. Do you mean a 32-bit machine, or do you mean x86-64? diegorusso commented on Sep 30, 2024 ContributorAuthor More actions I meant x86-64, I've edited the comment. pablogsal commented on Sep 30, 2024 pablogsal commented on Sep 30, 2024 Cannot reproduce on my aarch64 machines either:
./python.exe -m test test_perf_profiler -v
== CPython 3.14.0a0 (main, Sep 30 2024, 13:33:59) [GCC 14.2.1 20240912 (Red Hat 14.2.1-3)]
== Linux-6.6.32-linuxkit-aarch64-with-glibc2.39 little-endian
== Python build: release
== cwd: /src/build/test_python_worker_8383æ
== CPU count: 10
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests
Using random seed: 244045580
0:00:00 load avg: 4.84 Run 1 test sequentially in a single process
0:00:00 load avg: 4.84 [1/1] test_perf_profiler
test_pre_fork_compile (test.test_perf_profiler.TestPerfProfiler.test_pre_fork_compile) ... skipped 'Unwinding is unreliable with frame pointers'
test_python_calls_appear_in_the_stack_if_perf_activated (test.test_perf_profiler.TestPerfProfiler.test_python_calls_appear_in_the_stack_if_perf_activated) ... skipped 'Unwinding is unreliable with frame pointers'
test_python_calls_do_not_appear_in_the_stack_if_perf_deactivated (test.test_perf_profiler.TestPerfProfiler.test_python_calls_do_not_appear_in_the_stack_if_perf_deactivated) ... skipped 'Unwinding is unreliable with frame pointers'
test_python_calls_appear_in_the_stack_if_perf_activated (test.test_perf_profiler.TestPerfProfilerWithDwarf.test_python_calls_appear_in_the_stack_if_perf_activated) ... ok
test_python_calls_do_not_appear_in_the_stack_if_perf_deactivated (test.test_perf_profiler.TestPerfProfilerWithDwarf.test_python_calls_do_not_appear_in_the_stack_if_perf_deactivated) ... ok
test_sys_api (test.test_perf_profiler.TestPerfTrampoline.test_sys_api) ... ok
test_sys_api_get_status (test.test_perf_profiler.TestPerfTrampoline.test_sys_api_get_status) ... ok
test_sys_api_with_existing_trampoline (test.test_perf_profiler.TestPerfTrampoline.test_sys_api_with_existing_trampoline) ... ok
test_sys_api_with_invalid_trampoline (test.test_perf_profiler.TestPerfTrampoline.test_sys_api_with_invalid_trampoline) ... ok
test_trampoline_works (test.test_perf_profiler.TestPerfTrampoline.test_trampoline_works) ... ok
test_trampoline_works_with_forks (test.test_perf_profiler.TestPerfTrampoline.test_trampoline_works_with_forks) ... ok
----------------------------------------------------------------------
Ran 11 tests in 2.830s
OK (skipped=3)
== Tests result: SUCCESS ==
1 test OK.
Total duration: 5.1 sec
Total tests: run=11 skipped=3
Total test files: run=1/1
Result: SUCCESS
diegorusso commented on Sep 30, 2024 ContributorAuthor More actions With the help of a colleague, we understand where the problem is. On x86-64 and with the JIT enabled, the test needs DWARF unwind information for every frame and the JIT doesn't provide that.
On AArch64 it works fine because even without DWARF table, frame pointers can be used to unwind information.
Basically this test cannot work on x86 + JIT. I tested the #124636 but without JIT and it worked as expected.
But as I'm working on the JIT, all my builds have it enabled hence we discovered this issue.
On an x86-64 machine and the JIT compiled:
$ PYTHON_JIT=0 ./python ../Lib/test/test_perf_profiler.py
sss........
----------------------------------------------------------------------
Ran 11 tests in 3.677s
OK (skipped=3)
$ PYTHON_JIT=0 ./python ../Lib/test/test_perf_profiler.py
sss........
----------------------------------------------------------------------
Ran 11 tests in 3.677s
OK (skipped=3)
bot@aa62ba7f21ce:~/cpython/build$ PYTHON_JIT=1 ./python ../Lib/test/test_perf_profiler.py
sssF.......
======================================================================
FAIL: test_python_calls_appear_in_the_stack_if_perf_activated (__main__.TestPerfProfilerWithDwarf.test_python_calls_appear_in_the_stack_if_perf_activated)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/bot/cpython/build/../Lib/test/test_perf_profiler.py", line 355, in test_python_calls_appear_in_the_stack_if_perf_activated
self.assertIn(f"py::foo:{script}", stdout)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'py::foo:/tmp/tmpyccv_piu/perftest.py' not found in 'perf-exec ...
...
...
tart_main_impl+0x8a (inlined)\n\t 55b76e9ec984 _start+0x24 (/home/bot/cpython/build/python)\n\n'
----------------------------------------------------------------------
Ran 11 tests in 3.662s
FAILED (failures=1, skipped=3)
I guess for now we can skip (or xfail) this test if the JIT is enabled and also actually run this test on JIT builds.
Tagging @brandtbucher for his information. pablogsal commented on Sep 30, 2024
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|