GeezerVison is showing github.com — PPC64LE Fedora Stable Clang 3.x fails test_gdb · Issue #104736 · python/cpython
PPC64LE Fedora Stable Clang 3.x fails test_gdb #104736DescriptionBuildbot https://buildbot.python.org/all/#/builders/435 has been failing test_gdb with errors like these:
======================================================================
FAIL: test_pycfunction (test.test_gdb.PyBtTests.test_pycfunction) [_testcapi.meth_varargs]
Verify that "py-bt" displays invocations of PyCFunction instances
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-ppc64le.clang/build/Lib/test/test_gdb.py", line 944, in test_pycfunction
self.assertIn(f'<built-in method {func_name}', gdb_output)
AssertionError: '<built-in method meth_varargs' not found in 'Breakpoint 1 (meth_varargs) pending.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\n\nBreakpoint 1, meth_varargs (self=<module at remote 0x7fffea347230>, args=()) at ./Modules/_testcapimodule.c:2307\n2307\t return Py_BuildValue("NO", _null_to_none(self), args);\n#0 meth_varargs (self=<module at remote 0x7fffea347230>, args=()) at ./Modules/_testcapimodule.c:2307\n#1 PyModuleDef_Type ()\nBacktrace stopped: frame did not save the PC\nUnable to locate python frame\n'
It last succeeded on commit 373bca0 and first failed on fcd5fb4, suggesting that the culprit is one of these commits:
Neither looks likely to affect test_gdb though.
Linked PRs
Activityencukou commented on May 22, 2023 It looks like GDB can't extract the C stack here. I SSHd to the buildbot and tried:
$ gdb --args ./python -c 'import _testcapi; _testcapi.meth_varargs()'
[...]
Reading symbols from ./python...
(gdb) b meth_varargs
Function "meth_varargs" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (meth_varargs) pending.
(gdb) r
[...]
Breakpoint 1, meth_varargs (self=<module at remote 0x7ffff6f77d70>, args=())
at ./Modules/_testcapimodule.c:2307
2307 return Py_BuildValue("NO", _null_to_none(self), args);
(gdb) bt
#0 meth_varargs (self=<module at remote 0x7ffff6f77d70>, args=()) at ./Modules/_testcapimodule.c:2307
#1 0x00007ffff7cf7db0 in PyModuleDef_Type () from /home/pviktori/cpython/libpython3.12d.so.1.0
Backtrace stopped: frame did not save the PC
The stack looks corrupted: PyModuleDef_Type is data, not a function.
This looks like “miscommunication” between clang and gdb (bad debug info?)
The test seems very sensitive to platform/compiler, e.g.:
if 'Clang' in platform.python_compiler() and sys.platform == 'darwin':
raise unittest.SkipTest("test_gdb doesn't work correctly when python is"
" built with LLVM clang")
Note that the “installed” buildbots are green. (AFAIK these test Python after make install.) But when I make install, GDB still fails to extract the backtrace on the installed Python.
ping @vstinner, the PEP-11 ppc64le/clang maintainer encukou commented on May 22, 2023 See #65462 (comment):
Unless someone has a better idea, I suggest we just disable test_gdb on OS X or, somewhat more precisely, when Python has been compiled with LLVM clang.
Doesn't clang always use LLVM?
The skip is now 'Clang' in platform.python_compiler() and sys.platform == 'darwin', should it just be 'Clang' in platform.python_compiler()? stratakis commented on May 22, 2023 The failure started after the update of Clang to 16.0.0 from 15.0.7. thesamesam commented on May 23, 2023
See #65462 (comment):
Unless someone has a better idea, I suggest we just disable test_gdb on OS X or, somewhat more precisely, when Python has been compiled with LLVM clang.
Doesn't clang always use LLVM? The skip is now 'Clang' in platform.python_compiler() and sys.platform == 'darwin', should it just be 'Clang' in platform.python_compiler()?
I took it to mean "not Apple's Clang" which is distinct (has its own versioning and is essentially a fork). I might be wrong here though. vstinner commented on May 23, 2023 Maybe tets_gdb can skip a test if "frame did not save the PC" is found in the output. We already have many skips like that. carljm commented on Aug 30, 2023 It looks like multiple Clang buildbots are still failing test_gdb. vstinner commented on Sep 7, 2023 I closed issue #109053 as a duplicate of this issue. vstinner commented on Sep 13, 2023 I did tests on two ppc64le machines. In short, test_gdb only failed on ppc64le if Python is built with clang version 16.0.6 (clang -Og). Well, that's what we saw on buildbots. But I preferred to confirm manually. I built Python manually and ran test_gdb manually with -u all to no skip any test.
-
Machine 1:
- Fedora 38
Linux xxx 6.2.8-200.fc37.ppc64le #1 SMP Wed Mar 22 18:34:43 UTC 2023 ppc64le ppc64le ppc64le GNU/Linux- gcc (GCC) 13.2.1 20230728 (Red Hat 13.2.1-1)
- clang version 16.0.6 (Fedora 16.0.6-2.fc38)
- GNU gdb (GDB) Fedora Linux 13.2-3.fc38
-
Machine 2:
- Fedora 37
Linux localhost.localdomain 5.8.17-200.fc32.ppc64le #1 SMP Thu Oct 29 18:00:08 UTC 2020 ppc64le GNU/Linux- gcc (GCC) 12.3.1 2023050 (Red Hat 12.3.1-1)
- clang version 15.0.7 (Fedora 15.0.7-2.fc37)
- GNU gdb (GDB) Fedora Linux 13.2-2.fc37
GCC
Test: ./configure --with-pydebug && make && ./python -m test test_gdb -u all -v.
Build built with: gcc -Og.
LLVM clang
Test: ./configure --with-pydebug CC=clang && make && ./python -m test test_gdb -u all -v.
Build built with: clang -Og.
vstinner commented on Sep 13, 2023 Reproduce _testcapi.meth_varargs() issue.
script.py:
import _testcapi
def foo():
_testcapi.meth_varargs()
def bar():
foo()
bar()cmds:
file ./python
source Tools/gdb/libpython.py
set breakpoint pending yes
break meth_varargs
run script.py
where
py-bt
gdb output on ppc64le with clang -Og:
$ gdb -x cmds
GNU gdb (GDB) Fedora Linux 13.2-3.fc38
Function "meth_varargs" not defined.
Breakpoint 1 (meth_varargs) pending.
Breakpoint 1, meth_varargs (self=<module at remote 0x7fffea3d23f0>, args=()) at ./Modules/_testcapimodule.c:2060
2060 return Py_BuildValue("NO", _null_to_none(self), args);
#0 meth_varargs (self=<module at remote 0x7fffea3d23f0>, args=()) at ./Modules/_testcapimodule.c:2060
#1 0x0000000010625d78 in PyModuleDef_Type ()
Backtrace stopped: frame did not save the PC
Unable to locate python frame
The important message here is Backtrace stopped: frame did not save the PC. The gdb traceback is really just 2 frames and the 2nd frame PyModuleDef_Type () is wrong :-(
Comparison on x86-64, Python built with gcc -Og:
vstinner@mona$ gdb -x cmds
GNU gdb (GDB) Fedora Linux 13.2-3.fc38
Function "meth_varargs" not defined.
Breakpoint 1 (meth_varargs) pending.
Breakpoint 1, meth_varargs (self=self@entry=<module at remote 0x7fffea4c29f0>, args=args@entry=()) at ./Modules/_testcapimodule.c:2059
2059 {
#0 meth_varargs (self=self@entry=<module at remote 0x7fffea4c29f0>, args=args@entry=()) at ./Modules/_testcapimodule.c:2059
#1 0x00000000004f2dcc in cfunction_call (func=func@entry=<built-in method meth_varargs of module object at remote 0x7fffea4c29f0>, args=args@entry=(),
kwargs=kwargs@entry=0x0) at Objects/methodobject.c:551
#2 0x00000000004ab06f in _PyObject_MakeTpCall (tstate=tstate@entry=0x980358 <_PyRuntime+508728>,
callable=callable@entry=<built-in method meth_varargs of module object at remote 0x7fffea4c29f0>, args=args@entry=0x7ffff7fb9128, nargs=<optimized out>,
keywords=keywords@entry=0x0) at Objects/call.c:242
#3 0x00000000004ab282 in _PyObject_VectorcallTstate (tstate=0x980358 <_PyRuntime+508728>,
callable=callable@entry=<built-in method meth_varargs of module object at remote 0x7fffea4c29f0>, args=args@entry=0x7ffff7fb9128, nargsf=<optimized out>,
kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:185
#4 0x00000000004ab2d0 in PyObject_Vectorcall (callable=callable@entry=<built-in method meth_varargs of module object at remote 0x7fffea4c29f0>,
args=args@entry=0x7ffff7fb9128, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at Objects/call.c:327
#5 0x000000000059e417 in _PyEval_EvalFrameDefault (tstate=0x980358 <_PyRuntime+508728>, frame=0x7ffff7fb90d0, throwflag=0) at Python/generated_cases.c.h:3773
(...)
#20 0x000000000041b78f in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15
Traceback (most recent call first):
<built-in method meth_varargs of module object at remote 0x7fffea4c29f0>
File "/home/vstinner/python/main/script.py", line 3, in foo
_testcapi.meth_varargs()
File "/home/vstinner/python/main/script.py", line 5, in bar
foo()
File "/home/vstinner/python/main/script.py", line 6, in <module>
bar()
- gcc (GCC) 13.2.1 20230728 (Red Hat 13.2.1-1)
- GNU gdb (GDB) Fedora Linux 13.2-3.fc38
On x86-64 with gcc, I have 20 frames up to main() and the 2nd frame is cfunction_call() as expected. vstinner commented on Sep 13, 2023 script2.py:
import _testcapi
def foo():
_testcapi.meth_fastcall_keywords()
def bar():
foo()
bar()cmds2:
file ./python
source Tools/gdb/libpython.py
set breakpoint pending yes
break meth_fastcall_keywords
run script2.py
where
py-bt
gdb output on ppc64 with clang -Og:
$ gdb -x cmds2
GNU gdb (GDB) Fedora Linux 13.2-3.fc38
Function "meth_fastcall_keywords" not defined.
Breakpoint 1 (meth_fastcall_keywords) pending.
Breakpoint 1, meth_fastcall_keywords (self=<module at remote 0x7fffea3d23f0>, args=0x7fffea410128, nargs=0, kwargs=0x0) at ./Modules/_testcapimodule.c:2107
2107 PyObject *pyargs = _fastcall_to_tuple(args, nargs);
#0 meth_fastcall_keywords (self=<module at remote 0x7fffea3d23f0>, args=0x7fffea410128, nargs=0, kwargs=0x0) at ./Modules/_testcapimodule.c:2107
#1 0x0000000000000000 in ?? ()
Unable to locate python frame
Here, I think that the important part is #1 0x0000000000000000 in ?? (). When gdb says that, usually it's likely that the traceback is truncated or wrong.
For comparison, gdb on x86-64 on Python built with gcc -Og:
vstinner@mona$ gdb -x cmds2
GNU gdb (GDB) Fedora Linux 13.2-3.fc38
Function "meth_fastcall_keywords" not defined.
Breakpoint 1 (meth_fastcall_keywords) pending.
Breakpoint 1, meth_fastcall_keywords (self=<module at remote 0x7fffea4c2a50>, args=args@entry=0x7ffff7fb9128, nargs=nargs@entry=0, kwargs=kwargs@entry=0x0)
at ./Modules/_testcapimodule.c:2106
2106 {
#0 meth_fastcall_keywords (self=<module at remote 0x7fffea4c2a50>, args=args@entry=0x7ffff7fb9128, nargs=nargs@entry=0, kwargs=kwargs@entry=0x0)
at ./Modules/_testcapimodule.c:2106
#1 0x00000000004f259a in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<built-in method meth_fastcall_keywords of module object at remote 0x7fffea4c2a50>,
args=0x7ffff7fb9128, nargsf=<optimized out>, kwnames=0x0) at Objects/methodobject.c:441
#2 0x00000000004ab208 in _PyObject_VectorcallTstate (tstate=0x980358 <_PyRuntime+508728>,
callable=callable@entry=<built-in method meth_fastcall_keywords of module object at remote 0x7fffea4c2a50>, args=args@entry=0x7ffff7fb9128, nargsf=9223372036854775808,
kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:187
#3 0x00000000004ab2d0 in PyObject_Vectorcall (callable=callable@entry=<built-in method meth_fastcall_keywords of module object at remote 0x7fffea4c2a50>,
args=args@entry=0x7ffff7fb9128, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at Objects/call.c:327
#4 0x000000000059e417 in _PyEval_EvalFrameDefault (tstate=0x980358 <_PyRuntime+508728>, frame=0x7ffff7fb90d0, throwflag=0) at Python/generated_cases.c.h:3773
(...)
#19 0x000000000041b78f in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15
Traceback (most recent call first):
<built-in method meth_fastcall_keywords of module object at remote 0x7fffea4c2a50>
File "/home/vstinner/python/main/script2.py", line 3, in foo
_testcapi.meth_fastcall_keywords()
File "/home/vstinner/python/main/script2.py", line 5, in bar
foo()
File "/home/vstinner/python/main/script2.py", line 6, in <module>
bar()
Here the 19 frames are complete up to main().
- gcc (GCC) 13.2.1 20230728 (Red Hat 13.2.1-1)
- GNU gdb (GDB) Fedora Linux 13.2-3.fc38
vstinner commented on Sep 13, 2023 I wrote PR #109360 which fix test_gdb on Linux ppc64lc when Python is built with clang -Og (LLVM clang 16). encukou commented on Feb 14, 2025 The issue reappeared on Python 3.11 with clang 19 (f41). New failure:
=================================================================
FAIL: test_pycfunction_fastcall_keywords (test.test_gdb.test_cfunction.CFunctionTests.test_pycfunction_fastcall_keywords) [_testcapi.meth_fastcall_keywords]
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-ppc64le.clang/build/Lib/test/test_gdb/test_cfunction.py", line 67, in check_pycfunction
self.check(func_name, cmd)
File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-ppc64le.clang/build/Lib/test/test_gdb/test_cfunction.py", line 26, in check
self.assertIn(f'<built-in method {func_name}', gdb_output)
AssertionError: '<built-in method meth_fastcall_keywords' not found in 'Breakpoint 1 (meth_fastcall_keywords) pending.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\n\nBreakpoint 1, meth_fastcall_keywords (self=<module at remote 0x7fffe9bea9f0>, args=, nargs=0, kwargs=0x0) at /home/buildbot/buildarea/3.11.cstratak-fedora-stable-ppc64le.clang/build/Modules/_testcapimodule.c:5724\n5724\t PyObject *pyargs = _fastcall_to_tuple(args, nargs);\n#0 meth_fastcall_keywords (self=<module at remote 0x7fffe9bea9f0>, args=, nargs=0, kwargs=0x0) at /home/buildbot/buildarea/3.11.cstratak-fedora-stable-ppc64le.clang/build/Modules/_testcapimodule.c:5724\n#1 _PyRuntime ()\n#2 _PyObject_VectorcallTstate (tstate=, callable=<unknown at remote 0x105e0c88>, args=<optimized out>, nargsf=140737115783464, kwnames=<optimized out>) at ./Include/internal/pycore_call.h:92\n#3 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=, throwflag=<optimized out>) at ./Include/object.h:524\n#4 _PyEval_EvalFrame (tstate=<_PyRuntime+166328>, frame=, throwflag=0) at ./Include/internal/pycore_ceval.h:73\n#5 _PyEval_Vector (tstate=<_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6434\n#6 PyEval_EvalCode (co=<optimized out>, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x106728d0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7fffe9daf650>, \'_testcapi\': <module at remote 0x7fffe9bea9f0>, \'foo\': <function at remote 0x7fffe9bcf6a0>, \'bar\': <function at remote 0x7fffe9bf0ec0>}, locals=<optimized out>) at Python/ceval.c:1148\n#7 run_eval_code_obj (tstate=<_PyRuntime+166328>, co=, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x106728d0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remo
emote 0x106728d0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7fffe9daf650>, \'_testcapi\': <module at remote 0x7fffe9bea9f0>, \'foo\': <function at remote 0x7fffe9bcf6a0>, \'bar\': <function at remote 0x7fffe9bf0ec0>}, flags=) at Python/pythonrun.c:1632\n#10 PyRun_SimpleStringFlags (command="\\nimport _testcapi\\ndef foo():\\n _testcapi.meth_fastcall_keywords()\\ndef bar():\\n foo()\\nbar()\\n\\n", flags=) at Python/pythonrun.c:487\n#11 pymain_run_command (command=<optimized out>) at Modules/main.c:255\n#12 pymain_run_python (exitcode=<optimized out>) at Modules/main.c:596\n#13 Py_RunMain () at Modules/main.c:684\n#14 pymain_main (args=) at Modules/main.c:714\n#15 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:738\n#16 main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15\nTraceback (most recent call first):\n File "<string>", line 4, in foo\n File "<string>", line 6, in bar\n File "<string>", line 7, in <module>\n'
Stdout:
test call: _testcapi.meth_fastcall_keywords()
that is:
Breakpoint 1, meth_fastcall_keywords (self=<module at remote 0x7fffe9bea9f0>, args=, nargs=0, kwargs=0x0) at /home/buildbot/buildarea/3.11.cstratak-fedora-stable-ppc64le.clang/build/Modules/_testcapimodule.c:5724
5724 PyObject *pyargs = _fastcall_to_tuple(args, nargs);
#0 meth_fastcall_keywords (self=<module at remote 0x7fffe9bea9f0>, args=, nargs=0, kwargs=0x0) at /home/buildbot/buildarea/3.11.cstratak-fedora-stable-ppc64le.clang/build/Modules/_testcapimodule.c:5724
#1 _PyRuntime ()
#2 _PyObject_VectorcallTstate (tstate=, callable=<unknown at remote 0x105e0c88>, args=<optimized out>, nargsf=140737115783464, kwnames=<optimized out>) at ./Include/internal/pycore_call.h:92
#3 _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=, throwflag=<optimized out>) at ./Include/object.h:524
#4 _PyEval_EvalFrame (tstate=<_PyRuntime+166328>, frame=, throwflag=0) at ./Include/internal/pycore_ceval.h:73
#5 _PyEval_Vector (tstate=<_PyRuntime+166328>, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6434
#6 PyEval_EvalCode (co=<optimized out>, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x106728d0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fffe9daf650>, '_testcapi': <module at remote 0x7fffe9bea9f0>, 'foo': <function at remote 0x7fffe9bcf6a0>, 'bar': <function at remote 0x7fffe9bf0ec0>}, locals=<optimized out>) at Python/ceval.c:1148
#7 run_eval_code_obj (tstate=<_PyRuntime+166328>, co=, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x106728d0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remo
emote 0x106728d0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7fffe9daf650>, '_testcapi': <module at remote 0x7fffe9bea9f0>, 'foo': <function at remote 0x7fffe9bcf6a0>, 'bar': <function at remote 0x7fffe9bf0ec0>}, flags=) at Python/pythonrun.c:1632
#10 PyRun_SimpleStringFlags (command="\nimport _testcapi\ndef foo():\n _testcapi.meth_fastcall_keywords()\ndef bar():\n foo()\nbar()\n\n", flags=) at Python/pythonrun.c:487
#11 pymain_run_command (command=<optimized out>) at Modules/main.c:255
#12 pymain_run_python (exitcode=<optimized out>) at Modules/main.c:596
#13 Py_RunMain () at Modules/main.c:684
#14 pymain_main (args=) at Modules/main.c:714
#15 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:738
#16 main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15
Traceback (most recent call first):
File "<string>", line 4, in foo
File "<string>", line 6, in bar
File "<string>", line 7, in <module>
vstinner commented on Feb 19, 2025 encukou commented on Feb 20, 2025 vstinner commented on Feb 21, 2025
#2 _PyObject_VectorcallTstate (tstate=, callable=<unknown at remote 0x105e0c88>, args=, nargsf=140737115783464, kwnames=) at ./Include/internal/pycore_call.h:92
That's the classical <optimized out> problem. When the Python binary is optimized, sometimes gdb fails to read function arguments.
I'm not sure why this problem only occurs on the 3.11 branch. Python internals likely changed in the meanwhile, and so clang optimizes the binary differently. Maybe gdb is able to read function arguments on Python 3.12+.
The 3.11 branch no longer accepts bugfixes, so I don't think that it's worth it to investigate. encukou commented on Feb 21, 2025 Should we mark the builder as unstable, then? vstinner commented on Feb 21, 2025
Should we mark the builder as unstable, then?
It sounds like a reasonable tradeoff. Is it possible to only mark a builder as unstable on a specific branch? vstinner commented on Feb 26, 2025 I created #130593 for the 3.11 branch to skip test_gdb if Python if built with clang -Og (default optimization level when using --with-pydebug). It's a backport from main of #130550 change.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|