GeezerVison is showing github.com — Segfault during BOLT profile collection in `test_functools` (`fib`) when building with `--enable-bolt --enable-optimizations` · Issue #157062 · python/cpython
Segfault during BOLT profile collection in test_functools (fib) when building with --enable-bolt --enable-optimizations #157062DescriptionCrash report
What happened?
Environment
- OS: Linux (x86_64)
- CPython:
main branch
- Compiler: Clang / Clang++ (with
lld)
- BOLT version:
ca2b52862de907b4e4dca1d064098a28ab8269be
Steps to Reproduce
- Configure CPython:
CC=clang CXX=clang++ LDFLAGS='-fuse-ld=lld' ./configure --enable-bolt --enable-optimizations
- Run build:
Expected Behavior
Successful build and optimization of Python binary using BOLT.
Actual Behavior
Segmentation fault occurs while running instrumented binary to collect profile data (./python -m test --pgo --timeout=) during test_functools. The crash happens in the recursive fib function inside the BOLT runtime instrumentation layer.
Crash Log / Backtrace
Details
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: ca2b52862de907b4e4dca1d064098a28ab8269be
BOLT-INFO: first alloc address is 0x200000
BOLT-INFO: creating new program header table at address 0x800000, offset 0x600000
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling lite mode
BOLT-INFO: 0 out of 9012 functions in the binary (0.0%) have non-empty execution profile
BOLT-INFO: validate-mem-refs updated 2 object references
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 1949
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 9003
BOLT-INSTRUMENTER: Number of function descriptors: 9003
BOLT-INSTRUMENTER: Number of branch counters: 104776
BOLT-INSTRUMENTER: Number of ST leaf node counters: 52724
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 157500
BOLT-INSTRUMENTER: Total size of counters: 1260000 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 199375 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 10469868 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file /home/boss/projects/oss/cpython/main/python.bolt
BOLT-INFO: 73572 instructions were shortened
BOLT-INFO: removed 83 empty blocks
BOLT-INFO: UCE removed 606 blocks and 36973 bytes of code
BOLT-INFO: padding code to 0x1200000 to accommodate hot text
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0x1401b80
BOLT-INFO: clear procedure is 0x13fd310
BOLT-INFO: patched build-id (flipped last bit)
BOLT-INFO: setting __bolt_runtime_start to 0x1401b80
BOLT-INFO: setting __bolt_runtime_fini to 0x1401bd0
BOLT-INFO: setting __hot_start to 0xa00000
BOLT-INFO: setting __hot_end to 0x11a3ff8
BOLT-INFO: runtime library finalization was hooked via DT_FINI, set to 0x1401bd0
BOLT-INFO: runtime library initialization was hooked via ELF Header Entry Point, set to 0x1401b80
# Run instrumented binaries to collect data.
./python -m test --pgo --timeout=
Using random seed: 707753353
0:00:00 load avg: 3.07 Run 43 tests sequentially in a single process
0:00:00 load avg: 3.07 [ 1/43] test_array
0:00:04 load avg: 2.91 [ 1/43] test_array passed
0:00:04 load avg: 2.91 [ 2/43] test_base64
0:00:05 load avg: 2.91 [ 2/43] test_base64 passed
0:00:05 load avg: 2.91 [ 3/43] test_binascii
0:00:06 load avg: 2.91 [ 3/43] test_binascii passed
0:00:06 load avg: 2.91 [ 4/43] test_binop
0:00:06 load avg: 2.91 [ 4/43] test_binop passed
0:00:06 load avg: 2.91 [ 5/43] test_bisect
0:00:06 load avg: 2.91 [ 5/43] test_bisect passed
0:00:06 load avg: 2.91 [ 6/43] test_bytes
0:00:16 load avg: 2.61 [ 6/43] test_bytes passed
0:00:16 load avg: 2.61 [ 7/43] test_bz2
0:00:17 load avg: 2.61 [ 7/43] test_bz2 passed
0:00:17 load avg: 2.61 [ 8/43] test_cmath
0:00:17 load avg: 2.61 [ 8/43] test_cmath passed
0:00:17 load avg: 2.61 [ 9/43] test_codecs
0:00:19 load avg: 2.48 [ 9/43] test_codecs passed
0:00:19 load avg: 2.48 [10/43] test_collections
0:00:21 load avg: 2.48 [10/43] test_collections passed
0:00:21 load avg: 2.48 [11/43] test_complex
0:00:21 load avg: 2.48 [11/43] test_complex passed
0:00:21 load avg: 2.48 [12/43] test_dataclasses
0:00:23 load avg: 2.36 [12/43] test_dataclasses passed
0:00:23 load avg: 2.36 [13/43] test_datetime
0:00:31 load avg: 2.25 [13/43] test_datetime passed
0:00:31 load avg: 2.25 [14/43] test_decimal
0:00:36 load avg: 2.15 [14/43] test_decimal passed
0:00:36 load avg: 2.15 [15/43] test_difflib
0:00:38 load avg: 2.15 [15/43] test_difflib passed
0:00:38 load avg: 2.15 [16/43] test_float
0:00:38 load avg: 2.06 [16/43] test_float passed
0:00:38 load avg: 2.06 [17/43] test_fstring
0:00:41 load avg: 2.06 [17/43] test_fstring passed
0:00:41 load avg: 2.06 [18/43] test_functools
Fatal Python error: Segmentation fault
Current thread 0x00007f3bb89cb780 [python] (most recent call first):
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
Current thread's C stack trace (most recent call first):
Binary file "./python", at _Py_DumpStack+0x3d [0x104acd1]
Binary file "./python" [0x1045691]
Binary file "./python" [0x10453ec]
Binary file "/usr/lib/libc.so.6", at +0x3e6f0 [0x7f3bb863e6f0]
Binary file "./python" [0x14032ab]
Extension modules: _testcapi, _testinternalcapi, _testlimitedcapi (total: 3)
make[1]: *** [Makefile:1054: profile-bolt-stamp] Segmentation fault (core dumped)
make[1]: Leaving directory '/home/boss/projects/oss/cpython/main'
make: *** [Makefile:1081: bolt-opt] Error 2
CPython versions tested on:
3.16, CPython main branch
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.16.0a0 (heads/main:42de93a62e7, Sep 7 2026, 13:34:10) [Clang 22.1.8 ] Reacted by jp0000000000bt-alt Activitypicnixz commented on Sep 7, 2026 corona10 commented on Sep 7, 2026 I never consider that BOLT is non experimental, I need to reevaluate the current status. ashm-dev commented on Sep 7, 2026 ContributorAuthor More actions @picnixz Yes, the same issue occurs on both 3.15 and 3.14. Fidget-Spinner commented on Sep 7, 2026 @ashm-dev pretty easy way to test if this is indeed a clang inlining too much and causing stack overflow: just pass --with-tail-call-interp to your configure. If now the tests pass, that points to a high stack consumption bug on normal interpreter builds. ashm-dev commented on Sep 7, 2026 ContributorAuthor More actions @Fidget-Spinner
CC=clang CXX=clang++ LDFLAGS='-fuse-ld=lld' ./configure --enable-bolt --enable-optimizations --with-tail-call-interp
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: ca2b52862de907b4e4dca1d064098a28ab8269be
BOLT-INFO: first alloc address is 0x200000
BOLT-INFO: creating new program header table at address 0x800000, offset 0x600000
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling lite mode
BOLT-INFO: 0 out of 9260 functions in the binary (0.0%) have non-empty execution profile
BOLT-INFO: validate-mem-refs updated 2 object references
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 2305
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 9216
BOLT-INSTRUMENTER: Number of function descriptors: 9216
BOLT-INSTRUMENTER: Number of branch counters: 106963
BOLT-INSTRUMENTER: Number of ST leaf node counters: 54155
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 161118
BOLT-INSTRUMENTER: Total size of counters: 1288944 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 205747 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 10697116 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file /home/boss/projects/oss/cpython/main/python.bolt
BOLT-INFO: 75452 instructions were shortened
BOLT-INFO: removed 87 empty blocks
BOLT-INFO: UCE removed 615 blocks and 37565 bytes of code
BOLT-INFO: padding code to 0x1200000 to accommodate hot text
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0x140db80
BOLT-INFO: clear procedure is 0x1409310
BOLT-INFO: patched build-id (flipped last bit)
BOLT-INFO: setting __bolt_runtime_start to 0x140db80
BOLT-INFO: setting __bolt_runtime_fini to 0x140dbd0
BOLT-INFO: setting __hot_start to 0xa00000
BOLT-INFO: setting __hot_end to 0x11d7378
BOLT-INFO: runtime library finalization was hooked via DT_FINI, set to 0x140dbd0
BOLT-INFO: runtime library initialization was hooked via ELF Header Entry Point, set to 0x140db80
# Run instrumented binaries to collect data.
./python -m test --pgo --timeout=
Using random seed: 2596992137
0:00:00 load avg: 2.84 Run 43 tests sequentially in a single process
0:00:00 load avg: 2.84 [ 1/43] test_array
0:00:04 load avg: 2.77 [ 1/43] test_array passed
0:00:04 load avg: 2.77 [ 2/43] test_base64
0:00:06 load avg: 2.77 [ 2/43] test_base64 passed
0:00:06 load avg: 2.77 [ 3/43] test_binascii
0:00:06 load avg: 2.77 [ 3/43] test_binascii passed
0:00:06 load avg: 2.77 [ 4/43] test_binop
0:00:07 load avg: 2.77 [ 4/43] test_binop passed
0:00:07 load avg: 2.77 [ 5/43] test_bisect
0:00:07 load avg: 2.77 [ 5/43] test_bisect passed
0:00:07 load avg: 2.77 [ 6/43] test_bytes
0:00:26 load avg: 2.48 [ 6/43] test_bytes passed
0:00:26 load avg: 2.48 [ 7/43] test_bz2
0:00:27 load avg: 2.48 [ 7/43] test_bz2 passed
0:00:27 load avg: 2.48 [ 8/43] test_cmath
0:00:27 load avg: 2.48 [ 8/43] test_cmath passed
0:00:27 load avg: 2.48 [ 9/43] test_codecs
0:00:30 load avg: 2.37 [ 9/43] test_codecs passed
0:00:30 load avg: 2.37 [10/43] test_collections
0:00:34 load avg: 2.37 [10/43] test_collections passed
0:00:34 load avg: 2.37 [11/43] test_complex
0:00:34 load avg: 2.26 [11/43] test_complex passed
0:00:34 load avg: 2.26 [12/43] test_dataclasses
0:00:36 load avg: 2.26 [12/43] test_dataclasses passed
0:00:36 load avg: 2.26 [13/43] test_datetime
0:00:49 load avg: 2.55 [13/43] test_datetime passed
0:00:49 load avg: 2.55 [14/43] test_decimal
0:00:57 load avg: 2.46 [14/43] test_decimal passed
0:00:57 load avg: 2.46 [15/43] test_difflib
0:01:00 load avg: 2.34 [15/43] test_difflib passed
0:01:00 load avg: 2.34 [16/43] test_float
0:01:00 load avg: 2.34 [16/43] test_float passed
0:01:00 load avg: 2.34 [17/43] test_fstring
0:01:04 load avg: 2.34 [17/43] test_fstring passed
0:01:04 load avg: 2.34 [18/43] test_functools
Fatal Python error: Segmentation fault
Current thread 0x00007f0e3e0a2780 [python] (most recent call first):
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
Current thread's C stack trace (most recent call first):
Binary file "./python", at _Py_DumpStack+0x3d [0x1118dd1]
Binary file "./python" [0x1113791]
Binary file "./python" [0x11134ec]
Binary file "/usr/lib/libc.so.6", at +0x3e6f0 [0x7f0e3de3e6f0]
Binary file "./python" [0x140f2ab]
Extension modules: _testcapi, _testinternalcapi (total: 2)
make[1]: *** [Makefile:1054: profile-bolt-stamp] Segmentation fault (core dumped)
make[1]: Leaving directory '/home/boss/projects/oss/cpython/main'
make: *** [Makefile:1081: bolt-opt] Error 2
Fidget-Spinner commented on Sep 7, 2026 That's really suspicious 🤔 corona10 commented on Oct 21, 2026 I will take a look at this issue during the sprint. corona10 commented on Oct 21, 2026 ashm-dev commented on Oct 21, 2026 ContributorAuthor More actions @corona10
I’ve already dug into this. The issue is almost certainly on the LLVM-BOLT side. They attempted to fix it across several commits, but it’s still broken.
As a workaround, lifting the limits (like ulimit -s unlimited) allows BOLT to optimize successfully, but it fails out of the box with default limits.
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|