GeezerVison is showing github.com — `test_json` case `test_highly_nested_objects_decoding` fails under PGO on Clang 20 · Issue #140125 · python/cpython
test_json case test_highly_nested_objects_decoding fails under PGO on Clang 20 #140125
DescriptionBuilding 3.15.0a1 on aarch64 macOS with Clang 20 in python-build-standalone failed during the PGO test run in test_highly_nested_objects_decoding:
cpython-3.15> FAIL: test_highly_nested_objects_decoding (test.test_json.test_recursion.TestCRecursion.test_highly_nested_objects_decoding)
cpython-3.15> ----------------------------------------------------------------------
cpython-3.15> Traceback (most recent call last):
cpython-3.15> File "/private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmp7fxaapg7/Python-3.15.0a1/Lib/test/test_json/test_recursion.py", line 77, in test_highly_nested_objects_decoding
cpython-3.15> with self.assertRaises(RecursionError):
cpython-3.15> ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
cpython-3.15> AssertionError: RecursionError not raised
This failure is specific to macOS, and also occurs on x86-64. I haven't reproduced it outside of python-build-standalone yet. Jump to #140125 (comment) for the root cause.
Linked PRs
Activityvstinner commented on Oct 14, 2025 I cannot reproduce this issue on x86-64 Linux with clang 20. I used these commands:
./configure --enable-optimizations CC=clang LD=clang
time make
./python -m test -v test_json
My clang version: clang version 20.1.8 (Fedora 20.1.8-4.fc42). zanieb commented on Oct 14, 2025 ContributorAuthor More actions Thanks for trying to reproduce!
Just to be clear, my failure occurred while running the profile task, e.g., here, rather than during the test run after a successful build — so I'd expect it to fail during make.
We're specifically using clang 20.1.4 over in python-build-standalone. I tried to reproduce this on HEAD (2ca3c85) on my macOS machine with LLVM clang 20.1.3 from homebrew and could not — the build succeeded. I suspect there may be an interaction with another facet of our build system.
I've opened astral-sh/python-build-standalone#822 to reproduce in our CI matrix and determine if this is specific to aarch64 macOS (I encountered this locally) or occurs on other platforms. zanieb commented on Oct 14, 2025 ContributorAuthor More actions astral-sh/python-build-standalone#822 confirms that this is only an issue on aarch64 and x86-64 macOS.
I tried to reproduce in cpython with build flags that more closely resemble python-build-standalone's
export CFLAGS='-arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fPIC -Werror=unguarded-availability-new'
export CPPFLAGS='-arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fPIC -Werror=unguarded-availability-new'
export CFLAGS_JIT='-arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fPIC -Werror=unguarded-availability-new'
export LDFLAGS='-arch arm64 -mmacosx-version-min=11.0 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -LModules/_hacl -Wl,-headerpad,40'
./configure --build=aarch64-apple-darwin --host=aarch64-apple-darwin --prefix=/install --with-system-expat --with-system-libmpdec --without-ensurepip --with-tail-call-interp --enable-shared --with-mimalloc --enable-optimizations --enable-experimental-jit=yes-off ac_cv_lib_intl_textdomain=no ac_cv_func_ptsname_r=no ac_cv_func_explicit_bzero=no --with-dbmliborder=ndbm
which requires the following patch to allow LLVM 20 in the JIT (we are currently using this in 3.14 without problem)
diff --git a/Tools/jit/_llvm.py b/Tools/jit/_llvm.py
index bc3b50ffe61..c90abad38da 100644
--- a/Tools/jit/_llvm.py
+++ b/Tools/jit/_llvm.py
@@ -11,7 +11,7 @@
import _targets
-_LLVM_VERSION = "19"
+_LLVM_VERSION = "20"
_EXTERNALS_LLVM_TAG = "llvm-19.1.7.0"
_P = typing.ParamSpec("_P")Unfortunately, this failed with a segfault
./Programs/_freeze_module getpath ./Modules/getpath.py Python/frozen_modules/getpath.h
make[2]: *** [Python/frozen_modules/getpath.h] Segmentation fault: 11
make[2]: *** Deleting file `Python/frozen_modules/getpath.h'
make[1]: *** [profile-gen-stamp] Error 2
make: *** [profile-run-stamp] Error 2
If I use --enable-experimental-jit=no instead, the build succeeds. Unfortunately, that means our configure flags are not the sole cause of the test_json failure. I suspect the segfault is a JIT interaction with LLVM 20. I'm not sure why I get this segfault in cpython but not in python-build-standalone. It's worth noting my clang versions do not match exactly.
However, the test_json failure cannot be caused by the JIT because it is occurring on our x86-64 macOS and free-threaded builds, which have the JIT disabled. zanieb commented on Oct 14, 2025 ContributorAuthor More actions I cannot reproduce the segfault after running make clean, so I think we can disregard that for now. zanieb commented on Oct 14, 2025 ContributorAuthor More actions Looking at the test case... the failure is at
|
very_deep = 200000 |
|
# test that loading highly-nested objects doesn't segfault when C |
|
# accelerations are used. See #12017 |
|
with self.assertRaises(RecursionError): |
|
with support.infinite_recursion(): |
|
self.loads('{"a":' * very_deep + '1' + '}' * very_deep) |
which is checking that there are not segfaults when parsing a deep object.
The recursion limit should be set to 20,000 (+ current depth)
|
def infinite_recursion(max_depth=None): |
|
if max_depth is None: |
|
# Pick a number large enough to cause problems |
|
# but not take too long for code that can handle |
|
# very deep recursion. |
|
max_depth = 20_000 |
|
elif max_depth < 3: |
|
raise ValueError(f"max_depth must be at least 3, got {max_depth}") |
|
depth = get_recursion_depth() |
|
depth = max(depth - 1, 1) # Ignore infinite_recursion() frame. |
|
limit = depth + max_depth |
|
return set_recursion_limit(limit) |
The object is an order of magnitude deeper, so we should be hitting the recursion limit.
Specifically, we're failing in the cjson test, so it's just the C-accelerated JSON implementation that has different behavior
|
@unittest.skipUnless(cjson, 'requires _json') |
|
class CTest(unittest.TestCase): |
|
if cjson is not None: |
|
json = cjson |
|
loads = staticmethod(cjson.loads) |
|
dumps = staticmethod(cjson.dumps) |
|
JSONDecodeError = staticmethod(cjson.JSONDecodeError) |
zanieb commented on Oct 15, 2025 ContributorAuthor More actions Applying the following patch to revert 7016044 from #139232 (cc @rokm and @markshannon) resolves the issue
diff --git a/Python/ceval.c b/Python/ceval.c
index f48f412fab8..3bf1c098003 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -448,12 +448,6 @@ hardware_stack_limits(uintptr_t *top, uintptr_t *base)
ULONG guarantee = 0;
SetThreadStackGuarantee(&guarantee);
*base = (uintptr_t)low + guarantee;
-#elif defined(__APPLE__)
- pthread_t this_thread = pthread_self();
- void *stack_addr = pthread_get_stackaddr_np(this_thread); // top of the stack
- size_t stack_size = pthread_get_stacksize_np(this_thread);
- *top = (uintptr_t)stack_addr;
- *base = ((uintptr_t)stack_addr) - stack_size;
#else
/// XXX musl supports HAVE_PTHRED_GETATTR_NP, but the resulting stack size
/// (on alpine at least) is much smaller than expected and imposes undue limits
You can see the CI pass in astral-sh/python-build-standalone#823 rokm commented on Oct 18, 2025 @zanieb Do you perhaps have a problematic build with PGO+LTO available for testing? I imagine you would need to disable the cjson test that triggers segfault to get past the profiling stage for PGO...
But it would be interesting to see if a) such a build also causes segfault in my test case from #139231, b) if it does, what the stack depth increments were between recursions. Could be that LTO causes a problem similar to #137573, and the cost of (the last) recursion step ends up exceeding the margin. zanieb commented on Oct 18, 2025 ContributorAuthor More actions Indeed! I've disabled this test in order to release 3.15.0a1 in python-build-standalone so uv python install 3.15.0a1 will get you what you've requested. I'm not sure if the failure will occur when run outside of the profiling step, I haven't checked yet.
edit: I tried the questionable invocation
and the tests passed rokm commented on Oct 19, 2025
edit: I tried the questionable invocation
Can you try the example from #139231? (The recorded stack depth increments between recursion steps there are what is most interesting - i.e., whether they are suspiciously large).
Although if the test_json passes, it may be that the final PGO+LTO build is OK, and it is the intermediate (presumably just LTO-enabled) build that is problematic. zanieb commented on Oct 19, 2025 ContributorAuthor More actions ❯ uvx [email protected] test.py | tail
Recursion level: 4990, stack pointer: 0x16F3272B0, depth: 4451.33 kB
Recursion level: 4991, stack pointer: 0x16F326F20, depth: 4452.22 kB
Recursion level: 4992, stack pointer: 0x16F326B90, depth: 4453.11 kB
Recursion level: 4993, stack pointer: 0x16F326800, depth: 4454.00 kB
Recursion level: 4994, stack pointer: 0x16F326470, depth: 4454.89 kB
Recursion level: 4995, stack pointer: 0x16F3260E0, depth: 4455.78 kB
Recursion level: 4996, stack pointer: 0x16F325D50, depth: 4456.67 kB
Recursion level: 4997, stack pointer: 0x16F3259C0, depth: 4457.56 kB
Recursion level: 4998, stack pointer: 0x16F325630, depth: 4458.45 kB
Recursion Limit ok (reached level 4999)
❯ uvx [email protected] test.py | tail
Recursion level: 10007, stack pointer: 0x16C9D19F0, depth: 8137.52 kB
Recursion level: 10008, stack pointer: 0x16C9D16B0, depth: 8138.33 kB
Recursion level: 10009, stack pointer: 0x16C9D1370, depth: 8139.14 kB
Recursion level: 10010, stack pointer: 0x16C9D1030, depth: 8139.95 kB
Recursion level: 10011, stack pointer: 0x16C9D0CF0, depth: 8140.77 kB
Recursion level: 10012, stack pointer: 0x16C9D09B0, depth: 8141.58 kB
Recursion level: 10013, stack pointer: 0x16C9D0670, depth: 8142.39 kB
Recursion level: 10014, stack pointer: 0x16C9D0330, depth: 8143.20 kB
Recursion level: 10015, stack pointer: 0x16C9CFFF0, depth: 8144.02 kB
Recursion Limit ok (reached level 10016)
(note I compiled the stackpointer.dylib with gcc as suggested there)
It's also plausible that the instrumentation from PGO is causing the problem. rokm commented on Oct 19, 2025 Those increments are in line what I would expect (aside from the fact that stack size is 8 MB and not 16 MB - I assume you are overriding the setting in your builds)?
It's also plausible that the instrumentation from PGO is causing the problem.
That's true; the instrumentation is likely to result in increased stack consumption per recursion. It would be interesting to see how much (i.e., if it exceeds the value of the margin). Do you have an easy way to run test.py with the intermediate (instrumented) build? zanieb commented on Dec 2, 2025 ContributorAuthor More actions
Do you have an easy way to run test.py with the intermediate (instrumented) build?
I don't unfortunately. I presume it's doable via a patch but it doesn't sound trivial.
It looks like this affects 3.14.1 as well now. markshannon commented on Dec 3, 2025 Have you tried increasing the depth from 200k to, say, 500k? Is it much slower?
If that doesn't fail, it implies that PGO is thinking that deep recursion is a hot path, and maybe turning the calls into tailcalls.
We don't need the test to raise a recursion error, just not crash. So we could remove the assertion.
The test for the recursion error is just to check that the test is hitting the stack limit. zanieb commented on Dec 3, 2025 ContributorAuthor More actions added a commit that references this issue on Dec 8, 2025gh-140125: Increase object recursion depth for `test_json` from 200k …
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|