GeezerVison

Documentation build overview

📚 cpython-previews | 🛠️ Build #34907310 | 📁 Comparing ef0a101 against main (be87a85)

  🔍 Preview build  

On my Fedora 44, __clang_version__ is short: 22.1.8 (Fedora 22.1.8-4.fc44). But in issue gh-157368, way longer version has been seen: 22.1.3 (https://github.com/llvm/llvm-project e9846648fd6183ee6d8cbdb4502213fcf902a211). On Windows, PC/pyconfig.h no longer uses __clang_version__. But we may such long clang version on other platforms.

hey @vstinner, we ran into a side effect of moving the buffer to the heap - some debuggers and core-dump tools find the interpreter version by reading the Py_GetVersion() buffer out of process memory (py-spy for example). With only the 20-byte static_version left, they lose the tag and the "free-threading build" marker. Getting those back means following heap_version into the heap, which is fragile and not always possible.

can we make the static buffer bigger, at least enough to hold the version, tag, and free-threading marker (~64 bytes)? let me know if I should file a separate issue

py-spy:

    if let Some(&addr) = python_info.get_symbol("Py_GetVersion.version") {
        info!("Getting version from symbol address");
        if let Ok(bytes) = process.copy(addr as usize, 128) {
            if let Ok(version) = Version::scan_bytes(&bytes) {
                return Ok(version);
            }
        }
    }

I understand that in Python 3.15, py-spy copies 128 bytes from the static char version[300]; variable.

Would it be possible for py-spy to (1) call Py_GetVersion() function and (2) copy the return string? I suppose that it's better to not call functions in a debugger to leave the process state unchanged.

some debuggers and core-dump tools find the interpreter version by reading the Py_GetVersion() buffer out of process memory (py-spy for example)

I didn't think about these use cases. That's not how I expected the Python C API to be used. Well, they just inspect memory, they don't even call functions, if I understand correctly.

@itamaro:

can we make the static buffer bigger, at least enough to hold the version, tag, and free-threading marker (~64 bytes)? let me know if I should file a separate issue

Would you recommend to revert my change and add a comment explaining the debugger/crash reporter use cases?

Or just restore the static buffer to its Python 3.15 size (300 bytes)? I suppose that the static buffer name should be restored to version, so debuggers/crash reporters don't have to update their code.

Thanks for rising the issue!

Or there is another approach, build strings at build time, to avoid the complex code building strings at runtime: #158951.

I understand that in Python 3.15, py-spy copies 128 bytes from the static char version[300]; variable.

Would it be possible for py-spy to (1) call Py_GetVersion() function and (2) copy the return string? I suppose that it's better to not call functions in a debugger to leave the process state unchanged.

yeah, I think not calling function in the debugged process is the point. for core-dump tools, there isn't even a process around.

I didn't think about these use cases. That's not how I expected the Python C API to be used. Well, they just inspect memory, they don't even call functions, if I understand correctly.

sounds right. like you say, they're inspecting memories, not "using" the C API per se.

Would you recommend to revert my change and add a comment explaining the debugger/crash reporter use cases?

Or just restore the static buffer to its Python 3.15 size (300 bytes)? I suppose that the static buffer name should be restored to version, so debuggers/crash reporters don't have to update their code.

I don't think we need to revert, just restoring the version static buffer should be sufficient!

thanks for looking into this! I didn't look closely at your gh-158951, it looks like a bigger change than just restoring the static buffer, so if you think that's the better way to go, I may be able to test it in the next few days, or maybe next week.

thanks for looking into this! I didn't look closely at your #158951, it looks like a bigger change than just restoring the static buffer, so if you think that's the better way to go, I may be able to test it in the next few days, or maybe next week.

I created https://discuss.python.org/t/generate-modules-getbuildinfo-h-when-building-python-to-get-build-information-statically/109401 to discuss my change. The backup plan is to revert my change and just make the static version buffer larger (350 bytes?) to support very long Clang version strings.

thanks for making gh-158951 @vstinner ! I looked at it and did some testing, and here are my thoughts:

1. I don't have a strong opinion about generating the strings at build time, so I'll leave that to the issue/PR where there's already ongoing discussion.
2. if possible, I’d like to decouple fixing the regression from the rework. reinstating the version static buffer should be a quick, safe PR that doesn't need to wait on the bigger design discussion. happy to put a PR if it helps.
3. as implemented in gh-158948: Generate Modules/getbuildinfo.h #158951, the issue I raised isn't fixed. version is now a pointer rather than a buffer, and with gcc and clang at -O2 the variable is optimized out entirely, leaving only an unnamed string in .rodata.

to be more concrete about the use case: what broke for us is an lldb-based Python debugging extension that we use on live processes and on core dumps. on a core you can't call Py_GetVersion(), so it reads the string from memory. it can't rely on the symbol name either (with LTO the static becomes version.llvm.<hash>), so it scans .bss for the version pattern. a string in .rodata doesn't work for that, since it isn't in .bss, and default Linux core dumps don't include unmodified file-backed mappings like .rodata.

I tested this against our debugger test suite. with gh-158951 applied, it fails to detect the Python runtime. a named static const char version[] = VERSION; doesn't help either, since it's still in .rodata. a static char version[300] in .bss, filled at startup, passes everything.

what we actually use is the start of the string: version, local version suffix (3.16.0a0+meta), and the free-threading marker. a 20-byte fallback would cut off before the free-threading marker.

_Py_DebugOffsets covers most of this. I checked that _PyRuntime is in default core dumps with the cookie, version and free_threaded readable, and we already take free_threaded from there. what's missing is the suffix, which we use to pick build-specific struct offsets when debug info isn't available. if _Py_DebugOffsets exposed the PY_VERSION string, tools could move off the buffer for 3.16+, but that's a separate, bigger change, and older versions would still need the buffer.

so my concrete proposal is to keep static char version[~300] in .bss, filled at startup with the full/truncated string, as in 3.15. we only need roughly the first 64 bytes, but 300 matches what existing tools have always seen.

I was busy fixing other issues in the meanwhile, I didn't find time to finish PR gh-158951.

Since you seem to be affected by a regression, I created PR #159519 to simply revert my change allocating memory on the heap.