GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-90548: Make musl test skips smarter (fixes Alpine errors) by bitdancer · Pull Request #131313 · python/cpython

Conversation

@bitdancer

@bitdancer bitdancer commented Mar 15, 2025 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Make musl test skips smarter (fixes Alpine errors)

A relatively small number of tests fail when the underlying c library is
provided by musl. This was originally reported in bpo-46390 by
Christian Heimes. Among other changes, these tests were marked for
skipping in gh-31947/ef1327e3 as part of bpo-40280 (emscripten support),
but the skips were conditioned on the platform being emscripten (or
wasi, skips for which ere added in 9b50585).

In gh-131071 Victor Stinner added a linked_to_musl function to enable
skipping a test in test_math that fails under musl, like it does on a
number of other platforms. This check can successfully detect that
python is running under musl on Alpine, which was the original problem
report in bpo-46390.

This PR replaces Victor's solution with an enhancement to
platform.libc_ver that does the check more cheaply, and also gets the
version number. The latter is important because the math test being
skipped is due to a bug in musl that has been fixed, but as of this
checkin date has not yet been released. When it is, the test skip can
be fixed to check for the minimum needed version.

The enhanced version of linked_to_musl is also used to do the skips of
the other tests that generically fail under musl, as opposed to
emscripten or wasi only failures. This will allow these tests to be
skipped automatically on Alpine.

This PR does not enhance libc_ver to support emscripten and wasi, as
I'm not familiar with those platforms; instead it returns a version
triple of (0, 0, 0) for those platforms. This means the musl tests will
be skipped regardless of musl version, so ideally someone will add
support to libc_ver for these platforms.

📚 Documentation preview 📚: https://cpython-previews--131313.org.readthedocs.build/

  • Issue: Multiple test failures on Alpine 3.15 / musl-1.2.2-r7 #90548

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
mara004 and cmaureir reacted with thumbs up emoji
All reactions
  • 👍 2 reactions
bitdancer added 3 commits March 15, 2025 15:17
@bitdancer
Make musl test skips smarter (fixes Alpine errors)
2348711 
A relatively small number of tests fail when the underlying c library is
provided by musl.  This was originally reported in bpo-46390 by
Christian Heimes.  Among other changes, these tests were marked for
skipping in pythongh-31947/ef1327e3 as part of bpo-40280 (emscripten support),
but the skips were conditioned on the *platform* being emscripten (or
wasi, skips for which ere added in 9b50585).

In pythongh-131071 Victor Stinner added a linked_to_musl function to enable
skipping a test in test_math that fails under musl, like it does on a
number of other platforms.  This check can successfully detect that
python is running under musl on Alpine, which was the original problem
report in bpo-46390.

This PR replaces Victor's solution with an enhancement to
platform.libc_ver that does the check more cheaply, and also gets the
version number.  The latter is important because the math test being
skipped is due to a bug in musl that has been fixed, but as of this
checkin date has not yet been released.  When it is, the test skip can
be fixed to check for the minimum needed version.

The enhanced version of linked_to_musl is also used to do the skips of
the other tests that generically fail under musl, as opposed to
emscripten or wasi only failures.  This will allow these tests to be
skipped automatically on Alpine.

This PR does *not* enhance libc_ver to support emscripten and wasi, as
I'm not familiar with those platforms; instead it returns a version
triple of (0, 0, 0) for those platforms.  This means the musl tests will
be skipped regardless of musl version, so ideally someone will add
support to libc_ver for these platforms.
@bitdancer
Platform tests and bug fixes.
35960c1 
In adding tests for the new platform code I found a bug in the old code:
if a valid version is passed for version and it is greater than the
version found for an so *and* there is no glibc version, then the
version from the argument was returned.  The code changes here fix
that, as well as fixing my own broken additions.
@bitdancer
Add docs, including some existing is_xxx's.
417143e
@bitdancer
bitdancer requested a review from pganssle as a code owner March 15, 2025 22:31
@bitdancer bitdancer self-assigned this Mar 15, 2025
@bitdancer
bitdancer requested a review from abalkin as a code owner March 15, 2025 22:31
@bedevere-app bedevere-app Bot added the awaiting core review label Mar 15, 2025
@bitdancer bitdancer changed the title better musl test checks Make musl test skips smarter (fixes Alpine errors) Mar 15, 2025
@bitdancer
bitdancer requested review from tiran and vstinner March 15, 2025 22:40
@bitdancer bitdancer mentioned this pull request Mar 17, 2025
Multiple test failures on Alpine 3.15 / musl-1.2.2-r7 #90548
Closed
@vstinner vstinner mentioned this pull request Mar 17, 2025
test_re fails on Alpine Linux (with musl) #131342
Closed
vstinner
vstinner reviewed Mar 17, 2025
View reviewed changes
Comment thread Doc/library/test.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/test/test_math.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/test/test_os.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/test/test_platform.py
(b'GLIBC_2.9', ('glibc', '2.9')),
(b'libc.so.1.2.5', ('libc', '1.2.5')),
(b'libc_pthread.so.1.2.5', ('libc', '1.2.5_pthread')),
(b'/aports/main/musl/src/musl-1.2.5', ('musl', '1.2.5')),

@vstinner vstinner Mar 17, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add tests on version with 2 numbers and 4 numbers? Like /aports/main/musl/src/musl-1.2 and /aports/main/musl/src/musl-1.2.5.1.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@bitdancer bitdancer Mar 17, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could, but I don't believe musl will ever have such release numbers, so is it worth doing?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@vstinner vstinner Mar 17, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parsing a binary file using a regular expression is a strange thing for me. So I prefer to have tests to make sure that the implementation is reliable. The question is more if the regex will match versions with 2 members (x.y) or 4 members (x.y.z.a), or if it only matchs version with 3 members (x.y.z).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@bitdancer bitdancer Mar 17, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, though the existing tests don't do that, nor do they test the no-match cases. I could add such tests, but at that point I'd want to rewrite this test method to split it up into multiple tests, and I don't think that's in scope for this PR :) I'll add the couple of extra checks, though.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/test/test_support.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/test/test_support.py Outdated
linked = support.linked_to_musl()
self.assertIsInstance(linked, bool)
self.assertIsNotNone(linked)
if support.is_wasi:

@vstinner vstinner Mar 17, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about is_emscripten?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@bitdancer bitdancer Mar 17, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used is_wasi as an example, but it doesn't hurt to add emscripten as well.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/test/test_support.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@vstinner vstinner added the skip news label Mar 17, 2025
@vstinner

vstinner commented Mar 17, 2025

Copy link
Copy Markdown
Member

The code detecting the musl version works for me on Alpine Linux:

~/cpython # LD_LIBRARY_PATH=$PWD ./python 
Python 3.14.0a5+ (heads/main-dirty:fe186d76cfd, Mar 11 2025, 08:54:51) [GCC 14.2.0] on linux
>>> import test.support
>>> test.support.linked_to_musl()
(1, 2, 5)


~/cpython # LD_LIBRARY_PATH=$PWD ./python -m test.pythoninfo|grep libc
platform.libc_ver: musl 1.2.5


~/cpython # apk info musl
musl-1.2.5-r9 description:
the musl c library (libc) implementation

musl-1.2.5-r9 webpage:
https://musl.libc.org/

musl-1.2.5-r9 installed size:
646 KiB
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
picnixz
picnixz reviewed Mar 17, 2025
View reviewed changes

@picnixz picnixz left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my suggestions with dedents, please check them manually as I'm not entirely sure that I didn't mix tabs and spaces.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/platform.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/test/support/__init__.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/test/test_os.py
@unittest.skipIf(
support.linked_to_musl(),
'musl pathconf ignores the file descriptor and returns a constant',
)

@picnixz picnixz Mar 17, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
)
)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@bitdancer bitdancer Mar 17, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer the style I used because it lines up better with how python's statements work (first line only is outdented, lack of outdent signals return to outer logic block). This is accepted by PEP 8, are there additional style constrains on the cpython code base these days?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@picnixz picnixz Mar 17, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't seen lots of use of hanging indents in the lib. There is no additional constraints but usually we try to be consistent with the surrounding code when possible.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/test/test_platform.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/test/test_support.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/test/test_platform.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@picnixz

picnixz commented Mar 17, 2025

Copy link
Copy Markdown
Member

Btw, do we want to create an issue (maybe an umbrella one for musl issues) or do we make it a skip-issue?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bitdancer

bitdancer commented Mar 17, 2025

Copy link
Copy Markdown
Member Author

There is an issue (gh-90548). I forgot the gh issue existed when I mentioned the bpo issue :( I'm not actually sure how one does the link, as I'm still coming up to speed on the "modern" way of doing development ;)
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@picnixz picnixz changed the title Make musl test skips smarter (fixes Alpine errors) gh-90548: Make musl test skips smarter (fixes Alpine errors) Mar 17, 2025
@picnixz

picnixz commented Mar 17, 2025

Copy link
Copy Markdown
Member

I'm not actually sure how one does the link, as I'm still coming up to speed on the "modern" way of doing development ;)

No worries! the PR title should be gh-XXXXXX: title or GH-XXXXXX: title and the bot would link it.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
bitdancer added 3 commits March 17, 2025 14:24
@bitdancer
Address vstinner review comments.
ef36eb2
@bitdancer
Test some version number variants we accept.
909ea3b
@bitdancer
Add news item about libc_ver enhancement.
4e76d78
vstinner
vstinner approved these changes Mar 18, 2025
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but please fix the important typo 'musl' in binary.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/platform.py Outdated
pos = 0
while pos < len(binary):
if b'libc' in binary or b'GLIBC' in binary:
if b'libc' in binary or b'GLIBC' or 'musl' in binary:

@vstinner vstinner Mar 18, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if b'libc' in binary or b'GLIBC' or 'musl' in binary:
if b'libc' in binary or b'GLIBC' or b'musl' in binary:

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@bitdancer bitdancer Mar 18, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, that's a nasty one. The actual fix is b'GLIBC in binary or b'musl' in binary`. That's why it passed without the b...it just ran slower ;) (A lot slower, somewhat to my surprise.)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Mar 18, 2025
picnixz
picnixz approved these changes Mar 18, 2025
View reviewed changes

@picnixz picnixz left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I said, I won't block the PR for cosmetics only though I would have aligned with the surrounding/existing code when possible.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
serhiy-storchaka
serhiy-storchaka approved these changes Mar 18, 2025
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM in general.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/platform.py Outdated
Comment on lines +192 to +201
libc_search = re.compile(
br'(__libc_init)'
br'|'
br'(GLIBC_([0-9.]+))'
br'|'
br'(libc(_\w+)?\.so(?:\.(\d[0-9.]*))?)'
br'|'
br'(musl-([0-9.]+))'
br'',
re.ASCII)

@serhiy-storchaka serhiy-storchaka Mar 18, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, we can switch to the VERBOSE mode, which would be slightly more clear.

Suggested change
libc_search = re.compile(
br'(__libc_init)'
br'|'
br'(GLIBC_([0-9.]+))'
br'|'
br'(libc(_\w+)?\.so(?:\.(\d[0-9.]*))?)'
br'|'
br'(musl-([0-9.]+))'
br'',
re.ASCII)
libc_search = re.compile(br'''
(__libc_init)
| (GLIBC_([0-9.]+))
| (libc(_\w+)?\.so(?:\.(\d[0-9.]*))?)
| (musl-([0-9.]+))
''',
re.ASCII | re.VERBOSE)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@bitdancer bitdancer Mar 18, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/test/support/__init__.py Outdated
# emscripten (at least as far as we're concerned) and wasi use musl,
# but platform doesn't know how to get the version, so set it to zero.
if is_emscripten or is_wasi:
return (_linked_to_musl := (0, 0, 0))

@serhiy-storchaka serhiy-storchaka Mar 18, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not think that it is necessary to use the := operator here.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@bitdancer bitdancer Mar 18, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is never necessary to use the := operator, so do you mean you prefer the more verbose style (which I'm fine with changing to), or am I missing something?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@serhiy-storchaka serhiy-storchaka Mar 18, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer the more verbose style, which is also more sequential and clear.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@bitdancer bitdancer Mar 18, 2025

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/test/test_strptime.py
"musl libc issue on Emscripten, bpo-46390"
)
@unittest.skipIf(support.linked_to_musl(), "musl libc issue, bpo-46390")
@run_with_locales('LC_TIME', 'en_US', 'fr_FR', 'de_DE', 'ja_JP',

@serhiy-storchaka serhiy-storchaka Mar 18, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can add the linked_to_musl() check in run_with_locales() and run_with_locale(). This may be non-trivial, depending on arguments, a dummy run can be supported even if no suitable locale was found. For now, the proposed change is good as is.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
bitdancer added 2 commits March 18, 2025 14:49
@bitdancer
Fix 'in' phrase logic.
83add5d 
This didn't cause a test failure because the broken logic caused the
regex to always run...which is a lot slower.
@bitdancer
Prettify re expression using re.VERBOSE.
4d6aa92
@bitdancer
Use more verbose style rather than :=
111e2ec
serhiy-storchaka
serhiy-storchaka approved these changes Mar 19, 2025
View reviewed changes
@bitdancer
bitdancer merged commit 6146295 into python:main Mar 19, 2025
@bedevere-app bedevere-app Bot removed the awaiting merge label Mar 19, 2025
@bitdancer

bitdancer commented Mar 19, 2025

Copy link
Copy Markdown
Member Author

I'm thinking this is not worth backporting?
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner

vstinner commented Mar 19, 2025

Copy link
Copy Markdown
Member

I'm thinking this is not worth backporting?

I don't think that it's worth it to backport this change to stable branches. Supporting Alpine Linux can be seen as a new feature ;-)
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
This was referenced Apr 28, 2025
gh-87414: add musl support to platform.libc_ver #103784
Closed
platform.libc_ver() returns no information on Alpine Linux #87414
Closed
@bitdancer
bitdancer deleted the better_musl_test_checks branch May 26, 2025 17:06
@cdce8p cdce8p mentioned this pull request Aug 16, 2025
gh-90548: Fix musl version detection with --strip-all #137864
Merged
This was referenced Jun 29, 2026
Enable fma test on Alpine Linux (musl C library) #152563
Closed
gh-152563, test_math: Enable fma test on musl 1.2.6 #152564
Merged
@tonghuaroot tonghuaroot mentioned this pull request Aug 10, 2026
platform.libc_ver() raises TypeError on a threaded libc without a version #155454
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner approved these changes
@serhiy-storchaka serhiy-storchaka serhiy-storchaka approved these changes
@picnixz picnixz picnixz approved these changes
@pganssle pganssle Awaiting requested review from pganssle pganssle is a code owner
@abalkin abalkin Awaiting requested review from abalkin
@tiran tiran Awaiting requested review from tiran

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@bitdancer bitdancer

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@bitdancer @vstinner @picnixz @serhiy-storchaka
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.