I'm interested in working on this. I've posted a draft PR as earnest effort (in the sense of earnest money), but we should agree on the answers to these questions before serious review.
Do we fall back to stat?
The previous PR gh-19125 did not fall back to stat if statx was unavailable. A comment in this issue objected to this because statx was relatively new at the time. statx was introduced in Linux 4.11, released April 2017. The oldest currently-supported upstream LTS branch is 5.4, released November 2019. If I am reading the RHEL life cycle and RHEL release dates correctly, the oldest supported release is RHEL8, based on kernel 4.18. It is likely that all mainstream Linux distributions ship kernels supporting statx. On the other hand, there are probably embedded environments stuck on ancient kernels; do their users want to run the newest Python?
A complicating factor is default-deny seccomp syscall filters that have not been updated to allow statx. If they deny with ENOSYS and we fall back to stat, Python will keep working under those filters; if we don't fall back, Python will almost certainly fail during startup.
The code to fall back is simple enough, but testing the fallback path would require running on an old kernel or under a seccomp filter.
My draft PR has a fallback, but no tests for the fallback.
How do we call statx?
In order of increasing reliance on the build environment, here are the ways we can call statx:
-
Hardcode the syscall numbers for architectures we care about, provide our own definition of struct statx and related flags, and call via syscall(). Build is self-sufficient.
-
Use syscall numbers from sys/syscall.h, definitions of struct stat and flags from linux/stat.h, provide our own definitions of flags newer than statx itself, call via syscall(). Build requires Linux kernel userspace API headers >= 4.11. (New userspace API headers can be used on old kernels; they're distinct from the headers used to build kernel modules.)
-
Call using libc's statx wrapper (weakly linked), provide our own definitions of flags newer than statx itself. Requires glibc version 2.28 (August 2018) at build time and run time (but can fall back to stat if runtime libc is old).
My draft PR is option 3.
How do we indicate missing information?
We won't always get everything we ask for in struct statx, depending on the kernel, filesystem type, and filesystem instance (e.g., ext4 supports btime, but really old ext4 filesystems may not). For btime specifically, st_birthtime has never been exposed on Linux, so it's tempting to set it to None when it isn't available. Code that checks the platform before using st_birthtime and code using getattr(st, 'st_birthtime', None) will work, but code that relies on catching AttributeError may not expect st_birthtime to be None -- and there's an example of the latter in test_os.py. So we have to fill it in, either with 0 or with whatever statx returned (not necessarily 0).
The basic stats (also in struct stat) are always initialized, but their bits may not be set in stx_mask. linux/stat.h gives CIFS as an example that might clear STATX_UID/GID if they're being overridden client-side, though I haven't observed this. I have observed a CIFS mount on Linux 6.15.4 that doesn't set STATX_ATIME. We obviously can't set st_atime/uid/gid to None.
I think we should indicate to the application what fields are and aren't reliable, but I don't know how. Exposing stx_mask isn't extensible to other operating systems. We could make up our own mask and flags. We could have a set-like member for tests of the form 'st_birthtime' in st.present, though I don't know how to implement that performantly (and stat is likely hot in some programs). We could introduce int/float subclasses with an extra 'present' member, but I'm sure there are programs that expect exactly int/float. I don't like any of these options.
My draft PR does nothing here.
Which mount ID do we want?
There are two flags to ask for mount IDs, STATX_MNT_ID and STATX_MNT_ID_UNIQUE. The non-unique ID indexes /proc/self/mountinfo and is subject to reuse race conditions; the unique ID can be used with the statmount system call, which requires CAP_SYS_ADMIN and does not currently have a glibc wrapper. Unfortunately they both initialize stx_mnt_id, so we can only get one or the other (unless we call statx twice). Setting both flags is not an error, but I do not think the result is defined; my experiments have always obtained the unique ID.
My draft PR provides the non-unique ID because I think it will be more useful to general Python programs.
How do we test the new members?
ntfs-3g and cifs allow setting btime via the user.ntfs/cifs.creationtime magic xattr. I don't know offhand if ntfs-3g reports btime (being a FUSE filesystem). cifs does, but that requires a server to be available. We could also just create a file, wait, modify the file, and check its btime differs from its mtime and ctime.
We can test stx_mnt_id by looking for it in /proc/self/mountinfo.
Only btrfs and bcachefs currently support stx_subvol. bcachefs has an uncertain future, so we'd want btrfs.
Of the stx_attributes, only STATX_ATTR_MOUNT_ROOT is easily testable, by opening /proc/self/mountinfo and checking the flag is set on mount points. This flag should also be independent of both filesystems.
My draft PR adds no new tests.
Do we want to map attributes to st_flags?
The documentation for the os module claims st_flags "may be available on some Unix systems (such as Linux)", but as far as I can tell it has never been exposed on Linux. The previous PR gh-19125 mapped both Windows and Linux attribute bits to st_flags. Do we want that? Do we want to map st_flags back to attribute bits? (I don't know which systems currently have st_flags or how they are used. Many of them are macOS-specific. The os.chflags documentation says "Availability: Unix, not WASI" but it's not present on Linux. BSD?)
My draft PR doesn't do this.
What do we name stx_attributes?
My PR names it st_attributes, but maybe st_linux_attributes is safer.
Future work
statx supports the AT_STATX_FORCE_SYNC and AT_STATX_DONT_SYNC flags to ask for up-to-date or cached information. (AT_STATX_SYNC_AS_STAT is 0.) I'd like to expose this as a sync=True/False/None argument to os.stat, but I need to see what other operating systems offer first. In particular, the statx flags are hints; if cached info is not available, statx will hit the network/tape/etc. I can imagine that other operating systems might have a flag to say "cached if possible, else fail, never block", which would have to be handled differently.