GeezerVison

Error strings are locale specific so _locale.setlocale clears it.

A C library can change the LC_CTYPE locale. Or when Python is embedded in an application, the application can change the LC_CTYPE locale (without telling Python).

Such optimization would (only?) be safe if it checks if the LC_CTYPE locale changed. Either check setlocale(LC_CTYPE, NULL) string or check nl_langinfo(CODESET) string. I suggest checking nl_langinfo(CODESET).

This is looking like it needs two cache keys (CODESET + LC_MESSAGES) if we need to invalidate when things outside of python change locale but don't tell Python about it directly... which gets to significant added complexity.

This is safe in terms of "python won't crash". It's not great in "locale may not match exactly". Open to suggestions that keep the complexity from significantly expanding (cutting 20% off each FileNotFoundError when we start python and it looks for .pyc would be really nice)

I don't know if it's feasible, but I always wanted to defer when OSError message are formatted. In short, store components of the error message (like error number, optional path), and only render the message string if it's requested. In the majority of cases, the message is never requested.

One pain point is OSError.args which contains the message, so it should be a getter function. But BaseException constructor stores its positional arguments as the self->args attribute. I'm not sure how to deal with that.

If that would be possible, it should be faster than cached os.strerror().

I'll put this in draft and experiment a bit :). Planning to make the BaseException Argument Clinic Vectorcall shortly which moves performance relative to 3.15 a bit and changes the tuple args handling some.

As part of the posix -> _os I was looking at ntpath vs posixpath and other than some constants the key performance difference is ntpath uses Windows APIs to return "exists" and the like without requiring the exception while posixpath does not. A new primitive could potentially help there as well.... That said "exception on open" is the only safe/secure way to do a lot of things (check than open runs into timing attack issues)

Another different approach: optimize PyUnicode_DecodeLocale() for UTF-8 (most common locale encoding): vstinner#21. My hack makes the function 3.2x faster and makes os.strerror() 2.0x faster.

Have a PR that helps with that #161147 :). Although mainly focused on "We load large mainly-ascii text files a lot". Definitely very aligned here, just a matter of finding what is going to be shippable simplifications. For me generally aiming for do less work -> it runs faster. Eliminate a whole stack of things most direct, optimize them longer path but also very doable.

Making args lazy is hard because PyException_HEAD is publicly exposed and only soft deprecated (a member going from never NULL to starting NULL with helper feels like a bad silent break to me)

Making args lazy is hard because PyException_HEAD is publicly exposed and only soft deprecated (a member going from never NULL to starting NULL with helper feels like a bad silent break to me)

Are there projects reading the args member using the C API? Maybe we can put a placeholder value (like None or "error") until os.strerror() is called lazily.

os.strerror(2) 128 ns 29.1 ns: 4.41x faster

I'm honestlty surprised that calling nl_langinfo(CODESET) and setlocale(LC_MESSAGES, NULL) doesn't add overhead. Well, it's nice that it can be optimized!