GeezerVison is showing github.com — Support arbitrary code page encodings on Windows · Issue #123803 · python/cpython
Support arbitrary code page encodings on Windows #123803DescriptionFeature or enhancement
Python supports encodings that correspond to some code pages on Windows, like cp437 or cp1252. But every such encoding should be specially implemented. There are code pages that do not have corresponding codec implemented in Python.
But there are functions that allow to encode or decode using arbitrary code page: codecs.code_page_encode() and codecs.code_page_decode(). The only step left is to make them available as encodings, so they could be used in str.encode() and bytes.decode().
Currently this is already used for the current Windows (ANSI) code page. If the cpXXX encoding is not implemented in Python and XXX matches the value returned by GetACP(), "cpXXX" will be made an alias to the "mbcs" codec.
I propose to add support for arbitrary cpXXX encodings on Windows. If such encoding is not implemented directly, fall back to use the Windows-specific API.
Linked PRs
Activitymalemburg commented on Sep 7, 2024 FWIW, I don't think this is a good idea, since we'd lose the cross-platform compatibility, if codecs are only available on Windows and not on other platforms.
Overall, and as already stated in #123489, I don't think we should add more encodings to the stdlib set. Support for more esoteric encodings can easily be added via PyPI packages, if needed.
The stdlib already has good support for many encodings and we don't really need more. Only if new Unicode related codecs get standardized, we should add new ones, e.g. new transfer encodings. At the moment, the world is moving towards UTF-8 as the one and only encoding and that's good. In the future other transfer encodings may emerge, which are more efficient, so we should be open to add those, but for classic encodings ones we currently do not support, I think people can either use external tools such as iconv or one of the available PyPI packages. changed the title [-]Supports arbitrary code page encodings on Windows[/-] [+]Support arbitrary code page encodings on Windows[/+] on Sep 7, 2024serhiy-storchaka commented on Sep 9, 2024 This cat is already out of the bag. Currently, if cpXXX is not defined in Python, but matches the current Windows code page, it is mapped as an alias of the "mbcs" codec. So you can have different sets of supported codecs depending on the environment. I propose to make it this less environment depending -- provide the same set of encoding on all Windows machines (almost the same, as it may depend on the Windows version).
There are much more encodings in the world than Python supports. And it is not realistic to include all codecs. But some codecs are already here, provided by the OS. I think we should use codecs provided by the OS for better interoperability withing the platform, and also provide a set of codecs for inter-platform interoperability. We could even remove some codecs implemented in Python if they are well supported on all maintained platforms (but this is unlikely, because our own implementation may be more efficient). At least it will help us to reject requests for adding new codecs. zooba commented on Sep 9, 2024 Yeah, I understand MAL's concern, but I'm inclined to agree with Serhiy on this one. Specifically:
I think we should use codecs provided by the OS for better interoperability withing the platform
My request would be (and I haven't checked the PR to see if it's there yet) that we have an error message that clearly suggests the encoding is not available on this platform (doesn't have to specify the actual platform), as opposed to merely that the encoding doesn't exist. If it's easy, having it be different from the generic error would be great. e.g. (on POSIX):
s.encode('cp1234')
LookupError: encoding cp1234 is not available on this platform
s.encode('spamalot')
LookupError: unknown encoding: spamalot
serhiy-storchaka commented on Nov 5, 2024
My request would be (and I haven't checked the PR to see if it's there yet) that we have an error message that clearly suggests the encoding is not available on this platform (doesn't have to specify the actual platform), as opposed to merely that the encoding doesn't exist.
I afraid that this is impossible. How do you distinguish an encoding that is not available on this platform from an encoding that is not available on all platforms? We can change the error message for cpXXX exceptions, but it will be incorrect for code pages not supported on Windows, like cp999. We can maintain the list of currently supported Windows code pages, but for what Windows version? And it all should be hardcoded in C, to allow users to register custom search functions. zooba commented on Nov 5, 2024
How do you distinguish an encoding that is not available on this platform from an encoding that is not available on all platforms? We can change the error message for cpXXX exceptions, but it will be incorrect for code pages not supported on Windows, like cp999.
No, it'll be consistent. Both Windows and POSIX will raise "this codepage is not available on this system", as opposed to saying that it's an unknown encoding.
And the same mechanism on POSIX can raise "this codepage is not available on this system" for all cpXXX that don't have an alias, which is more accurate than saying that it's invalid.
But it's a minor thing to help out users when scripts get moved across platforms. If it's not easy to implement, then don't worry about it. serhiy-storchaka commented on Nov 20, 2024 I plan to add support for iconv codecs, so system encodings will also be supported on non-Windows platforms. But this is a much more complex issue. I am not sure that it is compatible with Python API for codecs. malemburg commented on Nov 20, 2024 Looking at the iconv docs (https://www.gnu.org/software/libiconv/), this should be possible. The only problem may be with allocating a suitable output buffer - estimating the needed size will likely be hard, since codecs will typically have very different behavior when it comes to encoding sizes.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|