GeezerVison is showing github.com — ElementTree should use UTF-8 for xml declaration. · Issue #91810 · python/cpython
ElementTree should use UTF-8 for xml declaration. #91810DescriptionFeature or enhancement
Currently, ElementTree.tostring(root, encoding="unicode", xml_declaration=True) uses locale encoding.
I think ElementTree should use UTF-8, instead of locale encoding.
Example:
$ LANG=ja_JP.eucJP ./python.exe
Python 3.11.0a7+ (heads/bytes-alloc-dirty:7fbc7f6128, Apr 19 2022, 16:53:54) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import xml.etree.ElementTree as ET
>>> et = ET.fromstring("<t>hello</t>")
>>> ET.tostring(et, encoding="unicode", xml_declaration=True)
"<?xml version='1.0' encoding='eucJP'?>\n<t>hello</t>"Code:
|
with _get_writer(file_or_filename, enc_lower) as write: |
|
if method == "xml" and (xml_declaration or |
|
(xml_declaration is None and |
|
enc_lower not in ("utf-8", "us-ascii", "unicode"))): |
|
declared_encoding = encoding |
|
if enc_lower == "unicode": |
|
# Retrieve the default encoding for the xml declaration |
|
import locale |
|
declared_encoding = locale.getpreferredencoding() |
|
write("<?xml version='1.0' encoding='%s'?>\n" % ( |
|
declared_encoding,)) |
Pitch
- UTF-8 is the most common encoding for XML.
- Locale encoding name (e.g.
cp932 or eucJP) would be different from XML encoding name recommended by w3c (e.g. Shift_JIS or EUC-JP).
Activityserhiy-storchaka commented on Apr 22, 2022 Look at dump(). It writes an element to stdout, which usually uses the locale encoding. I think it is the rationale of using the locale encoding here. You need to change dump() to use the stdout's encoding explicitly. serhiy-storchaka commented on Apr 22, 2022 Or maybe change write() to get the encoding from the output string if available. methane commented on Apr 23, 2022 dump don't use xml_declaration=True. So this issue doesn't affect it.
methane commented on Apr 25, 2022 @scoder would you give us an advice? (you are listed as etree expert in expert index).
There is no correct behavior, because output is Unicode and etree don't know what is real output encoding.
There are some cases that current behavior is better (e.g. using default encoding (e.g. open(filename, 'w')).
On the other hand, encoding="cp932" (in Japanese Windows) is non-portable (encoding="Shift_JIS" should be used), and UTF-8 is the most recommended encoding for XML.
I have two ideas:
a. Make UTF-8 default. This is simplest.
b. Keep using locale.getpreferredencoding() and wait PEP 686 accepted. (But it should be replaced with locale.getpreferredencoding(False) anyway.) serhiy-storchaka commented on Apr 25, 2022 Adding encoding="UTF-8" and using cp932 to encode the content would be even worse.
Maybe add a simple mapping from Python encodings to XML encodings (for example we need to write "ascii" as "us-ascii")? Later we can discuss adding a public API for this. serhiy-storchaka commented on Apr 25, 2022 I proposed to get the default encoding from the file object if available. #91812 (comment) methane commented on Apr 25, 2022
Adding encoding="UTF-8" and using cp932 to encode the content would be even worse.
Of course, we should recommend to use UTF-8.
Note that encoding='cp932' and using UTF-8 is possible bug for now already.
Any default value may cause bug. There is no one correct default. But UTF-8 may be the best for now.
Maybe add a simple mapping from Python encodings to XML encodings (for example we need to write "ascii" as "us-ascii")? Later we can discuss adding a public API for this.
We may not know Python encoding because output is Unicode (e.g. Unicode string or StringIO).
Such idea works only when output is TextIOWrapper. (And there are no guarantee that TextIOWrapper.encoding is really the final encoding.)
If we want to support arbitrary encoding, we should add another option like xml_declaration_encoding="Shift_JIS".
But this is not strict necessary.
User can chose xml_declaration=False and prepend <?xml version="1.0" encoding="Shift_JIS" ?> manually when they really need to use encoding other than UTF-8. serhiy-storchaka commented on Apr 25, 2022
Note that encoding='cp932' and using UTF-8 is possible bug for now already.
Yes, it is a bug, and #91903 fixes it. added a commit that references this issue on May 11, 2022gh-91810: ElementTree: Use text file's encoding by default in XML dec… added a commit that references this issue on Jun 14, 2022gh-91810: Fix regression with writing an XML declaration with encodin… added 2 commits that reference this issue on Jun 16, 2022gh-91810: Fix regression with writing an XML declaration with encodin… gh-91810: Fix regression with writing an XML declaration with encodin… added a commit that references this issue on Jul 26, 2022gh-91810: Fix regression with writing an XML declaration with encodin…
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|