GeezerVison is showing github.com — euc_kr char '0x3164' decode('ksx1001') cause UnicodeDecodeError · Issue #101863 · python/cpython
euc_kr char '0x3164' decode('ksx1001') cause UnicodeDecodeError #101863Labels interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-unicodetype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error Description
Bug report
char '0x3164' can be encode('ksx1001'), but can not decode('ksx1001')
def main():
code_point = 0x3164
c = chr(code_point)
raw = c.encode('ksx1001')
c2 = raw.decode('ksx1001') # <--- this cause error
print(f'{c} {c2}')
if __name__ == '__main__':
main()Traceback (most recent call last):
File "/Users/takwolf/Develop/FontDev/fusion-pixel-font/build.py", line 11, in <module>
main()
File "/Users/takwolf/Develop/FontDev/fusion-pixel-font/build.py", line 6, in main
c2 = raw.decode('ksx1001')
^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'euc_kr' codec can't decode bytes in position 0-1: incomplete multibyte sequence
The char is Hangul Compatibility Jamo -> Hangul Filler
https://unicode-table.com/en/3164/
The following code is get the zone in ks-x-1001:
def main():
code_point = 0x3164
c = chr(code_point)
raw = c.encode('ksx1001')
block_offset = 0xA0
zone_1 = raw[0] - block_offset
zone_2 = raw[1] - block_offset
print(f'{zone_1} {zone_2}')
if __name__ == '__main__':
main()https://en.wikipedia.org/wiki/KS_X_1001#Hangul_Filler
other chars in ksx1001 encode an decode is ok, but only this.
Your environment
- CPython versions tested on: Python 3.11.1
- Operating system and architecture: macOS 13.0
Linked PRs
Activityarhadthedev commented on Feb 13, 2023 corona10 commented on Feb 17, 2023 I will take a look at this issue as the internal OSS sprint issue.
I will delegate this issue to my mentees. b8choi commented on Feb 20, 2023 I will try to resolve this issue. Reacted by Donghee Na Reacted by Donghee Na TakWolf commented on Feb 23, 2023 corona10 commented on Feb 23, 2023
Is CP949 equals to ksx1001?
Not equals corona10 commented on Feb 23, 2023
I don't understand why the cp949encmap size not equals the ksx1001decmap size?
Because CP949 is a kind of extended version of KSX1001 TakWolf commented on Feb 23, 2023 iterate unicode :
def main():
encode_count = 0
decode_count = 0
for code_point in range(0, 0x10FFFF + 1):
c = chr(code_point)
try:
raw = c.encode('ksx1001')
encode_count += 1
try:
raw.decode('ksx1001')
decode_count += 1
except:
pass
except:
pass
print(f'encode: {encode_count}')
print(f'decode: {decode_count}')encode: 17176
decode: 17175
iterate ksx1001 :
def main():
decode_count = 0
for zone_1 in range(1, 94 + 1):
for zone_2 in range(1, 94 + 1):
try:
bytes([zone_1 + 0xA0, zone_2 + 0xA0]).decode('ksx1001')
decode_count += 1
except UnicodeDecodeError:
pass
print(f'decode: {decode_count}')TakWolf commented on Feb 23, 2023 def main():
not_2_bytes_alphabet = list()
not_2_bytes_count = 0
ksx1001_other_count = 0
ksx1001_syllable_count = 0
ksx1001_word_count = 0
ksx1001_but_not_a_char_count = 0
for code_point in range(0, 0x10FFFF + 1):
c = chr(code_point)
try:
raw = c.encode('ksx1001')
except UnicodeEncodeError:
continue
if len(raw) != 2:
not_2_bytes_count += 1
not_2_bytes_alphabet.append(c)
continue
zone_1 = raw[0] - 0xA0
if 1 <= zone_1 <= 12:
ksx1001_other_count += 1
elif 16 <= zone_1 <= 40:
ksx1001_syllable_count += 1
elif 42 <= zone_1 <= 93:
ksx1001_word_count += 1
else: # zone_1 is 13~15, 41, 94
ksx1001_but_not_a_char_count += 1
print(f'ksx other + syllable + word :{ksx1001_other_count + ksx1001_syllable_count + ksx1001_word_count}')
print(f'other: {ksx1001_other_count}')
print(f'syllable: {ksx1001_syllable_count}')
print(f'word: {ksx1001_word_count}')
print(f'in ksx but not a char: {ksx1001_but_not_a_char_count}')
print(f'not 2 len bytes: {not_2_bytes_count}')
print(f'not 2 len bytes + syllable: {not_2_bytes_count + ksx1001_syllable_count}')
print('unicode all syllable: 11172')
print(''.join(not_2_bytes_alphabet))ksx other + syllable + word :8226 # 8225 + 1(Hangul Filler)
other: 988
syllable: 2350
word: 4888
in ksx but not a char: 0
not 2 len bytes: 8950 # <---- what's this ? Include other syllables not in ksx1001 2350?
not 2 len bytes + syllable: 11300
unicode all syllable: 11172
not_2_bytes_alphabet:
�������
������������������ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~�갂갃갅갆갋 ...... # Behind them are all syllables
b8choi commented on Mar 2, 2023 From my investigation so far, there are some unclear aspects of the decoding process related to Korean characters encoded in EUC-KR.
FYI, the KS X 1001 standard defines the Korean character set, while EUC-KR is an encoding method for this set. In Python, "ksx1001" is an alias of "euc_kr".
Korean alphabet, also known as Hangul, expresses one syllable consisting of consonants and vowels as one letter. The KS X 1001 standard defines a precomposed set of characters that correspond to one syllable rather than individual consonants and vowels. However, to express a character not included in this set, a method of composing consonants and vowels is defined. The "Hangul Filler" character is used for this purpose, and it, along with the following three consonants and vowels, are arranged to express one character in a total of 8 bytes.
The KS X 1001 standard does not provide any specific guidelines other than the method for expressing one character using the "Hangul Filler".
Korean characters not included in this standard are decomposed into consonants and vowels, and the Korean alphabet code value of the special character area corresponding to each part is used for information exchange. To differentiate it from information exchange for individual consonant and vowel characters, one "Hangul Filler" is added in front of each syllable.
The main issue is whether to allow an independent "Hangul Filler" that is not followed by valid consonants and vowels to compose a single character. Currently, the EUC-KR encoder does not validate strings containing "Hangul Filler" and just convert "Hangul Filler" of KS X 1001 to the corresponding Unicode "Hangul Filler".
I think there are three possible solutions to this issue, which need to be discussed:
- Not fix anything: allow encoding to contain independent "Hangul Filler" characters, while decoding will treat an sequence starting with "Hangul Filler" as an error if it does not represent a valid combination of Korean characters.
- Fix the encoder: strings that contain independent "Hangul Filler" will be treated as an error (this may cause new errors in the existing codes).
- Fix the decoder: a valid sequence will be converted into one composed character, and in other cases, the code of each character will be simply converted into the corresponding code in Unicode.
I have tried the third method and confirmed that no errors have occurred.
For more information, please refer to the KS X 1001 specification and a detailed description of "Hangul Filler" character. corona10 commented on Mar 2, 2023 I prefer supporting the round-trip approach since we already support encoding.
But I would need to listen to the advice from the other core devs :)
cc @ezio-melotti @vstinner corona10 commented on Mar 2, 2023 @b8choi
I have tried the third method and confirmed that no errors have occurred.
Would you like to submit the patch anyway? We may need to analyze the side effect when we support the decoding for the character.
But please submit the patch with a nice test code. corona10 commented on Mar 14, 2023 Since the decoding implementation for this case is undefined and difficult to support decoding for this edge case due to streaming issues.
I close this issue as not a bug.
If someone think that this is a bug please let me know. TakWolf commented on Mar 14, 2023 So far, I still haven't got an explanation that chr(0x3164).encode('ksx1001').decode('ksx1001') will get a exception is or not a expected behavior (Why python do like this? What is the document or implements reference?).
At least in my understanding, the encoder should be idempotent (That if some char can be encoded, it should be decoded).
For example there are some control characters. They have special purposes in the display layer, but this does not affect that they can be encoded and decoded.
I also deal with other encoding standards ( gb2312, shitf-jis, big5), that have not the similar behaviors (They are all idempotent).
And this problem does not exist in Java. (I will test later in Rust , Go and node-Javascript)
EUC was used to solve computer coding in the early stage, if they can't convert with unicode(utf-8) equivalent, it seems incredible to me. TakWolf commented on Mar 14, 2023 My opinion, encoding is a storage structure, data should not be changed through coders but keep it as it is.
So for b8choi's proposal 2 and 3, they are not suitable.
If the char is in the scope, coder should not throw a exception. And coder should not changed the data.
(Imagine that you just browse the ksx file in a utf8 system, the data is changed, this is not reasonable)
Ksx can be fully mapped to unicode, convert the hangul is meaningless. There is also a Hangul Filler in Unicode. Should we convert all the Hangul Filler combin chars to hangul in utf-8 data layer?
An evidence that may support my opinion, there is a Emoji Combinations in Unicode.
For example: 👩❤️💋👨 = 👩+❤️+💋+👨
This just a display, on data they are equal.
Hangul Filler ’s role should be similar to Ideographic Description Characters only effect on display layer.
If the text engine support Hangul Filler, it will display hangul only on display. If not display as char is. pganssle commented on Oct 22, 2026 Following up on my discussion with @corona10 at the core dev sprint, I've taken a look at how other languages handle this by comparing how they render the filler character by itself and a full valid 8-byte sequence (A4 D4 A4 B6 A4 D0 A4 D4)
| Implementation |
Decode A4 D4 |
Encode U+3164 |
Decode the eight-byte sequence below |
Java, OpenJDK 21.0.12.1, EUC-KR |
U+3164 |
A4 D4 |
ㅤㅆㅠㅤ |
Ruby 3.4.10, EUC-KR |
U+3164 |
A4 D4 |
ㅤㅆㅠㅤ |
Rust, encoding_rs 0.8.28, EUC_KR |
U+3164 |
A4 D4 |
ㅤㅆㅠㅤ |
JavaScript, Node 26.11.1, TextDecoder("euc-kr") |
U+3164 |
Not exposed by TextEncoder |
ㅤㅆㅠㅤ |
C, glibc 2.44, iconv, EUC-KR |
U+3164 |
A4 D4 |
ㅤㅆㅠㅤ |
ICU 78.3, uconv, EUC-KR |
U+3164 |
A4 D4 |
ㅤㅆㅠㅤ |
Go, golang.org/x/text/encoding/korean.EUCKR (source inspection) |
U+3164 |
A4 D4 |
ㅤㅆㅠㅤ |
CPython, euc_kr |
Incomplete-sequence error |
A4 D4 |
쓔 |
Looks like they all accept A4 D4 as U+3164, though none of them implements Python's eight-byte Hangul composition behavior, which actually seems more correct than the four separate character behavior, which makes me think that complete compatibility is not something we should aim for.
The executed implementations also accepted A4 D4 41 as U+3164 followed by ASCII A, and A4 D4 A4 B6 as U+3164 followed by U+3146. They treat each two-byte pair independently, even when the input would be a prefix of a composition sequence in Python. The Java probes used CharsetDecoder and CharsetEncoder with their default error-reporting behavior, the Rust decoder used its no-replacement API, and the JavaScript decoder used fatal: true.
I think this supports accepting a standalone filler, but preserving Python's existing handling of complete composition sequences would still differ from every other implementation in this survey. There is also an ambiguity that a fallback for incomplete sequences cannot resolve: the four-character string U+3164 U+3146 U+3160 U+3164 encodes to exactly the same bytes as U+C4D4. Accepting incomplete prefixes as literal characters would fix the reported example, but cannot restore all string round trips while retaining both the existing encoder mappings and composition behavior.
I don't really know how these things are used in practice, so maybe the fact that we are very weird in doing this lossy one-way composition translation thing is actually very annoying to people and we should actually get rid of it.
I'm inclined to say that we should start with allowing standalone filler characters as a pareto improvement (makes us both more like other languages and the behavior less surprising), but I don't have strong opinions about the trade-offs involved in stopping the composition behavior.
MetadataMetadatainterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-unicodetype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DevelopmentNo branches or pull requestsIssue actions
|
