GeezerVison is showing github.com — codecs.StreamReader doesn't pass final=1 to the UTF-8 codec · Issue #56717 · python/cpython
codecs.StreamReader doesn't pass final=1 to the UTF-8 codec #56717Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2011-07-06.22:45:47.044>
labels = ['3.10', 'type-bug', '3.8', '3.9', 'expert-unicode']
title = "codecs.StreamReader doesn't pass final=1 to the UTF-8 codec"
updated_at = <Date 2020-11-11.19:26:15.934>
user = 'https://bugs.python.org/spatz123' bugs.python.org fields:
activity = <Date 2020-11-11.19:26:15.934>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2011-07-06.22:45:47.044>
creator = 'spatz123'
dependencies = []
files = ['22597', '26013']
hgrepos = []
issue_num = 12508
keywords = []
message_count = 6.0
messages = ['139954', '139956', '139957', '143473', '144373', '144376']
nosy_count = 4.0
nosy_names = ['lemburg', 'vstinner', 'ezio.melotti', 'spatz123']
pr_nums = []
priority = 'high'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue12508'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10'] Activityspatz123 commented on Jul 6, 2011 MannequinAuthor More actions The attached script produces the output
'A\ufffdBC\ufffd'
'A\ufffdBC'
although it seems to me that both lines should be the same. The first line is correct, I think, since the <F4> at the end is a maximal subpart of an ill-formed subsequence, according to the definition in the Unicode standard. vstinner commented on Jul 6, 2011 I confirm, there is a bug in codecs.StreamReader. vstinner commented on Jul 6, 2011 You should use the io module, it doesn't have the bug :) ezio-melotti commented on Sep 4, 2011 IIUC this happens because StreamReader calls codecs.utf_8_decode without passing final=1 [0], so when the decoder finds the trailing F4 it doesn't decode it yet because it waits from the other 3 bytes (F4 is the start byte of a 4-bytes UTF-8 sequence):
>>> b = b'A\xf5BC\xf4'
>>> chars, decnum = codecs.utf_8_decode(b, 'replace', 0) # final=0
>>> chars, decnum
('A�BC', 4) # F4 not decoded yet
>>> b = b[decnum:]
>>> b
b'\xf4' # F4 still here
>>> chars, decnum = codecs.utf_8_decode(b, 'replace', 0)
>>> chars, decnum
('', 0) # additional calls keep waiting for the other 3 bytes
>>> chars, decnum = codecs.utf_8_decode(b, 'replace', 1) # final=1
>>> chars, decnum
('�', 1) # when final=1 is passed F4 is decoded, but it never happensWhile passing 1 makes the attached script work as expected, it breaks several other test in test_codecs (apparently not all the decoders accept the 'final' argument).
Also passing 1 should be done only for the last call: read can be called several times with a specific size, and it shouldn't use final=1 until the last call to avoid errors mid-stream.
[0]: see Lib/codecs.py:482 malemburg commented on Sep 21, 2011 The final parameter is an extension to the decoder API signature,
so it's not surprising that not all codecs implement it.
The ones that do should use it for all calls, since that way
the actual consumed number of bytes is correctly reported
back to the StreamReader instance.
Note: The parameter name "final" is a bit misleading. What happens
is that the number of bytes consumed by the decoder were previously
always reported as len(buffer), since the C API for decoders did
not provide a way to report back the number of bytes consumed.
This was changed when stateful decoders were added to the C API,
since these do allow reporting back the consumed bytes. A more
appropriate name for the parameter would have been
"report_bytes_consumed". ezio-melotti commented on Sep 21, 2011 AFAIU final means:
- final=0: I'm passing in a few bytes, but there are more to come, so if the last byte(s) doesn't make sense on its own (e.g. it's a start byte but the continuation bytes are missing), wait for the others before raising an error;
- final=1: these are the last bytes, so if the last byte(s) doesn't make sense raise an error (or ignore/replace) because there won't be other bytes that might turn that in a well-formed byte sequence.
changed the title [-]Codecs Anomaly[/-] [+]codecs.StreamReader doesn't pass final=1 to the UTF-8 codec[/+] on Nov 11, 2020
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|