Bug description

With errors='surrogateescape', the incremental decoders of the CJK codecs that have sequences longer than two bytes (euc_jp, euc_jis_2004, euc_jisx0213, gb18030) lose data when the input ends in an incomplete sequence: the first byte is escaped, and whatever followed it is gone. The stateless decoder escapes the byte and keeps going:

>>> import codecs
>>> b'\x8f\x00'.decode('euc_jp', 'surrogateescape')
'\udc8f\x00'
>>> codecs.getincrementaldecoder('euc_jp')('surrogateescape').decode(b'\x8f\x00', final=True)
'\udc8f'
>>> b'\x806'.decode('gb18030', 'surrogateescape')
'\udc806'
>>> codecs.getincrementaldecoder('gb18030')('surrogateescape').decode(b'\x806', final=True)
'\udc80'

The 6 and the \x00 are plain ASCII; they are simply never looked at again. TextIOWrapper uses the incremental decoder, so this is reachable through ordinary file reading:

>>> import io
>>> io.TextIOWrapper(io.BytesIO(b'\xa8\x8fl'), encoding='euc_jp', errors='surrogateescape').read()
'\udca8\udc8f'

The other handlers do not lose anything because they consume the whole trailing range in one go (replace gives one � for it, backslashreplace escapes every byte). surrogateescape escapes a single byte and returns a resume position before the end of the input, and the final-flush path in _multibytecodec does not go back to the decoder from there: after the one multibytecodec_decerror(..., MBERR_TOOFEW) call it finishes up with whatever has been written so far, Modules/cjkcodecs/multibytecodec.c lines 1211-1218. The stateless entry point in the same file loops until the input is consumed, which is why it gets this right. That is from reading the code rather than tracing it, so treat the mechanism as a best guess. It also explains why the two-byte codecs (big5, euc_kr, gbk, shift_jis, ...) are not affected: with one pending byte there is nothing left to drop after escaping it.

Found by the Hypothesis tests in #160155, where it is pinned as MultibyteSurrogateescapeBug.test_trailing_bytes_after_escape under expectedFailure; the chunked-decoding tests there leave surrogateescape out for these four codecs until it is fixed.

CPython versions tested on

CPython main (6e1bd24)

Operating systems tested on

Linux

Linked PRs

• gh-160307: Complete final multibyte incremental decoding #160629