GeezerVison

cc @methane @serhiy-storchaka

What happens with truncated strings, like %.50s, if the are truncated in the middle of multibyte UTF-8 sequence?

What happens with truncated strings, like %.50s, if the are truncated in the middle of multibyte UTF-8 sequence?

There are two tests on that: UnicodeDecodeError is raised in this case.

Example of test: test_capi.test_unicode

        # test "%s" format with precision
        check_format('abc',
                     b'%.3s', b'abcdef')
        with self.assertRaises(UnicodeDecodeError):
            PyUnicode_FromFormat(b'%.5s', 'abc[\u20ac]'.encode('utf8'))
        check_format('abc[\u20ac',
                     b'%.7s', 'abc[\u20ac]'.encode('utf8'))

This is bad. Such formats are common in error formatting code (not only in CPython, but in third-party code), and now you will get a UnicodeDecodeError instead of the original error even if all was fine with encoding. In this case I think that it it is better to truncate the string before the truncated sequence.

But even without truncation, it may be better to get a replacement character in the error message of the correct exception than a UnicodeDecodeError.

On my PR gh-120248, @methane wrote:

I prefer "strict" because "hard to notice" is also hard to debug.

So I created this PR. @methane: What do you think?

I can modify the %.100s format ("%s" with precision) to truncate to 100 characters instead of 100 bytes, to avoid the risk of creating invalid UTF-8 strings.

I think that @methane's comment was only related to the format string (which currently is ASCII-only), not to arguments.

I think 100 codepoints is the best option.

About error handler, there is no correct answer. Theorically speaking, "strict" is "Errors should never pass silently."
But both of "replace" and "backslashreplace" are acceptable.

Precision should specify the length in bytes. This feature can be used to format not-null-teminated strings.

char buffer[100];
PyUnicode_FromFormat("%.100s", buffer);

If you start to count codepoints, you can read past the end of the array.

I abandon this PR. It seems like using "replace" error handler is more appropriate here.