GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-126004: fix positions handling in `codecs.backslashreplace_errors` by picnixz · Pull Request #127676 · python/cpython

gh-126004: fix positions handling in codecs.backslashreplace_errors #127676

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
picnixz merged 15 commits into python:main from picnixz:fix/codecs/backslashreplace-errors-126004
Jan 23, 2025

Uh oh!

There was an error while loading. Please reload this page.

Merged

gh-126004: fix positions handling in codecs.backslashreplace_errors #127676

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
c85e59c
fix `codecs.backslashreplace_errors`
picnixz Dec 6, 2024
cfc4c9b
blurb
picnixz Dec 6, 2024
2649ab7
fix warnings
picnixz Dec 6, 2024
9558928
cosmetic changes
picnixz Dec 6, 2024
11a9431
Merge remote-tracking branch 'upstream/main' into fix/codecs/backslas…
picnixz Jan 3, 2025
08be0a7
use internal `_PyUnicodeError_GetParams` helper
picnixz Jan 3, 2025
564ba70
Update Python/codecs.c
picnixz Jan 3, 2025
450af5d
Merge branch 'main' into fix/codecs/backslashreplace-errors-126004
picnixz Jan 14, 2025
e3ebbde
Merge branch 'main' into fix/codecs/backslashreplace-errors-126004
picnixz Jan 21, 2025
da54d9b
update usages of `_PyUnicodeError_GetParams`
picnixz Jan 21, 2025
98b4ec0
amend some cosmetic changes to be consistent
picnixz Jan 21, 2025
a19b4e3
fix bounds
picnixz Jan 21, 2025
b585bf0
Merge branch 'main' into fix/codecs/backslashreplace-errors-126004
picnixz Jan 23, 2025
3eff7b8
leave optimization to the compiler
picnixz Jan 23, 2025
5e5ecbc
remove magic constants
picnixz Jan 23, 2025
File filter
Filter by extension

Clear filters
Conversations
Failed to load comments.
Loading

Uh oh!

There was an error while loading. Please reload this page.

Jump to
Jump to file
Failed to load files.
Loading

Uh oh!

There was an error while loading. Please reload this page.

Diff view
Diff view

There are no files selected for viewing

3 changes: 2 additions & 1 deletion 3 Lib/test/test_capi/test_codecs.py
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -849,7 +849,8 @@ def test_codec_xmlcharrefreplace_errors_handler(self):

def test_codec_backslashreplace_errors_handler(self):
handler = _testcapi.codec_backslashreplace_errors
self.do_test_codec_errors_handler(handler, self.all_unicode_errors)
self.do_test_codec_errors_handler(handler, self.all_unicode_errors,
safe=True)

def test_codec_namereplace_errors_handler(self):
handler = _testlimitedcapi.codec_namereplace_errors
Expand Down
3 changes: 3 additions & 0 deletions 3 Misc/NEWS.d/next/Core_and_Builtins/2024-12-06-11-30-58.gh-issue-126004.-p8MAS.rst
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Fix handling of :attr:`UnicodeError.start` and :attr:`UnicodeError.end`
values in the :func:`codecs.backslashreplace_errors` error handler. Patch by
Bénédikt Tran.
127 changes: 62 additions & 65 deletions 127 Python/codecs.c
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -864,108 +864,105 @@ PyObject *PyCodec_XMLCharRefReplaceErrors(PyObject *exc)

PyObject *PyCodec_BackslashReplaceErrors(PyObject *exc)
{
PyObject *object;
Py_ssize_t i;
Py_ssize_t start;
Py_ssize_t end;
PyObject *res;
Py_UCS1 *outp;
int ressize;
Py_UCS4 c;

PyObject *obj;
Py_ssize_t objlen, start, end, slen;
if (PyObject_TypeCheck(exc, (PyTypeObject *)PyExc_UnicodeDecodeError)) {
const unsigned char *p;
if (PyUnicodeDecodeError_GetStart(exc, &start))
return NULL;
if (PyUnicodeDecodeError_GetEnd(exc, &end))
return NULL;
if (!(object = PyUnicodeDecodeError_GetObject(exc)))
if (_PyUnicodeError_GetParams(exc,
&obj, &objlen,
&start, &end, &slen, true) < 0)
{
return NULL;
p = (const unsigned char*)PyBytes_AS_STRING(object);
res = PyUnicode_New(4 * (end - start), 127);
}
PyObject *res = PyUnicode_New(4 * slen, 127);
if (res == NULL) {
Py_DECREF(object);
Py_DECREF(obj);
return NULL;
}
outp = PyUnicode_1BYTE_DATA(res);
for (i = start; i < end; i++, outp += 4) {
unsigned char c = p[i];
Py_UCS1 *outp = PyUnicode_1BYTE_DATA(res);
const unsigned char *p = (const unsigned char *)PyBytes_AS_STRING(obj);
for (Py_ssize_t i = start; i < end; i++, outp += 4) {
const unsigned char ch = p[i];
outp[0] = '\\';
outp[1] = 'x';
outp[2] = Py_hexdigits[(c>>4)&0xf];
outp[3] = Py_hexdigits[c&0xf];
outp[2] = Py_hexdigits[(ch >> 4) & 0xf];
outp[3] = Py_hexdigits[ch & 0xf];
}

assert(_PyUnicode_CheckConsistency(res, 1));
Py_DECREF(object);
Py_DECREF(obj);
return Py_BuildValue("(Nn)", res, end);
}
if (PyObject_TypeCheck(exc, (PyTypeObject *)PyExc_UnicodeEncodeError)) {
if (PyUnicodeEncodeError_GetStart(exc, &start))
return NULL;
if (PyUnicodeEncodeError_GetEnd(exc, &end))
return NULL;
if (!(object = PyUnicodeEncodeError_GetObject(exc)))
return NULL;
}
else if (PyObject_TypeCheck(exc, (PyTypeObject *)PyExc_UnicodeTranslateError)) {
if (PyUnicodeTranslateError_GetStart(exc, &start))
return NULL;
if (PyUnicodeTranslateError_GetEnd(exc, &end))
return NULL;
if (!(object = PyUnicodeTranslateError_GetObject(exc)))

if (
PyObject_TypeCheck(exc, (PyTypeObject *)PyExc_UnicodeEncodeError)
|| PyObject_TypeCheck(exc, (PyTypeObject *)PyExc_UnicodeTranslateError)
) {
if (_PyUnicodeError_GetParams(exc,
&obj, &objlen,
&start, &end, &slen, false) < 0)
{
return NULL;
}
}
else {
wrong_exception_type(exc);
return NULL;
}

if (end - start > PY_SSIZE_T_MAX / (1+1+8))
end = start + PY_SSIZE_T_MAX / (1+1+8);
for (i = start, ressize = 0; i < end; ++i) {
if (slen == 0) { // end <= start
Comment thread
picnixz marked this conversation as resolved.
Resolved

Uh oh!

There was an error while loading. Please reload this page.

Py_DECREF(obj);
return Py_BuildValue("(Nn)", Py_GetConstant(Py_CONSTANT_EMPTY_STR), end);
}
if (slen > PY_SSIZE_T_MAX / 10) {
end = start + PY_SSIZE_T_MAX / 10;
end = Py_MIN(end, objlen);
slen = Py_MAX(0, end - start);
}

Py_ssize_t ressize = 0;
for (Py_ssize_t i = start; i < end; ++i) {
/* object is guaranteed to be "ready" */
c = PyUnicode_READ_CHAR(object, i);
Py_UCS4 c = PyUnicode_READ_CHAR(obj, i);
if (c >= 0x10000) {
ressize += 1+1+8;
ressize += 10;
Comment thread
picnixz marked this conversation as resolved.
Resolved

Uh oh!

There was an error while loading. Please reload this page.

}
else if (c >= 0x100) {
ressize += 1+1+4;
ressize += 6;
}
else {
ressize += 4;
}
else
ressize += 1+1+2;
}
res = PyUnicode_New(ressize, 127);
PyObject *res = PyUnicode_New(ressize, 127);
if (res == NULL) {
Py_DECREF(object);
Py_DECREF(obj);
return NULL;
}
outp = PyUnicode_1BYTE_DATA(res);
for (i = start; i < end; ++i) {
c = PyUnicode_READ_CHAR(object, i);
Py_UCS1 *outp = PyUnicode_1BYTE_DATA(res);
for (Py_ssize_t i = start; i < end; ++i) {
Py_UCS4 c = PyUnicode_READ_CHAR(obj, i);
*outp++ = '\\';
if (c >= 0x00010000) {
*outp++ = 'U';
*outp++ = Py_hexdigits[(c>>28)&0xf];
*outp++ = Py_hexdigits[(c>>24)&0xf];
*outp++ = Py_hexdigits[(c>>20)&0xf];
*outp++ = Py_hexdigits[(c>>16)&0xf];
*outp++ = Py_hexdigits[(c>>12)&0xf];
*outp++ = Py_hexdigits[(c>>8)&0xf];
*outp++ = Py_hexdigits[(c >> 28) & 0xf];
*outp++ = Py_hexdigits[(c >> 24) & 0xf];
*outp++ = Py_hexdigits[(c >> 20) & 0xf];
*outp++ = Py_hexdigits[(c >> 16) & 0xf];
*outp++ = Py_hexdigits[(c >> 12) & 0xf];
*outp++ = Py_hexdigits[(c >> 8) & 0xf];
}
else if (c >= 0x100) {
*outp++ = 'u';
*outp++ = Py_hexdigits[(c>>12)&0xf];
*outp++ = Py_hexdigits[(c>>8)&0xf];
*outp++ = Py_hexdigits[(c >> 12) & 0xf];
*outp++ = Py_hexdigits[(c >> 8) & 0xf];
}
else
else {
*outp++ = 'x';
*outp++ = Py_hexdigits[(c>>4)&0xf];
*outp++ = Py_hexdigits[c&0xf];
}
*outp++ = Py_hexdigits[(c >> 4) & 0xf];
*outp++ = Py_hexdigits[c & 0xf];
}

assert(_PyUnicode_CheckConsistency(res, 1));
Py_DECREF(object);
Py_DECREF(obj);
return Py_BuildValue("(Nn)", res, end);
}

Expand Down
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.