GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-99593: Add tests for Unicode C API (part 2) by serhiy-storchaka · Pull Request #99868 · python/cpython

Conversation

@serhiy-storchaka

@serhiy-storchaka serhiy-storchaka commented Nov 29, 2022 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Add tests for lower-level functions.

  • Issue: Add tests for Unicode C API #99593

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@serhiy-storchaka
pythongh-99593: Add tests for Unicode C API (part 2)
f8754d7
Add tests for lower-level functions.
@bedevere-bot bedevere-bot mentioned this pull request Nov 29, 2022
Add tests for Unicode C API #99593
Closed
@bedevere-bot bedevere-bot added the awaiting core review label Nov 29, 2022
@serhiy-storchaka serhiy-storchaka added needs backport to 3.10 needs backport to 3.11 only security fixes labels Nov 29, 2022
This was referenced Nov 29, 2022
gh-93649: Split unicode tests from _testcapimodule.c & add some more #95819
Merged
gh-99593: Add tests for Unicode C API #99594
Closed
@serhiy-storchaka serhiy-storchaka added the skip news label Nov 29, 2022
@serhiy-storchaka
serhiy-storchaka marked this pull request as ready for review November 30, 2022 17:19
@serhiy-storchaka
serhiy-storchaka requested a review from vstinner November 30, 2022 17:20
vstinner
vstinner reviewed Dec 1, 2022
View reviewed changes
Comment thread Lib/test/test_capi/test_unicode.py Outdated
class CAPITest(unittest.TestCase):
# TODO: Test the following function:
#
# PyUnicode_ClearFreeList

@vstinner vstinner Dec 1, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyUnicode_ClearFreeList was removed in Python 3.9!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Dec 3, 2022

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, these tests were written when it was here.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/test_capi/test_unicode.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_capi/test_unicode.py
self.assertEqual(new(0, maxchar), '')
self.assertEqual(new(5, maxchar), chr(maxchar)*5)
self.assertEqual(new(0, 0x110000), '')
self.assertRaises(SystemError, new, 5, 0x110000)

@vstinner vstinner Dec 1, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this error should become a ValueError, but it can be changed outside this PR, since you seem to want to backport these news tests.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Dec 3, 2022

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that SystemError is better exception type here. It is a misuse of the C API, you cannot get this error from Python code.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Modules/_testcapi/unicode.c
if (!result) {
return NULL;
}
if (size > 0 && maxchar <= 0x10ffff &&

@vstinner vstinner Dec 1, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It sounds dangerous to return a string to the "Python space" if characters are not initialized when maxchar is greated than 0x10ffff. Can you remove maxchar <= 0x10ffff condition? PyUnicode_Fill() must fail if the fill character is too big (greater than 0x10ffff).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Dec 3, 2022

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a test for PyUnicode_New(), not for PyUnicode_Fill(). We should see exceptions raised by PyUnicode_New(), not PyUnicode_Fill().

It never happens, because PyUnicode_New() returns NULL if size > 0 and maxchar > 0x10ffff. If it will not return NULL, it is better to get a malformed string than get an exception raised by PyUnicode_Fill() and think that it was raised by PyUnicode_New().

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Modules/_testcapi/unicode.c
}

result = PyUnicode_WriteChar(to_copy, index, (Py_UCS4)character);
if (result == -1 && PyErr_Occurred()) {

@vstinner vstinner Dec 1, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me, -1 means an error. You shouldn't have to check if an exception was raised or not. Or it should become: assert(PyErr_Occurred()).

Same remark for other test wrapper like the one for PyUnicode_Resize().

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Dec 3, 2022

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If an exception was raised, but result is not -1, then what? The code will return NULL, and we will never know that something wrong happened with PyUnicode_WriteChar().

I do not use the C assert() in these tests, because I want to make tests working even in non-debug build.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Dec 5, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not use the C assert() in these tests, because I want to make tests working even in non-debug build.

C assert() are enabled on release (no-debug) builds: see the following code in Modules/_testcapi/parts.h:

// Always enable assertions
#undef NDEBUG

Many _testcapi tests are only implemented with assert().

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Modules/_testcapi/unicode.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/_testcapi/unicode.c Outdated

if (!PyUnicode_AsUCS4(unicode, buffer, buf_len, copy_null)) {
PyMem_Free(buffer);
PyMem_FREE(buffer);

@vstinner vstinner Dec 1, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyMem_FREE is a deprecated alias to PyMem_Free(). I would expect the opposite change, replace PyMem_FREE with PyMem_Free :-)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Dec 3, 2022

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted. The code in my branch was written a long time ago, when PyMem_FREE and PyMem_Free were different things. I just copied it over the current code.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/test_capi/test_unicode.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_capi/test_unicode.py
self.assertRaises(SystemError, append, 'abc', NULL)
# TODO: Test PyUnicode_Append() with modifiable unicode
# and with NULL as the address.
# TODO: Check reference counts.

@vstinner vstinner Dec 1, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's done automatically by Refleaks buildbots, no?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Dec 3, 2022

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe. But since this C API does unusual things with reference counts, it would be better to test it explicitly, if possible.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Lib/test/test_capi/test_unicode.py Outdated
if SIZEOF_WCHAR_T == 2:
if sys.byteorder == 'little':
encoding = 'utf-16le'
elif sys.byteorder == 'little':

@vstinner vstinner Dec 1, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This condition looks wrong. Maybe just use else:? I don't think that Python supports other endianness. Same a few lines below.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Dec 3, 2022

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. I though that explicit check can be better, but made an error in it.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@serhiy-storchaka
Address review comments.
1c1775c
@serhiy-storchaka

serhiy-storchaka commented Dec 3, 2022

Copy link
Copy Markdown
Member Author

Thank you for your review Victor.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka
Merge branch 'main' into test-unicode-capi-part2
4a60537
vstinner
vstinner approved these changes Dec 5, 2022
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for my addressing my review.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Modules/_testcapi/unicode.c
}

result = PyUnicode_WriteChar(to_copy, index, (Py_UCS4)character);
if (result == -1 && PyErr_Occurred()) {

@vstinner vstinner Dec 5, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not use the C assert() in these tests, because I want to make tests working even in non-debug build.

C assert() are enabled on release (no-debug) builds: see the following code in Modules/_testcapi/parts.h:

// Always enable assertions
#undef NDEBUG

Many _testcapi tests are only implemented with assert().

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels Dec 5, 2022
@hugovk hugovk removed the needs backport to 3.10 label Apr 7, 2023
@hugovk

hugovk commented Apr 7, 2023

Copy link
Copy Markdown
Member

@serhiy-storchaka Victor approved this, ready to merge?

(I removed the 3.10 backport label, it's now security only)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@arhadthedev
Merge branch 'main' into test-unicode-capi-part2
41e8a15
@vstinner
vstinner enabled auto-merge (squash) May 4, 2023 14:59
vstinner
vstinner approved these changes May 4, 2023
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@vstinner
Merge branch 'main' into test-unicode-capi-part2
36970c7
@vstinner

vstinner commented May 4, 2023

Copy link
Copy Markdown
Member

The CI test "DO-NOT-MERGE / unresolved review" was blocked on "Waiting for status to be reported". I tried to update the PR on main to see if it does unblock the PR.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
vstinner merged commit 2ba931f into python:main May 4, 2023
@miss-islington

miss-islington commented May 4, 2023

Copy link
Copy Markdown
Contributor

Thanks @serhiy-storchaka for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bedevere-bot bedevere-bot removed the awaiting merge label May 4, 2023
@miss-islington

miss-islington commented May 4, 2023

Copy link
Copy Markdown
Contributor

Sorry, @serhiy-storchaka and @vstinner, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 2ba931ff727395cf89b290ed313a8e15db0bfcf1 3.11

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington miss-islington assigned vstinner May 4, 2023
@vstinner vstinner removed the needs backport to 3.11 only security fixes label May 4, 2023
@vstinner

vstinner commented May 4, 2023

Copy link
Copy Markdown
Member

Merged, thanks @serhiy-storchaka.

@serhiy-storchaka: If you consider that this change should be backported to Python 3.11, go ahead. But the automated backport failed for an unknown reason.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

carljm added a commit to carljm/cpython that referenced this pull request May 5, 2023
@carljm
Merge branch 'main' into inlinecomp2
8a78a36
* main: (61 commits)
  pythongh-64595: Argument Clinic: Touch source file if any output file changed (python#104152)
  pythongh-64631: Test exception messages in cloned Argument Clinic funcs (python#104167)
  pythongh-68395: Avoid naming conflicts by mangling variable names in Argument Clinic (python#104065)
  pythongh-64658: Expand Argument Clinic return converter docs (python#104175)
  pythonGH-103092: port `_asyncio` freelist to module state (python#104196)
  pythongh-104051: fix crash in test_xxtestfuzz with -We (python#104052)
  pythongh-104190: fix ubsan crash (python#104191)
  pythongh-104106: Add gcc fallback of mkfifoat/mknodat for macOS (pythongh-104129)
  pythonGH-104142: Fix _Py_RefcntAdd to respect immortality (pythonGH-104143)
  pythongh-104112: link from cached_property docs to method-caching FAQ (python#104113)
  pythongh-68968: Correcting message display issue with assertEqual (python#103937)
  pythonGH-103899: Provide a hint when accidentally calling a module (pythonGH-103900)
  pythongh-103963: fix 'make regen-opcode' in out-of-tree builds (python#104177)
  pythongh-102500: Add PEP 688 and 698 to the 3.12 release highlights (python#104174)
  pythonGH-81079: Add case_sensitive argument to `pathlib.Path.glob()` (pythonGH-102710)
  pythongh-91896: Deprecate collections.abc.ByteString (python#102096)
  pythongh-99593: Add tests for Unicode C API (part 2) (python#99868)
  pythongh-102500: Document PEP 688 (python#102571)
  pythongh-102500: Implement PEP 688 (python#102521)
  pythongh-96534: socketmodule: support FreeBSD divert(4) socket (python#96536)
  ...
@serhiy-storchaka serhiy-storchaka unassigned vstinner Jul 10, 2023
@serhiy-storchaka serhiy-storchaka added the needs backport to 3.12 only security fixes label Jul 10, 2023
@miss-islington

miss-islington commented Jul 10, 2023

Copy link
Copy Markdown
Contributor

Thanks @serhiy-storchaka for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington

miss-islington commented Jul 10, 2023

Copy link
Copy Markdown
Contributor

Sorry, @serhiy-storchaka and @vstinner, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 2ba931ff727395cf89b290ed313a8e15db0bfcf1 3.12

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@miss-islington miss-islington assigned vstinner Jul 10, 2023
@serhiy-storchaka serhiy-storchaka removed the needs backport to 3.12 only security fixes label Jul 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@vstinner vstinner

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@serhiy-storchaka @hugovk @vstinner @miss-islington @bedevere-bot @arhadthedev
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.