GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-111495: Add tests for `PyCodec_*` C API by picnixz · Pull Request #123343 · python/cpython

Conversation

@picnixz

@picnixz picnixz commented Aug 26, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

By adding a test, I found a bug in PyUnicodeEncodeError_GetStart where constructing an UnicodeEncodeError with start = 0 and an empty string object raises an assertion error. While this should not happen at runtime, this is something that should still be avoided.

Test coverage is currently insufficient I think but I want an initial feedback first.

EDIT 1: Tests that are known to crash due to UnicodeError's start handling are disabled. I'd like to merge this one even though UnicodeError crashes (I removed the temporary fix from this PR).

  • Issue: Add more C API tests #111495

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
picnixz added 2 commits August 26, 2024 17:42
@picnixz
add tests for C API codecs
ce7c135
@picnixz
add Python tests for _codecs
f9e350a
@picnixz
picnixz requested a review from iritkatriel as a code owner August 26, 2024 15:44
@bedevere-app bedevere-app Bot mentioned this pull request Aug 26, 2024
Add more C API tests #111495
Closed
10 tasks done
@bedevere-app bedevere-app Bot added the awaiting review label Aug 26, 2024
@picnixz picnixz added skip news awaiting review and removed awaiting review labels Aug 26, 2024
@skirpichev

skirpichev commented Aug 27, 2024

Copy link
Copy Markdown
Member

I found a bug in PyUnicodeEncodeError_GetStart where constructing an UnicodeEncodeError with start = 0 and an empty string object raises an assertion error.

I think you should open an issue and create a separate pr with bugfix.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

picnixz commented Aug 27, 2024

Copy link
Copy Markdown
Member Author

I think you should open an issue and create a separate pr with bugfix.

Good idea. I separated the commit so that it can easily be cherry-picked (hehe).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

picnixz added 2 commits August 27, 2024 09:01
@picnixz
fix size bug
15b6811
@picnixz
rename test class
8048ae1
@picnixz
picnixz force-pushed the test/c-api-codec-111495 branch from 41c01a6 to 8048ae1 Compare August 27, 2024 07:02
@picnixz picnixz mentioned this pull request Aug 27, 2024
Incorrect handling of negative start values on PyUnicodeErrorObject #123378
Closed
picnixz added 3 commits September 25, 2024 15:04
@picnixz
Revert "fix size bug"
8487b46
This reverts commit 15b6811.
@picnixz
Merge branch 'main' into test/c-api-codec-111495
2dbe09a
@picnixz
Disable tests that are known to crash.
0097f2a
@picnixz
picnixz requested a review from vstinner September 25, 2024 14:39
vstinner
vstinner reviewed Sep 25, 2024
View reviewed changes
Comment thread Lib/test/test_capi/test_codecs.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_capi/test_codecs.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_capi/test_codecs.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_capi/test_codecs.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_capi/test_codecs.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_capi/test_codecs.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_capi/test_codecs.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_capi/test_codecs.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Modules/_testcapi/codec.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

picnixz added 2 commits September 25, 2024 17:24
@picnixz
address Victor's review
303b13c
@picnixz
update tests to reflect user errors
4f474dd
vstinner
vstinner reviewed Sep 26, 2024
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A test does crash if run more than once.

$ ./python -m test -m test_codec_error_handlers test_capi.test_codecs  
0:00:00 load avg: 2.46 [1/2] test_capi.test_codecs
0:00:00 load avg: 2.46 [2/2] test_capi.test_codecs

Fatal Python error: Segmentation fault

Current thread 0x00007f8ceb3a3740 (most recent call first):
  File "/home/vstinner/python/main/Lib/test/test_capi/test_codecs.py", line 800 in test_codec_error_handlers
  File "/home/vstinner/python/main/Lib/unittest/case.py", line 606 in _callTestMethod
  (...)

I was trying to check if there are refleaks.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@picnixz

picnixz commented Sep 26, 2024

Copy link
Copy Markdown
Member Author

So we need a way to unregister the codecs. Otherwise I don't know how to do it... (would reloading the module work? or do you want an API for explicitly unregistering an error handler?)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Sep 26, 2024

Copy link
Copy Markdown
Member

an API for explicitly unregistering an error handler?

That would be nice, yes. It should be a separated PR if you want to work on that.

I'm not sure if it's strictly needed right now if your test creates/loads a separated extension just for C tests. But the test should not leak :-)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

picnixz commented Sep 26, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

That would be nice, yes. It should be a separated PR if you want to work on that.

I'll work on that tomorrow (I wanted to work today but I have no energy at all).

I'm not sure if it's strictly needed right now if your test creates/loads a separated extension just for C tests

Before working on the above feature, I'll try to see how I can make it work without it. (at least we should be able to check if it leaks...).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz

picnixz commented Sep 27, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

So, the issue is with _testcapi.codec_namereplace_errors and the setUp(). I'll try to see what happens out there.

So something is wrong when I add those lines and uses -R ::

self.enterContext(import_helper.isolated_modules())
self.enterContext(import_helper.CleanImport('codecs'))

(Still searching where it leaks...)

So I managed to fix the leaks. The crash seems to be because of this line in PyCodec_NameReplaceErrors:

if (ucnhash_capi->getname(c, buffer, sizeof(buffer), 1))

I have no idea why no using fresh imports seem fine. However, the getname pointer is different just before it crashes.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz
Merge remote-tracking branch 'upstream/main' into test/c-api-codec-11…
d49743c
…1495
@vstinner

vstinner commented Sep 27, 2024

Copy link
Copy Markdown
Member

Why do you need a fresh import?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz
fix C API codec tests
87ee0d2
@picnixz

picnixz commented Sep 27, 2024

Copy link
Copy Markdown
Member Author

I don't need them anymore so I can remove them normally. I'll now work on the PR for removing a registered error handler so that we can run the test suite. For now, the PR is not ready since there will be a leak due to the error handler being held.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@picnixz
small hack to make the test suite correct
6a36eb0
@picnixz picnixz mentioned this pull request Sep 27, 2024
[C API] Add private _PyCodec_UnregisterError to un-register custom error handlers #124665
Closed
picnixz added 3 commits September 28, 2024 18:29
@picnixz
remove un-necessary imports
145b285
@picnixz
Merge remote-tracking branch 'upstream/main' into test/c-api-codec-11…
dc9af16
…1495
@picnixz
use _codecs._unregister_error to cleanup test state
7be1f55
@picnixz
picnixz force-pushed the test/c-api-codec-111495 branch from 850a441 to 7be1f55 Compare September 29, 2024 08:00
picnixz added 4 commits September 29, 2024 10:08
@picnixz
indicate some semantics for NULL case being tested
f72be5c
@picnixz
revert a cosmetic change
4d02c6c
@picnixz
Move PyCodec_NameReplaceErrors test to the _testlimitedcapi module
0f26ca7
@picnixz
add comment for why we do not test _PyCodec_UnregisterError
1399779
@picnixz
picnixz requested a review from erlend-aasland as a code owner September 29, 2024 08:33
picnixz added 3 commits September 29, 2024 10:34
@picnixz
update a comment
914151e
@picnixz
revert one cosmetic change
8dd7e8d
@picnixz
Fix Windows compilation
1e6a5ce
@picnixz
picnixz requested a review from a team as a code owner September 29, 2024 08:52
@picnixz
picnixz requested review from vstinner and removed request for a team and erlend-aasland September 29, 2024 09:01
vstinner
vstinner reviewed Sep 29, 2024
View reviewed changes
Comment thread Lib/test/test_capi/test_codecs.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_capi/test_codecs.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

@picnixz
address Victor's review
2ba5f03
vstinner
vstinner approved these changes Sep 29, 2024
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Sep 29, 2024
@vstinner
vstinner enabled auto-merge (squash) September 29, 2024 15:00
@vstinner
vstinner merged commit 6d0d26e into python:main Sep 29, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Sep 29, 2024
@picnixz
picnixz deleted the test/c-api-codec-111495 branch September 29, 2024 15:22
@picnixz

picnixz commented Sep 29, 2024

Copy link
Copy Markdown
Member Author

Mmh, apparently tests were backported for the other modules, but this one relies on the internal private feature of being able to unregister error handlers. I don't really want to add it 3.12 and 3.13 so I think we can keep these tests for 3.14+.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Sep 29, 2024

Copy link
Copy Markdown
Member

Yeah, it makes sense to not backport these tests.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Sep 29, 2024

Copy link
Copy Markdown
Member

Merged, thanks for this nice enhancement!

picnixz reacted with heart emoji
All reactions
  • ❤️ 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner approved these changes
@iritkatriel iritkatriel Awaiting requested review from iritkatriel

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

3 participants

@picnixz @skirpichev @vstinner
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.