GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — gh-94823: Improve coverage in tokenizer.c:valid_utf8 by mdboom · Pull Request #94856 · python/cpython

Conversation

mdboom commented Jul 14, 2022 •

edited by miss-islington

Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link

Copy Markdown

Contributor

When loading a source file from disk, there is a separate UTF-8 validator
distinct from the one in unicode_decode_utf8. This exercises that code path
with the same set of invalid inputs as we use for testing the "other" UTF-8
decoder.

Issue: [coverage] tokenizer.c: valid_utf8 #94823

Automerge-Triggered-By: GH:ericsnowcurrently

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

bedevere-bot added tests

Tests in the Lib/test dir

awaiting review labels

Jul 14, 2022

kumaraditya303 added the skip news label

Jul 14, 2022

brandtbucher added needs backport to 3.11 only security fixes labels

Jul 15, 2022


          pythongh-94823: Improve coverage in tokenizer.c:valid_utf8

e52f328

When loading a source file from disk, there is a separate UTF-8 validator
distinct from the one in `unicode_decode_utf8`. This exercises that code path
with the same set of invalid inputs as we use for testing the "other" UTF-8
decoder.

mdboom force-pushed the tokenizer-valid-utf8-coverage branch from 6db7d84 to e52f328 Compare

July 15, 2022 18:11

mdboom added 2 commits

July 20, 2022 11:41


          Perform all examples, since we are excluding Windows anyway

61cf58f


          Only run test on Linux

1f65938

mdboom requested a review from ericsnowcurrently

August 12, 2022 18:14

ericsnowcurrently requested changes

Aug 12, 2022

View reviewed changes

ericsnowcurrently left a comment

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this! It mostly looks good. I noticed a couple of small typos.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Comment thread

Lib/test/test_source_encoding.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

Lib/test/test_source_encoding.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

bedevere-bot added awaiting changes and removed awaiting review labels

Aug 12, 2022

bedevere-bot commented Aug 12, 2022

Copy link

Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.


          Fix typos in comment

1f90863

mdboom commented Aug 16, 2022

Copy link

Copy Markdown

Contributor Author

I have made the requested changes; please review again

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bedevere-bot added awaiting change review and removed awaiting changes labels

Aug 16, 2022

bedevere-bot commented Aug 16, 2022

Copy link

Copy Markdown

Thanks for making the requested changes!

@ericsnowcurrently: please review the changes made to this pull request.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bedevere-bot requested a review from ericsnowcurrently

August 16, 2022 14:15

ericsnowcurrently approved these changes

Aug 16, 2022

View reviewed changes

ericsnowcurrently left a comment

Copy link

Copy Markdown

Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

bedevere-bot added awaiting merge and removed awaiting change review labels

Aug 16, 2022

ericsnowcurrently added the 🤖 automerge label

Aug 16, 2022

miss-islington merged commit f215d7c into python:main

Aug 16, 2022

miss-islington commented Aug 16, 2022

Copy link

Copy Markdown

Contributor

Thanks @mdboom for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

bedevere-bot removed the awaiting merge label

Aug 16, 2022

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request

Aug 16, 2022


          pythongh-94823: Improve coverage in tokenizer.c:valid_utf8 (pythonGH-…

0425b23

…94856)

When loading a source file from disk, there is a separate UTF-8 validator
distinct from the one in `unicode_decode_utf8`. This exercises that code path
with the same set of invalid inputs as we use for testing the "other" UTF-8
decoder.
(cherry picked from commit f215d7c)

Co-authored-by: Michael Droettboom <[email protected]>

bedevere-bot removed the needs backport to 3.11 only security fixes label

Aug 16, 2022

bedevere-bot commented Aug 16, 2022

Copy link

Copy Markdown

GH-96029 is a backport of this pull request to the 3.11 branch.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ericsnowcurrently commented Aug 16, 2022

Copy link

Copy Markdown

Member

Thanks for the test!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

pablogsal pushed a commit that referenced this pull request

Aug 16, 2022


          [3.11] gh-94823: Improve coverage in tokenizer.c:valid_utf8 (GH-94856) (

2bb363c

#96029)

Co-authored-by: Michael Droettboom <[email protected]>

This was referenced Aug 22, 2022

test_invalid_utf8 test failing on some buildbots #96189

Closed

Assert and incorrect error message when loading source file containing invalid UTF-8 #96268

Closed

gh-96268: Fix loading invalid UTF-8 #96270

Merged

kumaraditya303 mentioned this pull request

Sep 17, 2022

[coverage] tokenizer.c: valid_utf8 #94823

Closed

Reviewers

ericsnowcurrently

ericsnowcurrently approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news tests

Tests in the Lib/test dir

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.

GeezerVison

Conversation

mdboom commented Jul 14, 2022 • edited by miss-islington Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bedevere-bot commented Aug 12, 2022

Uh oh!

mdboom commented Aug 16, 2022

Uh oh!

bedevere-bot commented Aug 16, 2022

Uh oh!

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Uh oh!

miss-islington commented Aug 16, 2022

Uh oh!

bedevere-bot commented Aug 16, 2022

Uh oh!

ericsnowcurrently commented Aug 16, 2022

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

mdboom commented Jul 14, 2022 •

edited by miss-islington

Loading

Uh oh!

There was an error while loading. Please reload this page.