GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-130197: Test various encodings with pygettext by tomasr8 · Pull Request #132244 · python/cpython

Conversation

@tomasr8

@tomasr8 tomasr8 commented Apr 7, 2025 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

For context: #131902 (comment)

[...] We need this to test with and without --escape option. In particularly, to see what encoding is used in the POT file and in the header. In file with non-UTF-8 encoding use also characters not encodable with the source encoding (\uXXXX).

We currently set the charset of the POT file as the default encoding on the system (fp.encoding):

cpython/Tools/i18n/pygettext.py

Lines 574 to 576 in f5639d8

def write_pot_file(messages, options, fp):
timestamp = time.strftime('%Y-%m-%d %H:%M%z')
encoding = fp.encoding if fp.encoding else 'UTF-8'

To have reproducible tests regardless of the OS they are running on, we set -X utf8 in the tests. As a consequence, the POT charset is always set to utf-8. I don't think there's an easy way to control that if we want to test other output encodings.. At least with these tests we know that non-utf8 input files can be read correctly.

cc @serhiy-storchaka Let me know if this is what you had in mind for the tests!

  • Issue: pygettext: Improve test coverage #130197

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@tomasr8
Test various encodings with pygettext
e28559c
@tomasr8 tomasr8 added the skip news label Apr 7, 2025
@bedevere-app bedevere-app Bot added the awaiting review label Apr 7, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Apr 7, 2025
pygettext: Improve test coverage #130197
Open
6 of 18 tasks
@tomasr8
Update test subdirs
e12bb90
@tomasr8
tomasr8 requested a review from erlend-aasland as a code owner April 7, 2025 22:01
serhiy-storchaka
serhiy-storchaka reviewed Apr 8, 2025
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not think that duplicating this test with multiple encodings is needed. It is enough to test with one encoding -- and it should not be Latin1 or Windows-1252, which are often the default encoding. The CPU time can be spent on different tests.

Please add also non-ASCII comments.

Finally, we need to add tests for non-ASCII filenames on non-UTF-8 locale. I afraid that i18n_data cannot be used for this -- we need to try several locales with different encodings and generate an input file with corresponding name.

We need also to test the stderr output for files with non-ASCII file name and non-ASCII source encoding on non-UTF-8 locale. It contains a file name and may contain a fragment of the source text.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
tomasr8 added 2 commits April 12, 2025 12:16
@tomasr8
Use latin2 encoding
cad1efd
@tomasr8
Lint fix
067807f
@tomasr8
tomasr8 marked this pull request as draft April 12, 2025 11:26
@bedevere-app bedevere-app Bot removed the awaiting review label Apr 12, 2025
@mtfreitasf mtfreitasf mentioned this pull request Oct 9, 2025
pygettext.py crashes with UnicodeEncodeError when writing .pot files containing emoji #139873
Open
@github-actions

github-actions Bot commented Apr 23, 2026

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Apr 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@serhiy-storchaka serhiy-storchaka serhiy-storchaka left review comments
@erlend-aasland erlend-aasland Awaiting requested review from erlend-aasland erlend-aasland is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner will be requested when the pull request is marked ready for review AA-Turner is a code owner
@emmatyping emmatyping Awaiting requested review from emmatyping emmatyping will be requested when the pull request is marked ready for review emmatyping is a code owner
@itamaro itamaro Awaiting requested review from itamaro itamaro will be requested when the pull request is marked ready for review itamaro is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

skip news stale Stale PR or inactive for long period of time.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@tomasr8 @serhiy-storchaka
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.