Description

serhiy-storchaka

opened on Oct 13, 2023

Issue body actions

Most of C strings in the C API are implied to be UTF-8 encoded. PyArg_ParseTupleAndKeywords() mostly works with non-ASCII keyword names as they are UTF-8 encoded. Except one case, when you pass argument by keyword with invalid non-ASCII name to a function that has optional parameter with non-ASCII UTF-8 encoded name. In this case you get a crash in the debug build.

It was caused by combination of f4934ea and a83a6a3 (bpo-28701, #72887). Before these changes you simply got inaccurate or even wrong error message.

Examples:

1. Parameters: "ä"
   Keyword arguments: "ë"
   Old behavior: TypeError "'ë' is an invalid keyword argument for this function"
   Current behavior: crash
   Expected behavior: TypeError "'ë' is an invalid keyword argument for this function"

2. Parameters: "ä"
   Keyword arguments: "ä"
   Old behavior: TypeError "invalid keyword argument for this function"
   Current behavior: crash
   Expected behavior: TypeError "'ä' is an invalid keyword argument for this function"

3. Parameters: "ä", "ë"
   Keyword arguments: "ä", "ë"
   Old behavior: TypeError "'ë' is an invalid keyword argument for this function"
   Current behavior: crash
   Expected behavior: TypeError "'ä' is an invalid keyword argument for this function"

In case 1 the pre-bpo-28701 behavior was correct, in case 2 it was correct but not precise (it failed to find the name of invalid keyword argument), in case 3 it was wrong (it found wrong name). In all cases there is a crash currently.

Linked PRs

• gh-110815: Support non-ASCII keyword names in PyArg_ParseTupleAndKeywords() #110816
• gh-110815: Improve tests for PyArg_ParseTupleAndKeywords() #110817
• [3.12] gh-110815: Improve tests for PyArg_ParseTupleAndKeywords() (GH-110817) #110825

Activity

1. serhiy-storchaka

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   3.11only security fixesonly security fixes

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   topic-C-API

   3.12only security fixesonly security fixes

   3.13only security fixesonly security fixes

   on Oct 13, 2023
2. serhiy-storchaka

   added a commit that references this issue on Oct 13, 2023

   pythongh-110815: Support non-ASCII keyword names in PyArg_ParseTupleA…

   6c154d3
3. serhiy-storchaka

   added

   topic-unicode

   on Oct 13, 2023
4. serhiy-storchaka commented on Oct 13, 2023

   

   serhiy-storchaka

   on Oct 13, 2023

   MemberAuthor

   More actions

   I am not sure how to classify this issue: as a bug or a new feature. On one hand, it was a regression in 3.7, crash is worse than wrong error message. On other hand, it was not explicitly documented that non-ASCII names are supported, there were no tests, and if it worked, it was only by accident. And it was a long time ago. So, official support of non-ASCII names can be a new feature.
5. serhiy-storchaka

   added 3 commits that reference this issue on Oct 13, 2023

   pythongh-110815: Support non-ASCII keyword names in PyArg_ParseTupleA…

   5153eb8

   pythongh-110815: Improve tests for PyArg_ParseTupleAndKeywords()

   c41cacb

   gh-110815: Improve tests for PyArg_ParseTupleAndKeywords() (GH-110817)

   548ce09
6. miss-islington

   added a commit that references this issue on Oct 13, 2023

   pythongh-110815: Improve tests for PyArg_ParseTupleAndKeywords() (pyt…

   dd99f04
7. serhiy-storchaka

   added a commit that references this issue on Oct 13, 2023

   [3.12] gh-110815: Improve tests for PyArg_ParseTupleAndKeywords() (GH…

   1c44f88
8. terryjreedy commented on Oct 13, 2023

   

   terryjreedy

   on Oct 13, 2023

   Member

   More actions

   Is the issue visible from Python code:

   def f(ä=''): pass
   
   f(ä="ë")
   f(ä="ä")
   

   works. If something like this were to crash, I would say definitely a bugfix to be backported.
9. serhiy-storchaka commented on Oct 14, 2023

   

   serhiy-storchaka

   on Oct 14, 2023

   MemberAuthor

   More actions

   It is an issue with PyArg_ParseTupleAndKeywords(), a C API used for parsing keyword arguments in extension functions. Python defined functions do not use it. Also, it is an issue with argument names, not values. Non-ASCII argument names are forbidden in the core interpreter and the stdlib, but third parties may have reasons to use them.
10. serhiy-storchaka

    added a commit that references this issue on Oct 14, 2023

    gh-110815: Support non-ASCII keyword names in PyArg_ParseTupleAndKeyw…

    7284e0e
11. serhiy-storchaka

    closed this as completedon Oct 14, 2023
12. terryjreedy commented on Oct 15, 2023

    

    terryjreedy

    on Oct 15, 2023

    Member

    More actions

    Thanks. Makes sense now.
13. aisk

    added 2 commits that reference this issue on Feb 11, 2024

    pythongh-110815: Improve tests for PyArg_ParseTupleAndKeywords() (pyt…

    3dd9087

    pythongh-110815: Support non-ASCII keyword names in PyArg_ParseTupleA…

    421748a
14. Glyphack

    added 2 commits that reference this issue on Sep 2, 2024

    pythongh-110815: Improve tests for PyArg_ParseTupleAndKeywords() (pyt…

    75f7150

    pythongh-110815: Support non-ASCII keyword names in PyArg_ParseTupleA…

    717a68a

Issue actions