Description

navytux

opened on Apr 30, 2023

Issue body actions

Hello up there. I've hit a discrepancy in how z flags is handled by % in unicode and bytes:

• for unicode % rejects it as "unsupported format character" according to original discussion in string formatting: normalize negative zero #90153 (= BPO-45995),
• hower for bytes % fully handles "z":

kirr@deca:~$ python3
Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> '%zf' % 1                                       <--   unicode
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: unsupported format character 'z' (0x7a) at index 1

>>> b'%zf' % 1                                      <--   bytes
b'1.000000'

>>> b'%zf' % 0.0                                    <--   +0 -> 0
b'0.000000'

>>> b'%zf' % -0.0                                   <--   -0 -> 0
b'0.000000'

>>> b'%f' % -0.0                                    <--   -0 -> -0 if run without 'z'
b'-0.000000'

In other words there is inconsistency in how 'z' is handled by '%' for unicode and bytes, and there is also inconsistency in how 'z' was supposed to be handled by .format and not handled by '%' as originally discussed on BPO-45995.

'z' handling was implemented in #30049 and indeed there I see b'%z' being fully handled:

b0b836b20cb5#diff-f6d440aad34e1c4535c0d898c0197a95490766c745991caace6f64b5dd1ece51

but u'%z' being only partly handled internally without corresponding frontend parsing that bytes has:

b0b836b20cb5#diff-34c966e7876d6f8bf801dd51896327e4f68bba02cddb95fbf3963f0b2e39c38a

In my view the fix should be either a) to add '%z' handling to unicode, or b) to remove '%z' handling from bytes.

Thanks beforehand,
Kirill

• CPython versions tested on: 3.11.2
• Operating system and architecture: Debian GNU/Linux 12 on AMD64

/cc @belm0, @mdickinson

Linked PRs

• gh-104018: disallow "z" format specifier in %-format of byte strings #104033
• [3.11] gh-104018: disallow "z" format specifier in %-format of byte strings (GH-104033) #104058
• gh-104018: remove unused format "z" handling in string formatfloat() #104107
• [3.11] gh-104018: remove unused format "z" handling in string formatfloat() (GH-104107) #104260

Reacted by John Belmonte

Activity

1. navytux

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Apr 30, 2023
2. arhadthedev

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on Apr 30, 2023
3. belm0 commented on May 1, 2023

   

   belm0

   on May 1, 2023

   Contributor

   More actions

   Good catch. I think having that enabled was left over from an early incarnation of the PR, before it was decided that %-format would not be supported.

   For now, I confirmed that the tests still pass after disable of case 'z' in _PyBytes_FormatEx().

   Fix should include tests to confirm that "z" format is not accepted for %-formatting.
4. belm0

   added a commit that references this issue on May 1, 2023

   pythongh-104018: disallow "z" format specifier in %-format of byte st…

   f697407
5. mdickinson

   added a commit that references this issue on May 1, 2023

   gh-104018: disallow "z" format specifier in %-format of byte strings (G…

   3ed8c88
6. miss-islington

   added a commit that references this issue on May 1, 2023

   pythongh-104018: disallow "z" format specifier in %-format of byte st…

   e18148a
7. mdickinson commented on May 1, 2023

   

   mdickinson

   on May 1, 2023

   Member

   More actions

   Fixed in #104018 and fix backported to 3.11 in #104033. Thanks @navytux for the report and @belm0 for the quick fix.
8. mdickinson

   closed this as completedon May 1, 2023
9. mdickinson

   added a commit that references this issue on May 1, 2023

   [3.11] gh-104018: disallow "z" format specifier in %-format of byte s…

   10db28b
10. carljm

    added a commit that references this issue on May 1, 2023

    Merge branch 'main' into inlinecomp2

    1274e2b
11. navytux commented on May 2, 2023

    

    navytux

    on May 2, 2023

    Author

    More actions

    @belm0, @mdickinson, thanks for the prompt fix.

    May I ask why static formatfloat() in bytesobject.c remains with F_NO_NEG_0 handling? Offhand it looks like that flag bit could never make it into that function, but I might be missing something. The same question applies to unicodeobject.c .

    Reacted by John Belmonte
12. belm0

    added a commit that references this issue on May 2, 2023

    pythongh-104018: remove unused format "z" handling in string objects

    35ba4e4
13. belm0 commented on May 2, 2023

    

    belm0

    on May 2, 2023

    Contributor

    More actions

    May I ask why static formatfloat() in bytesobject.c remains with F_NO_NEG_0 handling? Offhand it looks like that flag bit could never make it into that function, but I might be missing something. The same question applies to unicodeobject.c .

    Thank you, please see #104107
14. belm0

    added a commit that references this issue on May 2, 2023

    pythongh-104018: remove unused format "z" handling in string formatfl…

    a3be962
15. navytux commented on May 3, 2023

    

    navytux

    on May 3, 2023

    Author

    More actions

    Thanks
16. kumaraditya303

    added a commit that references this issue on May 7, 2023

    gh-104018: remove unused format "z" handling in string formatfloat() (#…

    69621d1
17. miss-islington

    added a commit that references this issue on May 7, 2023

    pythongh-104018: remove unused format "z" handling in string formatfl…

    eea8edc
18. kumaraditya303

    added a commit that references this issue on May 7, 2023

    [3.11] gh-104018: remove unused format "z" handling in string formatf…

    15ffcf7
19. jbower-fb

    added a commit that references this issue on May 8, 2023

    pythongh-104018: remove unused format "z" handling in string formatfl…

    06707b9

Issue actions