Description

loic-simon

opened on May 23, 2025

Issue body actions

Bug report

Bug description:

Found when investigating test.test_tokenize.TestRoundtrip.test_random_files failing on #134577

def test(code):
    tokens = list(tokenize.tokenize(iter([code]).__next__))
    from5 = tokenize.untokenize(tokens)
    print("from5   ", from5, eval(from5))

    tokens2 = [tok[:2] for tok in tokens]
    from2 = tokenize.untokenize(tokens2)
    print("from2   ", from2, eval(from2))

>>> test(b't"{ {} }"')
from5    b't"{ {} }"' Template(strings=('', ''), interpolations=(Interpolation({}, ' {}', None, ''),))
from2    b't"{{}}"'   Template(strings=('{}',), interpolations=())

>>> test(b'f"{ {} }"')
from5    b'f"{ {} }"' {}
from2    b'f"{ {} }"' {}

From what I understand, untokenize should round-trip correctly even with only the type and string of tokens.

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-134582: Fix t-strings untokenize() roundtrip removing space between braces #134603
• [3.14] gh-134582: Fix t-strings untokenize() roundtrip removing space between braces (GH-134603) #134659

Activity

1. loic-simon

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on May 23, 2025
2. picnixz

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   3.14bugs and security fixesbugs and security fixes

   3.15bugs and security fixesbugs and security fixes

   on May 23, 2025
3. pablogsal

   added a commit that references this issue on May 25, 2025

   gh-134582: Fix t-strings untokenize() roundtrip removing space betwee…

   52509cc
4. miss-islington

   added a commit that references this issue on May 25, 2025

   pythongh-134582: Fix t-strings untokenize() roundtrip removing space …

   057a41e
5. pablogsal

   added a commit that references this issue on May 25, 2025

   [3.14] gh-134582: Fix t-strings untokenize() roundtrip removing space…

   c61fd7c
6. loic-simon

   closed this as completedon May 25, 2025
7. Pranjal095

   added a commit that references this issue on Jul 12, 2025

   pythongh-134582: Fix t-strings untokenize() roundtrip removing space …

   4cfef61
8. taegyunkim

   added a commit that references this issue on Aug 4, 2025

   pythongh-134582: Fix t-strings untokenize() roundtrip removing space …

   c9d5680
9. gostak-dd

   added a commit that references this issue on Jun 2, 2026

   pythongh-134582: Fix t-strings untokenize() roundtrip removing space …

   e109768

Issue actions