GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — gh-134582: Fix t-strings untokenize() roundtrip removing space betwee… · miss-islington/cpython@52509cc

File tree

Expand file treeCollapse file tree

Open diff view settings

Filter options

Lib
- test
  - test_tokenize.py
- tokenize.py
Misc/NEWS.d/next/Library
- 2025-05-23-23-43-39.gh-issue-134582.9POq3l.rst

Expand file treeCollapse file tree

Open diff view settings

Collapse file

`‎Lib/test/test_tokenize.py‎`

Copy file name to clipboardExpand all lines: Lib/test/test_tokenize.py

+4Lines changed: 4 additions & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -1975,6 +1975,10 @@ def test_roundtrip(self):`
`1975`	`1975`	`for case in cases:`
`1976`	`1976`	`self.check_roundtrip(case)`
`1977`	`1977`
	`1978`	`+ self.check_roundtrip(r"t'{ {}}'")`
	`1979`	`+ self.check_roundtrip(r"t'{f'{ {}}'}{ {}}'")`
	`1980`	`+ self.check_roundtrip(r"f'{t'{ {}}'}{ {}}'")`
	`1981`	`+`
`1978`	`1982`
`1979`	`1983`	`def test_continuation(self):`
`1980`	`1984`	`# Balancing continuation`

Collapse file

`‎Lib/tokenize.py‎`

Copy file name to clipboardExpand all lines: Lib/tokenize.py

+7-7Lines changed: 7 additions & 7 deletions

Original file line number	Diff line number	Diff line change
`@@ -274,7 +274,7 @@ def compat(self, token, iterable):`
`274`	`274`	`toks_append = self.tokens.append`
`275`	`275`	`startline = token[0] in (NEWLINE, NL)`
`276`	`276`	`prevstring = False`
`277`		`- in_fstring = 0`
	`277`	`+ in_fstring_or_tstring = 0`
`278`	`278`
`279`	`279`	`for tok in _itertools.chain([token], iterable):`
`280`	`280`	`toknum, tokval = tok[:2]`
`@@ -293,10 +293,10 @@ def compat(self, token, iterable):`
`293`	`293`	`else:`
`294`	`294`	`prevstring = False`
`295`	`295`
`296`		`- if toknum == FSTRING_START:`
`297`		`- in_fstring += 1`
`298`		`- elif toknum == FSTRING_END:`
`299`		`- in_fstring -= 1`
	`296`	`+ if toknum in {FSTRING_START, TSTRING_START}:`
	`297`	`+ in_fstring_or_tstring += 1`
	`298`	`+ elif toknum in {FSTRING_END, TSTRING_END}:`
	`299`	`+ in_fstring_or_tstring -= 1`
`300`	`300`	`if toknum == INDENT:`
`301`	`301`	`indents.append(tokval)`
`302`	`302`	`continue`
`@@ -311,8 +311,8 @@ def compat(self, token, iterable):`
`311`	`311`	`elif toknum in {FSTRING_MIDDLE, TSTRING_MIDDLE}:`
`312`	`312`	`tokval = self.escape_brackets(tokval)`
`313`	`313`
`314`		`- # Insert a space between two consecutive brackets if we are in an f-string`
`315`		`- if tokval in {"{", "}"} and self.tokens and self.tokens[-1] == tokval and in_fstring:`
	`314`	`+ # Insert a space between two consecutive brackets if we are in an f-string or t-string`
	`315`	`+ if tokval in {"{", "}"} and self.tokens and self.tokens[-1] == tokval and in_fstring_or_tstring:`
`316`	`316`	`tokval = ' ' + tokval`
`317`	`317`
`318`	`318`	`# Insert a space between two consecutive f-strings`

Collapse file

`‎Misc/NEWS.d/next/Library/2025-05-23-23-43-39.gh-issue-134582.9POq3l.rst‎`

Copy file name to clipboard

+1Lines changed: 1 addition & 0 deletions

Display the source diff
Display the rich diff

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1 @@`
	`1`	`+Fix tokenize.untokenize() round-trip errors related to t-strings braces escaping`

0 commit comments

Comments

0 (0)

You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.