GeezerVison is showing github.com — Escaped quotes cause source comments to leak into debug f-string output and t-string metadata · Issue #154711 · python/cpython
Escaped quotes cause source comments to leak into debug f-string output and t-string metadata #154711Labels interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-parsertype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DescriptionBug report
Bug description:
When an f-string or t-string replacement expression contains a backslash-escaped quote, a following source comment can be copied into runtime output or t-string metadata.
Observed effects:
- A debug f-string includes the source comment in the resulting str.
- A debug t-string includes the comment in Template.strings.
- A debug t-string includes both the debug = and the comment inInterpolation.expression.
- A non-debug t-string also retains the comment inInterpolation.expression.
The replacement expression itself evaluates to the correct value. The problemappears to be in reconstruction of the replacement-field source text.
Reproducer
f_result = f"{'\'' = # c
}"
t_debug = t"{'a\'b' = # c
}"
t_plain = t"{'a\'b' # c
}"
print("debug f-string result:", repr(f_result))
print("\ndebug t-string strings:", repr(t_debug.strings))
print("\ndebug t-string expression:", repr(t_debug.interpolations[0].expression))
print("\nnon-debug t-string expression:", repr(t_plain.interpolations[0].expression))Actual behavior
On CPython main commit b86a41c, the output contains the source text # c in all four observed values.
debug f-string result: '\'\\\'\' = # c\n"\'"'
debug t-string strings: ("'a\\'b' = # c\n", '')
debug t-string expression: "'a\\'b' = # c"
non-debug t-string expression: "'a\\'b' # c"
Possible cause
The behavior appears to be caused by inconsistent escaped-character handling between two scans in _PyLexer_set_ftstring_expr() in Parser/lexer/string.c.
The initial scan that detects a comment outside a string literal explicitly skips a backslash and the character following it:
if (ch == '\\') {
i++;
continue;
}However, after a comment has been detected, the subsequent reconstruction pass that removes comments appears not to perform the equivalent skip. It changes in_string whenever it encounters a quote:
if (ch == '"' || ch == '\'') {
if (!in_string) {
in_string = 1;
quote_char = ch;
}
else if (ch == quote_char) {
in_string = 0;
}
}
else if (ch == '#' && !in_string) {
/* skip the comment */
}For:
the second pass may interpret the escaped quote as the end of the string and the real closing quote as the beginning of another string:
' begin string
\ treated as an ordinary character
' incorrectly treated as end of string
b
' incorrectly treated as beginning of another string
# incorrectly considered to be inside a string
Consequently, the # is not recognized by the comment-removal branch and the comment is copied into the reconstructed text.
This also explains why the same input pattern affects both f-strings and t-strings: both use the same replacement-expression reconstruction logic.
This is a proposed root cause based on the observed behavior and source review, not a confirmed diagnosis.
CPython versions tested on:
CPython main branch, 3.16
Operating systems tested on:
Linux
Linked PRs
Activitypicnixz commented on Jul 26, 2026 Mmh I think it is actually deliberate? at least it seems consistent. Does it reproduce in older versions?
EDIT: wait I think I missed something. It may be a legitimate issue. tekinertekin commented on Jul 29, 2026 @picnixz — about this being intentional: I don't think it can be. On 3.12 the comment is stripped correctly, both escaped and unescaped. I had to use the multiline form to compare, since the single-line version in the report is a SyntaxError before 3.14:
a = f"""{'ab' = # c
}"""
print(repr(a)) # "'ab' = \n'ab'"
b = f"""{'a\'b' = # c
}"""
print(repr(b)) # '\'a\\\'b\' = \n"a\'b"'
c = f"""{'a\'b' =
}"""
print(repr(c)) # same as b
print(b == c) # True
In the escaped case the result is byte-identical whether or not the comment is present, so stripping it is the established behaviour and the leak on main is new. I couldn't run 3.14/3.16 locally, so I haven't reproduced the reported output myself — that part is still the reporter's.
The pointer to _PyLexer_set_ftstring_expr() in Parser/lexer/string.c looks right to me: the two scans disagree on backslash handling when reconstructing the replacement-field source. I'd like to work on this if it's free. Does that match your reading? dqqFA8995 commented on Jul 29, 2026 Thanks for checking against older releases—that comparison is helpful.
If 3.12 produces identical output regardless of whether the comment is
present, that strongly suggests comment stripping has been the intended
behavior, and the behavior on main is likely a regression rather than a
deliberate change.
I also agree that _PyLexer_set_ftstring_expr() looks like the right place
to investigate. The mismatch in escaped-character handling between the
initial scan and the reconstruction pass would explain why the issue only
appears with escaped quotes and why it affects both f-strings and t-strings
through the shared reconstruction logic.
It would be good to confirm that by bisecting to the commit that introduced
the regression or by adding a regression test before changing the lexer
logic. That would help ensure the fix matches the intended behavior and
doesn't alter any newly introduced semantics.
…
MetadataMetadatainterpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)topic-parsertype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error DevelopmentNo branches or pull requestsIssue actions
|