GeezerVison is showing github.com — Raise non-silent warning for invalid escape sequences · Issue #77093 · python/cpython
Raise non-silent warning for invalid escape sequences #77093Description
| BPO |
32912 |
| Nosy |
@rhettinger, @terryjreedy, @gpshead, @mdickinson, @ncoghlan, @bitdancer, @njsmith, @ambv, @serhiy-storchaka, @zooba, @Vgr255, @CuriousLearner, @miss-islington, @asmeurer, @tirkarthi |
| PRs |
bpo-32912: Upgrade warning for invalid escape sequences from silent to non-silent #5849bpo-32912: Replace a DeprecationWarning with a SyntaxWarning #9652[3.8] bpo-32912: Revert warnings for invalid escape sequences. #15142bpo-32912: Revert SyntaxWarning on invalid escape sequences. #15195 |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2018-02-22.18:34:57.749>
labels = ['interpreter-core', 'type-bug', '3.9']
title = 'Raise non-silent warning for invalid escape sequences'
updated_at = <Date 2022-01-13.16:38:26.080>
user = 'https://github.com/Vgr255' bugs.python.org fields:
activity = <Date 2022-01-13.16:38:26.080>
actor = 'dansebcar'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2018-02-22.18:34:57.749>
creator = 'abarry'
dependencies = []
files = []
hgrepos = []
issue_num = 32912
keywords = ['patch']
message_count = 34.0
messages = ['312575', '312610', '312760', '326766', '328044', '344312', '344764', '345557', '345558', '345639', '345675', '345703', '346871', '348205', '348269', '348270', '348281', '348347', '348348', '348349', '348354', '348357', '348375', '349027', '349103', '349106', '349318', '349320', '349321', '349322', '349323', '349328', '349335', '410496']
nosy_count = 17.0
nosy_names = ['rhettinger', 'terry.reedy', 'gregory.p.smith', 'mark.dickinson', 'ncoghlan', 'r.david.murray', 'njs', 'lukasz.langa', 'serhiy.storchaka', 'steve.dower', 'abarry', 'CuriousLearner', 'miss-islington', 'asmeurer', 'xtreak', 'Eric Wieser', 'dansebcar']
pr_nums = ['5849', '9652', '15142', '15195']
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32912'
versions = ['Python 3.9'] ActivityVgr255 commented on Feb 22, 2018 MannequinAuthor More actions This is a follow-up to bpo-27364.
Back in Python 3.6, a silent warning was added for all invalid escape sequences in str and bytes. It was suggested that it would remain a silent warning (which still impacts tests, while not visually annoying the average user) for two releases (3.6 and 3.7), then would be upgraded to a non-silent warning for two subsequent releases (3.8 and 3.9) before becoming a full-on syntax error. With the 3.7 feature freeze on and going, I think it's time to evaluate the approach we take for 3.8 :)
I suggest upgrading the DeprecationWarning to a SyntaxWarning, which is visible by default, for 3.8 and 3.9. I have cross-linked bpo-27364 to this issue as well.
-Em ncoghlan commented on Feb 23, 2018 +1 for the DeprecationWarning->SyntaxWarning->SyntaxError approach from me (especially as 3.7 will make the existing deprecation warning visible in interactive shells and __main__ modules by default). serhiy-storchaka commented on Feb 24, 2018 PR 5849 changes not only the Python parser, but codecs. It shouldn't. The Python parser and codecs will go different ways. The warning in the Python parser will finally be upgraded to SyntaxError (it is already replaced with SyntaxError if the warning is raised as error). The warning in codecs will become UnicodeDecodeError or ValueError. This is why the code for emitting the warning is not shared between the parser and codecs at first place. serhiy-storchaka commented on Oct 1, 2018 Since the author of PR 5849 had not the time to work on it, I have created PR 9652 which properly replaces a DeprecationWarning with a SyntaxWarning. It also updates the documentation. serhiy-storchaka commented on Oct 19, 2018 rhettinger commented on Jun 2, 2019 I'm seeing these warnings pop up from time to time with various third party packages (such as pyparsing which is invoked by ensurepip).
Am wondering whether this should be deferred for another version or so until be have good confidence that the major third party packages have all had a chance to adapt. Otherwise, this will mostly be noise for end-users. Also, it gets in the way of the end-user strategy of "backslash anything that looks special" as preferred over "remember exactly which things are special". This may be a case of "the cure is worse than the disease". asmeurer commented on Jun 5, 2019 I agree with Raymond that third party libraries are not ready for this.
My biggest issue is that the way Python warns about this makes it very difficult for library authors to fix this. Most won't even notice. The problem is the warnings are only shown once, when the file is compiled. So if you run the code in any way that causes Python to compile it, a further run of 'python -Wd' will show nothing. I don't know if it's reasonable, but it would be nice if Python recompiled when given -Wd, or somehow saved the warning so it could be shown if that flag is given later.
As an anecdote, for SymPy's CI, we went through five (if I am counting correctly) iterations of trying to test this. Each of the first four were subtly incorrect, until we finally managed to find the correct one (for reference, 'python -We:invalid -m compileall -f -q module/'). So most library authors who will attempt to add tests against this will get it wrong. Simply adding -Wd as you would expect is wrong. If the code is already compiled (which it probably is, e.g., if you ran setup.py), it won't show the warnings. At the very least the "correct" way to test this should be documented.
Things would probably be improved if the warnings were always shown, as at least then devs will see the error once (although most will probably be confused when the warning doesn't repeat).
Another problem is the information in the warnings. It seems the line number of the string is now shown, which is an improvement (https://bugs.python.org/issue28128). It would be nice if it showed the actual line and column number in the file of the invalid escape. This is especially annoying when an escape appears in a docstring. It just shows """ as the offending line.
We have a lot of LaTeX in our docstrings in SymPy, so we had quite a few of these to fix. SymPy doesn't have invalid escapes anymore because I was proactive about it, but from what I've seen, most library authors haven't been.
By the way, this looks like a bug (python 3.8b1):
$ cat test.py
"""
a
\p
"""
$ python -We:invalid test.py
File "test.py", line 2
^
SyntaxError: invalid escape sequence \p
$It might not be possible to see in what I pasted, but it filled my terminal with spaces. serhiy-storchaka commented on Jun 14, 2019 In the particular case of pyparsing there was a bug in the docstring.
def sub(self, repl):
"""
Return Regex with an attached parse action to transform the parsed
result as if called using `re.sub(expr, repl, string) <https://docs.python.org/3/library/re.html#re.sub>`_.
Example::
make_html = Regex(r"(\w+):(.*?):").sub(r"<\1>\2</\1>")
print(make_html.transformString("h1:main title:"))
# prints "<h1>main title</h1>"
"""\1 and \2 were interpreted as control characters U+0001 and U+0002, so the user could see
make_html = Regex(r"(\w+):(.*?):").sub(r"<^A>^B</^A>") or
make_html = Regex(r"(\w+):(.*?):").sub(r"<☺>☻</☺>") or
make_html = Regex(r"(\w+):(.*?):").sub(r"<></>") depending on the output device.
So this examples proves the usefulness of the new warning. serhiy-storchaka commented on Jun 14, 2019 Aaron, you have reported several different issues in your message:
-
compiler warnings are emitted only once, therefore it is easy to miss them and hard to reproduce.
-
warnings for multiline expressions (and multiline string literals in particularly) could be better.
-
strange output of spaces in some cases.
And maybe more.
I think all of them deserve opening separate issues. asmeurer commented on Jun 14, 2019 I agree. Please someone else do that. I don't know what already has issues and I unfortunately don't have time right now to help out with any of this.
I simply mentioned all these things as arguments why Python should not (yet) make these warnings errors, which is the point of this issue.
Also, for the pyparsing example, you would have gotten lucky because it also contained \w, which is not a valid escape. If it didn't, you wouldn't be warned. So clearly this will help things, but it will also be good to have linting tools that, for example, warn about any escape sequences inside docstrings. asmeurer commented on Aug 9, 2019 Are there issues tracking the things I mentioned, which should IMO happen before this becomes a hard error (making the warnings reproduce even if the file has already been compiled, and making warning message point to the correct line in multiline strings)? And is it too late to potentially get some of those things in 3.8? gpshead commented on Aug 10, 2019 I haven't looked, so not that i'm aware of. I suggest filing one for each of those.
The warning not pointing to the right line in a multiline literal sounds like a bug to me so that one, if fixed, seems reasonable for 3.8. The release manager gets to decide.
Making the syntax warnings happen upon import of an already compiled pyc is more feature-ish, likely to be 3.9+. rhettinger commented on Aug 10, 2019 One possibility is to restrict the warning to a backslash followed by an alphabetic character or backslash, and that we define backslash followed by any other printable character as specifically allowed. This would catch the likely sources of errors without breaking the likes of \, or \-. Also, multiline strings have less of a need for a warning than a single line string. miss-islington commented on Aug 10, 2019 New changeset b4be87a by Miss Islington (bot) (Gregory P. Smith) in branch 'master':
bpo-32912: Revert SyntaxWarning on invalid escape sequences. (GH-15195)
b4be87a dansebcar commented on Jan 13, 2022 I'm not sure if this is an issue or by design, but this DeprecationWarning behaves differently to other DeprecationWarnings.
A normal DeprecationWarning triggered by code in __main__ is printed by default:
$ python -c 'import warnings; warnings.warn("test", DeprecationWarning)'
<string>:1: DeprecationWarning: testBut this one is silent:
$ python -c '"\,"'
[no output] To see this DeprecationWarning at all, I need to type:
$ python -Wdefault -c '"\,"'
<string>:1: DeprecationWarning: invalid escape sequence '\,' But that enables this DeprecationWarning for all modules, not just __main__ .
I've tested this with Python 3.9 on debian bullseye and the 3.10 docker image. vstinner commented on Nov 2, 2022 New attempt to replace DeprecationWarning with SyntaxWarning:
vstinner commented on Nov 3, 2022
Here's an example from the current version of Bottle (0.12.17):
/Users/raymond/Dropbox/Public/sj205/notes2/bottle.py:3433: SyntaxWarning: invalid escape sequence [
_re_tok += '|([[{(])'
The regex was fixed in the meanwhile. With Bottle 0.12.23, there is no warning anymore:
$ pycodestyle --select W605 env/lib/python3.12/site-packages/bottle.py
vstinner commented on Nov 3, 2022
Running Sphinx v2.1.0
/Users/raymond/npython/venv/lib/python3.8/site-packages/docutils/writers/latex2e/init.py:2978: SyntaxWarning: invalid escape sequence \l
I confirm that this issue is fixed in docutils 0.19. I tested Sphinx 5.3.0. There is no warning anymore:
$ pycodestyle --select W605 $(find env/lib/python3.12/site-packages/docutils/ -name "*.py")
$ pycodestyle --select W605 $(find env/lib/python3.12/site-packages/sphinx/ -name "*.py")
serhiy-storchaka commented on Jul 3, 2026 The SyntaxWarning was landed again in 3.12 (gh-98401). Closing as completed. Reacted by Victor Stinner
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|