GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-98401: Invalid escape sequences emits SyntaxWarning by vstinner · Pull Request #99011 · python/cpython

Conversation

@vstinner

@vstinner vstinner commented Nov 2, 2022 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

A backslash-character pair that is not a valid escape sequence now generates a SyntaxWarning, instead of DeprecationWarning. For example, re.compile("\d+.\d+") now emits a SyntaxWarning ("\d" is an invalid escape sequence), use raw strings for regular expression: re.compile(r"\d+.\d+"). In a future Python version, SyntaxError will eventually be raised, instead of SyntaxWarning.

Octal escapes with value larger than 0o377 (ex: "\477"), deprecated in Python 3.11, now produce a SyntaxWarning, instead of DeprecationWarning. In a future Python version they will be eventually a SyntaxError.

codecs.escape_decode() and codecs.unicode_escape_decode() are left unchanged: they still emit DeprecationWarning.

  • The parser only emits SyntaxWarning for Python 3.12 (feature version), and still emits DeprecationWarning on older Python versions.
  • Fix SyntaxWarning by using raw strings in Tools/c-analyzer/ and wasm_build.py.
  • Issue: Reject invalid escape sequences (and octal escape sequences) in bytes and Unicode strings #98401

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@vstinner
vstinner requested review from lysnikolaou and pablogsal as code owners November 2, 2022 16:21
@bedevere-bot bedevere-bot mentioned this pull request Nov 2, 2022
Reject invalid escape sequences (and octal escape sequences) in bytes and Unicode strings #98401
Closed
@bedevere-bot bedevere-bot added the awaiting core review label Nov 2, 2022
This was referenced Nov 2, 2022
gh-98401: Reject invalid escape sequences in strings #98404
Closed
Raise non-silent warning for invalid escape sequences #77093
Closed
lysnikolaou
lysnikolaou reviewed Nov 2, 2022
View reviewed changes
Comment thread Parser/string_parser.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Nov 2, 2022

Copy link
Copy Markdown
Member Author

Previous attempt in 2018:

  • Commit: 6543912
  • Serhiy only changed the Python parser, not codecs: rationale
  • The new SyntaxWarning was seen in multiple projects: docutils (fixed), bottle, SymPy (bug in a docstring)
  • SyntaxWarning are only emitted once, and only when a Python file is parsed: when a cached PYC file is used, not warning is emitted
  • Serhiy reverted his change
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Nov 3, 2022

Copy link
Copy Markdown
Member Author

I tested the Python test suite with SyntaxWarning treated as error: it does pass.

$ PYTHONWARNINGS=error::SyntaxWarning ./python -W error::SyntaxWarning -m test -j0 -r -u all,-gui
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Nov 3, 2022

Copy link
Copy Markdown
Member Author

This issue mostly hit code defining regular expressions. Example in BeautifulSoup 3.2.2:

$ pycodestyle --select W605 $(find -name "*.py")
./BeautifulSoup.py:455:52: W605 invalid escape sequence '\d'
./BeautifulSoup.py:455:73: W605 invalid escape sequence '\w'

./BeautifulSoup.py:574:47: W605 invalid escape sequence '\d'
./BeautifulSoup.py:574:66: W605 invalid escape sequence '\w'

./BeautifulSoup.py:1080:38: W605 invalid escape sequence '\s'

./BeautifulSoup.py:1589:36: W605 invalid escape sequence '\s'

./BeautifulSoup.py:1933:16: W605 invalid escape sequence '\?'
./BeautifulSoup.py:1933:46: W605 invalid escape sequence '\?'
./BeautifulSoup.py:1935:35: W605 invalid escape sequence '\s'

Examples of code, re.compile() calls:

    BARE_AMPERSAND_OR_BRACKET = re.compile("([<>]|"
                                           + "&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
                                           + ")")
(...)
                      (re.compile('<!\s+([^<>]*)>'),
(...)
        xml_encoding_match = re.compile(
            '^<\?.*encoding=[\'"](.*?)[\'"].*\?>').match(xml_data)
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

serhiy-storchaka
serhiy-storchaka reviewed Nov 3, 2022
View reviewed changes
Comment thread Doc/reference/lexical_analysis.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/test/test_codecs.py Outdated
b = bytes([i])
if b not in b'abfnrtvx':
with self.assertWarns(DeprecationWarning):
with self.assertWarns(SyntaxWarning):

@serhiy-storchaka serhiy-storchaka Nov 3, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SyntaxWarning is not related to codecs. It only should be emitted by the compiler.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Nov 3, 2022

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which warning should be emitted if SyntaxWarning is not the best choice? UnicodeWarning?

Does UnicodeWarning make sense for codecs.escape_decode()?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@serhiy-storchaka serhiy-storchaka Nov 3, 2022

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DeprecationWarning, as for all other deprecated features.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread Parser/string_parser.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka

serhiy-storchaka commented Nov 3, 2022

Copy link
Copy Markdown
Member

I tested the Python test suite with SyntaxWarning treated as error: it does pass.

Did you remove all pyc files and regenerate frozen modules before tests? Also try to regenerate generated code using the new Python binary as PYTHON_FOR_REGEN.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Nov 3, 2022

Copy link
Copy Markdown
Member Author

Did you remove all pyc files and regenerate frozen modules before tests? Also try to regenerate generated code using the new Python binary as PYTHON_FOR_REGEN.

I ran PYTHONWARNINGS=error::SyntaxWarning make regen-all PYTHON_FOR_REGEN=./python: it works as expected.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Nov 3, 2022

Copy link
Copy Markdown
Member Author

@serhiy-storchaka:

Perhaps SyntaxWarning could be raised for invalid octals too. It is a more obvious error, and less common, so it may have lesser deprecation period.

Ok, I updated my PR: Use SyntaxWarning for invalid octal sequence.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
gh-98401: Invalid escape sequences emits SyntaxWarning
069ef6c
A backslash-character pair that is not a valid escape sequence now
generates a SyntaxWarning, instead of DeprecationWarning.  For
example, re.compile("\d+\.\d+") now emits a SyntaxWarning ("\d" is an
invalid escape sequence), use raw strings for regular expression:
re.compile(r"\d+\.\d+"). In a future Python version, SyntaxError will
eventually be raised, instead of SyntaxWarning.

Octal escapes with value larger than 0o377 (ex: "\477"), deprecated
in Python 3.11, now produce a SyntaxWarning, instead of
DeprecationWarning. In a future Python version they will be
eventually a SyntaxError.

codecs.escape_decode() and codecs.unicode_escape_decode() are left
unchanged: they still emit DeprecationWarning.

* The parser only emits SyntaxWarning for Python 3.12 (feature
  version), and still emits DeprecationWarning on older Python
  versions.
* Fix SyntaxWarning by using raw strings in Tools/c-analyzer/ and
  wasm_build.py.
@vstinner

vstinner commented Nov 3, 2022

Copy link
Copy Markdown
Member Author

Hum, I messed up my PR so I squashed two commits and I fixed my PR:

  • invalid octal sequence and invalid escape sequence now both emit SyntaxWarning (and will emit SyntaxError in the future)
  • codecs.escape_decode() and codecs.unicode_escape_decode() are left unchanged

I also mentioned the future convertion to SyntaxError in the doc (What's New / NEWS entries).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner

vstinner commented Nov 3, 2022

Copy link
Copy Markdown
Member Author

@mdickinson @serhiy-storchaka @hugovk: Would you mind to review my PR?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@serhiy-storchaka

serhiy-storchaka commented Nov 3, 2022

Copy link
Copy Markdown
Member

Did you remove all pyc files?

find -name '*.py[co]' -exec rm -rf '{}' +
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

serhiy-storchaka
serhiy-storchaka approved these changes Nov 3, 2022
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels Nov 3, 2022
@vstinner

vstinner commented Nov 3, 2022

Copy link
Copy Markdown
Member Author

Let me try these commands:

export PYTHONWARNINGS=error::SyntaxWarning 
git clean -fdx
./configure --with-pydebug CFLAGS=-O0
make
make regen-all PYTHON_FOR_REGEN=./python
make

./python -c 's = "\477"'

The last command fails as expected with:

  File "<string>", line 1
    s = "\477"
        ^^^^^^
SyntaxError: invalid octal escape sequence '\477'
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
Fix test_string_literals
d24f799
@vstinner

vstinner commented Nov 3, 2022

Copy link
Copy Markdown
Member Author

Oops, test_string_literals didn't work when run with ./python -W error::SyntaxWarning: it's now fixed.

I ran the test suite with:

export PYTHONWARNINGS=error::SyntaxWarning 
./python -W error::SyntaxWarning -m test -j0 -r -u all,-gui 

Note: I would prefer to run the whole test suite with -Werror, but it fails even without this PR :-( Python currently emits many deprecation warnings for example :-(

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@vstinner
vstinner merged commit a60ddd3 into python:main Nov 3, 2022
@bedevere-bot bedevere-bot removed the awaiting merge label Nov 3, 2022
@vstinner
vstinner deleted the syntax_warning branch November 3, 2022 16:53
@chenhengqi chenhengqi mentioned this pull request Oct 31, 2023
tools/*.py: fix invalid escape sequences in regexps iovisor/bcc#4789
Merged
@benoit9126 benoit9126 mentioned this pull request Nov 9, 2023
Syntax warning because of "\S" in Python 3.12 keygen-sh/py-machineid#9
Closed
markmentovai added a commit to markmentovai/bits that referenced this pull request Jan 21, 2024
@markmentovai
faa_cs_aan.py: use r-strings for all regular expressions
f10a7b6
Fixes this warning experienced with Python 3.12
(python/cpython#98401,
python/cpython#99011):

faa_cs_aan.py:690: SyntaxWarning: invalid escape sequence '\.'
  email_match = re.search('For Inquiries: ([0-9a-z._-]+@[0-9a-z.-]+)\.?$',
@terryjreedy terryjreedy mentioned this pull request May 23, 2024
Invalid escape sequences raise SyntaxWarning #119458
Closed
@bebound bebound mentioned this pull request Sep 14, 2024
gh-123858: Improve Doc: SyntaxWarning is emitted during bytecode generation #122844
Merged
hubot pushed a commit to chromium/crashpad that referenced this pull request Feb 11, 2025
@markmentovai Crashpad LUCI CQ
Fix SyntaxWarning in build/run_tests.py
85ecbd2
Use r-strings for all regular expressions.

Fixes these warnings experienced with Python 3.12
(python/cpython#98401,
python/cpython#99011,
https://docs.python.org/3/whatsnew/3.12.html#other-language-changes
point 2):

run_tests.py:200: SyntaxWarning: invalid escape sequence '\d'
  FINAL_LINE_RE = re.compile('status=(\d+)$')
run_tests.py:441: SyntaxWarning: invalid escape sequence '\*'
  re.match('^\* daemon .+ \*$', line) or line == ''):

Change-Id: I71ddfb1a2ca62654378ae67a99e9aeb4ce7b7394
Reviewed-on: https://chromium-review.googlesource.com/c/crashpad/crashpad/+/6254063
Commit-Queue: Mark Mentovai <[email protected]>
Reviewed-by: Nico Weber <[email protected]>
gm4sl pushed a commit to backtrace-labs/crashpad that referenced this pull request Apr 1, 2025
@markmentovai @gm4sl
Fix SyntaxWarning in build/run_tests.py
f834d0a
Use r-strings for all regular expressions.

Fixes these warnings experienced with Python 3.12
(python/cpython#98401,
python/cpython#99011,
https://docs.python.org/3/whatsnew/3.12.html#other-language-changes
point 2):

run_tests.py:200: SyntaxWarning: invalid escape sequence '\d'
  FINAL_LINE_RE = re.compile('status=(\d+)$')
run_tests.py:441: SyntaxWarning: invalid escape sequence '\*'
  re.match('^\* daemon .+ \*$', line) or line == ''):

Change-Id: I71ddfb1a2ca62654378ae67a99e9aeb4ce7b7394
Reviewed-on: https://chromium-review.googlesource.com/c/crashpad/crashpad/+/6254063
Commit-Queue: Mark Mentovai <[email protected]>
Reviewed-by: Nico Weber <[email protected]>
aarongable pushed a commit to chromium/chromium that referenced this pull request Jun 16, 2025
@markmentovai Chromium LUCI CQ
bisect-builds: use r-string for regular expression
3738988
All other regular expressions in the file already used r-strings, but
this newer one added in beba95a did not.

Fixes this warning experienced with Python ≥3.12
(python/cpython#98401,
python/cpython#99011,
python/cpython@a60ddd31be7f):

bisect-builds.py:1260: SyntaxWarning: invalid escape sequence '\.'
  if not re.search("\.apks?$", apk_name):

Change-Id: Ia8f1981efde3b14d5c36c3dfc2bca7737fd4cb89
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6647645
Commit-Queue: Mark Mentovai <[email protected]>
Reviewed-by: Kuan Huang <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1474605}
kaidokert pushed a commit to kaidokert/depot_tools that referenced this pull request Jun 18, 2025
@markmentovai LUCI CQ
autoninja: fix strings with invalid escape sequences
22df6f8
Since 61fad56 (https://chromium-review.googlesource.com/c/5848450,
2024-09-11), autoninja under Python 3.12 presents these warnings:

```
…/autoninja.py:73: SyntaxWarning: invalid escape sequence '\s'
  m = re.match('instance\s*=\s*projects/([^/]*)/instances/.*', line)
…/autoninja.py:92: SyntaxWarning: invalid escape sequence '\s'
  m = re.match('SISO_PROJECT=\s*(\S*)\s*', line)
```

This warning appears in Python 3.12 ([1], [2], [3]).

'\s' and '\S' are not valid escape sequences in strings. r'\s' and r'\S'
are valid in regular expressions, but outside of raw strings, they would
need to be written as '\\s' and '\\S'.

There is no reason to not use raw strings in this case, so the new
regular expression pattern strings introduced in 61fad56 are
changed to raw strings.

[1] https://docs.python.org/3/whatsnew/3.12.html#:~:text=A%20backslash%2Dcharacter%20pair%20that%20is%20not%20a%20valid%20escape%20sequence%20now%20generates%20a%20SyntaxWarning%2C%20instead%20of%20DeprecationWarning.
[2] python/cpython#98401
[3] python/cpython#99011

Bug: b/364318216
Change-Id: I0f237976fe9c39208541ae78205f5bdbf126fa82
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5859159
Commit-Queue: Mark Mentovai <[email protected]>
Reviewed-by: Philipp Wollermann <[email protected]>
Auto-Submit: Mark Mentovai <[email protected]>
@braingram braingram mentioned this pull request Nov 13, 2025
ignore reverted warning issued in python 3.11 spacetelescope/romancal#2059
Merged
10 tasks
@coderabbitai coderabbitai Bot mentioned this pull request Jul 13, 2026
♻️ Centralize and structure code generation/infrastructure munich-quantum-toolkit/qecc#751
Merged
8 of 9 tasks
This was referenced Jul 27, 2026
Feature units concrete ch8 Blueprints-org/blueprints#1093
Merged
Remove obsolete docs and simplify the homepage hero surajmandalcell/switch#18
Closed
@coderabbitai coderabbitai Bot mentioned this pull request Aug 5, 2026
Fix Gutenberg contents-list ingest and green up CI on Python 3.14 gilesknap/somnia#1
Merged
@coderabbitai coderabbitai Bot mentioned this pull request Aug 16, 2026
fix: keep backslashes in the generated SQL constants rayakame/sqlc-gen-better-python#262
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lysnikolaou lysnikolaou lysnikolaou left review comments
@serhiy-storchaka serhiy-storchaka serhiy-storchaka approved these changes
@pablogsal pablogsal Awaiting requested review from pablogsal pablogsal is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@vstinner @serhiy-storchaka @lysnikolaou @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.