GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-151464: exclude '<>' token from tokenize output by skirpichev · Pull Request #154854 · python/cpython

gh-151464: exclude '<>' token from tokenize output #154854

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pablogsal merged 14 commits into python:main from skirpichev:save-python/151464
Sep 25, 2026

Uh oh!

There was an error while loading. Please reload this page.

Merged

gh-151464: exclude '<>' token from tokenize output #154854

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
73f2d90
gh-151464: exclude '<>' token from tokenize output
skirpichev Jul 29, 2026
c53a17f
+1
skirpichev Jul 29, 2026
e525b8f
Merge branch 'master' into save-python/151464
skirpichev Aug 10, 2026
f4dd644
address review: lowercase and move invalid rule
skirpichev Aug 11, 2026
ba5843d
address review: news
skirpichev Aug 11, 2026
9ea138f
address review: move test_guido_as_bdfl_ineq_tokens()
skirpichev Aug 11, 2026
3f9c57e
address review: revert _PyTokenizer_From* changes
skirpichev Aug 11, 2026
a966092
Merge branch 'main' into save-python/151464
skirpichev Aug 21, 2026
d81b19c
Merge branch 'master' into save-python/151464
skirpichev Sep 1, 2026
3ff8480
+ revert unrelated change
skirpichev Sep 1, 2026
d134772
Merge branch 'master' into save-python/151464
skirpichev Sep 15, 2026
e1fc4cc
Merge branch 'master' into save-python/151464
skirpichev Sep 19, 2026
8b5f1e4
address review: remove whatsnew entry
skirpichev Sep 19, 2026
76b2ee6
Merge branch 'master' into save-python/151464
skirpichev Sep 25, 2026
File filter
Filter by extension

Clear filters
Conversations
Failed to load comments.
Loading

Uh oh!

There was an error while loading. Please reload this page.

Jump to
Jump to file
Failed to load files.
Loading

Uh oh!

There was an error while loading. Please reload this page.

Diff view
Diff view

There are no files selected for viewing

8 changes: 8 additions & 0 deletions 8 Grammar/python.gram
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -789,6 +789,7 @@ compare_op_bitwise_or_pair[CmpopExprPair*]:
| (tok='!=' { _PyPegen_check_barry_as_flufl(p, tok) ? NULL : tok }) a=bitwise_or {
_PyPegen_cmpop_expr_pair(p, NotEq, a) }
| '<=' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, LtE, a) }
| invalid_noteq
| '<' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, Lt, a) }
| '>=' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, GtE, a) }
| '>' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, Gt, a) }
Expand Down Expand Up @@ -1615,3 +1616,10 @@ invalid_bitwise_or:
? RAISE_SYNTAX_ERROR_KNOWN_RANGE(b, c, "invalid syntax. Maybe you meant 'or' or '|' instead of '||'?")
: NULL
}

invalid_noteq:
| a='<' b='>' {
_PyPegen_tokens_are_adjacent(a, b)
? RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "invalid syntax. Maybe you meant '!=' instead of '<>'?")
: NULL
}
2 changes: 1 addition & 1 deletion 2 Include/internal/pycore_token.h
Show comments
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

25 changes: 25 additions & 0 deletions 25 Lib/test/test_syntax.py
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3644,6 +3644,31 @@ def test_ifexp_body_stmt_else_stmt(self):
]:
self._check_error(f"x = {lhs_stmt} if 1 else {rhs_stmt}", msg)

def test_diamond_operator(self):
self._check_error(
"1<>2",
r"Maybe you meant '!=' instead of '<>'\?",
lineno=1,
end_lineno=1,
offset=2,
end_offset=4,
)

def test_diamond_operator_barry_as_flufl(self):
# Under barry_as_FLUFL, '<>' is the valid "not equal" operator
compile(
"from __future__ import barry_as_FLUFL\n1<>2",
"<test>", "exec",
)
self._check_error(
"from __future__ import barry_as_FLUFL\na != b",
"with Barry as BDFL, use '<>' instead of '!='",
lineno=2,
end_lineno=2,
offset=3,
end_offset=5,
)

def test_double_ampersand(self):
self._check_error(
"a && b",
Expand Down
14 changes: 14 additions & 0 deletions 14 Lib/test/test_tokenize.py
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1234,6 +1234,20 @@ def test_multiline_non_ascii_fstring_with_expr(self):
FSTRING_END \'"\' (2, 2) (2, 3)
""")

def test_ineq_tokens(self):
self.check_tokenize("1 != 2", """\
NUMBER '1' (1, 0) (1, 1)
OP '!=' (1, 2) (1, 4)
NUMBER '2' (1, 5) (1, 6)
""")
self.check_tokenize("1 <> 2", """\
NUMBER '1' (1, 0) (1, 1)
OP '<' (1, 2) (1, 3)
OP '>' (1, 3) (1, 4)
NUMBER '2' (1, 5) (1, 6)
""")


class GenerateTokensTest(TokenizeTest):
def check_tokenize(self, s, expected):
# Format the tokens in s in a table format.
Expand Down
2 changes: 2 additions & 0 deletions 2 Misc/NEWS.d/next/Core_and_Builtins/2026-07-29-08-43-14.gh-issue-151464.o_mtmz.rst
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Exclude invalid token ``<>`` from :mod:`tokenize` output. :exc:`SyntaxError`
for ``<>`` now suggests ``!=``.
2 changes: 2 additions & 0 deletions 2 Parser/action_helpers.c
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

#include "pegen.h"
#include "string_parser.h" // _PyPegen_decode_string()
#include "lexer/state.h" // tok_state


void *
Expand Down Expand Up @@ -2133,6 +2134,7 @@ _PyPegen_checked_from_import(Parser *p, asdl_seq *dots, expr_ty module_name,
alias_ty alias = asdl_seq_GET(names, i);
if (PyUnicode_CompareWithASCIIString(alias->name, "barry_as_FLUFL") == 0) {
p->flags |= PyPARSE_BARRY_AS_BDFL;
p->tok->barry_as_bdfl = 1;
}
}
}
Expand Down
2 changes: 1 addition & 1 deletion 2 Parser/lexer/lexer.c
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -409,7 +409,7 @@ _PyLexer_get_normal(struct tok_state *tok, ftstring_state *current, struct token
/* Check for two-character token */
{
int c2 = tok_nextc(tok);
int current_token = _PyToken_TwoChars(c, c2);
int current_token = _PyToken_TwoChars(c, c2, tok->barry_as_bdfl);
if (current_token != OP) {
int c3 = tok_nextc(tok);
int current_token3 = _PyToken_ThreeChars(c, c2, c3);
Expand Down
1 change: 1 addition & 0 deletions 1 Parser/lexer/state.h
Show comments
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ struct tok_state {
#ifdef Py_DEBUG
int debug;
#endif
int barry_as_bdfl;
};

static inline ftstring_state *
Expand Down
Loading
Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Loading
Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.