GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — [3.14] gh-151464: exclude '<>' token from tokenize output (GH-154854) by pablogsal · Pull Request #158190 · python/cpython

Conversation

@pablogsal

@pablogsal pablogsal commented Sep 25, 2026 •
edited by bedevere-app Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Manual backport of #154854 to 3.14. The comparison grammar differs from main, so this adds the diagnostic to the 3.14 rules and regenerates the parser. The tokenizer, syntax, Barry, and future-import suites pass locally.

  • Issue: Tokenizer produces token '<>', missing in the language grammar (regardless on __future__.barry_as_FLUFL settings) #151464

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@pablogsal
pablogsal requested a review from lysnikolaou as a code owner September 25, 2026 18:56
@bedevere-app bedevere-app Bot mentioned this pull request Sep 25, 2026
Tokenizer produces token '<>', missing in the language grammar (regardless on __future__.barry_as_FLUFL settings) #151464
Closed
@bedevere-app bedevere-app Bot added the awaiting core review label Sep 25, 2026
@bedevere-app bedevere-app Bot mentioned this pull request Sep 25, 2026
gh-151464: exclude '<>' token from tokenize output #154854
Merged
@pablogsal
pablogsal force-pushed the gh-154854-3.14 branch from 3cf2fdf to 70568b6 Compare September 25, 2026 18:59
@pablogsal
pablogsal enabled auto-merge (squash) September 25, 2026 19:02
@pablogsal

pablogsal commented Sep 25, 2026

Copy link
Copy Markdown
Member Author

The required-check aggregate is red because the two Windows ARM jobs fail before compiling CPython: this runner has Visual Studio 18 but lacks the v143 toolset requested by the 3.14 branch. ABI, lint, review gates, and the other required platform jobs passed. The focused tokenizer, syntax, Barry, and future-import suites also pass locally. I am merging this backport with the infrastructure failure noted here.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@skirpichev @pablogsal
[3.14] pythongh-151464: exclude '<>' token from tokenize output (pyth…
1ed73a8
…onGH-154854)

* pythongh-151464: exclude '<>' token from tokenize output

Was:
```
$ echo '1 <> 2' | python -m tokenize
1,0-1,1:            NUMBER         '1'
1,2-1,4:            OP             '<>'
1,5-1,6:            NUMBER         '2'
1,6-1,7:            NEWLINE        '\n'
2,0-2,0:            ENDMARKER      ''
```

Now (regardless on ``__future__.barry_as_FLUFL`` import):
```
$ echo '1 <> 2' | ./python -m tokenize
1,0-1,1:            NUMBER         '1'
1,2-1,3:            OP             '<'
1,3-1,4:            OP             '>'
1,5-1,6:            NUMBER         '2'
1,6-1,7:            NEWLINE        '\n'
2,0-2,0:            ENDMARKER      ''
```
in accordance with the Grammar:
https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters

Also adds a custom error message for ``<>`` ("not equal" in Pascal
and Python 2).

* +1

* address review: lowercase and move invalid rule

* address review: news

* address review: move test_guido_as_bdfl_ineq_tokens()

* address review: revert _PyTokenizer_From* changes

* + revert unrelated change

* address review: remove whatsnew entry

(cherry picked from commit 198bc76)
@pablogsal
pablogsal force-pushed the gh-154854-3.14 branch from 70568b6 to 1ed73a8 Compare September 25, 2026 20:10
@pablogsal

pablogsal commented Sep 25, 2026

Copy link
Copy Markdown
Member Author

The VS 2026 toolset and UWP fixes have now merged in #158188. I rebased this backport onto the updated 3.14 branch, and its checks are running again.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal
pablogsal merged commit 66f8dbe into python:3.14 Sep 25, 2026
49 checks passed

Uh oh!

There was an error while loading. Please reload this page.

@pablogsal
pablogsal deleted the gh-154854-3.14 branch September 25, 2026 20:38
@bedevere-app bedevere-app Bot removed the awaiting core review label Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@pablogsal @skirpichev
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.