GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-131507: Add support for syntax highlighting in PyREPL by ambv · Pull Request #133247 · python/cpython

Conversation

@ambv

@ambv ambv commented May 1, 2025 •
edited by github-actions Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

This is a much improved version of gh-131562. It uses the tokenizer for better speed and pattern matching for more robust handling of soft keywords. While it still won't hit all cases correctly, it's better than idlelib's regular expression-based colorizer and unlike our glorious PEG parser it supports incomplete input, which is crucial for an interactive shell.

Relatedly, pasting support was tweaked to be way faster. Now the entire contents of Frankenstein can be pasted within 3 seconds both on Unix and Windows as long as bracketed pasting is supported by the terminal. This is a necessary tweak for syntax highlighting not to cripple performance of pastes above 5kB.

There is experimental support for theming through _colorize.set_theme() that's mentioned in "What's New" but otherwise undocumented so far.

  • Issue: Syntax highlighting in PyREPL #131507

📚 Documentation preview 📚: https://cpython-previews--133247.org.readthedocs.build/

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
ZeroIntensity, chris-eibl, and loureq177 reacted with rocket emoji
All reactions
  • 🚀 3 reactions
ambv added 15 commits April 29, 2025 18:05
@ambv
pythongh-131507: Add support for syntax highlighting in PyREPL
e921a80
@ambv
Add Blurb
fb95911
@ambv
Fix irrelevant Windows tests
b428513
@ambv
Replace idlelib.colorizer with a faster solution
2bdcd06
@ambv
Slurp the entire input buffer before refreshing display during bracke…
8c70c45 
…ted paste
@ambv
Implement getpending() on Windows
4d7ae36
@ambv
Adapt tests
9585bd6
@ambv
Support soft keywords (fight fire with fire)
b1f2557
@ambv
Fix test
01e1129
@ambv
Remove unnecessary import
20eff49
@ambv
Add test for prev_next_window
8d3648a
@ambv
Windows: bracketed pasting of 448692 chars done in 2.38s ✊
656fea3
@ambv
Remove colors from Windows low-level console tests
dac8961
@ambv
Fix lint and stuff
7891fa7
@ambv
Add experimental theming support for syntax highlighting and the prompt
362a21b
@ambv
ambv requested review from hugovk, lysnikolaou and pablogsal as code owners May 1, 2025 11:02
@bedevere-app bedevere-app Bot mentioned this pull request May 1, 2025
Syntax highlighting in PyREPL #131507
Closed
@bedevere-app bedevere-app Bot added the awaiting core review label May 1, 2025
@ambv
Fix lint
ffebbbe
@ambv ambv added 🔨 test-with-buildbots Test PR w/ buildbots; report in status section topic-repl Related to the interactive shell labels May 1, 2025
@bedevere-bot

bedevere-bot commented May 1, 2025

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @ambv for commit ffebbbe 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133247%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 1, 2025
tomasr8
tomasr8 reviewed May 1, 2025
View reviewed changes

@tomasr8 tomasr8 left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for being too nitpicky, but I still find the blue used for keywords to be too dark when coupled with dark-themed terminals (e.g. #131562 (comment)). Do you think we could use a more contrasting color?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/_pyrepl/utils.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/_pyrepl/utils.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@ambv

ambv commented May 1, 2025

Copy link
Copy Markdown
Contributor Author

I disagree about the blue color, the fact that Ubuntu colors it too dim is not actionable for the Python project. Adjust your terminal.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ambv

ambv commented May 1, 2025

Copy link
Copy Markdown
Contributor Author

The refleak failures are unrelated, see #133258.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
ambv added 2 commits May 1, 2025 19:17
@ambv
Merge branch 'main' into pyrepl-syntax-highlighting-tokens
388e494
@ambv
Add t-string support to syntax highlighting
9b60382
@ambv
Remove - and + from first sets matching for match
ff1f92b
@chris-eibl

chris-eibl commented May 2, 2025

Copy link
Copy Markdown
Member

@chris-eibl I simplified the Unix case to remove the additional buffering since it complicates the codebase. Your PR was adding that same buffering to Windows.

Sorry for only spotting the windows_console.py change - I must be biased :)

Yeah, simple is always better 👍

Given that we now achieve the same performance without double-buffering, I'd say we don't need the other PR.

I can confirm: this PR is now way faster pasting in the virtual terminal mode on Windows 🚀
ambv reacted with thumbs up emoji tomasr8 reacted with rocket emoji
All reactions
  • 👍 1 reaction
  • 🚀 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@chris-eibl chris-eibl mentioned this pull request May 2, 2025
GH-130328: further speedup of pasting in new REPL on Windows by reading in chunks #132889
Closed
pablogsal
pablogsal reviewed May 2, 2025
View reviewed changes
Comment thread Lib/_pyrepl/utils.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
pablogsal
pablogsal reviewed May 2, 2025
View reviewed changes
Comment thread Lib/_pyrepl/utils.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/_pyrepl/utils.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/_pyrepl/commands.py
done = "\x1b[201~"
data = ""
import time
start = time.time()

@pablogsal pablogsal May 2, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leftover from testing?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@ambv ambv May 2, 2025

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The trace below shows time. I can move the import up.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
pablogsal reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@pablogsal pablogsal May 2, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then use perf_counter please

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/_pyrepl/utils.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/_pyrepl/utils.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
pablogsal
pablogsal reviewed May 2, 2025
View reviewed changes
Comment thread Lib/_pyrepl/reader.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
pablogsal
pablogsal reviewed May 2, 2025
View reviewed changes
Comment thread Lib/_pyrepl/unix_console.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
@ambv
Address Pablo's review
bd84cd8
pablogsal
pablogsal reviewed May 2, 2025
View reviewed changes
Comment thread Lib/_pyrepl/windows_console.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
pablogsal
pablogsal reviewed May 2, 2025
View reviewed changes
Comment thread Lib/_pyrepl/utils.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
pablogsal
pablogsal reviewed May 2, 2025
View reviewed changes
Comment thread Lib/_pyrepl/utils.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
pablogsal
pablogsal approved these changes May 2, 2025
View reviewed changes

@pablogsal pablogsal left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a bunch of questions and nitpicks but overall this looks fantastic.

I've reviewed the tokenizer-related parts of the implementation and I'm comfortable with the current approach. We have already discussed this offline but for everyone else reading: while the soft keyword detection uses heuristics, this is a reasonable compromise as bringing in a more correct solution would require a full parser run, which would be much heavier and slower for this use case and we don't even have the technology now to do partial input in the PEG parser so whatever we do will be in python and much slower.

I've also run some performance benchmarks and things look really good — the impact on responsiveness is minimal, even with syntax highlighting enabled by default. And on Windows seem to be super fast.

I'm happy to explore further optimizations in the future, such as avoiding repeated tokenization during screen refreshes. But as it stands, this looks solid. Great work!

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
chris-eibl, hugovk, and tgt843 reacted with hooray emoji
All reactions
  • 🎉 3 reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels May 2, 2025
@pablogsal

pablogsal commented May 2, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

The only open question is what to do for setting the theme officially (right now is a "experimental" API) but I think since that affects other parts of the interpreter (such as tracebacks) this is out of scope of this particular PR, so I propose to discuss this separately
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ambv
Exclude applying soft keywords inside any bracket pairs, not just par…
080f300 
…entheses
@chris-eibl

chris-eibl commented May 2, 2025

Copy link
Copy Markdown
Member

In a legacy Windows console, the prompt is no longer colored, but e.g. a SyntaxError still is.

Also, syntax highlighting is turned off (this maybe is wanted behaviour?)

image
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@chris-eibl

chris-eibl commented May 2, 2025

Copy link
Copy Markdown
Member

can_colorize is True, but _colorize.theme has empty strings for all values.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ambv

ambv commented May 2, 2025

Copy link
Copy Markdown
Contributor Author

Good catch, @chris-eibl. I'll fix it forward in a subsequent PR.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ambv
ambv merged commit fac41f5 into python:main May 2, 2025
@bedevere-app bedevere-app Bot removed the awaiting merge label May 2, 2025
@chris-eibl

chris-eibl commented May 2, 2025

Copy link
Copy Markdown
Member

Calling _colorize.set_theme() right after virtual terminal processing is enabled via

cpython/Lib/_pyrepl/windows_console.py

Lines 150 to 155 in fac41f5

SetConsoleMode(
OutHandle,
ENABLE_WRAP_AT_EOL_OUTPUT
| ENABLE_PROCESSED_OUTPUT
| ENABLE_VIRTUAL_TERMINAL_PROCESSING,
)

does fix it for me.
ambv reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@ambv
ambv deleted the pyrepl-syntax-highlighting-tokens branch May 3, 2025 09:07
This was referenced May 4, 2025
GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe #132440
Merged
Pasting in new REPL is slow #130328
Closed
@hugovk hugovk mentioned this pull request May 5, 2025
Support completion for sqlite3 command-line interface #133390
Closed
@stonebig stonebig mentioned this pull request May 26, 2025
Problems with some characters: [ ] { } ... winpython/winpython#1606
Closed
@tomasr8 tomasr8 mentioned this pull request Jun 3, 2025
Python 3.14 REPL blanks links above terminal rows when pasting #135055
Open
Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request Jul 12, 2025
@ambv @Viicos
@hugovk @Pranjal095
pythongh-131507: Add support for syntax highlighting in PyREPL (pytho…
be13396 
…nGH-133247)

Co-authored-by: Victorien <[email protected]>
Co-authored-by: Hugo van Kemenade <[email protected]>
@breindt4 breindt4 mentioned this pull request Sep 24, 2025
Add more granularity to _pyrepl syntax theming for "keywords" #134953
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hugovk hugovk hugovk left review comments
@tomasr8 tomasr8 tomasr8 left review comments
@pablogsal pablogsal pablogsal approved these changes
@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner
+1 more reviewer
@Viicos Viicos Viicos left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

topic-repl Related to the interactive shell

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

8 participants

@ambv @bedevere-bot @chris-eibl @skirpichev @pablogsal @hugovk @tomasr8 @Viicos
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.