GeezerVison is showing github.com — Pasting in new REPL is slow · Issue #130328 · python/cpython
Pasting in new REPL is slow #130328DescriptionBug report
Bug description:
Pasting large blocks of text in the new REPL is slow. Tested by pasting the following string (3035 characters):
test value
"""1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890"""
When using the old repl (started via $env:PYTHON_BASIC_REPL="true"; python in powershell), pasting the string is near immediate. When using the new repl (started via python in powershell), pasting the string takes 15 seconds.
The closed issue #119517 also mentioned some pasting performance issues, though that was for Linux with significantly more characters.
CPython versions tested on:
3.13
Operating systems tested on:
Windows
Linked PRs
ActivityStanFromIreland commented on Feb 20, 2025 Possibly a windows issue, something to do with powershell? On linux (main branch build) it pastes as expected, instantly.
cc windows expert @zooba zooba commented on Feb 20, 2025 It's no good looking at me, I know nothing about the new repl (other than that I've suggested a couple of times that it should stop using ctypes and use native code instead, which could be the cause of any performance issue).
@pablogsal @lysnikolaou @ambv are code owners for the REPL. natedogith1 commented on Feb 21, 2025 I also see this when python is started from the win-r run dialog.
Tested some other environments, using the first 604 lines of this copy of Frankenstein (32,003 characters) (same as #119517 did). WSL old REPL takes ~2 seconds. WSL new REPL takes ~4 seconds, Powershell old REPL takes ~1 second. Powershell new REPL takes ~55 seconds for just the first 25 lines to appear; when pasting only the first 25 lines (659 characters) it takes ~2 seconds. devdanzin commented on Feb 22, 2025 I think it's caused by the Windows console reading one character at a time from input:
|
rec = INPUT_RECORD() |
|
read = DWORD() |
|
if not ReadConsoleInput(InHandle, rec, 1, read): |
|
raise WinError(GetLastError()) |
But I don't know enough about ctypes or Windows API to know how to increase that value, naively using larger ints results in errors.
cc @DinoV, who implemented the Windows console. devdanzin commented on Feb 22, 2025 I was able to make ReadConsoleInput read 1024 chars at a time and also reduced self.console.wait() in Reader.handle1 from 100 to 0.1. That reduced time to paste the content from first post from 10s to 6s, which isn't a significant improvement given the numbers from WSL and basic REPL. devdanzin commented on Feb 22, 2025 #124119 seems to improve things, but nowhere near WSL numbers.
Edit: With that PR, time to past the sample from first post goes from 10s to 2.5s. Reading 32 chars of input instead of 1 and reducing Reader.console.wait from 100 to 10 further reduces time to 1.7s. So we can paste 5x faster than current version, which pales compared to the near instant paste in WSL. devdanzin commented on Feb 23, 2025 Can you check whether devdanzin#1 speeds up pasting significantly for you?
It's a quick and dirty POC built on top of #124119 that:
- Batches event processing;
- Reads 32 chars from input instead of 1;
- Enables virtual terminal (from the mentioned PR);
- Reduces
Reader.console.wait() from 100 to 10.
It fails some tests and the code is ugly, but if it makes things better we can try to improve on this. devdanzin commented on Feb 28, 2025 Please test the #130677 PR to check whether it significantly improves pasting time for you. chris-eibl commented on Apr 24, 2025 The reason why pasting is so slow on Windows (especially in the legacy console case where virtual terminal mode - and thus bracketed paste - is disabled):
|
while True: |
|
# We use the same timeout as in readline.c: 100ms |
|
self.run_hooks() |
|
self.console.wait(100) |
|
event = self.console.get_event(block=False) |
and
|
def wait(self, timeout: float | None) -> bool: |
|
"""Wait for an event.""" |
|
# Poor man's Windows select loop |
|
start_time = time.time() |
|
while True: |
|
if msvcrt.kbhit(): # type: ignore[attr-defined] |
|
return True |
|
if timeout and time.time() - start_time > timeout / 1000: |
|
return False |
|
time.sleep(0.01) |
It is interesting, that msvcrt.kbhit() returns True in case of pasting, but if that weren't the case, we'd hit the 100ms timeout and would be even way slower. However, msvcrt.kbhit() is very slow in case of the legacy console, and it is called at least twice as often, because we get a key up and a key down event - but only a key down event in the virtual terminal case. If the pasted input contains upper case letters, we additionally get a "shift key pressed" event - so in the worst case, msvcrt.kbhit() gets called 3 times more often (and each call is slower).
Output of python.bat -m cProfile -m _pyrepl when pasting the "test value" given in the OP for a legacy console:
ncalls tottime percall cumtime percall filename:lineno(function)
50/1 0.010 0.000 17.202 17.202 {built-in method builtins.exec}
1 0.000 0.000 17.202 17.202 <frozen runpy>:201(run_module)
1 0.000 0.000 17.199 17.199 <frozen runpy>:65(_run_code)
1 0.000 0.000 17.199 17.199 __main__.py:1(<module>)
1 0.000 0.000 17.112 17.112 main.py:24(interactive_console)
1 0.000 0.000 17.112 17.112 simple_interact.py:99(run_multiline_interactive_console)
5 0.000 0.000 17.111 3.422 readline.py:375(multiline_input)
5 0.004 0.001 17.111 3.422 reader.py:741(readline)
3090 0.036 0.000 17.103 0.006 reader.py:694(handle1)
6210 0.013 0.000 13.714 0.002 windows_console.py:523(wait)
6236 13.433 0.002 13.433 0.002 {built-in method msvcrt.kbhit}
Here the relevant line in case of virtual terminal:
3270 0.554 0.000 0.554 0.000 {built-in method msvcrt.kbhit}
The fix is to use WaitForSingleObject
def wait(self, timeout: float | None) -> bool:
"""Wait for an event."""
ret = WaitForSingleObject(InHandle, int(timeout))
if ret == WAIT_FAILED:
raise WinError(ctypes.get_last_error())which speeds up especially the legacy console (times in seconds):
|
legacy terminal |
virtual terminal |
| before |
15.6 |
0.89 |
| after |
1.8 |
0.26 |
Note, see also #132440 (comment):
- legacy terminal: manually start
cmd.exe. Timings gotten by pasting
import time
t1 = time.time()
<"test value" given in the OP>
print(time.time() - t1)
- virtual terminal: power shell via Windows terminal. Timings gotton by temporarily patching
commands.py:
import time
class enable_bracketed_paste(Command):
def do(self) -> None:
self.reader.bp_begin = time.perf_counter()
self.reader.paste_mode = True
self.reader.in_bracketed_paste = True
class disable_bracketed_paste(Command):
def do(self) -> None:
print("bracketed_paste took %5.2f s" % (time.perf_counter() - self.reader.bp_begin, ))
self.reader.paste_mode = False
self.reader.in_bracketed_paste = False
self.reader.dirty = Truechris-eibl commented on Apr 24, 2025 Because pasting is so slow (especially in case of a legacy console), I've marked this as a release blocker. chris-eibl commented on May 9, 2025 @ambv I gave it a try and came up with #133728. Works for me in Windows and WSL in legacy and vt terminals and is as fast as bracketed pasting in paste mode (F3). stonebig commented on May 26, 2025 will it be backported to 3.13 too ? ambv commented on May 26, 2025 No, this is a performance optimization with pretty large internal changes required.
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|