GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — GH-130328: pasting in new REPL is slow on Windows by chris-eibl · Pull Request #132884 · python/cpython Conversation
|
| while True: | |
| # We use the same timeout as in readline.c: 100ms | |
| self.run_hooks() | |
| self.console.wait(100) | |
| event = self.console.get_event(block=False) |
and
cpython/Lib/_pyrepl/windows_console.py
Lines 523 to 532 in 9f5994b
| def wait(self, timeout: float | None) -> bool: | |
| """Wait for an event.""" | |
| # Poor man's Windows select loop | |
| start_time = time.time() | |
| while True: | |
| if msvcrt.kbhit(): # type: ignore[attr-defined] | |
| return True | |
| if timeout and time.time() - start_time > timeout / 1000: | |
| return False | |
| time.sleep(0.01) |
It is interesting, that msvcrt.kbhit() returns True in case of pasting, but if that weren't the case, we'd hit the 100ms timeout and would be even way slower. However, msvcrt.kbhit() is very slow in case of the legacy console, and it is called at least twice as often, because we get a key up and a key down event - but only a key down event in the virtual terminal case. If the pasted input contains upper case letters, we additionally get a "shift key pressed" event - so in the worst case, msvcrt.kbhit() gets called 3 times more often (and each call is slower).
Output of python.bat -m cProfile -m _pyrepl when pasting the "test value" given in the OP for a legacy console:
ncalls tottime percall cumtime percall filename:lineno(function)
50/1 0.010 0.000 17.202 17.202 {built-in method builtins.exec}
1 0.000 0.000 17.202 17.202 <frozen runpy>:201(run_module)
1 0.000 0.000 17.199 17.199 <frozen runpy>:65(_run_code)
1 0.000 0.000 17.199 17.199 __main__.py:1(<module>)
1 0.000 0.000 17.112 17.112 main.py:24(interactive_console)
1 0.000 0.000 17.112 17.112 simple_interact.py:99(run_multiline_interactive_console)
5 0.000 0.000 17.111 3.422 readline.py:375(multiline_input)
5 0.004 0.001 17.111 3.422 reader.py:741(readline)
3090 0.036 0.000 17.103 0.006 reader.py:694(handle1)
6210 0.013 0.000 13.714 0.002 windows_console.py:523(wait)
6236 13.433 0.002 13.433 0.002 {built-in method msvcrt.kbhit}
Here the relevant line in case of virtual terminal:
3270 0.554 0.000 0.554 0.000 {built-in method msvcrt.kbhit}
The fix is to use WaitForSingleObject
def wait(self, timeout: float | None) -> bool: """Wait for an event.""" ret = WaitForSingleObject(InHandle, int(timeout)) if ret == WAIT_FAILED: raise WinError(ctypes.get_last_error())
which speeds up especially the legacy console (times in seconds):
| legacy terminal | virtual terminal | |
|---|---|---|
| before | 15.6 | 0.89 |
| after | 1.8 | 0.26 |
Note, see also #132440 (comment):
- legacy terminal: manually start
cmd.exe. Timings gotten by pasting
import time
t1 = time.time()
<"test value" given in the OP>
print(time.time() - t1)
- virtual terminal: power shell via Windows terminal. Timings gotton by temporarily patching
commands.py:
import time class enable_bracketed_paste(Command): def do(self) -> None: self.reader.bp_begin = time.perf_counter() self.reader.paste_mode = True self.reader.in_bracketed_paste = True class disable_bracketed_paste(Command): def do(self) -> None: print("bracketed_paste took %5.2f s" % (time.perf_counter() - self.reader.bp_begin, )) self.reader.paste_mode = False self.reader.in_bracketed_paste = False self.reader.dirty = True
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
|
Good analysis and speedup! 👍 |
All reactions
-
❤️ 1 reaction
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| if not events.value: | ||
| return None | ||
| if not block and not self.wait(timeout=0): | ||
| return None |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now we are nicely in sync with unix_console.py
cpython/Lib/_pyrepl/unix_console.py
Lines 400 to 401 in e1c09ff
| if not block and not self.wait(timeout=0): | |
| return None |
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
All reactions
Uh oh!
There was an error while loading. Please reload this page.
|
We generally don't backport perf fixes unless they lead to a denial-of-service problem (so usually like an exponential-time algorithm), or result in a serious bug. So I'm removing the backport label. |
All reactions
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
On second thought, "slow" is perhaps an understatement. So once this is merged, let's ping Thomas to ask for his permission to backport to 3.13 |
All reactions
-
👍 1 reaction
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
chris-eibl
commented
Apr 24, 2025
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
edited
Uh oh!
There was an error while loading. Please reload this page.
|
Yeah, in case of the legacy console pasting of the 30 + 3 lines given by the OP takes 15 seconds, and this brings it down to 1.8. Most people will hopefully use a virtual terminal, where the situation is much less severe. After all, the new REPL can be disabled using https://docs.python.org/3/using/cmdline.html#envvar-PYTHON_BASIC_REPL. 3 more lines added by me for easier benchmarking #130328 (comment) "test value" given in the OP
"""1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890 |
All reactions
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
If @ambv has not checked this this in a couple of days ping me and I will do a pass and land it. Thanks a lot for looking into this @chris-eibl ❤️ |
All reactions
-
❤️ 1 reaction
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
in class Reader
This reverts commit bba26c9.
|
Thanks for the fix, Chris! |
All reactions
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
@Yhg1s seeking permission to backport this to 3.13. According to Chris, it takes 15 seconds to paste 58 lines in the new repl in 3.13. Do you think that's slow enough to be worth backporting? |
All reactions
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Reviewers
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assignedLabels
Projects
Milestone
No milestoneDevelopment
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
6 participants
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.