GeezerVison is showing github.com — gh-129005: Align FileIO.readall allocation by cmaloney · Pull Request #129424 · python/cpython
Conversation
cmaloney
commented
Jan 29, 2025
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
Both _io and _pyio now use a pre-allocated buffer of length bufsize, fill it using a os.readinto / _Py_read, and have matching "expand buffer" logic.
On my machine (Linux, Debug build) this takes: ./python -m test -M8g -uall test_largefile -m test_large_read -v from ~3.7 seconds to ~3.3 seconds
_pyio still uses 2x the memory, there are two remaining copies
- the
bytes(result) currently copies. I'd like to either just rely on "duck typing" / bytearray is close enough to bytes, or would need to do something similar to C++ "move" semantics where the bytes could take ownership of the data buffer from the bytearray without copying... Not sure what is most Pythonic
_pyio.BufferedIO._read_unlocked in the read-all case where no buffer has been allocated always does return buf[:pos] + chunk which causes another copy. Patch for that case / "if buf is length 0, just return" coming shortly.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Both now use a pre-allocated buffer of length `bufsize`, fill it using
a readinto, and have matching "expand buffer" logic.
On my machine this takes:
`./python -m test -M8g -uall test_largefile -m test_large_read -v`
from ~3.7 seconds to ~3.3 seconds
Uh oh!
There was an error while loading. Please reload this page.
cmaloney
commented
Jan 29, 2025
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
|
Not sure how to get the Tests / Ubuntu / build and test unbroken, they seem to be auto-cancelling fast... which isn't happening on any of my other open PRs.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
rebased on main and opened #129458 as a replacement that hopefully will revoid the cancelled test runs.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.