GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-129005: Align FileIO.readall between _pyio and _io by cmaloney · Pull Request #129705 · python/cpython

Conversation

@cmaloney

@cmaloney cmaloney commented Feb 5, 2025 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

Utilize bytearray.resize() and os.readinto() to reduce copies and match behavior of _io.FileIO.readall().

There is still an extra copy which means twice the memory required compared to FileIO because there isn't a zero-copy path from bytearray -> bytes currently.

On my system reading a 2GB file
./python -m test -M8g -uall test_largefile -m test.test_largefile.PyLargeFileTest.test_large_read -v

Goes from ~2.7 seconds -> ~2.2 seconds. The C _io implementation is ~1.2 seconds, so still some performance gap, but less.

  • Issue: Reduce copies when reading files in pyio, match behavior of _io #129005

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@cmaloney
pythongh-12005: Align FileIO.readall between _pyio and _io
50389df
Utilize `bytearray.resize()` and `os.readinto()` to reduce copies
and match behavior of `_io.FileIO.readall()`.

There is still an extra copy which means twice the memory required
compared to FileIO because there isn't a zero-copy  path from
`bytearray` -> `bytes` currently.

On my system reading a 2GB file
`./python -m test -M8g -uall test_largefile -m test.test_largefile.PyLargeFileTest.test_large_read -v`

Goes from ~2.7 seconds -> ~2.2 seconds
@bedevere-app bedevere-app Bot added the awaiting review label Feb 5, 2025
@cmaloney cmaloney changed the title gh-12005: Align FileIO.readall between _pyio and _io gh-129005: Align FileIO.readall between _pyio and _io Feb 5, 2025
@bedevere-app bedevere-app Bot mentioned this pull request Feb 5, 2025
Reduce copies when reading files in pyio, match behavior of _io #129005
Closed
vstinner
vstinner reviewed Feb 5, 2025
View reviewed changes
Comment thread Lib/_pyio.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/_pyio.py
result += chunk

bytes_read += n
result.resize(bytes_read)

@vstinner vstinner Feb 5, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

result = memoryview(result)[bytes_read:] would avoid a truncation which can imply a memory copy in the worst case, no?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@cmaloney cmaloney Feb 5, 2025

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the resize "shrink" in bytearray doesn't actually resize unless the buffer's "capacity" is 2x the requested size (https://github.com/python/cpython/blob/main/Objects/bytearrayobject.c#L201-L214). Just updates its internal "this is how long the bytes is" counter (which for things like full-file readall with known size, this should already be just one byte over the right size).

My plan currently is to make it so bytes(bytearray(10)) and bytearray(b'\0' * 10) both don't copy (Ongoing discussion in https://discuss.python.org/t/add-zero-copy-conversion-of-bytearray-to-bytes-by-providing-bytes/79164). Having a memoryview would mean there's more than one reference to the bytearray, and I couldn't do / use that optimization.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@vstinner vstinner Feb 6, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'm fine with using result.resize() here.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@cmaloney @vstinner
Update Lib/_pyio.py
4520ecc
Co-authored-by: Victor Stinner <[email protected]>
@cmaloney

cmaloney commented Feb 6, 2025

Copy link
Copy Markdown
Contributor Author

Hypothesis test failure in binascii / pretty sure unrelated

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner
vstinner reviewed Feb 6, 2025
View reviewed changes
Comment thread Lib/_pyio.py Outdated
bufsize += max(bufsize, DEFAULT_BUFFER_SIZE)
n = bufsize - len(result)
if bytes_read >= bufsize:
# Parallels _io/fileio.c new_buffersize

@vstinner vstinner Feb 6, 2025

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the C code, new_buffersize() argument is bytes_read, not bufsize. You may keep new_buffersize() as a private module-level function.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

cmaloney reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

@cmaloney cmaloney Feb 6, 2025

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the loop to no longer use bufsize at all, this is the only line that used it, and it feels more Pythonic to me to just use len(result).

That enables rewriting to:

        try:
            # Read until EOF (n == 0)
            while n := os.readinto(self._fd, memoryview(result)[bytes_read:]):
                bytes_read += n
                if bytes_read >= len(result):
                    result.resize(_new_buffersize(bytes_read))
        except BlockingIOError:
            if not bytes_read:
                return None
        assert len(result) - bytes_read >= 1, \
            "os.readinto buffer size 0 will result in erroneous EOF / returns 0"
        result.resize(bytes_read)
        return bytes(result)

which feels cleaner, but also starts changing structure relative to _io version.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@cmaloney cmaloney Feb 7, 2025

Copy link
Copy Markdown
Contributor Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

decided to refactor to this. Control flow feels a lot simpler to me and a lot more readable than the branches and breaks.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@cmaloney
Use len(result) rather than bufsize, _new_buffersize, make in terms o…
6c3ac57
…f bytes_read
@cmaloney

cmaloney commented Feb 6, 2025

Copy link
Copy Markdown
Contributor Author

Tests / Windows / build and test (Win32) failure is a urllib.error.HTTPError : HTTP error 504: Gateway Timeout [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj], believe unrelated

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

cmaloney and others added 4 commits February 6, 2025 14:41
@cmaloney
Merge branch 'main' into fileio_readall
0e15dee
@cmaloney
Simplify control structure
b50fb66
@cmaloney
Fix whitespace
4da4e31
@cmaloney
Merge branch 'main' into fileio_readall
09b52ed
vstinner
vstinner approved these changes Feb 7, 2025
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting review labels Feb 7, 2025
@vstinner
vstinner merged commit a3d5aab into python:main Feb 7, 2025
@bedevere-app bedevere-app Bot removed the awaiting merge label Feb 7, 2025
@vstinner

vstinner commented Feb 7, 2025

Copy link
Copy Markdown
Member

Merged, thank you.

cmaloney reacted with hooray emoji
All reactions
  • 🎉 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@cmaloney
cmaloney deleted the fileio_readall branch February 7, 2025 18:46
@cmaloney cmaloney mentioned this pull request Oct 22, 2026
PEP 399: exception to remove io module Python implementation _pyio python/steering-council#372
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner vstinner approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

@cmaloney @vstinner
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.