GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |||||||||||||||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |||||||||||||||||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — gh-140135: use `PyBytesWriter` in `io.RawIOBase.readall`; 3.98x faster by maurycy · Pull Request #140139 · python/cpython Conversation
|
| Benchmark | main | pybytes-iobase-readall |
|---|---|---|
| rawiobase_readall_1MB_chunk4096 | 673 us | 111 us: 6.04x faster |
| rawiobase_readall_4MB_chunk4096 | 2.49 ms | 434 us: 5.74x faster |
| rawiobase_readall_8MB_chunk4096 | 5.21 ms | 907 us: 5.75x faster |
| rawiobase_readall_16MB_chunk4096 | 12.1 ms | 2.45 ms: 4.95x faster |
| rawiobase_readall_32MB_chunk4096 | 25.1 ms | 10.2 ms: 2.45x faster |
| rawiobase_readall_64MB_chunk4096 | 51.0 ms | 20.4 ms: 2.50x faster |
| rawiobase_readall_128MB_chunk4096 | 101 ms | 38.9 ms: 2.60x faster |
| Geometric mean | (ref) | 3.98x faster |
The environment:
% ./python -c "import sysconfig; print(sysconfig.get_config_var('CONFIG_ARGS'))"
'--enable-optimizations' '--with-lto'
sudo ./python -m pyperf system tune ensured.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
PyBytesWriter in io.RawIOBase.readall, 2.34x fasterPyBytesWriter in io.RawIOBase.readall
PyBytesWriter in io.RawIOBase.readallPyBytesWriter in io.RawIOBase.readall; 3.98x faster
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Please add a NEWS entry, IMO this speedup is significant enough to be documented!
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Impressive speedup! |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Overall looking good
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Thank you! My biggest worry that the benchmark is wrong. I'm more than happy to run a completely different benchmark on the same machine!
Done: f0ae824 |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
|
The CI failed. I'm not exactly sure why. The same jobs succeeded before:
No meaningful changes since the last run (b8a7f89 is just style, and f0ae824 just adds a NEWS entry.) From what I see, some flakiness was reported: I'm rerunning by merging the |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Nice optimization!
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
I think this looks good, the only other thing is I would verify this doesn't regress performance for sizes <1MB (e.g. 1K and 4K maybe). I expect the performance differences to be much smaller there if at all present, but that's OK, we just want to make sure they aren't regressing. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Great point. The benchmark (the same, just tweaked `SIZES_KB`):import io import pyperf CHUNK_SIZE = 1024 SIZES_KB = [1, 4, 8] class ChunkedRaw(io.RawIOBase): def __init__(self, data, chunk_size): self._buf = memoryview(data) self._pos = 0 self._chunk_size = chunk_size def readable(self): return True def read(self, n: int = -1): if self._pos >= len(self._buf): return b"" to_read = ( self._chunk_size if (n is None or n < 0) else min(n, self._chunk_size) ) end = min(self._pos + to_read, len(self._buf)) out = bytes(self._buf[self._pos : end]) self._pos = end return out def generate_bytes(total): block = b"abcdefghijklmnopqrstuvwxyz0123456789" * 128 return (block * (total // len(block) + 1))[:total] def _bench_readall(data, chunk_size): r = ChunkedRaw(data, chunk_size) out = r.readall() if len(out) != len(data): raise RuntimeError("what is going on...???") def main(): runner = pyperf.Runner() for size_kib in SIZES_KB: total_bytes = size_kib * 1024 data = generate_bytes(total_bytes) name = f"rawiobase_readall_{size_kib}KB_chunk{CHUNK_SIZE}" runner.bench_func(name, _bench_readall, data, CHUNK_SIZE) if __name__ == "__main__": main() The result:
|
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
Great thank you!
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.