GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-121313: multiprocessing: change connection buffer size to 64KiB by methane · Pull Request #123559 · python/cpython

Conversation

@methane

@methane methane commented Sep 1, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Windows:

Current buffer size is 8KiB since multiprocessing is introduced.
It seems small for recent Python usages.

e711caf#diff-2c54a007d7fe1d9ac5ca008fe2d054394c39a4f521eea2cb580101a284d7b7ecR28

macOS/BSD:

They use 64KiB buffer for pipes. Current 16 pages (256KiB) buffer makes ~10% slowdown compared to 64KiB on M1 mac.

Linux:

I don't have 16k/64k page Linux. But when I change the pipe buffer size via fcntl, 256KiB buffer doesn't make notable performance benefit.

64KiB seems good default buffer size.
If it is not suitable, user can try other size by changing multiprocessing.connection.BUFSIZE.

  • Issue: Limit the reading size from pipes to their default buffer size on Unix systems #121313

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
methane added 2 commits September 1, 2024 07:16
@methane
pythongh-121313: multiprocessing: change connection buffer size
88f46e4
@methane
update changelog
b24b46f
@methane
methane requested a review from gpshead as a code owner September 1, 2024 07:24
@bedevere-app bedevere-app Bot mentioned this pull request Sep 1, 2024
Limit the reading size from pipes to their default buffer size on Unix systems #121313
Closed
@bedevere-app bedevere-app Bot added the awaiting core review label Sep 1, 2024
@methane methane mentioned this pull request Sep 1, 2024
gh-123557: Limit the reading size from Unix sockets to same limit pipes use #123558
Closed
@methane

methane commented Sep 1, 2024

Copy link
Copy Markdown
Member Author

Previous discussion:
#121315 (comment)

quick bench code:
https://gist.github.com/methane/a6cb799704a11f2bb2f64b16c0b830cc

64k vs 256k quick bench:

$ time python hello.py # 64k

real	0m25.999s
user	0m9.753s
sys	0m29.687s

real	0m25.964s
user	0m9.553s
sys	0m29.854s

$ time python hello.py # 256k

real	0m25.927s
user	0m9.421s
sys	0m29.958s

real	0m25.933s
user	0m9.330s
sys	0m30.108s

How to change the pipe buffer size:

diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py
index ede5596e4fc..9c1bfb93728 100644
--- a/Lib/multiprocessing/connection.py
+++ b/Lib/multiprocessing/connection.py
@@ -544,7 +544,10 @@ def Pipe(duplex=True):
             c1 = Connection(s1.detach())
             c2 = Connection(s2.detach())
         else:
+            import fcntl
             fd1, fd2 = os.pipe()
+            fcntl.fcntl(fd1, fcntl.F_SETPIPE_SZ, 256*1024)
+            fcntl.fcntl(fd2, fcntl.F_SETPIPE_SZ, 256*1024)
             c1 = Connection(fd1, writable=False)
             c2 = Connection(fd2, readable=False)

I confirmed that this setting works by checking n = len(chunk) size.

So I don't think having Linux pipe only optimization is worth enough.
Static 64KiB size seems good enough.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@methane methane mentioned this pull request Sep 1, 2024
gh-121313: Limit the reading size from pipes to their default buffer size on Unix systems #121315
Merged
picnixz
picnixz reviewed Sep 1, 2024
View reviewed changes
Comment thread Lib/multiprocessing/connection.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
methane added 2 commits September 2, 2024 13:28
@methane
update comment
0e0933d
@methane
remove unnecessary code
205ec7b
picnixz
picnixz approved these changes Sep 2, 2024
View reviewed changes

@picnixz picnixz left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really small nitpick. But otherwise LGTM.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
Comment thread Lib/multiprocessing/connection.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
methane and others added 3 commits September 2, 2024 19:28
@methane @picnixz
Update Lib/multiprocessing/connection.py
bfbf9bd 
Co-authored-by: Bénédikt Tran <[email protected]>
@methane
fix news category
5ea7d02
@methane
revice changelog
3dbf77d
picnixz
picnixz reviewed Sep 2, 2024
View reviewed changes
Comment thread Misc/NEWS.d/next/Library/2024-07-03-10-11-53.gh-issue-121313.D7gARW.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
vstinner
vstinner reviewed Sep 2, 2024
View reviewed changes
Comment thread Lib/multiprocessing/connection.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Lib/multiprocessing/connection.py Resolved

Uh oh!

There was an error while loading. Please reload this page.
Comment thread Misc/NEWS.d/next/Library/2024-07-03-10-11-53.gh-issue-121313.D7gARW.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.
methane and others added 2 commits September 2, 2024 20:32
@methane @vstinner
Apply suggestions from code review
3535586 
Co-authored-by: Victor Stinner <[email protected]>
@methane
fix link
7dc489b
vstinner
vstinner approved these changes Sep 2, 2024
View reviewed changes

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@bedevere-app bedevere-app Bot added awaiting merge and removed awaiting core review labels Sep 2, 2024
@methane

methane commented Sep 3, 2024

Copy link
Copy Markdown
Member Author

I tested this PR on Windows but no speedup. It is because Windows can read whole remaining data at once via this function.

cpython/Lib/multiprocessing/connection.py

Line 350 in 7dc489b

def _get_more_data(self, ov, maxsize):

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
gpshead
gpshead approved these changes Sep 3, 2024
View reviewed changes
@gpshead
gpshead merged commit 13f61bf into python:main Sep 3, 2024
@bedevere-app bedevere-app Bot removed the awaiting merge label Sep 3, 2024
@methane
methane deleted the mp-bufsize branch January 15, 2025 04:19
@methane methane mentioned this pull request Feb 4, 2025
performance: Update io.DEFAULT_BUFFER_SIZE to make python IO faster? #117151
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@picnixz picnixz picnixz approved these changes
@gpshead gpshead gpshead approved these changes
@vstinner vstinner vstinner approved these changes

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@methane @gpshead @vstinner @picnixz
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402520 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.