GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-38061: subprocess uses closefrom() on FreeBSD by vstinner · Pull Request #19697 · python/cpython

Conversation

@vstinner

@vstinner vstinner commented Apr 23, 2020 •
edited by bedevere-bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Optimize the subprocess module on FreeBSD using closefrom().
A single close(fd) syscall is cheap, but when sysconf(_SC_OPEN_MAX)
is high, the loop calling close(fd) on each file descriptor can take
several milliseconds.

The workaround on FreeBSD to improve performance was to load and
mount the fdescfs kernel module, but this is not enabled by default.

Initial patch by Kyle Evans with the help of Kubilay Kocak:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274

https://bugs.python.org/issue38061

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@vstinner
vstinner requested a review from gpshead as a code owner April 23, 2020 23:59
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Apr 23, 2020
@bedevere-bot bedevere-bot added the awaiting core review label Apr 23, 2020
@vstinner

vstinner commented Apr 24, 2020

Copy link
Copy Markdown
Member Author

cc @koobs
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner

vstinner commented Apr 24, 2020

Copy link
Copy Markdown
Member Author

PR created from this FreeBSD downstream patch:
https://svnweb.freebsd.org/ports/head/lang/python38/files/patch-Modules___posixsubprocess.c?revision=518640&view=markup
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
gpshead
gpshead approved these changes Apr 24, 2020
View reviewed changes
@bedevere-bot bedevere-bot added awaiting merge and removed awaiting core review labels Apr 24, 2020
@vstinner
bpo-38061: subprocess uses closefrom() on FreeBSD
0c8b6d8 
Optimize the subprocess module on FreeBSD using closefrom().
A single close(fd) syscall is cheap, but when sysconf(_SC_OPEN_MAX)
is high, the loop calling close(fd) on each file descriptor can take
several milliseconds.

The workaround on FreeBSD to improve performance was to load and
mount the fdescfs kernel module, but this is not enabled by default.

Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans
(kevans) and Kubilay Kocak (koobs):
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242274
@vstinner

vstinner commented Apr 24, 2020

Copy link
Copy Markdown
Member Author

I will check with FreeBSD developers if the authors are correct and check that the fix works as expected.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
koobs
koobs reviewed Apr 24, 2020
View reviewed changes
Comment thread Misc/NEWS.d/next/Library/2020-04-24-01-55-00.bpo-38061.XmULB3.rst
fdescfs kernel module, but this is not enabled by default.

Initial patch by Ed Maste (emaste), Conrad Meyer (cem), Kyle Evans (kevans) and
Kubilay Kocak (koobs):

@koobs koobs Apr 24, 2020

Copy link
Copy Markdown

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can remove this, full credit to Ed, Conrad & Kyle

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions

@vstinner vstinner Apr 24, 2020

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even if you didn't write the patch directly, you helped to get it packaged. Thanks @koobs, you deserved to be credited ;-)

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
All reactions
@jlevon

jlevon commented Apr 24, 2020

Copy link
Copy Markdown

This change could also add sun to get this supported under Solaris and illumos
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner

vstinner commented Apr 24, 2020

Copy link
Copy Markdown
Member Author

I tested this change functionaly using truss:

vstinner@freebsd$ truss -o log -f ./python -c 'import subprocess, sys, os; os.dup2(0, 8); args=[sys.executable, "-c", "pass"]; subprocess.run(args, pass_fds=[8])' 

vstinner@freebsd$ grep closefrom log
  947: closefrom(0x9)				 = 0 (0x0)

vstinner@freebsd$ wc -l log   # total number of syscalls (parent+child)
    1586 log

It works as expected: closefrom() is called

Without this change, there are way more syscalls (+229 274) :-)

vstinner@freebsd$ wc -l log
  230860 log

... which is very close to the maximum number of file descriptor:

>>> os.sysconf("SC_OPEN_MAX")
229284
cemeyer reacted with thumbs up emoji cemeyer reacted with hooray emoji
All reactions
  • 👍 1 reaction
  • 🎉 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
@vstinner
vstinner merged commit e6f8abd into python:master Apr 24, 2020
@bedevere-bot bedevere-bot removed the awaiting merge label Apr 24, 2020
@vstinner
vstinner deleted the subprocess_closefrom branch April 24, 2020 10:07
@vstinner

vstinner commented Apr 24, 2020

Copy link
Copy Markdown
Member Author

This change could also add sun to get this supported under Solaris and illumos

Sure. But please propose a separated PR.
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.
CuriousLearner added a commit to CuriousLearner/cpython that referenced this pull request Apr 27, 2020
@CuriousLearner
Merge branch 'master' of github.com:python/cpython into fix-issue21914
42c39a3 
* 'master' of github.com:python/cpython: (2949 commits)
  Add files in tests/test_peg_generator to the install target lists (pythonGH-19723)
  bpo-40398: Fix typing.get_args() for special generic aliases. (pythonGH-19720)
  bpo-40348: Fix typos in the programming FAQ (pythonGH-19729)
  bpo-38387: Formally document PyDoc_STRVAR and PyDoc_STR macros (pythonGH-16607)
  bpo-40401: Remove duplicate pyhash.h include from pythoncore.vcxproj (pythonGH-19725)
  bpo-40387: Improve queue join() example. (pythonGH-19724)
  bpo-40396: Support GenericAlias in the typing functions. (pythonGH-19718)
  Fix typo in Lib/typing.py (pythonGH-19717)
  Fix typo in object.__format__ docs (pythonGH-19504)
  bpo-40275: Avoid importing logging in test.support (pythonGH-19601)
  bpo-40275: Avoid importing socket in test.support (pythonGH-19603)
  bpo-40275: Avoid importing asyncio in test.support (pythonGH-19600)
  bpo-40279: Add some error-handling to the module initialisation docs example (pythonGH-19705)
  closes bpo-40385: Remove Tools/scripts/checkpyc.py (pythonGH-19709)
  bpo-40334: Add What's New sections for PEP 617 and PEP 585 (pythonGH-19704)
  bpo-40340: Separate examples more clearly in the programming FAQ (pythonGH-19688)
  bpo-40360: Deprecate lib2to3 module in light of PEP 617 (pythonGH-19663)
  bpo-40334: Rewrite test_c_parser to avoid memory leaks (pythonGH-19694)
  bpo-38061: subprocess uses closefrom() on FreeBSD (pythonGH-19697)
  bpo-38061: os.closerange() uses closefrom() on FreeBSD (pythonGH-19696)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead gpshead approved these changes
+1 more reviewer
@koobs koobs koobs left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

6 participants

@vstinner @jlevon @gpshead @koobs @the-knights-who-say-ni @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.