GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — gh-124694: Add concurrent.futures.InterpreterPoolExecutor by ericsnowcurrently · Pull Request #124548 · python/cpython

Conversation

@ericsnowcurrently

@ericsnowcurrently ericsnowcurrently commented Sep 25, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

This is an implementation of InterpreterPoolExecutor that builds on ThreadPoolExecutor.

This assumes that we're okay adding the executor separately from PEP 734. That PEP is about adding a new stdlib module, which is a separate matter from adding the new executor.

Possible future improvements:

  • support passing (most) arbitrary functions without pickling
  • support passing closures
  • optionally exec functions against __main__ instead of the their original module

CC @brianquinlan

  • Issue: Add concurrent.futures.InterpreterPoolExecutor #124694

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

temeddix reacted with heart emoji
All reactions
  • ❤️ 1 reaction
ericsnowcurrently added 4 commits September 27, 2024 12:04
@ericsnowcurrently
Make ThreadPoolExecutor extensible.
5c69d38
@ericsnowcurrently
Add InterpreterPoolExecutor.
01789be
@ericsnowcurrently
Clean up the interpreter if initialize() fails.
6def4be
@ericsnowcurrently
Add a missing import.
84993a5
@ericsnowcurrently
ericsnowcurrently force-pushed the interpreter-pool-executor branch from 46b5388 to 84993a5 Compare September 27, 2024 18:08
@ericsnowcurrently ericsnowcurrently changed the title Add concurrent.futures.InterpreterPoolExecutor gh-124694: Add concurrent.futures.InterpreterPoolExecutor Sep 27, 2024
@bedevere-app bedevere-app Bot mentioned this pull request Sep 27, 2024
Add concurrent.futures.InterpreterPoolExecutor #124694
Closed
ericsnowcurrently added 2 commits September 27, 2024 14:57
@ericsnowcurrently
Fix some typos.
c540cf0
@ericsnowcurrently
Add more tests.
45d584d
@ericsnowcurrently
ericsnowcurrently force-pushed the interpreter-pool-executor branch from 28f948b to 45d584d Compare September 27, 2024 20:57
ericsnowcurrently added 4 commits September 27, 2024 15:32
@ericsnowcurrently
Add docs.
c90c016
@ericsnowcurrently
Add a NEwS entry.
1cb4657
@ericsnowcurrently
Fix the last test.
4dc0989
@ericsnowcurrently
Add more tests.
57b2db6
@ericsnowcurrently
ericsnowcurrently marked this pull request as ready for review September 27, 2024 22:37
@bedevere-app bedevere-app Bot added the awaiting core review label Sep 27, 2024
ZeroIntensity
ZeroIntensity reviewed Sep 29, 2024
View reviewed changes
Comment thread Doc/library/concurrent.futures.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/library/concurrent.futures.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/library/concurrent.futures.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/library/concurrent.futures.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/library/concurrent.futures.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/library/concurrent.futures.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Doc/library/concurrent.futures.rst Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread Lib/concurrent/futures/interpreter.py Resolved

Uh oh!

There was an error while loading. Please reload this page.

ericsnowcurrently added 5 commits September 30, 2024 16:11
@ericsnowcurrently
Simplify ExecutionFailed.
75e11d2
@ericsnowcurrently
Fix the signature of resolve_task().
69c2b8e
@ericsnowcurrently
Capture any uncaught exception.
f03c314
@ericsnowcurrently
Add TODO comments.
4806d9f
@ericsnowcurrently
Docs fixes.
efc0395
@ericsnowcurrently

ericsnowcurrently commented Sep 30, 2024

Copy link
Copy Markdown
Member Author

@ZeroIntensity, I've fixed all those Docs things.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ZeroIntensity
ZeroIntensity approved these changes Sep 30, 2024
View reviewed changes

@ZeroIntensity ZeroIntensity left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. A small nitpick is that it might be a good idea to mention textwrap.dedent in the docs for initializer -- I'm worried that users might run into pesky indentation problems when passing scripts, and textwrap.dedent is a nice way to deal with that.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@ericsnowcurrently

ericsnowcurrently commented Sep 30, 2024

Copy link
Copy Markdown
Member Author

Good point. It would make sense to automatically call it for users.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

ericsnowcurrently added 2 commits September 30, 2024 17:39
@ericsnowcurrently
Automatically apply textwrap.dedent() to scripts.
a29aee3
@ericsnowcurrently
Fix the WASI build.
8bab457
@ericsnowcurrently

ericsnowcurrently commented Sep 30, 2024

Copy link
Copy Markdown
Member Author

@brianquinlan, what are your thoughts on this?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@brianquinlan

brianquinlan commented Oct 1, 2024

Copy link
Copy Markdown
Contributor

@brianquinlan, what are your thoughts on this?

@ericsnowcurrently I haven't been active for yours but this seems like an excellent addition.

ericsnowcurrently reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently
wasi
cd29914
@bedevere-app bedevere-app Bot removed the awaiting merge label Oct 16, 2024
@ericsnowcurrently
ericsnowcurrently deleted the interpreter-pool-executor branch October 16, 2024 22:50
@ericsnowcurrently

ericsnowcurrently commented Oct 17, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

This may have introduced a refleak crash on one of the refleak buildbots: https://buildbot.python.org/#/builders/259/builds/1528. I'm taking a look.

willingc reacted with confused emoji
All reactions
  • 😕 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ZeroIntensity

ZeroIntensity commented Oct 17, 2024

Copy link
Copy Markdown
Member

Yeah, I noticed the tests for InterpreterPoolExecutor segfaulting on someone elses PR yesterday on something seemingly unrelated. Since this is pure-Python, _interpreters might be buggier than we think :(

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently

ericsnowcurrently commented Oct 17, 2024

Copy link
Copy Markdown
Member Author

In the crash I linked to, it happened when calling _interpqueues.create(), so not _interpreters.

willingc and ZeroIntensity reacted with thumbs up emoji
All reactions
  • 👍 2 reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently

ericsnowcurrently commented Oct 17, 2024

Copy link
Copy Markdown
Member Author

FWIW, the only other failure I've seen with this on refleak buildbots is AMD64 FreeBSD Refleaks 3.x, with 3 successful runs after one failure, which looks like it hung.

The other one I mentioned was on AMD64 RHEL8 Refleaks 3.x, and has had multiple successful runs to go with that one crash.

I don't see any other failures for this on other stable buildbots.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently

ericsnowcurrently commented Oct 17, 2024

Copy link
Copy Markdown
Member Author

I also haven't been able to reproduce any crash or hang locally yet.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ZeroIntensity

ZeroIntensity commented Oct 17, 2024

Copy link
Copy Markdown
Member

I guess it's possible that this only affects AMD?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently

ericsnowcurrently commented Oct 17, 2024

Copy link
Copy Markdown
Member Author

It's probably a race due to some load profile that has only shown up there.

ZeroIntensity reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ZeroIntensity

ZeroIntensity commented Oct 17, 2024

Copy link
Copy Markdown
Member

I'll run the tests under valgrind to see if it picks anything up. That will take a while, though. Do you want to revert this in the meantime?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ZeroIntensity

ZeroIntensity commented Oct 17, 2024 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member

Ah, @ericsnowcurrently, I think I found the problem. In most of the _interpqueues methods, something like this is passed to PyArg* as a converter:

qidarg_converter_data qidarg;

As far as I can tell, default struct initialization is not C standard, so it's values are just junk stack memory. I'm guessing that x86 has some sort of detail that sets stack-allocated structs to NULL, but not on ARM--that's why it's only failing there.

Edit: Oh wait, it's AMD, not ARM. I guess it's a chip-specific issue then, but the point is that it's UB.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently

ericsnowcurrently commented Oct 17, 2024

Copy link
Copy Markdown
Member Author

Hmm, I'll take a look.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently

ericsnowcurrently commented Oct 17, 2024

Copy link
Copy Markdown
Member Author

Yeah, that could very well be it. The "label" field would sometimes be a problem if not initialized. I'll fix that.

ZeroIntensity reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently ericsnowcurrently mentioned this pull request Oct 17, 2024
qidarg_converter_data Values Not Properly Initialized #125667
Closed
@ericsnowcurrently

ericsnowcurrently commented Oct 17, 2024

Copy link
Copy Markdown
Member Author

I've opened gh-125667.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ZeroIntensity

ZeroIntensity commented Oct 17, 2024

Copy link
Copy Markdown
Member

Great, I'll review your PR whenever you get to it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently

ericsnowcurrently commented Oct 17, 2024

Copy link
Copy Markdown
Member Author

gh-125668

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently

ericsnowcurrently commented Oct 18, 2024

Copy link
Copy Markdown
Member Author

I also noticed a failure on the Android buildbot: https://buildbot.python.org/#/builders/1594/builds/338. I'm looking into it.

ZeroIntensity reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently

ericsnowcurrently commented Oct 18, 2024

Copy link
Copy Markdown
Member Author

#125708

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently

ericsnowcurrently commented Oct 18, 2024

Copy link
Copy Markdown
Member Author

Looks like there's more to do: https://buildbot.python.org/#/builders/1610/builds/198 (AMD64 CentOS9 NoGIL Refleaks). I'm looking into it.

ZeroIntensity reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@ericsnowcurrently

ericsnowcurrently commented Oct 18, 2024

Copy link
Copy Markdown
Member Author

I've opened a new issue to deal with this, so we don't keep going on here: #125716.

willingc reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jakirkham jakirkham mentioned this pull request Oct 22, 2024
concurrent.futures.InterpreterPoolExecutor #81474
Closed
This was referenced Oct 23, 2024
ENH: Please support subinterpreters numpy/numpy#24755
Open
[Python] Support for SubInterpreters and InterpreterPoolExecutors apache/arrow#44511
Closed
[ENH] Subinterpreter Support and 3.14 / InterpreterPoolExecutors cython/cython#6445
Closed
@ericsnowcurrently ericsnowcurrently mentioned this pull request Jan 7, 2025
gh-76785: Add concurrent.futures.InterpreterPoolExecutor #116430
Closed
ebonnal pushed a commit to ebonnal/cpython that referenced this pull request Jan 12, 2025
@ericsnowcurrently @ebonnal
pythongh-124694: Add concurrent.futures.InterpreterPoolExecutor (pyth…
843c78d
…ongh-124548)

This is an implementation of InterpreterPoolExecutor that builds on ThreadPoolExecutor.

(Note that this is not tied to PEP 734, which is strictly about adding a new stdlib module.)

Possible future improvements:

* support passing a script for the initializer or to submit()
* support passing (most) arbitrary functions without pickling
* support passing closures
* optionally exec functions against __main__ instead of the their original module
@donbarbos donbarbos mentioned this pull request Apr 4, 2025
gh-129667: Update annotation in documentation #129669
Open
@ZeroIntensity ZeroIntensity mentioned this pull request May 31, 2025
gh-134939: Add the interpreters Module #133958
Merged
@cdce8p cdce8p mentioned this pull request Jun 11, 2025
Update ThreadPoolExecutor attributes for Python 3.14 python/typeshed#14263
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@1st1 1st1 1st1 approved these changes
@willingc willingc willingc approved these changes
@ZeroIntensity ZeroIntensity ZeroIntensity approved these changes
@asvetlov asvetlov Awaiting requested review from asvetlov asvetlov is a code owner
@kumaraditya303 kumaraditya303 Awaiting requested review from kumaraditya303 kumaraditya303 is a code owner
@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner is a code owner
+1 more reviewer
@bluss bluss bluss left review comments
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

7 participants

@ericsnowcurrently @brianquinlan @ZeroIntensity @kumaraditya303 @1st1 @willingc @bluss
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.