GeezerVison is showing github.com — Executing code in thread or process pools: run_in_executor example · Issue #85299 · python/cpython
Executing code in thread or process pools: run_in_executor example #85299Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2020-06-26.15:24:56.727>
labels = ['3.8', 'docs']
title = 'Executing code in thread or process pools: run_in_executor example'
updated_at = <Date 2020-06-26.15:24:56.727>
user = 'https://github.com/aaraney' bugs.python.org fields:
activity = <Date 2020-06-26.15:24:56.727>
actor = 'aaraney'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2020-06-26.15:24:56.727>
creator = 'aaraney'
dependencies = []
files = []
hgrepos = []
issue_num = 41127
keywords = []
message_count = 1.0
messages = ['372428']
nosy_count = 2.0
nosy_names = ['docs@python', 'aaraney']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue41127'
versions = ['Python 3.8'] Activityaaraney commented on Jun 26, 2020 MannequinAuthor More actions slateny commented on Apr 22, 2022 Not getting errors on my end for python 3.8 and 3.10, could you reconfirm the error? AlexWaygood commented on Apr 22, 2022 @slateny FYI I believe, due to some oddities of the github migration, some "mannequin" users won't necessarily receive notifications on issues unless you actually @ them, even if they show up as having previously participated in the thread. (You can safely assume that non-mannequin users are still subscribed to all the issues they were "nosy" to on BPO.) slateny commented on Apr 22, 2022
Not getting errors on my end for python 3.8 and 3.10, could you reconfirm the error?
@aaraney ^ I know it's been a little while, but if you have time could you take another look? aaraney commented on Apr 22, 2022 Certainly, thanks for pinging me, @slateny. Admittedly, the original bug report is not very descriptive, so Ill try to clarify the issue.
In the python 3.8 concurrent futures documentation the provided executor example (see below) throws a RuntimeError if you copy and paste the example into a file and run it. It appears the documentation for 3.9 and 3.10 also have this issue. This is because the asyncio.run(main()) statement at the bottom of the example is not guarded by if __name__ == "__main__":.
import asyncio
import concurrent.futures
def blocking_io():
# File operations (such as logging) can block the
# event loop: run them in a thread pool.
with open('/dev/urandom', 'rb') as f:
return f.read(100)
def cpu_bound():
# CPU-bound operations will block the event loop:
# in general it is preferable to run them in a
# process pool.
return sum(i * i for i in range(10 ** 7))
async def main():
loop = asyncio.get_running_loop()
## Options:
# 1. Run in the default loop's executor:
result = await loop.run_in_executor(
None, blocking_io)
print('default thread pool', result)
# 2. Run in a custom thread pool:
with concurrent.futures.ThreadPoolExecutor() as pool:
result = await loop.run_in_executor(
pool, blocking_io)
print('custom thread pool', result)
# 3. Run in a custom process pool:
with concurrent.futures.ProcessPoolExecutor() as pool:
result = await loop.run_in_executor(
pool, cpu_bound)
print('custom process pool', result)
asyncio.run(main())The example should be updated to the following:
import asyncio
import concurrent.futures
def blocking_io():
# File operations (such as logging) can block the
# event loop: run them in a thread pool.
with open('/dev/urandom', 'rb') as f:
return f.read(100)
def cpu_bound():
# CPU-bound operations will block the event loop:
# in general it is preferable to run them in a
# process pool.
return sum(i * i for i in range(10 ** 7))
async def main():
loop = asyncio.get_running_loop()
## Options:
# 1. Run in the default loop's executor:
result = await loop.run_in_executor(
None, blocking_io)
print('default thread pool', result)
# 2. Run in a custom thread pool:
with concurrent.futures.ThreadPoolExecutor() as pool:
result = await loop.run_in_executor(
pool, blocking_io)
print('custom thread pool', result)
# 3. Run in a custom process pool:
with concurrent.futures.ProcessPoolExecutor() as pool:
result = await loop.run_in_executor(
pool, cpu_bound)
print('custom process pool', result)
if __name__ == "__main__":
asyncio.run(main())slateny commented on Apr 23, 2022 Thanks for the clarification - I tried the code and it seems to run for me without error. Are you running this from another file or is this the only file? aaraney commented on Apr 25, 2022 Im running it from a standalone script. So, not from the IDLE. slateny commented on Apr 26, 2022 Hmm interesting, I ran it on ubuntu 20 w/ python 3.8 and 3.10 and no errors for me - maybe a version/platform thing? aaraney commented on Apr 26, 2022 So, I was not able to reproduce the issue in a docker container. I used the following base images in attempt to be more complete: python:3.8-slim-buster, python:3.8.10-slim-buster, and continuumio/miniconda3:4.11.0. However, I am beginning to think this is a platform issue not a python version issue. More specifically, something to do with the spawn semantics used vs fork exec. This behavior was changed in 3.8 for OSX.
Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725.
Instead of droning on about why I think that is the case, I threw together a repo that uses github actions and a macos 11 runner. Below ive included the gh action yaml for convenience. I was able to reproduce the issue using gh actions. Please find the failing job here.
name: Run Unit Tests
on: push
jobs:
cpython_85299:
runs-on: macos-11
steps:
- uses: actions/checkout@v2
- name: Set up Python 3.8
uses: actions/setup-python@v2
with:
python-version: '3.8'
- name: Reproduce Runtime Error
run: |
python3 main.py
slateny commented on Apr 30, 2022 Hmm, if it's a platform issue then do you think that the docs still need updating? If the file's the only one being run, then if __name__ == "__main__": seems equivalent to leaving it out aaraney commented on May 2, 2022 Yeah I think that is a fair point, @slateny. Ive not seen it stated in python docs that examples are runnable as their own script, but I think that is kind of implicit to an example snippet. I would prefer that if __name__ == "__main__": is included where examples are expected to be run in a standalone context. slateny commented on May 14, 2022 The third example uses ProcessPoolExecutor, defined here, which then seems to get the start method, which as you mentioned uses spawn since 3.8. A bit down below, there is a note about protecting the program's entry point via if __name__ == '__main__' if using the spawn method.
Taking all that into account, I think the doc change should be instead to add a note/link to the warning for macOS, advising the entrypoint guard as needed, instead of adding it directly into the example as there might be confusion on why the guard's there when it may not be necessary. gvanrossum commented on Oct 13, 2022 The problem seems to be with how the OP is running the code, not with the example. Nevertheless it is fine to add a __main__ check to the example, since that is customary for all main programs. No notes about platforms though. added a commit that references this issue on Oct 16, 2022gh-85299: Add note warning about entry point guard for asyncio example ( added 4 commits that reference this issue on Oct 16, 2022pythongh-85299: Add note warning about entry point guard for asyncio … pythongh-85299: Add note warning about entry point guard for asyncio … gh-85299: Add note warning about entry point guard for asyncio example ( gh-85299: Add note warning about entry point guard for asyncio example ( added a commit that references this issue on Oct 22, 2022gh-85299: Add note warning about entry point guard for asyncio example (
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|