Description

assignee = None
closed_at = None
created_at = <Date 2017-02-17.22:10:23.402>
labels = ['3.8', 'type-feature', 'library']
title = 'Expose max_queue_size in ThreadPoolExecutor'
updated_at = <Date 2021-10-14.21:59:47.298>
user = 'https://github.com/prayerslayer'

activity = <Date 2021-10-14.21:59:47.298>
actor = 'iforapsy'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-02-17.22:10:23.402>
creator = 'prayerslayer'
dependencies = []
files = []
hgrepos = []
issue_num = 29595
keywords = ['patch']
message_count = 6.0
messages = ['288043', '289082', '290253', '290949', '314699', '314743']
nosy_count = 11.0
nosy_names = ['rhettinger', 'pitrou', 'python-dev', 'davin', 'xiang.zhang', 'tomMoral', 'Jim Fasarakis-Hilliard', 'prayerslayer', 'stephen.oneal.04', 'iforapsy', 'tianc777']
pr_nums = ['143', '23864', '23865']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29595'
versions = ['Python 3.8']

Activity

1. prayerslayer commented on Feb 17, 2017

   

   prayerslayermannequin

   on Feb 17, 2017

   MannequinAuthor

   More actions

   Hi!

   I think the ThreadPoolExecutor should allow to set the maximum size of the underlying queue.

   The situation I ran into recently was that I used ThreadPoolExecutor to parallelize AWS API calls; I had to move data from one S3 bucket to another (~150M objects). Contrary to what I expected the maximum size of the underlying queue doesn't have a non-zero value by default. Thus my process ended up consuming gigabytes of memory, because it put more items into the queue than the threads were able to work off: The queue just kept growing. (It ran on K8s and the pod was rightfully killed eventually.)

   Of course there ways to work around this. One could use more threads, to some extent. Or you could use your own queue with a defined maximum size. But I think it's more work for users of Python than necessary.
2. prayerslayer

   added

   type-featureA feature request or enhancementA feature request or enhancement

   on Feb 17, 2017
3. Mariatta

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   3.7 (EOL)end of lifeend of life

   on Feb 18, 2017
4. prayerslayer commented on Mar 6, 2017

   

   prayerslayermannequin

   on Mar 6, 2017

   MannequinAuthor

   More actions

   Hello again, there's a reviewed PR open for this issue and it hasn't even received authoritative feedback yet (ie whether or not you intend to support this feature at all). I would be very happy if a core dev could look over this change before everyone forgets about it :)
5. prayerslayer commented on Mar 24, 2017

   

   prayerslayermannequin

   on Mar 24, 2017

   MannequinAuthor

   More actions

   Ping. That's really a two-line change, can be easily reviewed in 15 minutes :)
6. rhettinger commented on Mar 31, 2017

   

   rhettinger

   on Mar 31, 2017

   Contributor

   More actions

   Prayslayer, please don't shove. Your PR request was responded to by Mariatta so it wasn't ignored.

   Making decisions about API expansions takes a while (making sure it fits the intended use, that it isn't a bug factory itself, that it is broadly useful, that it is the best solution to the problem, that is doesn't complicate the implementation or limit future opportunities, that there are unforeseen problems). Among the core developers, there are only a couple part-time contributors who are qualified to make these assessments for the multi-processing module (those devs don't include me).
7. rhettinger

   assigned

   applio

   on Mar 31, 2017
8. stephenoneal commented on Mar 30, 2018

   

   stephenonealmannequin

   on Mar 30, 2018

   Mannequin

   More actions

   My project we're going into the underlying _work_queue and blocking adding more elements based on unfinished_tasks to accomplish this, bubbling this up to the API would be a welcome addition.
9. pitrou commented on Mar 31, 2018

   

   pitrou

   on Mar 31, 2018

   Member

   More actions

   Please note the PR here has some review comments that need addressing.
   Also, it needs its conflicts with git master resolved.

   I'm cc'ing Thomas Moreau, who has done a lot of work recently on the concurrent.futures internals.
10. pitrou

    added

    3.8 (EOL)end of lifeend of life

    and removed

    3.7 (EOL)end of lifeend of life

    on Mar 31, 2018
11. pitrou

    unassigned

    applio

    on Mar 31, 2018
12. ezio-melotti

    transferred this issue fromon Apr 10, 2022
13. serhiy-storchaka

    added this to Threading issues 🧵on May 20, 2022

Issue actions