GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |||||||||||||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |||||||||||||||||||||||||||||||||||||||||||||
GeezerVison is showing github.com — bpo-29842: Make Executor.map less eager so it handles large/unbounded… by graingert · Pull Request #18566 · python/cpython Conversation
|
| except StopIteration: | ||
| argsiter = None | ||
| else: | ||
| fs.append(self.submit(fn, *args)) |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
If the executor has been shut down, this will raise:
cannot schedule new futures after shutdown
But, also the base executor holds no state, so at this level it'll be pretty hard to tell if the executor has been shut down or not.
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
@rdarder
this could catch that RuntimeError and just return, or the executor could keep a weakset of result_iterators and close them on shutdown
Sorry, something went wrong.
There was an error while loading. Please reload this page.
| .. versionchanged:: 3.5 | ||
| Added the *chunksize* argument. | ||
|
|
||
| .. versionchanged:: 3.9 |
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
| .. versionchanged:: 3.9 | |
| .. versionchanged:: 3.11 |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
|
While waiting for this to be included in Python, one can easily copy&paste the map function out of this version and call it with the executor as the first argument. Hoping that you get this merged soon! |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
|
@graingert Tests are failing, can you fix them and rebase to main? I'll then review and run through the buildbots. |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
| yield fs.pop().result() | ||
| res = fs.popleft().result() | ||
| else: | ||
| yield fs.pop().result(end_time - time.monotonic()) |
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
The reason will be displayed to describe this comment to others. Learn more.
this doesn't cancel the currently waited on fut correctly: #95166
import contextlib import functools import concurrent.futures import threading import sys def fn(num, stop_event): if num == 1: stop_event.wait() return "done 1" if num == 2: return "done 2" def main(): stop_event = threading.Event() log = [] with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool: def print_n_wait(ident): log.append(f"{ident=} started") try: stop_event.wait() finally: log.append(f"{ident=} stopped") fut = pool.submit(print_n_wait, ident="first") try: with contextlib.closing(pool.map(print_n_wait, ["second", "third"], timeout=1)) as gen: try: next(gen) except concurrent.futures.TimeoutError: print("timed out") else: raise RuntimeError("timeout expected") finally: stop_event.set() assert log == ["ident='first' started", "ident='first' stopped"], f"{log=} is wrong" if __name__ == "__main__": sys.exit(main())
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
@graingert You can run |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
|
Hey all, I took the inspiration from this PR and continued the work in #114975 |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
|
Hi, fyi here is a follow up PR: #125663 🙏🏻 |
Sorry, something went wrong.
There was an error while loading. Please reload this page.
There was an error while loading. Please reload this page.
Successfully merging this pull request may close these issues.
There was an error while loading. Please reload this page.