GeezerVison is showing github.com — bpo-37334: Add a cancel method to asyncio Queues by tecki · Pull Request #14227 · python/cpython
Conversation
tecki
commented
Jun 19, 2019
•
edited
Loading
Uh oh!
There was an error while loading. Please reload this page.
When working with queues, it is not uncommon that at some point the producer stops producing data for good, or the consumer stops consuming, for example because a network connection broke down or some user simply closed the session.
In this situation it is very useful to simply cancel all the waiting getters and putters. A simple method can do that, Queue.close.
https://bugs.python.org/issue37334
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Do you want to allow calling queue.close() multiple times?
Call get() / put() after close()?
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
While I think it does not make much sense, I do not see a good reason to forbid calling close several times. You can also call cancel several times on a future. Also I do not think we should forbid calling get or put after a close. We would have to artificially add some flags.
Currently, what close does is pretty simple: it cancels all waiting getters and putters. Not more, not less. I think users will be happy to have this very simple interface, as then they know what's going on.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
I think the proposal makes the queues API more error-prone: concurrent put() and close() produces unpredictable result on get() side.
@1st1 we need your opinion here.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
I think the proposal makes the queues API more error-prone: concurrent put() and close() produces unpredictable result on get() side.
Is that so? I am not aware of such a behavior, but if there is one, I'm happy to fix it. Could you give an example?
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
@tecki Also I do not think we should forbid calling get or put after a close. We would have to artificially add some flags.
Currently, what close does is pretty simple: it cancels all waiting getters and putters. Not more, not less.
What about a consumers or producers doing something like:
async def consumer(queue):
while not_done:
job = await queue.get()
result = await do_stuff(job)
async def producer(queue):
while not_done:
job = await make_job()
await queue.put(job)
(This is the actual case I had, that led me to wish queues had cancellation, that led me to this issue).
In either case, they may not be waiting on the queue itself at the moment close (or cancel) is called, but they should still be shut down. Semantically, I think it makes the most sense of close means: "no more items can be put on this queue, so trying to do so raises an exception, and trying to get once the queue is empty also raises an exception."
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
assert that already enqueued values are still processed after the queue
is closed.
|
@jpetkau your case can actually be solved pretty easily by setting your not_done variable to False once you cancel/close the queue. Sure, this feature could also be added to the cancel/close itself, but I am not sure whether it is worth the additional complexity in the standard library vs. additional work for users.
So, I would add this "close is close forever" functionality if there is a general agreement about it, but if not I would rather let it be as it is.
This PR has been open for almost two years now, it would be cool if it could be finished...
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
I propose to close this in favor of a shutdown() method with slightly different behavior -- it would allow the readers to continue draining the queue until it is empty. It would still stop writers whenever they attempt to write (put).
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|