GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — bpo-37334: Add a cancel method to asyncio Queues by tecki · Pull Request #14227 · python/cpython

Conversation

@tecki

@tecki tecki commented Jun 19, 2019 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Contributor

When working with queues, it is not uncommon that at some point the producer stops producing data for good, or the consumer stops consuming, for example because a network connection broke down or some user simply closed the session.

In this situation it is very useful to simply cancel all the waiting getters and putters. A simple method can do that, Queue.close.

https://bugs.python.org/issue37334

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@tecki
tecki requested a review from rhettinger as a code owner June 19, 2019 07:43
@the-knights-who-say-ni the-knights-who-say-ni added the CLA signed label Jun 19, 2019
@bedevere-bot bedevere-bot added the awaiting review label Jun 19, 2019
@mangrisano

mangrisano commented Jun 20, 2019

Copy link
Copy Markdown
Contributor

/cc @asvetlov @1st1

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@asvetlov

asvetlov commented Jun 24, 2019

Copy link
Copy Markdown
Contributor

Do you want to allow calling queue.close() multiple times?
Call get() / put() after close()?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tecki

tecki commented Jun 24, 2019

Copy link
Copy Markdown
Contributor Author

While I think it does not make much sense, I do not see a good reason to forbid calling close several times. You can also call cancel several times on a future. Also I do not think we should forbid calling get or put after a close. We would have to artificially add some flags.
Currently, what close does is pretty simple: it cancels all waiting getters and putters. Not more, not less. I think users will be happy to have this very simple interface, as then they know what's going on.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tecki
tecki force-pushed the cancelqueue branch from f1cc034 to d259118 Compare November 21, 2019 10:08
@asvetlov

asvetlov commented Nov 21, 2019

Copy link
Copy Markdown
Contributor

I think the proposal makes the queues API more error-prone: concurrent put() and close() produces unpredictable result on get() side.

@1st1 we need your opinion here.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tecki

tecki commented Nov 21, 2019

Copy link
Copy Markdown
Contributor Author

I think the proposal makes the queues API more error-prone: concurrent put() and close() produces unpredictable result on get() side.

Is that so? I am not aware of such a behavior, but if there is one, I'm happy to fix it. Could you give an example?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jpetkau

jpetkau commented Jul 10, 2020

Copy link
Copy Markdown

@tecki Also I do not think we should forbid calling get or put after a close. We would have to artificially add some flags.
Currently, what close does is pretty simple: it cancels all waiting getters and putters. Not more, not less.

What about a consumers or producers doing something like:

async def consumer(queue):
    while not_done:
      job = await queue.get()
      result = await do_stuff(job)

async def producer(queue):
    while not_done:
      job = await make_job()
      await queue.put(job)

(This is the actual case I had, that led me to wish queues had cancellation, that led me to this issue).

In either case, they may not be waiting on the queue itself at the moment close (or cancel) is called, but they should still be shut down. Semantically, I think it makes the most sense of close means: "no more items can be put on this queue, so trying to do so raises an exception, and trying to get once the queue is empty also raises an exception."

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@tecki
tecki force-pushed the cancelqueue branch from b07a9e1 to 02701cd Compare May 12, 2021 11:45
tecki and others added 7 commits May 12, 2021 13:59
@tecki
Add a cancel method to asyncio Queues.
ce81fc4
@tecki
test asyncio.Queue.cancel
ad2c8a6
@tecki
document asyncio.Queue.cancel
da02bb4
@tecki
rename asyncio.queue.cancel to .close
909ce14
@tecki
improve testing of asyncio.queue.close
9e895b3
assert that already enqueued values are still processed after the queue
is closed.
@tecki
add a news entry
6522e52
@tecki
remove loop parameter in asyncio tests
ca06730
@tecki
tecki force-pushed the cancelqueue branch from 02701cd to ca06730 Compare May 12, 2021 12:55
@tecki

tecki commented May 12, 2021

Copy link
Copy Markdown
Contributor Author

@jpetkau your case can actually be solved pretty easily by setting your not_done variable to False once you cancel/close the queue. Sure, this feature could also be added to the cancel/close itself, but I am not sure whether it is worth the additional complexity in the standard library vs. additional work for users.

So, I would add this "close is close forever" functionality if there is a general agreement about it, but if not I would rather let it be as it is.

This PR has been open for almost two years now, it would be cool if it could be finished...

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@rhettinger
rhettinger removed their request for review May 3, 2022 06:34
@tecki tecki mannequin mentioned this pull request Apr 10, 2022
Add a shutdown method to threading and asyncio Queue classes #81515
Closed
@ezio-melotti ezio-melotti removed the CLA signed label Jul 13, 2022
@gvanrossum

gvanrossum commented Aug 28, 2022

Copy link
Copy Markdown
Member

I propose to close this in favor of a shutdown() method with slightly different behavior -- it would allow the readers to continue draining the queue until it is empty. It would still stop writers whenever they attempt to write (put).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@gvanrossum gvanrossum closed this Aug 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

awaiting review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

8 participants

@tecki @mangrisano @asvetlov @jpetkau @gvanrossum @the-knights-who-say-ni @ezio-melotti @bedevere-bot
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.