GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — bpo-34831: Asyncio tutorial by cjrh · Pull Request #9748 · python/cpython

bpo-34831: Asyncio tutorial #9748

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

cjrh wants to merge 31 commits into python:main from cjrh:bpo-34831-asyncio-tutorial

Uh oh!

There was an error while loading. Please reload this page.

Closed

bpo-34831: Asyncio tutorial #9748

Show file tree

Hide file tree

Changes from all commits

Commits

Show all changes

31 commits

Select commit Hold shift + click to select a range

16d3b94

Create basic structure of the asyncio tutorial

cjrh Oct 7, 2018

50a901e

Begun work on the case study for the server

cjrh Oct 14, 2018

dfede40

Incorporate review comments from @willingc

cjrh Oct 21, 2018

a11e659

Refine language around threads and processes

cjrh Oct 21, 2018

7e205d2

Incorporate message handling into server code

cjrh Oct 21, 2018

7f2f149

Add message receiving to server code.

cjrh Oct 21, 2018

61402e1

Added skeleton suggestions for the cookbook section

cjrh Oct 21, 2018

550bdbf

Further notes in the cookbook

cjrh Oct 21, 2018

e7bc56d

Further work on describing how async def functions work

cjrh Nov 4, 2018

3d4cdae

Fix review comment from @tirkarthi

Clarify the "What is async" section

cjrh Jun 15, 2019

0de2748

Flesh out the sync-versus-async functions section

Write "Executing Async Functions"

cjrh Jun 15, 2019

69190b8

Fix spurious backtick

cjrh Jun 15, 2019

89f7ca2

Make the case study (server) a little neater.

cjrh Jun 15, 2019

36fc743

Some refactoring and finishing off the server.

cjrh Jun 15, 2019

d55d8fb

Cleaned up the last bit of the chat server code sample.

cjrh Jun 16, 2019

34306f0

Further progress - got a CLI chat client working using prompt-toolkit.

cjrh Jun 16, 2019

0c82755

Include chat client code in the text.

Clarify switching behaviour

cjrh Jun 17, 2019

a8a801d

Add async generators and async context managers discussion.

cjrh Jun 17, 2019

8e6dcfd

Add some comparison with JavaScript async/await and asyncio.create_task

cjrh Jun 17, 2019

0e5ed3f

Fix "no good read" typo

cjrh Jun 17, 2019

4714ed2

Fix "do not required" typo

Removing the GUI case study section

cjrh Jun 19, 2019

9530021

Remove problematic backticks inside a code-block

cjrh Sep 11, 2019

File filter

Filter by extension .py (10) .rst (10) All 2 file types selected

Viewed files

Clear filters

Conversations

Failed to load comments.

Uh oh!

There was an error while loading. Please reload this page.

Jump to

Jump to file

Failed to load files.

Uh oh!

There was an error while loading. Please reload this page.

Diff view

Unified

Split

Hide whitespace

Diff view

Unified

Split

Hide whitespace

There are no files selected for viewing

485 changes: 485 additions & 0 deletions 485 Doc/library/asyncio-tutorial/async-functions.rst

Show comments

View file

Open in desktop

Large diffs are not rendered by default.

Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

258 changes: 258 additions & 0 deletions 258 Doc/library/asyncio-tutorial/asyncio-cookbook.rst

Show comments

View file

Open in desktop

Original file line number Diff line number Diff line change

@@ -0,0 +1,258 @@

Asyncio Cookbook

================

Let's look at a few common situations that will come up in your

``asyncio`` programs, and how best to tackle them.

[There's a lot more we can do if we're able to refer to

3rd party packages here. We could show a websockets example,

and other things.]

Using A Queue To Move Data Between Long-Lived Tasks

---------------------------------------------------

TODO

Comment thread

asvetlov Sep 12, 2019

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Using A Queue To Control A Pool of Resources

--------------------------------------------

- show example with a pool of workers

- show example with a connection pool

Best Practices For Timeouts

---------------------------

- start with ``asyncio.wait_for()``

Comment thread

asvetlov Sep 12, 2019

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

	- start with ``asyncio.wait_for()``
	- start with :func:`asyncio.wait_for()`

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

- also look at ``asyncio.wait()``, and what to do if not all tasks

Comment thread

asvetlov Sep 12, 2019

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

	- also look at ``asyncio.wait()``, and what to do if not all tasks
	- also look at :func:`asyncio.wait()`, and what to do if not all tasks

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

are finished when the timeout happens. Also look at the different

termination conditions of ``asyncio.wait()``

How To Handle Cancellation

--------------------------

- app shutdown

- how to handle CancelledError and then close sockets

Comment thread

asvetlov Sep 12, 2019

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

	- how to handle CancelledError and then close sockets
	- how to handle :exc:`~asyncio.CancelledError` and then close sockets

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

- also, when waiting in a loop on ``await queue.get()`` is it better to

handle CancelledError, or use the idiom of putting ``None`` on the

queue? (``None`` would be better because it ensures the contents of the

queue get processed first, but I don't think we can prevent

CancelledError from getting raised so it must be handled anyway. I

can make an example to explain better.)

Keeping Track Of Many Connections

---------------------------------

- example using a global dict

- show how a weakref container can simplify cleanup

- show how to access connection info e.g. ``get_extra_info()``

- this kind of thing:

.. code-block:: python3

import asyncio

from weakref import WeakValueDictionary

CONNECTIONS = WeakValueDictionary()

async def client_connected_cb(reader, writer):

addr = writer.get_extra_info('peername')

print(f'New connection from {addr}')

# Every new connection gets added to the global dict.

# Actually, *writer* objects get added. This makes

# it easy to look up a connection and immediately

# send data to it from other async functions.

CONNECTIONS[addr] = writer

...

async def main():

server = await asyncio.start_server(

Comment thread

asvetlov Sep 12, 2019

Copy link

Copy Markdown

Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New API:

async with asyncio.StreamServer(client_connected_cb, 'localhost', 9011) as server:
    await server.serve_forever()

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

client_connected_cb=client_connected_db,

host='localhost',

port='9011',

)

async with server:

await server.serve_forever()

if __name__ == '__main__':

asyncio.run(main())

Handling Reconnection

---------------------

- Example is a client app that needs to reconnect to a server

if the server goes down, restarts, or there is a network partition

or other general kind of error

Async File I/O

--------------

- mention that disk I/O is still IO

- Python file operations like ``open()``, etc. are blocking

- I think all we can do here is refer to the 3rd party *aiofiles*

package?

- I suppose we could show how to do file IO in thread, driven

by ``run_in_executor()``...

Wait For Async Results In Parallel

----------------------------------

TODO

- show an example with gather

- show another example with wait

- maybe throw in an example with gather that also uses

"wait_for" for timeout

- either include "return_exceptions" here or in a different question

.. code-block:: python3

import asyncio

async def slow_sum(x, y):

result = x + y

await asyncio.sleep(result)

return result

async def main():

results = await asyncio.gather(

slow_sum(1, 1),

slow_sum(2, 2),

)

print(results) # "[2, 4]"

if __name__ == '__main__':

asyncio.run(main())

- we should also include a brief discussion of "when to use asyncio.gather and

when to use asyncio.wait"

Secure Client-Server Networking

-------------------------------

- built-in support for secure sockets

- you have to make your own secret key, and server certificate

.. code-block:: bash

:caption: Create a new private key and certificate

$ openssl req -newkey rsa:2048 -nodes -keyout chat.key \

-x509 -days 365 -out chat.crt

This creates ``chat.key`` and ``chat.crt`` in the current dir.

.. code-block:: python3

:caption: Secure server

import asyncio

import ssl

async def main():

ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)

ctx.check_hostname = False

# These must have been created earlier with openssl

ctx.load_cert_chain('chat.crt', 'chat.key')

server = await asyncio.start_server(

client_connected_cb=client_connected_cb,

host='localhost',

port=9011,

ssl=ctx,

)

async with server:

await server.serve_forever()

async def client_connected_cb(reader, writer):

print('Client connected')

received = await reader.read(1024)

while received:

print(f'received: {received}')

received = await reader.read(1024)

if __name__ == '__main__':

asyncio.run(main())

.. code-block:: python3

:caption: Secure client

import asyncio

import ssl

async def main():

print('Connecting...')

ctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)

ctx.check_hostname = False

# The client must only have access to the cert *not* the key

ctx.load_verify_locations('chat.crt')

reader, writer = await asyncio.open_connection(

host='localhost',

port=9011,

ssl=ctx

)

writer.write(b'blah blah blah')

await writer.drain()

writer.close()

await writer.wait_closed()

if __name__ == '__main__':

asyncio.run(main())

Correctly Closing Connections

-----------------------------

- from the client side

- from the server side

- Yury I need your help here. What is the actual "correct" way

to do this? Streams API preferable, if possible.

Handling Typical Socket Errors

------------------------------

- Maybe describe the situations in which they can occur? Not sure.

- ``ConnectionError``

- ``ConnectionResetError``

- ``ConnectionAbortedError``

- ``ConnectionRefusedError``

Might also want to show some examples of ``asyncio.IncompleteReadError``.

Also link/refer to the socket programming HOWTO in the docs.

Graceful Shutdown on Windows

----------------------------

TODO

Run A Blocking Call In An Executor

----------------------------------

- show example with default executor

- show example with a custom executor (thread-based)

- show example with a custom executor (process-based)

Adding Asyncio To An Existing Sync (Threaded) Application

---------------------------------------------------------

- Imagine an existing app that uses threading for concurrency,

but we want to make use of asyncio only for, say, a large

number of concurrent GET requests, but leave the rest of the

app unchanged.

- Plan would be to run the asyncio loop in another thread

- Can show how to safely communicate between that thread and

the main thread (or others).

Notes:

- My thinking here was a Q&A style, and then each section has

a code snippet demonstrating the answer.

Comment thread

cjrh marked this conversation as resolved.

Resolved

Uh oh!

There was an error while loading. Please reload this page.

19 changes: 19 additions & 0 deletions 19 Doc/library/asyncio-tutorial/case-study-chat-client-cli.rst

Show comments

View file

Open in desktop

Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,19 @@
		Asyncio Case Study: Chat Application (Client)
		=============================================

		WIP

		.. literalinclude:: client05.py
		:caption: client.py
		:language: python3


		TODO

		Notes:

		- using the streams API
		- first show the client.
		- will have to explain a message protocol. (But that's easy)
		- then show the server
		- spend some time on clean shutdown.

Oops, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.