GeezerVison

Python 3.8 is out, with new features and changes. The themes for this release have been performance, ABI/internals, and static typing, along with a smattering of new syntax. Given the recent community statement on Python support, we should be staying up to date with the current changes in Python. As Python 2 sunsets, we are finally in an era where we can hope to someday use the features we see coming out of Python release again!

Past changes in Python

Let’s start by a quick overview of the current state of Python 3. With Python 3.5 disappearing, we will simply mention that it brought the @ operator to Python, and was one of the first versions that really started to tempt Python 2 users over. But let’s really start with Python 3.6.

Python 3.6 was a big update, and possibly the most important Python 3 version to date. It was the very first version in the 3 series that was touted as “just as fast as Python 2.7” - the 3 series started life notoriously slow, due to the unicode strings and heavy internal changes. Python 3.6 brought f-strings, which were wildly popular and finally brought Python up to the level of Bash and Ruby for string interpolation. It has ordered dictionaries, simpler pathlib usage, and much more. It has a massive list of improvements and changes, feel free to check the official document.

Python 3.7 was the first version actually claiming to be faster than 2.7. It also brought some nice changes, like officially ordered dictionaries to the language (not just a CPython implementation detail as in 3.6), and some nice additions to the standard library. There was also an OpenSSL update, which seems to have affected adoption rates. It has a fairly big list of improvements and changes, feel free to check the official document.

And, of course, Python 3.8, which was released just a few days ago. The changes in general are a bit smaller than the last two versions, possibly due to the large change this year in how Python is governed. But let’s see what’s new!

Positional-only arguments

Let’s take a function/class you’ve seen in the standard library, the humble dict:

d = dict(one=1, two=2)
d = dict({"one": 1, "two": 2})

Here’s your challenge: write dict yourself. Your first attempt might be:

def dict(arg=None, **kargs): ...

Okay, let’s try to use it:

dict(arg=3)  # OOPS!

You have to use *args, and limit the input to 1 argument yourself. Or, you can use the new positional argument syntax!

If you look at the signatures of dict, pow, or some other builtins, you will see something kind of like this:

def dict(arg=None, /, **kargs): ...

That is now valid in a function definition in Python 3.8! Anything before the / is positional only, and cannot be matched with a keyword. Why is this useful?

• You can allow kwargs and positional arguments without overlap in names.
• You can force positional arguments without names.
• You can change the internal name – the name is not part of your external API.

This is now the full syntax for arguments:

def f(pos_required, /, pos_or_kw_required, *, kw_required): ...


def f(pos_optional=None, /, pos_or_kw_optional=None, *, kw_optional=None): ...

The walrus operator

Assignment in Python is a statement, which means it is quite limited. These are the only allowed ways you can make an assignment:

item = ...  # simple
item[...] = ...  # item
item.attr = ...  # attr
(a, b) = ...  # tuple
a = b = ...  # chained (special case)

But what about assigning in other places, like in the C languages? Up till now, you could not do it. And there was a problem with adding it; if you just opened up the = operator, you would run afoul in several areas of Python:

if x = True:  # Will never be allowed, too easy to make mistake
f(x=True)  # Keyword argument

The solution? A new operator!

• Spelling: := (looks like a sideways walrus)
• Works almost anywhere normal = doesn’t (one way to do things)
• Often requires parenthesis for clarity

Examples:

if res := check():
    print(res)

a = [None, 0, 1, 2]
while a := b.pop():
    print(a)

You should use it carefully; this could make code harder to read. Also note that the scope leaks, which is useful in some cases and Pythonic, but means you can’t limit the scope of a variable using this syntax (which is one of the reasons C++17/C++20 added variable defines in several new places).

f-string debugging

In Python 3.6, f-strings make string interpolation easy, and were a runaway hit:

>>> x = 3
>>> print(f"x = {x}")
x = 3

Debugging a value, however, still requires you type it twice. This is now much more DRY with the = specifier:

>>> print(f"{x = }")
x = 3

A few notes:

• Spaces around the = are respected.
• Mix with complex expressions, the entire expression is printed on the left, while the output is on the right.
• Formatting specifiers are allowed (after a :), as well.

Static typing

Static type hints are a big feature of Python 3, and now they are much more powerful:

Literals

You can make make-shift enums now using Literals; these limit the values allowed for a variable in the typechecker:

def f(val: Literal["yes", "no", "auto"]): ...

Note this really is just a Union of Literals, with a shortcut syntax for creating them.

Final

You can specify a “const” variable, one that is not allowed to be changed to something else later:

x: Final[bool] = True
x = False  # Invalid in type checker like mypy

Protocols

This is the C++20 Concepts / Java Interface idea; you define what methods and such a value should have:

class HasName(Protocol):
    name: str

Now you can use HasName as a type; it will require a name attribute.

Runtime helpers

Runtime users of type hints get two big helpers: typing.get_origin gets the runtime type associated with a type hint, like a list for List[...]. And typing.get_args gets the [] arguments to the type. For example, get_args(Literal["a", "b"]) will give you ("a", "b"). These are also available in typing_extensions.

TypedAST

TypedAST was merged into Python! The AST parser has gained a feature_version selector as well, supporting 3.4+. Let’s take a look at an example parse with a type comment, which would not have been accessible before:

import ast

s = ast.parse("x = 2 # type: Int", type_comments=True)
ast.dump(s)

"Module(body=[Assign(
  targets=[Name(id='x', ctx=Store())],
  value=Constant(value=2, kind=None),
  type_comment='Int')],
type_ignores=[])"

• ast.get_source_segment gets the source for a bit of ast, if location information is present.

Other features

Here are a few other features of note:

• TypedDict gives types to dict parts
• importlib.metadata gives you info from installed packages (like importlib.resources)
• math and statistics have new functions
• namedtuple, pickle, and more are faster
• SyntaxError messages are more detailed in some common cases
• multiprocessing.shared_memory – can avoid pickle transfer of objects (possible before, but now more visible)
• reversed works on dicts
• Unpacking in return/yield
• TemporaryDirectory now cleans up without throwing an error on Windows if a read-only file is added to it.

Other developer changes

Library developers may need to be aware of the following changes:

• --libs no longer include libpython
• Single ABI for debug/release
• Runtime audit hooks
• New C API for initialization
• Provisional vectorcall protocol – fast calling of C functions
• Pickle support out-of-band data (multiple streams) (Protocol 5)
• __code__ now has .replace, like __signature__
• PYTHONPATH is no longer used to search for DLLs

Final Words

Python.org downloads and Docker images were released on launch day. You can try it for yourself with docker run --rm -it python:3.8. The Scikit-HEP GCC 9.2 ManyLinux1 containers were updated a couple of days later, and boost-histogram supported it with binary wheels on macOS and manylinux2010 in the first beta. Conda-forge followed quickly. Azure and GitHub actions have now been updated.

Sources

This blog post was originally given as a talk at PyHEP 2019. Inspiration for the material within, along with good sources for further study:

• Official docs
• RealPython article
• DeepSource