GeezerVison

This guide covers four things that help, and none of them are about making the program work. It already works. They're about what happens next:

• Saying what you mean, with type hints and docstrings, so the next person reading your code doesn't have to guess. That person is usually you.
• Saying what happened, with logging instead of print, so a failure in the middle of the night leaves evidence behind.
• Proving it works, with tests, so a change you make in March doesn't quietly break something you wrote in January.
• Finding out why, by reading tracebacks properly and using a debugger.

Analogy: think of a car. Getting it to start is "making the program work". But a car you can rely on also has a manual (type hints and docstrings), a dashboard with warning lights (logging), an inspection that checks the brakes before every long trip (tests), and a mechanic's diagnostic tool for when something goes wrong anyway (tracebacks and the debugger). None of those make the engine run. All of them make the car trustworthy.

Chapter 9 introduced assert and logging and said this chapter would cover them properly. Here's the rest of the story.

Here is what we will cover:

1. Type hints, and the fact that Python ignores them
2. Running a type checker, which doesn't
3. Docstrings, and what belongs in one
4. logging: levels, loggers, and why print isn't enough
5. assert versus real validation, and the flag that deletes your asserts
6. pytest: writing tests, running them, reading a failure
7. Reading a traceback, and breakpoint()
8. PEP 8, briefly

Setting Up

This chapter writes a few small files and tests them, so it needs a folder to work in. shutil.rmtree deletes anything left over from a previous run (and ignore_errors=True means "don't complain if there's nothing to delete"), which keeps the output the same every time:

import os
import shutil
from pathlib import Path

shutil.rmtree("reliable_demo", ignore_errors=True)
os.makedirs("reliable_demo", exist_ok=True)

print("working in", Path("reliable_demo").resolve().name)

working in reliable_demo

Path(...).resolve().name turns the folder into its full path and then takes just the last part, so the output doesn't show your computer's folder structure.

Part 1: Type Hints

A type hint says what kind of value a function expects and what kind it gives back. It goes after a colon for each argument, and after an arrow (->) for the return value:

def average(numbers: list[float]) -> float:
    return sum(numbers) / len(numbers)

Read it as: "average takes numbers, which should be a list of floats, and returns a float."

Apart from the fields of chapter 15's dataclass, everything in this course has been written without them, and it all ran fine. So what are they for?

Python Doesn't Check Them

This is the first thing to understand, and it surprises a lot of people:

def average(numbers: list[float]) -> float:
    return sum(numbers) / len(numbers)


print(average([10.0, 20.0, 30.0]))
print(average([1, 2, 3]))                   # ints, not floats. Fine.
print(average((4, 5, 6)))                   # a tuple, not a list. Also fine.

20.0
2.0
5.0

None of those calls match the hint exactly:

• [1, 2, 3] is a list of integers, not floats.
• (4, 5, 6) is a tuple, not a list.

And Python didn't care. Type hints are not checked when the code runs. Python stores them on the function and otherwise ignores them:

print(average.__annotations__)

{'numbers': list[float], 'return': <class 'float'>}

__annotations__ is where Python keeps the hints, as a dictionary: argument name to hint, plus 'return' for the return value. That's all Python does with them.

Analogy: a type hint is like the "Fragile: this way up" label on a parcel. The label doesn't stop anyone from turning the box upside down. It tells people what the box expects, and a careful inspector at the depot (the type checker in Part 2) can spot when the instructions are being ignored.

So a type hint isn't a safety net. It's two other things.

It's documentation that stays up to date. A comment saying "pass a list of floats" goes stale the moment someone changes the function. A hint sits right in the function's first line, where the change happens.

It can be checked by a tool. That's the next section, and the main reason hints are worth writing.

(There's a third, everyday benefit too: code editors like VS Code and PyCharm read hints to give you better autocomplete and to underline mistakes as you type.)

The Hints You'll Actually See

That "or" hint is written str | None, with the | character meaning "or". (It's the same symbol you met for sets and regex alternation, given a new job.)

str | None (Python 3.10 and later) is the one you'll see most, because it describes every function that might not find what it was looking for. Chapter 18's text_of is exactly that kind of function, so here it is with hints:

from bs4 import BeautifulSoup


def text_of(parent, selector: str, default: str | None = None) -> str | None:
    found = parent.select_one(selector)
    return found.get_text(strip=True) if found else default


soup = BeautifulSoup(Path("sample_data/shop_page_2.html").read_text(encoding="utf-8"),
                     "html.parser")
article = soup.select("article.product")[1]

print(repr(text_of(article, "h3 a")))
print(repr(text_of(article, "p.price")))
print(repr(text_of(article, "p.price", default="")))

'Effective Pandas'
None
''

The three calls show the three possibilities:

• the title exists, so you get a string: 'Effective Pandas',
• the price is missing, so you get None,
• the price is missing but we passed default="", so you get an empty string.

The first line of the function now tells you what the output already showed: this function can give you None, so check for it. That's the whole point.

parent has no hint, on purpose. Its type is a BeautifulSoup Tag, and naming it would mean importing a class just for the hint. Hints are optional, and you can add them to some arguments and not others. Add them where they help.

In older code you'll often see Optional[float] (imported from the typing module), which is the older way of writing float | None. Likewise List[int] and Dict[str, float] (with capital letters, from typing) are the older spellings of list[int] and dict[str, float].

Part 2: A Tool That Does Check

mypy is a type checker: it reads the hints and reports the places where they don't add up, without running anything.

Analogy: mypy is like a spell checker for types. It reads your writing and underlines the places where the words don't fit together, before anyone else reads it. It doesn't run your program, just as a spell checker doesn't post your letter.

To try it we need real files. %%writefile at the top of a notebook cell saves the cell's contents to a file instead of running it, as in chapter 11. Here's a small module with two functions, both with hints:

%%writefile reliable_demo/prices.py
"""Turning scraped price text into numbers."""
import re


def to_price(text: str) -> float | None:
    """Return the number in a price string, or None if there is not one."""
    digits = re.sub(r"[^\d.]", "", text or "")
    return float(digits) if digits else None


def total(prices: list[float]) -> float:
    """Add up a list of prices."""
    return sum(prices)

Writing reliable_demo/prices.py

And here's a file that uses it, with two real mistakes in it:

%%writefile reliable_demo/uses_prices.py
from prices import to_price, total

total(["12.50", "8.00"])                # strings, not floats
count: int = to_price("£12.50")         # to_price can return None, and a float

Writing reliable_demo/uses_prices.py

• total expects a list of floats, but we're passing a list of strings.
• to_price can return a float or None, but we're storing the result in a variable we've declared as int (count: int = ... is a type hint on a variable).

mypy is a command-line tool, so the next cell runs it with subprocess.run, which runs a terminal command from Python and captures what it prints. sys.executable (chapter 11) makes sure it uses the same Python as this notebook:

import subprocess
import sys

result = subprocess.run(
    [sys.executable, "-m", "mypy", "uses_prices.py",
     "--no-color-output", "--no-error-summary"],
    cwd="reliable_demo", capture_output=True, text=True)

print(result.stdout or "(mypy found nothing)")

uses_prices.py:3: error: List item 0 has incompatible type "str"; expected "float"  [list-item]
uses_prices.py:3: error: List item 1 has incompatible type "str"; expected "float"  [list-item]
uses_prices.py:4: error: Incompatible types in assignment (expression has type "float | None", variable has type "int")  [assignment]

The arguments say: run mypy on uses_prices.py, without colours (--no-color-output) and without the summary line at the end (--no-error-summary), inside the reliable_demo folder. capture_output=True, text=True collects what it prints as a string in result.stdout.

Each error tells you the file, the line number, what's wrong, and a code in square brackets (like [list-item]) naming the kind of error.

Two mistakes, reported as three errors, because the first bad call has two bad items in its list. mypy found both mistakes without running the file.

If you did run it, the first one would crash inside sum when it tried to add up strings. The second wouldn't crash at all: count would quietly hold a float, or None if the price were missing, until something later tried to do maths with it. That's exactly the bug chapter 18's missing price would cause.

   your code ──► mypy reads the hints ──► reports mismatches ──► you fix them ──► then run the code
                 (nothing is executed)

You run mypy the same way you run tests: from the terminal, or automatically before every commit. It's already in this course's requirements.txt:

python3 -m pip install mypy         # if you haven't installed requirements.txt
mypy your_package/

Hints are optional, and you can add as many or as few as you like. Add them to the functions other code calls, skip them in a three-line script, and don't let anyone tell you it's all or nothing.

One thing to know about that "add as few as you like" freedom: by default, mypy doesn't check the inside of a function that has no hints at all. It assumes an unhinted function is code you haven't opted in yet. You'll see this in action in Part 6.

Part 3: Docstrings

You met docstrings in chapter 13: a string written as the first statement of a function, class or module. Python stores it, help() prints it, and code editors show it when you hover over the name.

import re


def to_price(text: str) -> float | None:
    """Return the number in a price string, or None if there is not one.

    Strips currency symbols, spaces and thousands separators, so
    "£1,299.00" becomes 1299.0 and "free" becomes None.
    """
    digits = re.sub(r"[^\d.]", "", text or "")
    return float(digits) if digits else None


print(to_price.__doc__.splitlines()[0])
print()
help(to_price)

Return the number in a price string, or None if there is not one.

Help on function to_price in module __main__:

to_price(text: str) -> float | None
    Return the number in a price string, or None if there is not one.

    Strips currency symbols, spaces and thousands separators, so
    "£1,299.00" becomes 1299.0 and "free" becomes None.

• to_price.__doc__ is where Python keeps the docstring. .splitlines()[0] takes just the first line.
• help(to_price) prints the function's signature (including the type hints) and its whole docstring, the same way help() works on any built-in function.

How to Write a Good One

The first line is a one-sentence summary, written as a command: "Return the number", not "This function returns the number". Then a blank line, then anything a caller needs to know that the function's first line doesn't already say.

Analogy: a docstring is like the label on a medicine bottle. The first line says what it's for. The rest covers what you need to know to use it safely: the dose, the side effects, when not to take it. It doesn't explain the chemistry of how it was made.

What belongs in one, and what doesn't:

A docstring that says """Converts text to price.""" above def to_price(text) adds nothing; the name already said that. """...or None if there is not one.""" tells the caller something they really needed to know.

Part 4: Logging

Chapter 9 introduced the logging module and logger.exception(). Here's the fuller picture, starting with why print isn't enough. print is fine in a notebook, but wrong for anything that runs without you watching:

Analogy: print is like shouting a message across the room: fine if someone's there, lost if nobody is. Logging is like a ship's logbook: every entry has a time, a level of importance, and who wrote it, and it's still there to read the next morning.

The Five Levels

They exist so you can turn the volume down without deleting anything.

import logging
import sys

logger = logging.getLogger("shop.scraper")          # a name, usually __name__
logger.setLevel(logging.DEBUG)
logger.handlers.clear()                             # so re-running this cell is clean

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logging.Formatter("%(levelname)-8s %(name)s | %(message)s"))
logger.addHandler(handler)
logger.propagate = False

logger.debug("parsed 3 products from page 1")
logger.info("page 1 of 3 done")
logger.warning("no price for sku 9781098139482")
logger.error("page 4 returned 500, giving up on it")
logger.critical("no pages could be fetched at all")

DEBUG    shop.scraper | parsed 3 products from page 1
INFO     shop.scraper | page 1 of 3 done
WARNING  shop.scraper | no price for sku 9781098139482
ERROR    shop.scraper | page 4 returned 500, giving up on it
CRITICAL shop.scraper | no pages could be fetched at all

The Pieces of the Setup

There are three objects involved, and it helps to know which does what:

• A logger is what your code talks to: logger.info(...). logging.getLogger("shop.scraper") creates (or fetches) the logger with that name.
• A handler decides where the messages go. StreamHandler(sys.stdout) sends them to the screen. A FileHandler would send them to a file. One logger can have several handlers.
• A formatter decides what each line looks like.

   logger.warning("no price ...")
          │
          ▼
   ┌───────────────┐  level high enough?  ┌───────────────┐   formatter   ┌───────────────┐
   │ logger        │ ───────────────────► │ handler(s)    │ ────────────► │ screen / file │
   │ "shop.scraper"│        (else         │ where it goes │  what it      │               │
   └───────────────┘      dropped)        └───────────────┘  looks like   └───────────────┘

The rest of the setup, line by line:

• logger.setLevel(logging.DEBUG) lets every level through, for now.
• logger.handlers.clear() removes any handler from a previous run of the cell, so messages don't appear twice.
• %(levelname)-8s, %(name)s and %(message)s are logging's own fill-in-the-blank codes. -8 pads the level name to 8 characters so the lines line up.
• logger.propagate = False stops the message also being passed up to Python's main logger (the "root" logger). If that main logger has been given a handler of its own (for example by basicConfig, below), the message would otherwise be printed a second time.
• In a real script, the format would start with %(asctime)s for a timestamp. This notebook leaves it out so the output is the same every time it runs.

Turning the Volume Down

The level works as a threshold. Raise it, and the less important messages disappear without any code changing:

logger.setLevel(logging.WARNING)

logger.debug("parsed 3 products from page 1")       # gone
logger.info("page 1 of 3 done")                     # gone
logger.warning("no price for sku 9781098139482")
logger.error("page 4 returned 500, giving up on it")

print("\nonly two of the four messages got through")

WARNING  shop.scraper | no price for sku 9781098139482
ERROR    shop.scraper | page 4 returned 500, giving up on it

only two of the four messages got through

With the level at WARNING, debug and info messages are dropped; warning, error and critical still get through.

Analogy: it's like the volume knob on a radio. Every station is still broadcasting, but you only hear the ones above the level you've set. Turn it back up while debugging, down again in production.

This is the whole argument in one example. Every print you've ever deleted before sharing your code could have been a logger.debug you left in.

One Logger per Module

The usual line at the top of a module is:

logger = logging.getLogger(__name__)

__name__ is the module's name, from chapter 11. In a package called shop, shop/scraper.py gets a logger called shop.scraper. Loggers form a family tree based on those dotted names, so you can quieten the whole package with one call on shop, or turn up just the scraper.

   root
   └── shop                ← setLevel(WARNING) here quietens everything below
       ├── shop.scraper    ← ...unless this one has its own level, e.g. DEBUG
       └── shop.cleaner

Logging an Exception

As chapter 9 showed, logger.exception() inside an except block records your message and the traceback. It's one of the most useful calls in the module:

logger.setLevel(logging.DEBUG)


def parse_row(row):
    return float(row["price"])


try:
    parse_row({"price": "not a number"})
except ValueError:
    logger.exception("could not parse row 12")

ERROR    shop.scraper | could not parse row 12
Traceback (most recent call last):
  File "/var/folders/6t/w0q7sf4n28n8p23_b3rqs2x80000gn/T/ipykernel_58214/1405366784.py", line 9, in <module>
    parse_row({"price": "not a number"})
  File "/var/folders/6t/w0q7sf4n28n8p23_b3rqs2x80000gn/T/ipykernel_58214/1405366784.py", line 5, in parse_row
    return float(row["price"])
           ^^^^^^^^^^^^^^^^^^^
ValueError: could not convert string to float: 'not a number'

The traceback is in the log, and the program carried on. (The file name in it is the temporary one Jupyter gives each notebook cell, which is why this is one of the few examples whose output will look different on your machine. Newer Python versions also draw the ^^^^ markers under the code slightly differently.)

logger.exception logs at the ERROR level, and only makes sense inside an except block, where there's an exception to attach.

Compare that with except ValueError: pass, which chapter 9 warned leaves no evidence at all, or with print("error!"), which tells you nothing you can act on.

Setting It Up in a Script

The setup above shows each piece separately. A real script calls basicConfig once, at the start, and every module's logger then feeds into it (chapter 9 showed this too):

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)-8s %(name)s | %(message)s",
    handlers=[logging.FileHandler("scraper.log"),
              logging.StreamHandler()],          # file and screen
)

This sets the level to INFO, adds a timestamp to every line, and sends each message to two places at once: a file called scraper.log and the screen.

Two rules worth following from the start:

• Never log a secret, because logs get copied, emailed and sent to other services. Chapter 17 said the same about API keys.
• Log the values you'll want later, not just the fact that something failed. Logging uses %s as a placeholder: logger.warning("no price for sku %s", sku) puts the actual sku into the message. That's useful. "parse failed" on its own isn't.

(Why %s with a separate argument, instead of an f-string? Logging only fills in the blanks if the message is actually going to be shown, so a debug message that's switched off costs almost nothing. Either works; the %s style is the convention.)

Part 5: assert, and the Flag That Deletes It

Chapter 9 explained assert, and its rule: use it for things you believe are true, never for checking input. Here's a quick reminder of how it works:

def average_price(prices):
    assert prices, "average_price got an empty list"
    return sum(prices) / len(prices)


print(average_price([10.0, 20.0]))

try:
    average_price([])
except AssertionError as e:
    print("AssertionError:", e)

15.0
AssertionError: average_price got an empty list

assert prices, "..." means "I'm sure prices isn't empty; if it is, stop with this message". With an empty list, it raised AssertionError.

That looks like a perfectly good way to check input. Chapter 9 said it isn't, because of one flag, and promised to show you. Here it is.

Running Python with -O (for "optimise") removes every assert statement from the code before it runs. It doesn't skip them, it deletes them. Here's a function whose only protection is an assert:

%%writefile reliable_demo/guard.py
def withdraw(balance, amount):
    assert amount > 0, "amount must be positive"
    return balance - amount


print("balance after withdrawing -50:", withdraw(100, -50))

Writing reliable_demo/guard.py

Now let's run that file twice, once normally and once with -O:

for flags in ([], ["-O"]):
    result = subprocess.run([sys.executable, *flags, "guard.py"],
                            cwd="reliable_demo", capture_output=True, text=True)
    label = "python3 " + " ".join(flags) if flags else "python3"
    print(f"{label:<12} ->", (result.stdout or result.stderr).strip().splitlines()[-1])

python3      -> AssertionError: amount must be positive
python3 -O   -> balance after withdrawing -50: 150

The loop runs python3 guard.py and then python3 -O guard.py, and prints the last line of whatever each one printed (its normal output, or its error).

Without the flag, the guard works. With it, withdrawing minus fifty increases the balance to 150, and nothing complains.

Analogy: an assert is like a safety note pinned to a machine: "never run this with the guard off". -O is like a cleaner who removes every note to make the room look tidier. The machine still runs, but the warning is gone, and nobody notices until someone gets hurt.

You might never use -O yourself, but you don't always control how your code gets run, and some setups switch it on for a small speed boost (the PYTHONOPTIMIZE environment variable does the same thing).

So the rule is:

Here's the same function, written properly:

def withdraw(balance: float, amount: float) -> float:
    if amount <= 0:
        raise ValueError(f"amount must be positive, got {amount}")
    return balance - amount


try:
    withdraw(100, -50)
except ValueError as e:
    print("ValueError:", e)

ValueError: amount must be positive, got -50

An if with raise is ordinary code, so -O can't remove it. And the error message now includes the actual bad value (got -50), which makes it much easier to track down.

Where assert does belong is inside tests, which is the next section, and as an internal check on something you've already validated.

Part 6: Testing with pytest

A test is a small function that runs your code and checks the answer. pytest is a tool that finds your tests and runs them for you.

Analogy: tests are like the answer key a teacher uses to mark homework. Every time you change your code, you hand it in again, and the answer key instantly tells you which questions it now gets wrong. Without one, you'd have to re-check every answer by hand after every change, and you'd soon stop bothering.

Its naming rules are the whole setup:

• files named test_*.py,
• functions named test_*,
• and a plain assert for each check.

There's no class to inherit from and nothing to register. pytest finds anything that follows those names.

Here's a test file for the prices.py we wrote in Part 2:

%%writefile reliable_demo/test_prices.py
"""Tests for prices.py. Run with: pytest"""
from prices import to_price


def test_plain_number():
    assert to_price("12.50") == 12.50


def test_strips_currency_and_separators():
    assert to_price("£1,299.00") == 1299.0


def test_missing_price_is_none():
    assert to_price("") is None
    assert to_price("free") is None


def test_none_input_is_none():
    assert to_price(None) is None

Writing reliable_demo/test_prices.py

Each test checks one behaviour, and its name says which: a plain number, stripping currency symbols and commas, a missing price, and None as input. If one fails, the name alone tells you what broke.

Now let's run it. The helper below runs pytest the same way you would in a terminal:

import re


def run_pytest(*args):
    result = subprocess.run([sys.executable, "-m", "pytest",
                             "--no-header", "--color=no",
                             "-p", "no:cacheprovider", *args],
                            cwd="reliable_demo", capture_output=True, text=True,
                            env={**os.environ, "COLUMNS": "80"})
    # the duration is different every run, so blank it out
    print(re.sub(r"in \d+\.\d+s", "in Xs", result.stdout))


run_pytest("test_prices.py", "-q")

....                                                                     [100%]
4 passed in Xs

• --no-header and --color=no keep the output plain.
• -p no:cacheprovider stops it leaving a cache folder behind.
• COLUMNS=80 fixes the width of the output.
• pytest also prints how long it took, which changes every run, so the helper uses chapter 16's re.sub to hide that one number.
• -q means "quiet": one character per test.

Four dots, four passing tests. In a terminal, that's just:

pytest

Hints Versus Tests: a Small Surprise

Look at the last test: to_price(None). But to_price's hint says text: str, not str | None. So why didn't mypy complain about this file? Let's ask it, with and without one extra option:

for extra in ([], ["--check-untyped-defs"]):
    result = subprocess.run(
        [sys.executable, "-m", "mypy", "test_prices.py",
         "--no-color-output", "--no-error-summary", *extra],
        cwd="reliable_demo", capture_output=True, text=True)
    label = " ".join(extra) or "plain mypy"
    print(f"{label:<22} ->", result.stdout.strip() or "no errors reported")

plain mypy             -> no errors reported
--check-untyped-defs   -> test_prices.py:19: error: Argument 1 to "to_price" has incompatible type "None"; expected "str"  [arg-type]

Plain mypy reported nothing, because the test functions have no hints of their own, and by default mypy doesn't look inside unhinted functions (Part 2). With --check-untyped-defs, it checks them anyway, and spots the mismatch.

Both tools are telling the truth. The test proves to_price copes with None (thanks to text or ""), and mypy points out that the hint doesn't admit it. The honest fix would be to change the hint to text: str | None, so the first line of the function says what the code really accepts. Hints and tests check different things, which is why reliable code uses both.

Reading a Failure

A passing test doesn't tell you much. A failing one is where the useful output is, so here's a test file with a deliberately wrong expectation in it:

%%writefile reliable_demo/test_broken.py
from prices import to_price, total


def test_total_of_two_prices():
    assert total([12.50, 8.00]) == 20.50


def test_price_keeps_the_pence():
    assert to_price("£19.99") == 19.0        # wrong on purpose

Writing reliable_demo/test_broken.py

run_pytest("test_broken.py", "-q")

.F                                                                       [100%]
=================================== FAILURES ===================================
__________________________ test_price_keeps_the_pence __________________________

    def test_price_keeps_the_pence():
>       assert to_price("£19.99") == 19.0        # wrong on purpose
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       AssertionError: assert 19.99 == 19.0
E        +  where 19.99 = to_price('£19.99')

test_broken.py:9: AssertionError
=========================== short test summary info ============================
FAILED test_broken.py::test_price_keeps_the_pence - AssertionError: assert 19...
1 failed, 1 passed in Xs

How to read that output:

• .F means the first test passed (.) and the second failed (F).
• Under FAILURES, pytest shows the test function, with > pointing at the line that failed.
• The E lines explain why: assert 19.99 == 19.0. The function returned 19.99, and the test expected 19.0.
• + where 19.99 = to_price('£19.99') even shows which call produced that value.
• test_broken.py:9 is the file and line number.
• The short test summary at the bottom lists every failure in one line each.

pytest showing the actual values on both sides of == is a big reason people use it, and all it needs is a plain assert.

Here, the test is wrong (19.99 is the right answer). In real life it's usually the code, but a failing test always means "the code and the expectation disagree", and you decide which one to fix.

Testing That Something Raises an Error

The other half of testing is checking that bad input gets rejected. A test file can only import functions that live in a .py file, so Part 5's checked withdraw gets a file of its own first. Notice its docstring has a Raises section, from Part 3:

%%writefile reliable_demo/banking.py
"""The validated withdraw from section 5, in a module so it can be tested."""


def withdraw(balance: float, amount: float) -> float:
    """Return the balance after withdrawing amount.

    Raises:
        ValueError: if amount is zero or negative.
    """
    if amount <= 0:
        raise ValueError(f"amount must be positive, got {amount}")
    return balance - amount

Writing reliable_demo/banking.py

Then pytest.raises works as a context manager, from chapter 14:

%%writefile reliable_demo/test_raises.py
import pytest

from banking import withdraw


def test_rejects_a_negative_amount():
    with pytest.raises(ValueError):
        withdraw(100, -50)


def test_says_what_was_wrong():
    with pytest.raises(ValueError, match="must be positive"):
        withdraw(100, 0)


def test_a_normal_withdrawal_works():
    assert withdraw(100, 30) == 70

Writing reliable_demo/test_raises.py

• with pytest.raises(ValueError): means "the code inside must raise ValueError". If it does, the test passes. If it doesn't, the test fails.
• match="must be positive" also checks the error message contains that text (it's a regex search, from chapter 16).
• The third test checks the normal case still works, which matters just as much.

run_pytest("test_raises.py", "-q")

...                                                                      [100%]
3 passed in Xs

These tests pass when the error is raised, and fail when it isn't. That's the opposite of everywhere else in this course, where an exception meant something went wrong.

One Test, Many Cases

Writing the same test five times with different numbers is a quick way to get bored of testing. @pytest.mark.parametrize is a decorator (chapter 13) that runs one test once for each case:

%%writefile reliable_demo/test_many.py
import pytest

from prices import to_price


@pytest.mark.parametrize("text, expected", [
    ("12.50", 12.50),
    ("£12.50", 12.50),
    ("  12.50  ", 12.50),
    ("£1,299.00", 1299.0),
    ("USD 45", 45.0),
    ("", None),
    ("free", None),
    (None, None),
])
def test_to_price(text, expected):
    assert to_price(text) == expected

Writing reliable_demo/test_many.py

• The first argument, "text, expected", names the test's parameters.
• The list holds one tuple per case: the input text, and the expected result.
• pytest calls test_to_price eight times, once with each pair.

run_pytest("test_many.py", "-v")

============================= test session starts ==============================
collecting ... collected 8 items

test_many.py::test_to_price[12.50-12.5] PASSED                           [ 12%]
test_many.py::test_to_price[\xa312.50-12.5] PASSED                       [ 25%]
test_many.py::test_to_price[  12.50  -12.5] PASSED                       [ 37%]
test_many.py::test_to_price[\xa31,299.00-1299.0] PASSED                  [ 50%]
test_many.py::test_to_price[USD 45-45.0] PASSED                          [ 62%]
test_many.py::test_to_price[-None] PASSED                                [ 75%]
test_many.py::test_to_price[free-None] PASSED                            [ 87%]
test_many.py::test_to_price[None-None] PASSED                            [100%]

============================== 8 passed in Xs ===============================

-v means "verbose": one line per test. Eight cases, one test function, and each case is named in the output (in square brackets), so a failure tells you exactly which input broke. The \xa3 in two of those names is pytest writing £ in a safe, plain-text form.

Analogy: parametrize is like a mail merge: one letter template, many names and addresses, and a separate letter printed for each.

Running Everything

With no filename, pytest finds and runs every test_*.py file in the folder. That includes the broken one, on purpose:

run_pytest("-q")

.F...............                                                        [100%]
=================================== FAILURES ===================================
__________________________ test_price_keeps_the_pence __________________________

    def test_price_keeps_the_pence():
>       assert to_price("£19.99") == 19.0        # wrong on purpose
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       AssertionError: assert 19.99 == 19.0
E        +  where 19.99 = to_price('£19.99')

test_broken.py:9: AssertionError
=========================== short test summary info ============================
FAILED test_broken.py::test_price_keeps_the_pence - AssertionError: assert 19...
1 failed, 16 passed in Xs

17 tests in total: four in test_prices.py, two in test_broken.py, three in test_raises.py and eight in test_many.py. Three files pass, one has a failure, and the summary at the bottom tells you exactly where to look.

What's Worth Testing

You can't test everything, and trying to is how people give up on testing. Aim at:

• The edge cases you've already thought about. Empty input, None, zero, a missing field. Chapter 18's missing price is a test waiting to be written.
• Anything you've fixed once. A test written the day you fix a bug stops that bug coming back. (This is often called a regression test: it catches the code going backwards.)
• The parts with rules in them. Parsing, validation, calculations. Not the code that just connects things together.

What not to test: the standard library (it's already well tested), the network (fake it or use a saved response, as chapter 17 did with sample_data/), and things whose answer keeps changing, like the current time.

Part 7: When It Breaks Anyway

Reading a Traceback

A traceback is the report Python prints when an error stops the program. Read it from the bottom up. The bottom line is the error. Above it is the call that caused it, and above that, the call that made that call.

Here's a small program with a hidden problem:

%%writefile reliable_demo/report.py
from prices import to_price


def load_prices(rows):
    return [to_price(row["price"]) for row in rows]


def report(rows):
    prices = load_prices(rows)
    return sum(prices) / len(prices)


print(report([{"price": "£12.50"}, {"price": "free"}]))

Writing reliable_demo/report.py

And here's what happens when it runs:

result = subprocess.run([sys.executable, "report.py"],
                        cwd="reliable_demo", capture_output=True, text=True)

# Python reports absolute paths; shorten them so this reads like your own screen
print(result.stderr.replace(str(Path.cwd()) + "/", ""))

Traceback (most recent call last):
  File "reliable_demo/report.py", line 13, in <module>
    print(report([{"price": "£12.50"}, {"price": "free"}]))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "reliable_demo/report.py", line 10, in report
    return sum(prices) / len(prices)
           ^^^^^^^^^^^
TypeError: unsupported operand type(s) for +: 'float' and 'NoneType'

(The cell shortens the file paths so the output looks like your own screen. Newer Python versions also show this traceback in colour and draw the markers under the code slightly differently, but the content is the same.)

Read it in the order it happened:

1. The last line says what went wrong: unsupported operand type(s) for +: 'float' and 'NoneType'. Something tried to add a float and None.
2. The frame above it (each File ... line ... pair is called a frame) shows sum(prices), so prices contains a None.
3. The frame above that is just the call at the bottom of the file.

   Traceback (most recent call last):
     File "report.py", line 13, in <module>   ← 3. who called report()
     File "report.py", line 10, in report     ← 2. the line that crashed: sum(prices)
   TypeError: ... 'float' and 'NoneType'      ← 1. START HERE: what went wrong

   not shown: load_prices() ── it already finished, after putting None in the list

Now notice what's not there. load_prices doesn't appear in the traceback at all, and load_prices is where the None came from. to_price("free") returned None, exactly as its docstring said it would, and load_prices put it in the list and finished without any problem.

That's worth remembering about tracebacks: they show you where the program crashed, not where the mistake was made. The frames tell you which calls were still running. The mistake is often in a function that had already finished.

Analogy: a traceback is like finding a flat tyre. It tells you exactly where the car stopped. It doesn't tell you where you drove over the nail, which might have been ten kilometres earlier.

Two habits make tracebacks much easier:

• Read the last line first, then look for the highest frame in your own code, skipping frames inside libraries (their file paths usually contain site-packages).
• Don't paste the error into a search engine before reading it. It usually tells you exactly what's wrong.

breakpoint()

Putting print everywhere works. Pausing the program and looking around works better. breakpoint() stops the program at that line and opens Python's debugger, called pdb:

def report(rows):
    prices = load_prices(rows)
    breakpoint()                  # execution stops here
    return sum(prices) / len(prices)

The debugger waits for you to type commands, so it isn't included as a runnable example here: an automated run would just sit there waiting. In a script run from a terminal, you get a (Pdb) prompt, with the program paused and every local variable available to look at. (In Jupyter, the same prompt appears as an input box under the cell.)

Typing p prices at that prompt would have shown [12.5, None] straight away, and that's the whole investigation done.

Analogy: print debugging is like putting security cameras around a building and watching the footage afterwards. A debugger is like pausing time and walking around the building yourself, opening any drawer you want.

In VS Code or PyCharm, the same thing is a red dot you click in the margin (a breakpoint), and the variables appear in a panel. Either way, the skill is the same: stop close to the problem and look at the actual values.

Remember to remove breakpoint() calls when you're done. (Setting the environment variable PYTHONBREAKPOINT=0 also switches them all off, which is a useful safety net.)

Part 8: PEP 8, Briefly

PEP 8 is Python's official style guide. ("PEP" stands for Python Enhancement Proposal: the documents in which changes and conventions for Python are written down.) Following it matters for one main reason: code that looks like every other Python project is code other people can read quickly.

Analogy: it's like the rules of the road. Driving on the left or the right isn't better in itself, but everyone agreeing on one makes traffic flow. Consistent style lets readers focus on what the code does, not on how it's laid out.

The parts that come up all the time:

Nobody applies this by hand. black reformats a file into a consistent style, and ruff finds problems (unused imports, undefined names, style issues), and both are very fast:

python3 -m pip install black ruff
black your_package/
ruff check your_package/

Run them before you commit and you'll never argue about formatting again. The naming rules are the part worth memorising, because no formatter will rename anything for you.

Part 9: Common Mistakes

Quick Reference Summary

Type hints

def to_price(text: str) -> float | None: ...
def parse(rows: list[dict[str, str]]) -> dict[str, float]: ...
def save(path: str, rows: list[dict]) -> None: ...

python3 -m pip install mypy
mypy your_package/            # hints are checked here, never when the code runs

Docstring

def to_price(text: str) -> float | None:
    """Return the number in a price string, or None if there is not one.

    Strips currency symbols and separators, so "£1,299.00" becomes 1299.0.

    Raises:
        TypeError: if text is neither a string nor None.
    """

Logging

import logging
logger = logging.getLogger(__name__)

logger.debug("detail while developing")
logger.info("normal progress")
logger.warning("odd, but carried on")
logger.error("this piece of work failed")
logger.exception("same as error, plus the traceback")   # inside except only

logging.basicConfig(level=logging.INFO,
                    format="%(asctime)s %(levelname)-8s %(name)s | %(message)s")

assert versus raise

assert len(rows) > 0            # an assumption. -O deletes this.
if not rows:                    # a check. always runs.
    raise ValueError("no rows")

pytest

# test_prices.py
import pytest
from prices import to_price
from banking import withdraw

def test_plain_number():
    assert to_price("12.50") == 12.50

def test_rejects_negatives():
    with pytest.raises(ValueError, match="must be positive"):
        withdraw(100, -50)

@pytest.mark.parametrize("text, expected", [("12.50", 12.5), ("free", None)])
def test_cases(text, expected):
    assert to_price(text) == expected

pytest                  # everything
pytest -v               # one line per test
pytest test_prices.py   # one file
pytest -k price         # tests whose name contains "price"
pytest -x               # stop at the first failure

Debugging

breakpoint()            # then: p name, n, s, c, l, w, q

Read tracebacks from the bottom up. Last line first, then your own frames.

Style

black your_package/     # reformat
ruff check your_package/   # find problems