• strategy_is_a_function.py

Design Patterns

The “design patterns” movement was an important step forward in object-oriented design. Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides carried it into the mainstream with their 1994 book Design Patterns. The four became known as the “Gang of Four”¹. I refer to that book as GoF Design Patterns, and use design patterns for the concept.

GoF Design Patterns shows 23 solutions to particular classes of problems, along with one or more examples for each, typically in C++ but sometimes in Smalltalk. Many of those examples inspired the ones in this part of the book.

This chapter introduces the concepts. Its one listing shows a pattern shrinking to a function. The chapters that follow supply the rest of the code.

What Is a Pattern?

At first, think of a pattern as an especially clever and insightful way to solve a particular class of problems. Many people have worked out all the angles of a problem and arrived at the most general, flexible solution. You may have seen and solved something like it before, but your solution probably lacks the completeness a pattern embodies.

That completeness has a failure mode. When you know a catalog of patterns, that catalog tempts you to treat it as a checklist, and to install patterns as proof of sophistication. A pattern belongs in your program only when you have the problem it solves. Machinery for isolating variation needs a variation to isolate.

Despite the name “design patterns,” patterns apply beyond design. Because a pattern translates into code, you might expect it to appear first in low-level design. But a pattern embodies a complete idea within a program, so it can appear at the analysis phase or high-level design phase, where you are still describing what the system does rather than how to build it. A pattern appears at every level. Often you first discover that you need one when you reach the code.

The basic concept of a pattern is also the basic concept of program design: adding a layer of abstraction. Whenever you abstract something, you isolate particular details. One of the most compelling motivations for abstraction is to separate things that change from things that stay the same. Once you find a part of your program that’s likely to change, patterns can keep those changes from reaching the rest of your code. That isolation confines each change to one place, which makes the code easier to maintain and usually simpler to understand.

What an Abstraction Erases

Isolating what changes discards information. An abstraction is a guess about which details no caller will need. A good abstraction does more than hide those details. It erases them. Code outside the boundary cannot recover information the boundary discards, so a caller that turns out to need an erased detail cannot work around the interface. Someone must change the interface. The art of design lies in guessing well about which details you can hide and which you must expose.

Erased details are also the source of scaling limits. Every abstraction hides some work that takes negligible time at the size you built it: a copy, a sort, a lookup behind an attribute. A growing system eventually reaches the size where one piece of hidden work dominates the running time. The layer that made the code simple now hides the work you need to change. That hidden work is one reason Performance tells you to measure at a realistic size rather than trust a small trial.

The Vector of Change

Often, the most difficult part of developing an elegant and easy-to-maintain design is discovering what I call “the vector of change” (here, “vector” means a direction of change, not an array of numbers). You look for the most important thing that changes, because most of your maintenance effort goes into that change. Once you discover the vector of change, you have the focal point around which to structure your design.

You discover a vector of change. You do not predict it. Guessing at it up front often adds complexity to allow flexibility in a direction that goes unused. The second time a requirement changes the same part of the design, you have evidence.

Patterns You Have Already Seen

You have seen some design patterns in this book. For example, you can think of inheritance as a design pattern (albeit one built into the language). Inheritance lets you express differences in behavior in objects that all have the same interface. The behavior changes from one subclass to the next, while the interface stays the same. Composition also qualifies as a pattern, since it lets you change the objects to which your class delegates, and with them the way that class works. You can make that change dynamically or statically.

Another pattern that appears in GoF Design Patterns is the Iterator. An iterator lets you hide the particular implementation of the container as you step through it. You can write generic code that operates on all the elements in a sequence, whether the sequence holds them in an array, a linked list, or a tree. The code works with any object that produces an iterator.

Pattern Evolution

A pattern develops in stages, each more general than the last:

1. Idiom: how you write code in a particular language to do this particular type of thing. This can be as common as the way you step through an array in C (stopping at its last element).
2. Specific Design: the solution that arose to solve this particular problem. This might be a clever design, but it stays specific to that problem.
3. Standard Design: a specific design made general through reuse, so it solves this kind of problem wherever it appears in your programs.
4. Design Pattern: the shape several standard designs share, stated apart from any one program or language. This usually appears only after you apply a standard design several times, and then see a common pattern across those uses.

In Python terms: with open(...) for guaranteed cleanup is an idiom, stage one, tied to a language that provides with. A dictionary mapping one program’s shape names to its shape classes is a specific design, stage two. The same dictionary becomes a standard design, stage three, when each subclass registers itself as its class statement runs. A new type then adds its entry to the dictionary. Factory builds both designs. Template Method is a design pattern, stage four: a shape of solution you can build in any language with polymorphism.

The stages are a history, not a ranking. Patterns are subtle and appear over time, so leave a solution at the stage it has reached rather than forcing it toward a design pattern.

The progression also runs in reverse. A pattern that a language builds in becomes an idiom again, stage one. Programmers who arrive next learn it as syntax rather than as a design. Stepping through a container is stage one in Python and was stage four in the GoF Design Patterns examples.

When a Pattern Dissolves

A pattern is often a sign of something missing in a language. Programmers wrote the same supporting code often enough that it acquired a name. The pattern exists because the language leaves that code for them to write.

A language can supply that missing piece in two ways. Sometimes a language adds the feature, and the feature replaces the pattern². Iterator is the clear case. It was implicit in the for loop from the start, and Python 2.2 made it a protocol whose methods the for loop calls on your behalf. More often the language had the piece all along, and the pattern came from a language that didn’t. Strategy and Command shrink to passing a function, because a Python function is an object. A Factory becomes a dictionary, because a class is an object too. Singleton becomes a module, because Python imports each module once and caches it.

Here is the whole of a Strategy in Python:

# strategy_is_a_function.py
from collections.abc import Callable

def apply(nums: list[int],
          strategy: Callable[[list[int]], int]) -> int:
    return strategy(nums)
nums = [3, 1, 2]
for strategy in (max, sum):
    print(strategy.__name__, apply(nums, strategy))
#: max 3
#: sum 6

The classic form declares a Strategy interface, writes one class per algorithm, and adds a context class to hold the chosen algorithm. Here the Callable annotation on strategy replaces the interface, existing functions such as max() and sum() replace the algorithm classes, and apply() replaces the context class, holding the chosen algorithm in strategy for one call.

This listing shows only the shape. max() and sum() are too small to justify a Strategy class hierarchy. Function Objects works through three root-finding algorithms that a caller chooses between at runtime.

A function with a Callable parameter can replace a pattern’s interface and classes, so the chapters ahead keep asking a question from Rethinking Objects: how much of each pattern’s machinery does Python still need, and how much of it becomes functions, data, and protocols?

Pattern Taxonomy

GoF Design Patterns discusses 23 patterns and sorts them under three purposes, each named for what its patterns do:

1. Creational: how to create an object. When you isolate the details of object creation, your code stops depending on which object types exist. Adding a type then leaves that code unchanged. Singleton counts as a Creational pattern. Factory covers the other four: Factory Method, Abstract Factory, Prototype, and Builder.
2. Structural: how objects connect to other objects, arranged so that changes leave those connections alone. The structural patterns in this book appear in Surrogate, Changing the Interface, Flyweight, Decorators, and the Composite half of Composite and Interpreter.
3. Behavioral: objects that handle particular types of actions within a program. These encapsulate processes such as interpreting a language, fulfilling a request, moving through a sequence (as in an iterator), or implementing an algorithm. The largest group of patterns in this book is behavioral: Iterator, Template Method, Function Objects (Command, Strategy, and Chain of Responsibility), Observer, Visitor, Memento, State, and Interpreter. State appears beside Proxy, and Interpreter beside Composite, for the reason that closes this section.

The catalog above comes from GoF. Patterns from outside it, like the Null Object and the Data Transfer Object, appear in the Pattern Catalog.

I’ve found the GoF Design Patterns classification too obscure, and not always helpful. Certainly, the Creational patterns are straightforward. How will you create objects? This is a normal question, and the name tells you which group of patterns answers it. But I find Structural and Behavioral far less useful distinctions. I have not been able to look at a problem and say “clearly, you need a structural pattern here,” so that classification doesn’t lead me to a solution (I’ll readily admit that I may be missing something here).

Patterns often resemble each other more in their implementations than the GoF Design Patterns categories suggest. This book groups them by that resemblance. Surrogate treats Proxy and State as one front-object structure. Function Objects treats Command, Strategy, and Chain of Responsibility as one function-passing structure. Composite and Interpreter treats both of its patterns as one recursive-data structure. When two patterns share a structure, learning one teaches you most of the other. The remaining difference is intent.

Coupling

Every pattern in GoF Design Patterns answers one question: what does one part of a program know about another? GoF’s first chapter lists eight “common causes of redesign” and names the patterns that address each one. Six of the eight are a dependence of one part on another: on a class name, on a specific operation, on a platform, on an object’s representation, on an algorithm, or on each other, which the list calls tight coupling. The other two are about what a dependence makes hard once it exists: extending by subclassing, and altering a class.

The glossary defines coupling as “the degree to which software components depend on each other.” Both of that first chapter’s design principles tell you to loosen one particular dependence. “Program to an interface, not an implementation” says stop specifying concrete classes. “Favor object composition over class inheritance” says stop inheriting implementations. GoF Design Patterns has a name for the result, abstract coupling. A class that holds a reference to an abstract class “refers to a type of object, not a concrete object.” Read this way, the catalog is one idea applied twenty-three times.

GoF’s own discussion of the behavioral patterns compares four of them by their coupling. Command, Observer, Mediator, and Chain of Responsibility each decouple a sender from a receiver, “but with different trade-offs.” Observer “defines a looser sender-receiver binding than Command.”

A diagram makes coupling visible. Once you can see coupling, the patterns in the chapters ahead stop being twenty-three shapes to memorize, and become a few moves applied to one kind of diagram.

What One Part Knows About Another

Coupling has degrees. Every working program has parts that depend on each other, so the question is how much of the other part a dependence reaches.

Each rung is something the dependent part knows. A subclass knows its parent’s internals: which methods call which, the attributes’ names, what a super() call expects to find. A change to any of that can reach the subclass, so GoF Design Patterns repeats the saying that “inheritance breaks encapsulation.” A caller that writes Circle(2) knows a name and a constructor signature. A caller that writes isinstance(s, Circle) knows the name and tests an object’s type against it. A caller written against an abstract base class knows a set of method names and signatures, and every class that implements them says so in its class line. A caller written against a Protocol knows the same set of names, but no class declares that it implements them. The type checker compares each class’s method names and signatures with those of the protocol. A caller that takes a Callable knows one signature. A caller that takes a value knows the value’s type and nothing about who produced it.

The rung a dependence sits on decides how many parts a change can reach. If you rename a method on a class that other code subclasses, every subclass that overrides or calls the old name must change. If you rename it on a class that other code reaches through a Protocol, every caller’s source stays as it was, because each caller names the protocol’s method. The renamed class stops matching the protocol, and the type checker reports each place that passes it to a caller.

In Python, an implementer meets the Protocol and Callable rungs through its signatures alone. An abstract base class is a class you write and every implementer inherits. A Protocol is a class you write and no implementer inherits. A Callable annotation is not a class.

The Protocol and Callable rungs are the mechanism behind When a Pattern Dissolves. In a language where every implementer must specify its interface, a pattern exists to build a declared interface. In a language with the Protocol and Callable rungs, those rungs supply the interface, so the pattern dissolves.

A Pattern Moves an Edge

In a design drawn as parts and the edges between them, each edge says what the dependent part knows. A heavy edge names a concrete class. A thin edge names an interface. A dashed edge with a hollow head says the part satisfies that interface. With those three edges alone, six GoF patterns look like this:

The red box in each panel marks the part the pattern protects from change. Read the heavy edges first, because a change to the class at the head of a heavy edge can force an edit in the part at its tail.

Strategy has none. The context specifies an interface and each algorithm satisfies it, so adding an algorithm leaves every existing part as it was.

Observer has one, and it points from the observer to the subject. An observer specifies its subject’s class in update(), whose subject parameter receives the subject that changed. The subject knows each observer through the Observer interface: “All a subject knows is that it has a list of observers, each conforming to the simple interface of the abstract Observer class.” The heavy edge points at the part that changes least, which is where a heavy edge belongs.

Factory Method keeps a heavy edge too. Something must specify a class to construct it. The pattern moves that choice out of the client and into a creator. Naming the concrete product is the creator’s whole job. The client knows the creator and the product through their interfaces.

Adapter moves the heavy edge inside the adapter. The client specifies the target interface. The adapter alone knows the adaptee’s real name, so a change to the adaptee reaches one file.

Decorator, like Strategy, points every edge at its interface. A topping satisfies the component interface and also holds a component, so it can wrap a pizza or another topping, knowing each one through that interface.

Visitor is the reverse of the others. The visitor specifies every concrete element. The pattern concentrates the heavy edges in the visitors on purpose. Adding an operation means writing one new visitor and leaving every other part as it was, while adding an element type means changing every visitor. That choice defines Visitor. The diagram shows where the choice puts the heavy edges.

Seen this way, the patterns differ less in how much coupling they carry than in where they put it and which way it points. A heavy edge is acceptable when it sits in a part that is easy to change, or points at a part that rarely changes. The Managed Coupling principle asks for this: coupling you can see and have placed on purpose.

The Reach of a Change

Coupling decides which existing parts you edit when you add a part. A Report renders itself through a writer, and main assembles the pieces. On the left, Report names each writer class and chooses between them. On the right, Report specifies a Writer protocol. Then you add a Markdown writer.

The dashed box is the new class. The shaded parts are the existing ones whose source changes when you add MdWriter. Each heavy edge into the dashed box is a new place that specifies MdWriter. As in the gallery, the red box marks the part the design protects from the change.

On the left, Report gains a branch and main gains a case. On the right, Report keeps its source, because Report specifies the Writer protocol, which the new writer satisfies. main still changes, since something must construct the new class. A registry factory removes that last change. In a registry factory, the new class registers itself as its class statement runs.

A change to a part can force an edit in every part with a heavy edge pointing at it. A pattern moves those edges, usually out of the parts that change often and into one part you can find.

The Edge Python Deletes

One more diagram explains why so many of the pattern chapters subtract code as they go. Take a caller, a function area() that needs the area of whatever it receives, and write the same dependence three ways.

With an abstract base class, two edges exist in source: the caller specifies Shape, and Circle lists Shape in its class line. With a Protocol, one edge exists in source. Circle does not mention Shape. The type checker connects the two when it compares Circle’s members with the protocol’s members. The figure draws that connection as a dashed edge with a hollow head. A dashed edge exists only at check time, and neither of its ends specifies the other. With a Callable, the interface has no name and no file of its own. The caller’s signature declares it, and any function whose parameters and return type match satisfies it.

The edge a Protocol deletes is the one that a C++ base class or a Java interface makes mandatory. The implementer must specify the interface. That edge makes every implementer depend on the interface, so renaming or moving the interface means editing every file that names it. Rethinking Objects states the consequence: a protocol works with any type that has matching members, “including types in libraries you cannot edit.” In general, the fewer edges in source, the fewer files a change can reach.

One question serves for every pattern chapter. Which edge does this pattern move, and where does it put it?

Design Principles

Design principles are at least as important as design patterns, but they do a different job. A pattern is a shape of solution. A principle is a test you apply to the shape you chose: a claim the design either satisfies or violates.

Most of the principles below hold for any code, but Reflexivity and the Law of Demeter assume classes and objects. The list is a reference. Later chapters name a few of these principles when a design depends on them. The rest are here for your own designs.

• Principle of least astonishment (don’t be astonishing).
• Make common things easy, and rare things possible.
• Consistency. Every inconsistency in a design is one more arbitrary rule to remember. The more rules you impose beyond what the problem at hand needs, the slower the programmer works. The slowdown grows faster than the rule count, because the rules interact.
• Liskov Substitution Principle (LSP). A subtype must work anywhere code expects its base type.
• Law of Demeter: a.k.a. “Don’t talk to strangers.” A method should talk to its own object (self), that object’s attributes, its parameters, and objects it creates, not to the internals of objects it reached through something else. The Law of Demeter is another way to say “minimize coupling.”
• Independence or Orthogonality. Express independent ideas independently. Orthogonality complements separating things that change from things that stay the same, and is part of the Low-Coupling-High-Cohesion message. Coupling is the degree to which one part depends on another, so low coupling means few connections between parts. Cohesion is the degree to which the contents of one part serve a single subject, so high cohesion means one subject per part. The two measure the same design from opposite sides. Changing a part with low cohesion means editing code for an unrelated subject in the same place. Changing a tightly coupled part means editing the parts that depend on it. Rethinking Objects argues for composition on those grounds.
• Managed Coupling. Declaring that a design should have “low coupling” is usually too vague. Every design has coupling, so acknowledge it and control it. Find each dependence, and use a design or pattern to put it where a change reaches the fewest parts.
• Design the communication, not the parts. Alan Kay, on the purpose of object orientation: “The key in making great and growable systems is much more to design how its modules communicate rather than what their internal properties and behaviors should be.”³ A Protocol is that design made literal. It describes what a caller needs from the object on the other side of a call, and leaves open what that object is. Rethinking Objects builds on this. Stateless moves that description into a function’s signature, where Need specifies what the function requires of its surroundings. Designing the communication gives Managed Coupling its target: not the least coupling, but coupling you can read.
• Subtraction. A design is complete when you cannot take anything else away⁴.
• Simplicity before generality⁵. A common problem we find in frameworks is that they aim to be general purpose without reference to actual systems. This leads to a dizzying array of options that are often unused, misused or not useful. However, most developers work on specific systems, and the quest for generality does not always serve them well. The best route to generality is through understanding well-defined specific examples. This principle acts as the tie breaker between otherwise equally viable design alternatives. The simpler solution may also turn out to be the more general one. Pattern Refactoring works through a case of this, one requirement at a time.
• Reflexivity. One abstraction per class, one class per abstraction. Also goes by Isomorphism.
• Once and once only. When two pieces of code express the same intent for the same reason, write that logic and structure once.
• Make things as immutable as possible.
• Make functions pure whenever you can.

Reading the Chapters Ahead

Most of the chapters ahead take a pattern, or a family of patterns that share a structure, and ask three questions of it. What changes and what stays the same? That describes the problem the pattern exists to solve. How much of the answer does Python supply on its own? That decides how much remains for you to write. What remains after you subtract Python’s share? That remainder is the part to learn, and it is usually the intent rather than the structure.

A pattern that subtracts to nothing is not a mistake. It is the right answer for a language missing the piece Python has.

From Iterator through Memento, each pattern chapter includes a figure of its pattern at work, drawn from that chapter’s own listings. Data Transfer Objects is the one chapter in that range without a figure. The gallery in A Pattern Moves an Edge shows which way each edge points, while each chapter’s figure shows what happens, in what order, and which part knows about which.

Part III closes with a Pattern Catalog, a name-and-intent index of the wider literature, with a link to this book’s coverage wherever it exists.

Exercises

Try each exercise before opening its solution.

1. Pick a program you have written that changed more than once. Name its vector of change: the thing that shifted every time. Say which part of the design absorbed the change, and which parts you edited.
2. Take a pattern you know from another language and list its parts: the classes, the interfaces, and the methods its usual form requires. Cross out every part Python supplies for you. Describe what remains in one sentence.
3. Apply Subtraction to a design of your own. Remove one class, one interface, or one level of inheritance, and say what stopped working. If everything still works, leave it out.
4. Write the Report design from The Reach of a Change twice, with a PDF writer and an HTML writer: once where Report specifies each writer class, and once where Report specifies a Writer protocol. Add a Markdown writer to both versions. For each version, list the existing classes and functions you edited.