GeezerVison

There's also a very practical reason to learn this. Once you understand iterators and generators, you can work with data that is bigger than your computer's memory. That happens all the time in data science: log files with millions of rows, datasets that don't fit in RAM, websites that hand you results one page at a time. So it's worth understanding this before you touch a real dataset.

Here is what we will cover:

1. What a for loop really does, step by step, and the difference between an iterable and an iterator
2. Writing your own iterator with __iter__ and __next__
3. Generators: the same thing with much less code
4. Generator expressions, and the other kinds of comprehension
5. Reading big files a little at a time
6. When a generator is the wrong choice
7. A real-world example: going through an API's results page by page

Part 1: What a for Loop Really Does

Let's start with something you've written a hundred times:

shows = ["Stranger Things", "Dark", "Wednesday"]

for show in shows:
    print(show)

Stranger Things
Dark
Wednesday

It looks simple. But what is Python actually doing here? It is not looking inside the list by itself, and the for loop is not keeping its own counter like 0, 1, 2. Instead, it follows three steps:

1. It asks the list for an iterator. An iterator is a small helper object whose only job is to walk through the list and remember where it is.
2. It asks that iterator for the next item, again and again.
3. It stops when the iterator says there's nothing left.

Analogy: think of a book and a bookmark. The book (the list) holds all the pages. The bookmark (the iterator) doesn't hold any pages itself; it just remembers where you are. Every time you want to read on, you look at the bookmark, read the next page, and move the bookmark forward. When the bookmark reaches the back cover, you're done.

   shows = ["Stranger Things", "Dark", "Wednesday"]       ← the iterable (the book)

   it = iter(shows)                                        ← the iterator (the bookmark)
         │
         ▼
   next(it) ──► "Stranger Things"     bookmark moves forward
   next(it) ──► "Dark"                bookmark moves forward
   next(it) ──► "Wednesday"           bookmark moves forward
   next(it) ──► StopIteration!        bookmark is at the back cover: "I'm done"

You can do all three steps yourself, by hand. The built-in function iter() does step 1, and next() does step 2:

it = iter(shows)

print(next(it))
print(next(it))
print(next(it))

Stranger Things
Dark
Wednesday

Three items, three calls to next(), three answers. So what happens if we call it a fourth time, when there's nothing left?

try:
    next(it)
except StopIteration:
    print("StopIteration: the iterator is exhausted")

StopIteration: the iterator is exhausted

That's the whole trick. An iterator says "I'm finished" by raising an exception called StopIteration. It's not an error in the scary sense. It's just the iterator's polite way of saying "there's nothing more to give you".

Analogy: think of a vending machine for a single snack column. Each time you press the button (next()), one snack drops out. When the column is empty, a small "SOLD OUT" light turns on. StopIteration is that light.

A for Loop Is Just This Pattern, Done for You

A for loop is exactly the same pattern, with the exception handling written for you. These two cells do the same thing:

# The for loop you write
for show in shows:
    print(show)

Stranger Things
Dark
Wednesday

# What Python effectively runs
it = iter(shows)
while True:
    try:
        show = next(it)
    except StopIteration:
        break
    print(show)

Stranger Things
Dark
Wednesday

Read the second version slowly, line by line:

• it = iter(shows) gets the iterator (the bookmark).
• while True: keeps going forever, until something breaks out.
• show = next(it) asks for the next item.
• except StopIteration: break means "when the iterator says it's done, leave the loop".
• print(show) is the body of your loop.

Nobody actually writes the second version in real code. But knowing that this is what happens behind the scenes explains a lot of behaviour that would otherwise seem random, including a couple of bugs you'll meet later in this guide.

Almost Everything Is Iterable

Anything you can put after in in a for loop is called iterable. That's a lot more than just lists:

print(list(iter("Dark")))                       # strings, character by character
print(list(iter((1, 2, 3))))                    # tuples
print(list(iter({"a": 1, "b": 2})))             # dicts -> the keys
print(sorted(iter({"x", "y"})))                 # sets -- sorted, because a set has no fixed order
print(list(iter(range(4))))                     # range objects

['D', 'a', 'r', 'k']
[1, 2, 3]
['a', 'b']
['x', 'y']
[0, 1, 2, 3]

Notice two small details:

• Looping over a dictionary gives you its keys, not its values.
• A set has no fixed order, so the example uses sorted() to make the output the same every time.

range Doesn't Build a List

range is worth a closer look, because it surprises people. range(1_000_000) does not build a list of a million numbers.

(The underscores in 1_000_000 are only there to make the number easier to read, like the commas in 1,000,000. Python ignores them, so 1_000_000 and 1000000 are exactly the same number.)

r = range(1_000_000)

print(r)                    # it prints as a range, not as [0, 1, 2, ...]
print(type(r))
print(r[500])               # you can still index it
print(sum(r))               # and consume it -- the numbers are produced on demand

range(0, 1000000)
<class 'range'>
500
499999500000

A range only stores three numbers: where to start, where to stop, and the step size. It works out each value only when you ask for it. Even r[500] is calculated on the spot (start + 500 × step), not looked up in a stored list.

Analogy: a range is like a recipe for numbers, not a box of numbers. A recipe for 1,000 cookies is one small card, not 1,000 cookies. You bake each cookie only when someone wants one.

That idea, making values when you need them instead of storing them all in advance, is what this whole guide is about. You've been using range for a long time without noticing that it works like this.

enumerate() and zip() work the same way. They don't build a list; they hand out values as you ask for them:

e = enumerate(["a", "b"])

print(type(e))              # not a list
print(list(e))              # it only becomes one when you ask

<class 'enumerate'>
[(0, 'a'), (1, 'b')]

Iterable vs Iterator

These two words sound almost the same, which makes them confusing. But they mean different things:

Going back to the analogy: the book is the iterable. You can read it as many times as you like. The bookmark is the iterator. It only moves forward, and once it reaches the end, it stays at the end.

Notice that an open file is in the iterator column. Loop over a file object once, and a second loop over the same file object finds nothing left, because the "bookmark" is already at the end of the file.

That last row of the table is the one that causes bugs. You can loop over a list again and again:

numbers = [1, 2, 3]

print(sum(numbers))
print(sum(numbers))         # same answer, the list is unchanged

6
6

An iterator can't do that. It has no way to go back to the start:

it = iter(numbers)

print(sum(it))
print(sum(it))              # already exhausted -- there is nothing left to add

6
0

You got 0, not 6. There was no error and no warning. The second sum() simply found an empty iterator and added up nothing.

Analogy: an iterator is like a one-time-use train ticket. Once it has been punched for a journey, showing it again won't get you anywhere. The ticket still exists, but it's used up.

Remember this one. It's the most common bug with iterators and generators, and you'll see it again later in a more realistic form.

iter() on a List vs iter() on an Iterator

One more difference worth seeing for yourself. Calling iter() on a list makes a new iterator every time (a fresh bookmark at page one). But calling iter() on an iterator just gives you the same object back:

numbers = [1, 2, 3]

print(iter(numbers) is iter(numbers))     # two separate iterators over one list

it = iter(numbers)
print(iter(it) is it)                     # an iterator is its own iterator

False
True

• The first line is False because each iter(numbers) call creates a separate, brand-new iterator. (is checks whether two things are the very same object.)
• The second line is True because an iterator's __iter__ simply returns itself.

That second line is why you can hand an iterator straight to a for loop. The loop always calls iter() on whatever you give it, and an iterator basically answers "that's already me, go ahead".

Part 2: Writing Your Own Iterator

To make your own class work in a for loop, you give it the two methods that a for loop calls behind the scenes:

• __iter__, which returns the iterator, and
• __next__, which gives back the next value, or raises StopIteration when there's nothing left.

These are dunder methods (short for "double underscore"), from the same family as __init__ and __str__ that you met with classes.

Here's a countdown, written the long way:

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self                     # this object is its own iterator

    def __next__(self):
        if self.current <= 0:
            raise StopIteration         # tell the loop we are done
        value = self.current
        self.current -= 1
        return value

Let's walk through it:

• __init__ stores the starting number in self.current. This is the iterator's "bookmark": it remembers where we are.
• __iter__ returns self, meaning "this object is its own iterator".
• __next__ is called once per loop step. If current has reached 0, it raises StopIteration to end the loop. Otherwise it saves the current value, moves the bookmark down by one, and returns the saved value.

for n in Countdown(5):
    print(n, end=" ")

5 4 3 2 1 

(end=" " tells print() to put a space after each number instead of starting a new line, so they all come out on one line.)

It works, and it works anywhere a for loop works, including list(), sum() and max(), because they all use the same two methods behind the scenes:

print(list(Countdown(4)))
print(sum(Countdown(4)))
print(max(Countdown(4)))

[4, 3, 2, 1]
10
4

A More Realistic Iterator: Results Split into Pages

Here's something closer to real life: going through search results page by page, the way many websites and APIs make you do it. You get one page at a time, and you don't know in advance how many pages there are.

Analogy: think of Google search results. You see results 1 to 10 on page 1, then click "Next" to see page 2, and so on. You'd really like to just read through all results in one smooth list, without thinking about pages.

class SearchResults:
    # Walks every result across every page, fetching one page at a time.
    def __init__(self, pages):
        self.pages = pages
        self.page_index = 0
        self.item_index = 0

    def __iter__(self):
        return self

    def __next__(self):
        while self.page_index < len(self.pages):
            page = self.pages[self.page_index]
            if self.item_index < len(page):
                item = page[self.item_index]
                self.item_index += 1
                return item
            self.page_index += 1        # this page is done, move to the next
            self.item_index = 0
        raise StopIteration

This iterator has to remember two positions at once: which page it's on (page_index) and which item on that page it's on (item_index). In __next__:

• If there's still an item left on the current page, it returns that item and moves item_index forward.
• If the current page is finished, it moves to the next page and resets item_index to 0.
• If there are no pages left at all, it raises StopIteration.

pages = [["Dune", "Arrival"], ["Sicario"], ["Blade Runner 2049", "Prisoners"]]

for title in SearchResults(pages):
    print(title)

Dune
Arrival
Sicario
Blade Runner 2049
Prisoners

That's more than fifteen lines of bookkeeping (two index variables, a while loop and a hand-written StopIteration) just to say something simple: give me each item from each page, one after another.

That's a lot of code for a simple idea, and it's exactly the problem generators solve.

Part 3: Generators, the Same Thing with Much Less Code

A generator function is a function that hands back a series of values, one at a time, instead of returning just one value. You write it like a normal function, with one change: you use yield instead of return.

Here's the whole SearchResults class from above, rewritten as a generator function:

def search_results(pages):
    for page in pages:
        for item in page:
            yield item

for title in search_results(pages):
    print(title)

Dune
Arrival
Sicario
Blade Runner 2049
Prisoners

Four lines instead of more than fifteen. No index variables, no hand-written StopIteration, and the code now reads just like the sentence we used to describe it: for each page, for each item in the page, hand over the item.

yield from: Passing Along Everything

That inner loop, "yield every item from this thing", is so common that Python has a shortcut for it, yield from:

def search_results(pages):
    for page in pages:
        yield from page         # hand over every item in page, one at a time

print(list(search_results(pages)))

['Dune', 'Arrival', 'Sicario', 'Blade Runner 2049', 'Prisoners']

yield from page does exactly what the inner for loop did. Use it whenever a generator just needs to pass along everything from something else.

What Do You Get When You Call a Generator Function?

Notice that we never wrote __iter__ or __next__. Python builds the iterator for you. When you call a function that contains yield, you don't get a normal result back. You get a generator object, and none of the function's code has run yet (you'll see that for yourself in a moment):

gen = search_results(pages)

print(type(gen))

<class 'generator'>

If you print the generator itself, you get something like <generator object search_results at 0x7f9c1a2b3c40>. That hex number is just where it happens to sit in memory, so it changes every time you run it. It tells you nothing about the values, because right now there aren't any values yet.

From here on, "generator" means that generator object: the thing you get back when you call a generator function. A generator is a kind of iterator, so everything from Part 1 applies to it: next() works on it, for loops work on it, and it gets used up after one pass.

What yield Does: Pause, Not Stop

return ends a function for good. yield only pauses it. It hands a value back and keeps all the local variables exactly as they were, so the function can carry on from exactly where it stopped.

Analogy: think of watching a show on Netflix. return is like the end credits: the show is over. yield is like pressing pause: the picture freezes on the exact second you stopped, and when you press play again, it continues from that same second, not from the beginning.

The easiest way to see this is to put some print lines around the yield:

def countdown(n):
    print(f"    [starting at {n}]")
    while n > 0:
        print(f"    [about to yield {n}]")
        yield n
        print("    [resumed, now decrementing]")
        n -= 1
    print("    [function finished]")

gen = countdown(3)
print("generator created -- notice nothing above has run yet")

generator created -- notice nothing above has run yet

The function hasn't started yet. Not even the first print ran. Now ask for one value:

print("first value:", next(gen))

    [starting at 3]
    [about to yield 3]
first value: 3

It ran up to the first yield and stopped there. The function is paused in the middle of its loop, with n still equal to 3, waiting for you.

Ask for another value and it carries on from the line right after the yield:

print("second value:", next(gen))

    [resumed, now decrementing]
    [about to yield 2]
second value: 2

print("third value:", next(gen))

try:
    next(gen)
except StopIteration:
    print("StopIteration -- the function ran off the end")

    [resumed, now decrementing]
    [about to yield 1]
third value: 1
    [resumed, now decrementing]
    [function finished]
StopIteration -- the function ran off the end

Here's the whole story as a timeline:

   your code                   inside countdown(3)
   ─────────                   ───────────────────
   gen = countdown(3)          (nothing runs yet)

   next(gen) ───────────────►  [starting at 3]
                               [about to yield 3]
             ◄──── 3 ───────── yield 3        ⏸ paused, n = 3

   next(gen) ───────────────►  [resumed, now decrementing]   ▶ continues
                               [about to yield 2]
             ◄──── 2 ───────── yield 2        ⏸ paused, n = 2

   next(gen) ───────────────►  [resumed ...]  [about to yield 1]
             ◄──── 1 ───────── yield 1        ⏸ paused, n = 1

   next(gen) ───────────────►  [resumed ...]  [function finished]
             ◄── StopIteration (the function ran off the end)

So a generator is a function that can be paused and resumed, and when the function finally ends, Python raises StopIteration for you. That's really all there is to it.

What About return in a Generator?

Earlier we said a generator uses yield instead of return. That's not quite the full story. A generator can still contain a return, and there it simply means stop now. Unlike in a normal function, it doesn't hand a value to your for loop.

def countdown_to_zero(n):
    while True:
        if n == 0:
            return              # ends the generator here; produces no value
        yield n
        n -= 1

print(list(countdown_to_zero(3)))

[3, 2, 1]

When n reaches 0, return ends the generator, so the for loop inside list() sees StopIteration and finishes. So the rule is: yield gives a value, and return ends the generator. You'll see return used as an early exit like this in the API example at the end of this guide.

Getting a Value Without the Exception

Wrapping every next() in try/except gets messy when all you want is "the next value, or something else if there isn't one". next() takes an optional second argument for exactly this: a default value to return when the iterator is empty.

gen = iter([10, 20])

print(next(gen, "nothing left"))
print(next(gen, "nothing left"))
print(next(gen, "nothing left"))    # exhausted -- returns the default instead of raising

10
20
nothing left

Generators Are Lazy

A generator only runs its code when you ask for a value. Programmers call this being lazy, and here it's a good thing: it means no work gets done until it has to be.

Analogy: compare a buffet with a cook-to-order kitchen. A buffet (a list) cooks every dish in advance and lays it all out, whether anyone eats it or not. A cook-to-order kitchen (a generator) cooks each dish only when a customer orders it. Nothing is wasted, and the kitchen never runs out of table space.

Because of this, a generator can describe a sequence that never ends, which you simply can't do with a list (a list that never ends would need infinite memory):

def ticket_numbers(start):
    n = start
    while True:                 # no exit condition at all
        yield n
        n += 1

Think of the token machine at a bank or a hospital. It hands out ticket 1001, then 1002, then 1003, for as long as people keep arriving. It doesn't print every ticket for the whole year in the morning. It just prints the next one when someone presses the button.

tickets = ticket_numbers(1001)

for t in tickets:
    print(t, end=" ")
    if t >= 1005:
        break                   # we decide when to stop, not the generator

1001 1002 1003 1004 1005 

That's a loop with no end, and it still doesn't freeze, because nothing gets worked out until you ask for it. We decide when to stop, using break. The tickets generator is still sitting there paused, ready to carry on from 1006 if we ask for more.

Here's the practical difference between a list and a generator:

Part 4: Generator Expressions

You already know list comprehensions, the one-line way to build a list. There's a generator version too, and the only difference is the brackets: square brackets make a list, round brackets make a generator.

squares_list = [x * x for x in range(10)]      # square brackets -> a list
squares_gen  = (x * x for x in range(10))      # parentheses     -> a generator

print(squares_list)
print(type(squares_gen))

[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
<class 'generator'>

The list worked out all ten values straight away and stored them. The generator hasn't worked out any of them yet.

With ten items, that doesn't matter. With millions of items, it really starts to:

import sys

big_list = [x * x for x in range(1_000_000)]
big_gen  = (x * x for x in range(1_000_000))

print("list:     ", f"{sys.getsizeof(big_list):>9,} bytes")
print("generator:", f"{sys.getsizeof(big_gen):>9,} bytes")

list:      8,448,728 bytes
generator:       200 bytes

The list holds a million numbers. The generator only holds the instructions for making them, so its size doesn't depend on how many there are. It would be about the same size for a billion.

   big_list  ███████████████████████████████████████████  8,448,728 bytes
   big_gen   ▏                                                   200 bytes

Analogy: filling a swimming pool versus having a tap. The list fills the whole pool before you can use any water. The generator is a tap: water comes out only when you turn it on, and the tap itself takes up almost no space.

Two small notes: the exact byte counts depend on your Python version and your computer. And sys.getsizeof on a list only measures the list itself (the "shelf" of slots), not the numbers stored inside it. So the real difference is even bigger than this. The file example later in this guide measures both.

Both of them still add up to the same total:

print(sum(big_list))
print(sum(big_gen))

333332833333500000
333332833333500000

Same answer. And you don't even need a variable for the generator. When a generator expression is the only thing you pass to a function, you can leave out its own round brackets:

print(sum(x * x for x in range(1_000_000)))

333332833333500000

That's why you'll see sum(...), max(...), any(...) and all(...) written this way all the time.

A simple way to choose:

• If you need the values more than once, or need len() or indexing, use a list.
• If you're passing the values straight into sum(), max(), a for loop or another generator, use a generator.
• If the data is big enough that you'd notice the memory, use a generator.

The Comprehension Family

While we're here: comprehensions come in four kinds, not just one. They're all written the same way; only the brackets change.

titles = ["Stranger Things", "The Crown", "Wednesday", "Dark", "The Crown"]

as_list = [t.lower() for t in titles]                 # [ ] -> list, keeps duplicates
as_set  = {t.lower() for t in titles}                 # { } -> set, drops duplicates
as_dict = {t: len(t) for t in titles}                 # {k: v} -> dict
as_gen  = (t.lower() for t in titles)                 # ( ) -> generator

print("list:", as_list)
print("set: ", sorted(as_set))
print("dict:", as_dict)
print("gen: ", type(as_gen).__name__)

list: ['stranger things', 'the crown', 'wednesday', 'dark', 'the crown']
set:  ['dark', 'stranger things', 'the crown', 'wednesday']
dict: {'Stranger Things': 15, 'The Crown': 9, 'Wednesday': 9, 'Dark': 4}
gen:  generator

Look at what each one did with the duplicate "The Crown":

• The list kept both copies.
• The set dropped the duplicate, because a set only keeps unique values.
• The dict used each title as a key. A key can only appear once, so the second "The Crown" simply overwrote the first one (with the same value).
• The generator hasn't produced anything yet.

After lists, the dict version is the one you'll use most. It looks like {key: value for ...}, and the colon inside the curly braces is what makes it a dict instead of a set:

runtimes = {"Dune": 155, "Arrival": 116, "Sicario": 121, "Prisoners": 153}

# Convert to hours, keeping only the long ones
long_films = {title: round(mins / 60, 2) for title, mins in runtimes.items() if mins > 120}

print(long_films)

{'Dune': 2.58, 'Sicario': 2.02, 'Prisoners': 2.55}

Reading it left to right: for each title, mins pair in the dictionary, if mins is more than 120, put title as the key and mins / 60 (rounded to 2 decimals) as the value. Arrival (116 minutes) didn't pass the if, so it was left out.

And the set version is an easy way to collect the unique values in one line:

watch_log = [
    {"user": "amit", "genre": "Sci-Fi"},
    {"user": "sara", "genre": "Drama"},
    {"user": "amit", "genre": "Sci-Fi"},
    {"user": "raj",  "genre": "Comedy"},
]

genres = {entry["genre"] for entry in watch_log}
print(sorted(genres))

['Comedy', 'Drama', 'Sci-Fi']

In short:

Part 5: Reading Big Files a Little at a Time

Now it gets practical. First we need a file big enough to show the difference, so this cell writes 200,000 rows of made-up music streaming history, like a tiny version of the listening logs a music app keeps.

(The _ in for _ in range(200_000) is a normal variable name. Programmers use _ when they need a loop variable but never use its value. It's a way of saying "I don't care about this one".)

import random
import shutil
from pathlib import Path

demo = Path("generators_demo")
if demo.exists():
    shutil.rmtree(demo)
demo.mkdir()

random.seed(7)
artists = ["Arijit Singh", "Taylor Swift", "The Weeknd", "A.R. Rahman", "Drake"]

log = demo / "play_history.csv"
with open(log, "w") as f:
    f.write("artist,ms_played\n")
    for _ in range(200_000):
        f.write(f"{random.choice(artists)},{random.randint(5_000, 240_000)}\n")

print(f"{log} created: {log.stat().st_size / 1_000_000:.1f} MB, 200,000 rows")

generators_demo/play_history.csv created: 3.5 MB, 200,000 rows

The Way That Uses the Most Memory

The first approach is to read everything into memory first, and then work on it:

with open(log) as f:
    next(f)                             # skip the header
    all_lines = f.readlines()           # every remaining line, in memory, right now

total_bytes = sys.getsizeof(all_lines) + sum(sys.getsizeof(line) for line in all_lines)
print(f"{len(all_lines):,} lines held in memory: ~{total_bytes / 1_000_000:.1f} MB")

200,000 lines held in memory: ~13.3 MB

That's almost four times the size of the 3.5 MB file. Why? Each line becomes its own Python string object, and every string carries some extra bookkeeping on top of its characters (its type, its length, and so on). So holding everything in memory costs more than the file itself. A 10 GB file read this way could need several times that much memory, which most laptops simply don't have.

The Lazy Way

A file object is already an iterator over its lines. So if you loop over it directly, Python reads one line at a time and never holds the whole file:

with open(log) as f:
    next(f)                             # skip the header (a file supports next() because it is an iterator)
    line_count = sum(1 for _ in f)      # count without storing anything

print(f"counted {line_count:,} lines, holding one at a time")

counted 200,000 lines, holding one at a time

sum(1 for _ in f) adds up a 1 for every line, without keeping any of the lines. Same count, and the memory you use doesn't depend on how big the file is. A 50 GB file would work exactly the same way; it would just take longer.

Analogy: reading the whole file at once is like photocopying an entire phone book before you look up a single name. Reading it line by line is like running your finger down the pages: you only ever look at one line at a time.

Building a Pipeline

Generators get really useful when you chain them together. Each step takes an iterator in, and yields a new one out, so the full data never has to sit in memory at once. Rows flow through the steps one at a time.

Analogy: think of a factory assembly line. One worker unpacks a part, passes it to the next worker who cleans it, who passes it to the next worker who checks its quality. Each part moves down the line on its own. Nobody waits for all 200,000 parts to be unpacked before cleaning starts.

   play_history.csv
        │  one line at a time
        ▼
   ┌─────────────┐    "Drake,183000"     ┌─────────────┐   ("Drake", 183000)   ┌──────────────────┐
   │ read_rows   │ ────────────────────► │ parse       │ ───────────────────► │ long_plays       │ ──► only plays
   │ skip header │                       │ split + int │                      │ keep ms >= 180000│     of 3+ minutes
   └─────────────┘                       └─────────────┘                      └──────────────────┘
         only ONE row is inside the pipeline at any moment

def read_rows(path):
    # Stage 1: yield each line, skipping the header.
    with open(path) as f:
        next(f)
        for line in f:
            yield line.rstrip("\n")


def parse(rows):
    # Stage 2: turn each line into a (artist, ms) tuple.
    for row in rows:
        artist, ms = row.split(",")
        yield artist, int(ms)


def long_plays(records, minimum_ms):
    # Stage 3: keep only the ones that played for a while.
    for artist, ms in records:
        if ms >= minimum_ms:
            yield artist, ms

Each function is small and does one job:

• read_rows opens the file, skips the header line, and yields each line with the newline character removed.
• parse splits each line at the comma and turns the milliseconds into an int.
• long_plays only lets through plays of at least minimum_ms milliseconds.

Connecting them reads like a description of the task, and still nothing has been worked out:

rows = read_rows(log)
records = parse(rows)
serious_listens = long_plays(records, 180_000)

print(type(serious_listens).__name__)   # a generator -- the file has not been read yet

generator

At this point, the file hasn't even been opened. We've only connected the pipes; no water is flowing yet. The work only happens when something asks for the values:

from collections import defaultdict

totals = defaultdict(int)
for artist, ms in serious_listens:
    totals[artist] += ms

for artist, ms in sorted(totals.items(), key=lambda pair: pair[1], reverse=True):
    print(f"{artist:<15} {ms / 3_600_000:>6.1f} hours")

Arijit Singh     601.8 hours
Taylor Swift     598.2 hours
A.R. Rahman      596.8 hours
The Weeknd       593.9 hours
Drake            591.0 hours

Here's what happens in that loop:

• defaultdict(int) is a dictionary that starts every new key at 0, so totals[artist] += ms works even for an artist we haven't seen yet.
• sorted(..., key=lambda pair: pair[1], reverse=True) sorts the artists by their total time, biggest first.
• ms / 3_600_000 turns milliseconds into hours (1 hour = 3,600,000 ms).

That's two hundred thousand rows added up, with only one row in memory at a time. If the file had 200 million rows, the pipeline code wouldn't need to change at all. It would just take longer.

Data-cleaning pipelines in the real world look exactly like this, which is why generators show up everywhere in data engineering.

Part 6: When a Generator Is the Wrong Choice

Generators aren't free. You give up three things in exchange for saving memory.

1. You Can Only Go Through It Once

This is the bug that catches everyone:

scores = (n for n in [4.5, 3.8, 4.9, 4.1])

print("average:", sum(scores) / 4)
print("highest:", max(scores, default="nothing left!"))

average: 4.325
highest: nothing left!

sum() used up the whole generator. By the time max() runs, there's nothing left to look at. Without that default=, it would have raised a ValueError instead. (default= tells max() what to return if there's nothing to compare.)

Notice something else too: the code divided by 4 by hand, because a generator can't tell you how many items it has. That's a hint that a list would have been a better fit here.

If you need the values twice, turn them into a list first:

scores = (n for n in [4.5, 3.8, 4.9, 4.1])
scores = list(scores)                  # store the values once, then reuse them as often as you like

print("average:", sum(scores) / len(scores))
print("highest:", max(scores))

average: 4.325
highest: 4.9

2. No len() and No Indexing

A generator doesn't know how many values it will make (it hasn't made them yet!), and it can't jump straight to the fifth one without making the first four:

gen = (x for x in range(10))

try:
    len(gen)
except TypeError as e:
    print("len(gen) ->", e)

try:
    gen[3]
except TypeError as e:
    print("gen[3]   ->", e)

len(gen) -> object of type 'generator' has no len()
gen[3]   -> 'generator' object is not subscriptable

Analogy: asking a generator for its length is like asking a storyteller "how many sentences are in the story you haven't told yet?" They can't know until they tell it.

3. Debugging Is Harder

Printing a generator shows <generator object ...>, not what's inside it. And if you call list() on it to take a peek, you've just used it up, so the rest of your code will find it empty. Looking inside it changes it.

The Middle Ground: A Reusable Iterable

There's also a middle ground, for when you need to loop more than once and the data is too big to keep in a list. Write a small iterable class whose __iter__ makes a new generator every time it's called:

class PlayHistory:
    # An iterable, not an iterator: __iter__ hands back a brand new generator
    # on every call, so this object can be looped over as often as you like.
    def __init__(self, path):
        self.path = path

    def __iter__(self):
        with open(self.path) as f:
            next(f)                             # skip the header
            for line in f:
                artist, ms = line.rstrip("\n").split(",")
                yield artist, int(ms)

The trick is that __iter__ here contains yield, so it is itself a generator function. Every time a for loop (or sum(), or max()) calls iter() on the object, it gets a brand-new generator that opens the file again from the top.

Analogy: a generator is like a newspaper: once you've read it and thrown it away, it's gone. PlayHistory is like a newspaper subscription: every time you ask, a fresh copy arrives.

history = PlayHistory(log)

print(f"rows:         {sum(1 for _ in history):,}")            # one full pass
print(f"longest play: {max(ms for _, ms in history):,} ms")    # another full pass, no problem

rows:         200,000
longest play: 239,999 ms

Two full passes over the file, still only one row in memory at a time, and no "used up" bug, because each loop gets its own fresh generator. Libraries use this same trick to give you big datasets that you can loop over again and again.

So, use a list when you need the values more than once, need to count them, need to index or slice them, or the data is small enough that none of this matters.

Use a generator when the data is big, the source is slow, the sequence never ends, or you're building a pipeline that passes values straight through.

Where You'll See This

This isn't a rare feature you'll use once in a while. Values made on demand are all over the tools you're about to learn:

• File objects. for line in f uses exactly the iterator steps we just covered.
• csv.reader is an iterator too. It hands you rows one at a time instead of building a list.
• range, enumerate, zip, map and filter are all lazy. They make values when you ask for them.
• pandas: read_csv(..., chunksize=...) gives you back an iterator of chunks (small DataFrames), so you can work through a file that's bigger than your memory.
• PyTorch and TensorFlow give you training data as iterables that hand over one batch at a time. Like PlayHistory, you can loop over them again for every round of training.

Once you know this pattern, you'll spot it everywhere. When a library gives you something that prints as an object instead of your data, it's almost always an iterator or an iterable waiting for you to loop over it.

Real-World Example: Going Through an API Page by Page

This is the example that makes generators click for a lot of people.

An API is a way for your program to ask another service (like a social network or a weather service) for data over the internet. Most APIs don't send you everything at once. They send results one page at a time: page 1 has the first few users, page 2 has the next few, and so on, until a page comes back empty.

You'd like to just write for user in all_users() and forget that pages exist. A generator hides the paging completely:

def fetch_page(page_number):
    # Stands in for a real network call -- chapter 17 replaces this with requests.
    fake_data = {
        1: ["amit", "sara", "raj"],
        2: ["neha", "vikram"],
        3: [],                          # an empty page means we have reached the end
    }
    print(f"    [fetching page {page_number}]")
    return fake_data.get(page_number, [])


def all_users():
    page = 1
    while True:
        users = fetch_page(page)
        if not users:                   # no more results
            return                      # ends the generator
        for user in users:
            yield user
        page += 1

Let's read all_users() carefully:

• It starts at page = 1.
• while True keeps fetching pages until something stops it.
• If a page comes back empty, return ends the generator (remember: in a generator, return means "stop now").
• Otherwise, it yields each user on that page, one at a time, and then moves on to the next page.

fetch_page is a stand-in for a real network call. In real code, it would use a library like requests to download each page from the internet.

The code that uses the generator just sees one long run of users, and has no idea there are pages:

for user in all_users():
    print(user)

    [fetching page 1]
amit
sara
raj
    [fetching page 2]
neha
vikram
    [fetching page 3]

Look at the order of that output. The "fetching" messages are mixed in with the results: page 2 isn't requested until every user from page 1 has been handed over. The generator only fetches the next page when the loop actually needs more users.

Analogy: a good waiter doesn't bring all ten courses to your table at once. They bring the next course only when you've finished the current one. If you say "I'm full" after the second course, the kitchen never cooks the rest.

And that's exactly what happens if the loop stops early. The remaining pages are never fetched at all:

for user in all_users():
    print(user)
    if user == "sara":
        print("found who we were looking for, stopping")
        break

    [fetching page 1]
amit
sara
found who we were looking for, stopping

Only one page fetched instead of three. With a real API, that's a network request you didn't have to make (and real APIs often limit how many requests you can make, or charge for them). On a big dataset, it's the difference between a script that starts working straight away and one that sits there downloading for ten minutes first.

Quick Reference Summary

The two steps behind every loop

iter(x)      # get an iterator from an iterable
next(it)     # get the next value; raises StopIteration when empty

A for loop calls iter() once, then calls next() again and again, and catches StopIteration for you.

Writing an iterator the long way (a sketch: done and value stand for your own logic)

class MyIterator:
    def __iter__(self):
        return self
    def __next__(self):
        if done:
            raise StopIteration
        return value

Writing the same thing as a generator

def my_generator():
    while not done:
        yield value          # pauses here, resumes on the next request

Key words

Comprehensions

Choosing

The bug to remember

gen = (x for x in [1, 2, 3])
sum(gen)     # 6
sum(gen)     # 0  -- already exhausted, and nothing warns you