Generators & Iterators — Memory Magic

Generators & Iterators — Memory Magic

Lists compute everything upfront and hold it all in memory. Generators produce values one at a time, on demand — which makes them the tool for huge datasets, infinite sequences, and streaming data.

The problem: lists are greedy

# A list of the first 10 million squares? That's ~400MB of RAM.
squares = [n ** 2 for n in range(10_000_000)]

You probably only look at a few of those values — but the list computed and stored all ten million immediately. Wasteful.

The generator: values on demand

Swap the brackets for parentheses:

squares = (n ** 2 for n in range(10_000_000))
print(squares)       # <generator object <genexpr> at 0x...>  ← no numbers yet!

print(next(squares))   # 0    ← computed NOW
print(next(squares))   # 1    ← computed NOW
print(next(squares))   # 4

Nothing is calculated until you ask. Each next() computes one value and pauses. Ten million potential values, ~zero memory — they’re computed and discarded one at a time.

Looping works naturally (generators are iterable):

squares = (n ** 2 for n in range(5))
for value in squares:
    print(value)      # 0 1 4 9 16

One warning: generators are single-use. Once exhausted, they’re empty — loop twice and the second loop gets nothing. Need the values twice? Make a list.

yield — writing your own generator

The yield keyword turns any function into a generator function:

def countdown(start):
    while start > 0:
        yield start          # pause here, hand out a value
        start -= 1           # resume here on the next next()

for n in countdown(3):
    print(n)

Output:

3
2
1

When Python hits yield, it pauses the function and hands out the value. The next next() (or loop pass) resumes from that exact line — the function remembers where it was. Regular functions run start-to-finish; generators run in chapters.

Why generators matter: the real use cases

1. Huge files, line by line:

# A 2GB log file — reading it all would eat RAM
def error_lines(path):
    with open(path) as f:
        for line in f:
            if "ERROR" in line:
                yield line.strip()

for line in error_lines("server.log"):
    print(line)        # only error lines, one at a time, tiny memory

2. Infinite sequences (impossible with lists!):

def fibonacci():
    a, b = 0, 1
    while True:          # infinite — and totally fine
        yield a
        a, b = b, a + b

fib = fibonacci()
for _ in range(8):
    print(next(fib), end=" ")    # 0 1 1 2 3 5 8 13

A list can’t be infinite. A generator doesn’t mind — values are born only when asked for.

3. Streaming pipelines:

numbers = (n for n in range(1_000_000))
squared = (n ** 2 for n in numbers)
big = (n for n in squared if n % 2 == 0)

print(next(big))     # 0 — three lazy stages, still tiny memory

Generators chaining into generators — each stage transforms lazily. This is how Python processes data streams.

The one-line summary

✅ Checkpoint

1. What’s the difference between [x for x in xs] and (x for x in xs)? (List vs generator — eager vs lazy)
2. What does yield do that return doesn’t? (Pauses the function; it resumes where it left off)
3. Loop a generator twice — what happens the second time? (Nothing — it’s exhausted)
4. When do generators shine? (Huge data, infinite sequences, streaming pipelines)

Next: modules and imports — organizing code across files and meeting the standard library.