Tech With Galvan
Back to Articles
TechOct 4, 2026 • 5 min read

Python Generators and itertools: Lazy, Memory-Efficient Code

Python generators, yield, generator expressions, and itertools — lazy evaluation, memory efficiency, and the patterns that matter.

Galvan
Galvan

Founder & Creator

Introduction

Lists load everything into memory. Generators produce values one at a time, on demand. For large datasets, this is the difference between 4GB of RAM and 4MB.

This post covers generators, yield, generator expressions, and the itertools module — the tools for memory-efficient Python.

The problem: lists eat memory

# Loads 1 million numbers into memory
numbers = [x * 2 for x in range(1_000_000)]   # ~8MB + overhead

# Generator: produces on demand, ~0 memory
numbers = (x * 2 for x in range(1_000_000))   # generator object, ~100 bytes

The generator doesn’t compute anything until you iterate. And it never stores all values at once.

Generator functions with yield

def count_up(n):
    i = 0
    while i < n:
        yield i
        i += 1

gen = count_up(5)
print(next(gen))   # 0
print(next(gen))   # 1
for x in gen:
    print(x)       # 2, 3, 4

yield pauses the function and returns a value. The next next() resumes from where it left off. State is preserved.

Generator vs list: memory

import sys

# List: 8MB for 1M ints
lst = [x for x in range(1_000_000)]
print(sys.getsizeof(lst))   # 8,000,056 bytes

# Generator: 200 bytes
gen = (x for x in range(1_000_000))
print(sys.getsizeof(gen))   # 200 bytes

Generator expressions

Like list comprehensions but with parentheses:

# List comprehension (eager)
squares_list = [x**2 for x in range(1000)]

# Generator expression (lazy)
squares_gen = (x**2 for x in range(1000))

# Use in functions that accept iterables
total = sum(x**2 for x in range(1000))   # no brackets needed

Real example: reading large files

def read_large_file(path):
    with open(path) as f:
        for line in f:       # file is already a generator
            yield line.strip()

# Process a 10GB file in 10MB of RAM
for line in read_large_file("huge.log"):
    if "ERROR" in line:
        print(line)

No loading the whole file. Process line by line, forever.

Chaining generators

def read_lines(path):
    with open(path) as f:
        for line in f:
            yield line.strip()

def filter_errors(lines):
    for line in lines:
        if "ERROR" in line:
            yield line

def parse_timestamp(lines):
    for line in lines:
        yield line.split(" ")[0]

# Pipeline: read → filter → parse, all lazy
pipeline = parse_timestamp(filter_errors(read_lines("app.log")))
for ts in pipeline:
    print(ts)

Each stage processes one item at a time. The entire pipeline uses O(1) memory regardless of file size.

itertools — the power tools

import itertools

# Infinite sequences
itertools.count(0, 2)          # 0, 2, 4, 6, ... (infinite)
itertools.cycle("ABC")        # A, B, C, A, B, C, ...
itertools.repeat("x", 3)      # x, x, x

# Combinatorics
itertools.combinations("ABC", 2)   # AB, AC, BC
itertools.permutations("ABC", 2)   # AB, AC, BA, BC, CA, CB
itertools.product("AB", "12")      # A1, A2, B1, B2

# Grouping
data = [("a", 1), ("a", 2), ("b", 3)]
for key, group in itertools.groupby(data, key=lambda x: x[0]):
    print(key, list(group))
# a [('a',1), ('a',2)]
# b [('b',3)]

# Chaining
itertools.chain([1,2], [3,4])   # 1, 2, 3, 4

# Slicing infinite sequences
itertools.islice(itertools.count(), 5)   # 0, 1, 2, 3, 4

itertools for data processing

import itertools

# Batch processing
def batched(iterable, n):
    it = iter(iterable)
    while chunk := list(itertools.islice(it, n)):
        yield chunk

for batch in batched(range(100), 10):
    process(batch)   # 10 items at a time

# Running total
list(itertools.accumulate([1, 2, 3, 4]))   # [1, 3, 6, 10]

# Drop while condition
itertools.dropwhile(lambda x: x < 5, [1, 2, 3, 5, 6, 7])   # 5, 6, 7

send() and throw() — advanced generators

def accumulator():
    total = 0
    while True:
        value = yield total
        if value is not None:
            total += value

gen = accumulator()
next(gen)           # prime the generator (advance to first yield)
print(gen.send(5))  # 5
print(gen.send(3))  # 8
print(gen.send(10)) # 18

send() lets you push values into a running generator. Used for coroutines and pipelines.

Common mistakes

  • Iterating a generator twice — generators are exhausted after one pass. Convert to list if you need multiple passes: list(gen).
  • Forgetting to consume — gen = (x for x in range(5)) does nothing until iterated. If you never iterate, nothing runs.
  • Mixing sync and async generators — async def + yield creates an async generator; use async for.
  • Overusing generators — for small data, lists are simpler and faster. Generators shine on large/streaming data.
  • Not handling StopIteration — next(gen) raises StopIteration when exhausted. Use for loops to handle it automatically.

Key Concepts

  • Generator = function with yield, produces values lazily
  • Generator expression = lazy list comprehension with ()
  • yield = pause and return a value, resume on next call
  • itertools = memory-efficient tools for infinite/large sequences
  • Pipeline = chain generators for O(1) memory processing

What to Try Next

  • Convert your file organizer to stream files with generators.
  • Use itertools.groupby to group CSV data without loading it all.
  • Combine with asyncio for async generators.
  • Use lru_cache + generators for memoized lazy computation.

FAQ

Generator vs list comprehension — which is faster?

For small data (<1000 items), lists are faster (no generator overhead). For large data, generators are faster and use less memory. Rule: if you need all values and the data is small, use a list. If you process one at a time or data is large, use a generator.

Can I get the length of a generator?

No — generators don’t know their length until consumed. Convert to list first: len(list(gen)). Or count manually: sum(1 for _ in gen).

yield from — what is it?

Delegates to another iterable:

def flatten(nested):
    for sublist in nested:
        yield from sublist

list(flatten([[1,2],[3,4]]))   # [1, 2, 3, 4]

Cleaner than nested for loops. Great for recursive flattening.