Introduction
Lists load everything into memory. Generators produce values one at a time, on demand. For large datasets, this is the difference between 4GB of RAM and 4MB.
This post covers generators, yield, generator expressions, and the itertools module — the tools for memory-efficient Python.
The problem: lists eat memory
# Loads 1 million numbers into memory
numbers = [x * 2 for x in range(1_000_000)] # ~8MB + overhead
# Generator: produces on demand, ~0 memory
numbers = (x * 2 for x in range(1_000_000)) # generator object, ~100 bytes
The generator doesn’t compute anything until you iterate. And it never stores all values at once.
Generator functions with yield
def count_up(n):
i = 0
while i < n:
yield i
i += 1
gen = count_up(5)
print(next(gen)) # 0
print(next(gen)) # 1
for x in gen:
print(x) # 2, 3, 4
yield pauses the function and returns a value. The next next() resumes from where it left off. State is preserved.
Generator vs list: memory
import sys
# List: 8MB for 1M ints
lst = [x for x in range(1_000_000)]
print(sys.getsizeof(lst)) # 8,000,056 bytes
# Generator: 200 bytes
gen = (x for x in range(1_000_000))
print(sys.getsizeof(gen)) # 200 bytes
Generator expressions
Like list comprehensions but with parentheses:
# List comprehension (eager)
squares_list = [x**2 for x in range(1000)]
# Generator expression (lazy)
squares_gen = (x**2 for x in range(1000))
# Use in functions that accept iterables
total = sum(x**2 for x in range(1000)) # no brackets needed
Real example: reading large files
def read_large_file(path):
with open(path) as f:
for line in f: # file is already a generator
yield line.strip()
# Process a 10GB file in 10MB of RAM
for line in read_large_file("huge.log"):
if "ERROR" in line:
print(line)
No loading the whole file. Process line by line, forever.
Chaining generators
def read_lines(path):
with open(path) as f:
for line in f:
yield line.strip()
def filter_errors(lines):
for line in lines:
if "ERROR" in line:
yield line
def parse_timestamp(lines):
for line in lines:
yield line.split(" ")[0]
# Pipeline: read → filter → parse, all lazy
pipeline = parse_timestamp(filter_errors(read_lines("app.log")))
for ts in pipeline:
print(ts)
Each stage processes one item at a time. The entire pipeline uses O(1) memory regardless of file size.
itertools — the power tools
import itertools
# Infinite sequences
itertools.count(0, 2) # 0, 2, 4, 6, ... (infinite)
itertools.cycle("ABC") # A, B, C, A, B, C, ...
itertools.repeat("x", 3) # x, x, x
# Combinatorics
itertools.combinations("ABC", 2) # AB, AC, BC
itertools.permutations("ABC", 2) # AB, AC, BA, BC, CA, CB
itertools.product("AB", "12") # A1, A2, B1, B2
# Grouping
data = [("a", 1), ("a", 2), ("b", 3)]
for key, group in itertools.groupby(data, key=lambda x: x[0]):
print(key, list(group))
# a [('a',1), ('a',2)]
# b [('b',3)]
# Chaining
itertools.chain([1,2], [3,4]) # 1, 2, 3, 4
# Slicing infinite sequences
itertools.islice(itertools.count(), 5) # 0, 1, 2, 3, 4
itertools for data processing
import itertools
# Batch processing
def batched(iterable, n):
it = iter(iterable)
while chunk := list(itertools.islice(it, n)):
yield chunk
for batch in batched(range(100), 10):
process(batch) # 10 items at a time
# Running total
list(itertools.accumulate([1, 2, 3, 4])) # [1, 3, 6, 10]
# Drop while condition
itertools.dropwhile(lambda x: x < 5, [1, 2, 3, 5, 6, 7]) # 5, 6, 7
send() and throw() — advanced generators
def accumulator():
total = 0
while True:
value = yield total
if value is not None:
total += value
gen = accumulator()
next(gen) # prime the generator (advance to first yield)
print(gen.send(5)) # 5
print(gen.send(3)) # 8
print(gen.send(10)) # 18
send() lets you push values into a running generator. Used for coroutines and pipelines.
Common mistakes
- Iterating a generator twice — generators are exhausted after one pass. Convert to list if you need multiple passes:
list(gen). - Forgetting to consume —
gen = (x for x in range(5))does nothing until iterated. If you never iterate, nothing runs. - Mixing sync and async generators —
async def+yieldcreates an async generator; useasync for. - Overusing generators — for small data, lists are simpler and faster. Generators shine on large/streaming data.
- Not handling
StopIteration—next(gen)raisesStopIterationwhen exhausted. Useforloops to handle it automatically.
Key Concepts
- Generator = function with
yield, produces values lazily - Generator expression = lazy list comprehension with
() yield= pause and return a value, resume on next callitertools= memory-efficient tools for infinite/large sequences- Pipeline = chain generators for O(1) memory processing
What to Try Next
- Convert your file organizer to stream files with generators.
- Use
itertools.groupbyto group CSV data without loading it all. - Combine with asyncio for async generators.
- Use
lru_cache+ generators for memoized lazy computation.
FAQ
Generator vs list comprehension — which is faster?
For small data (<1000 items), lists are faster (no generator overhead). For large data, generators are faster and use less memory. Rule: if you need all values and the data is small, use a list. If you process one at a time or data is large, use a generator.
Can I get the length of a generator?
No — generators don’t know their length until consumed. Convert to list first: len(list(gen)). Or count manually: sum(1 for _ in gen).
yield from — what is it?
Delegates to another iterable:
def flatten(nested):
for sublist in nested:
yield from sublist
list(flatten([[1,2],[3,4]])) # [1, 2, 3, 4]
Cleaner than nested for loops. Great for recursive flattening.