Introduction
The generators post covered the basics. This post is the cookbook — copy-paste itertools recipes for the patterns you use every week.
Recipe 1: Batched (chunk into groups)
import itertools
def batched(iterable, n):
"""Yield successive n-sized chunks."""
it = iter(iterable)
while chunk := list(itertools.islice(it, n)):
yield chunk
# Usage
for batch in batched(range(25), 5):
print(batch)
# [0,1,2,3,4], [5,6,7,8,9], ...
Python 3.12+ has itertools.batched built in.
Recipe 2: Sliding window
def sliding_window(iterable, n):
"""Yield sliding windows of size n."""
it = iter(iterable)
window = list(itertools.islice(it, n))
if len(window) == n:
yield tuple(window)
for x in it:
window = window[1:] + [x]
yield tuple(window)
# Usage
list(sliding_window([1,2,3,4,5], 3))
# [(1,2,3), (2,3,4), (3,4,5)]
Great for time-series (moving averages, n-gram extraction).
Recipe 3: Flatten nested lists
def flatten(nested):
"""Flatten one level of nesting."""
return list(itertools.chain.from_iterable(nested))
flatten([[1, 2], [3, 4], [5]])
# [1, 2, 3, 4, 5]
# Deep flatten (any depth)
def deep_flatten(nested):
for item in nested:
if isinstance(item, list):
yield from deep_flatten(item)
else:
yield item
list(deep_flatten([1, [2, [3, 4]], 5]))
# [1, 2, 3, 4, 5]
Recipe 4: Running total / cumulative sum
import itertools
data = [1, 2, 3, 4, 5]
running = list(itertools.accumulate(data))
# [1, 3, 6, 10, 15]
# With custom function
running_product = list(itertools.accumulate(data, lambda a, b: a * b))
# [1, 2, 6, 24, 120]
# Max so far
running_max = list(itertools.accumulate(data, max))
# [1, 2, 3, 4, 5]
Recipe 5: Round-robin
def round_robin(*iterables):
"""Interleave multiple iterables."""
pending = len(iterables)
nexts = itertools.cycle(iter(it).__next__ for it in iterables)
while pending:
try:
for next_ in nexts:
yield next_()
except StopIteration:
pending -= 1
nexts = itertools.cycle(itertools.islice(nexts, pending))
list(round_robin("ABC", "1234", "xy"))
# ['A', '1', 'x', 'B', '2', 'y', 'C', '3', '4']
Great for fair scheduling, merging streams.
Recipe 6: Take first N (safe for infinite)
def take(n, iterable):
"""Take first n items from any iterable (even infinite)."""
return list(itertools.islice(iterable, n))
take(5, itertools.count()) # [0, 1, 2, 3, 4]
take(3, itertools.cycle("AB")) # ['A', 'B', 'A']
Recipe 7: Drop while condition
# Skip leading items while condition is True
data = [0, 0, 0, 1, 2, 3, 0, 4]
result = list(itertools.dropwhile(lambda x: x == 0, data))
# [1, 2, 3, 0, 4]
# Take while condition is True
result = list(itertools.takewhile(lambda x: x < 3, [1, 2, 3, 4, 5]))
# [1, 2]
Recipe 8: Group by (with sorting)
data = [
{"name": "Alice", "dept": "eng"},
{"name": "Bob", "dept": "sales"},
{"name": "Carol", "dept": "eng"},
{"name": "Dave", "dept": "sales"},
]
# MUST sort first — groupby only groups consecutive items
data.sort(key=lambda x: x["dept"])
for dept, group in itertools.groupby(data, key=lambda x: x["dept"]):
print(dept, [p["name"] for p in group])
# eng ['Alice', 'Carol']
# sales ['Bob', 'Dave']
Recipe 9: Cartesian product (all combinations)
# All combinations of sizes and colors
sizes = ["S", "M", "L"]
colors = ["red", "blue"]
variants = list(itertools.product(sizes, colors))
# [('S','red'), ('S','blue'), ('M','red'), ('M','blue'), ('L','red'), ('L','blue')]
# With repeat (dice rolls)
list(itertools.product("123456", repeat=2)) # all 36 dice combinations
Recipe 10: Unique (preserve order)
def unique(iterable):
"""Remove duplicates while preserving order."""
seen = set()
for item in iterable:
if item not in seen:
seen.add(item)
yield item
list(unique([3, 1, 2, 1, 3, 4, 2]))
# [3, 1, 2, 4]
Python 3.7+ dicts preserve insertion order, so list(dict.fromkeys(iterable)) also works.
Recipe 11: Pairwise (adjacent pairs)
# Python 3.10+ has itertools.pairwise
list(itertools.pairwise([1, 2, 3, 4]))
# [(1, 2), (2, 3), (3, 4)]
# For older Python:
def pairwise(iterable):
a, b = itertools.tee(iterable)
next(b, None)
return zip(a, b)
Great for diffing, delta calculation, consecutive comparisons.
Recipe 12: Repeat with function
# Call a function N times
def call_repeatedly(func, n):
return [func() for _ in itertools.repeat(None, n)]
# Or with starmap
results = list(itertools.starmap(pow, [(2, 3), (3, 2), (4, 1)]))
# [8, 9, 4] → 2^3, 3^2, 4^1
Performance tip
itertools functions return iterators — they don’t materialize lists. Chain them for O(1) memory:
# Process a 10GB file in constant memory
pipeline = itertools.islice(
(line.strip() for line in open("huge.log")),
1000
)
for line in pipeline:
process(line)
Key Concepts
islice= slice any iterable (even infinite)chain= concatenate iterablesgroupby= group consecutive items (sort first!)accumulate= running totalsproduct= Cartesian productpairwise= adjacent pairs
What to Try Next
- Use
batchedin your data pipeline. - Use
sliding_windowfor time-series analysis. - Combine with asyncio for async iteration.
- Use
groupbyto group API responses.
FAQ
itertools vs list comprehensions — which is faster?
For small data, list comprehensions are faster (no iterator overhead). For large/infinite data, itertools is essential (O(1) memory). Use itertools when memory matters or data is unbounded.
Python 3.12 itertools.batched — should I use it?
Yes — it’s built in and optimized. For older Python, use the recipe above. The recipe also handles the last partial batch (which itertools.batched does too).
tee vs list for multiple passes?
itertools.tee creates independent iterators without storing all data (but buffers internally). For small data, list() is simpler. For large data, tee saves memory. Use list for clarity unless memory is a concern.