Comprehensions and Generators

Intermediate⏱75 minutes📖16 min readLesson 5 of 10

Comprehensions and Generators

Express transformations as comprehensions and stream data with generators: understand evaluation order and scope, keep memory flat, build lazy pipelines, delegate with yield from, and use the generator protocol deliberately

The log analyser worked perfectly in development, where the sample file had forty thousand lines. In production it ran against a directory of forty gigabytes, built a list of every line, and was killed by the operating system before it printed its first result. The transform itself was correct. The container had simply run out of memory.

The rewrite kept the same stages and changed one decision: each stage would yield one item at a time instead of returning a list. Peak memory dropped from tens of gigabytes to a few megabytes, and the first result appeared in milliseconds instead of minutes. Nothing about the logic changed, only the moment at which it ran.

This lesson covers the two features that make that rewrite possible. Comprehensions express a transformation over a collection in one expression. Generators express a transformation as a sequence of paused steps. Used well, they make data pipelines shorter and cheaper. Used carelessly, they make code that is impossible to debug and generators that are silently empty.

Key terms

Comprehension

An expression that builds a list, dict or set by iterating over one or more iterables.

Filter clause

An if clause inside a comprehension that decides whether the current item reaches the expression.

Generator function

A function that contains yield and returns a generator object instead of running its body immediately.

Generator object

The iterator produced by a generator function or expression, which resumes at the last yield.

Lazy evaluation

Computing a value only when it is requested, instead of building the whole result up front.

Exhausted

The state of a generator after it has yielded every value, when further next calls raise StopIteration.

yield from

A delegation expression that forwards iteration and protocol calls to a sub-iterator and evaluates to its return value.

send

A generator method that resumes execution and makes the argument the value of the paused yield expression.

A generator method that raises GeneratorExit at the suspension point so cleanup code can run.

Iterator

An object with a next method that produces values one at a time and signals the end with StopIteration.

Pipeline

A chain of iterators where each stage consumes the previous stage and yields transformed items.

1. The Shape of a Comprehension

A comprehension has three parts: an output expression, one or more for clauses, and optional if clauses. The output expression is evaluated once per surviving item, and the result is collected into a list, dict or set depending on the brackets.¹

python

matrix = [[1, 2], [3, 4]]
flat = [number for row in matrix for number in row]
print(flat)

The output is [1, 2, 3, 4]. The clauses read left to right, exactly like nested for loops written in the same order: the first for is the outer loop, and the second for runs fully for each value of the first.

python

pairs = [(a, b) for a in range(2) for b in range(2)]
print(pairs)

The output is [(0, 0), (0, 1), (1, 0), (1, 1)], which confirms that b varies fastest. Reversing the two clauses would produce the same pairs in a different order, and the difference becomes visible as soon as the expression has side effects or the result feeds a stateful consumer.

The diagram below shows the same order as a flow. The inner loop is re-entered for every value produced by the outer loop, and the filter runs before the output expression.

💡 Tip: Use Ctrl + Mouse Wheel to zoom, or click the fullscreen icon

Description

A flowchart of a nested list comprehension. The outer for clause takes one row, the if clause filters it, the inner for clause takes one item, the output expression is evaluated, and the item is appended to the result. The inner loop repeats, then the outer loop advances, and the result is returned when both are exhausted.

The bracket decides the container. Square brackets build a list, curly braces with a colon build a dict, and curly braces without a colon build a set.

python

text = "hello world"
letters = {character for character in text if character != " "}
print(sorted(letters))

The output is ['d', 'e', 'h', 'l', 'o', 'r', 'w']. The set removes the duplicate l and o, which is the whole reason to choose it over a list. A dict comprehension has the same shape but needs a key and a value separated by a colon.

python

words = ["apple", "fig", "banana"]
lengths = {word: len(word) for word in words if len(word) > 3}
print(lengths)

The output is {'apple': 5, 'banana': 6}. When two rows produce the same key, the later value wins, because a dict cannot hold duplicates.

python

last_wins = {key: value for key, value in [("a", 1), ("a", 2)]}
print(last_wins)

The output is {'a': 2}. That behaviour is deterministic and worth knowing, because it turns a dict comprehension into a compact way to build a lookup table from an ordered stream.

ℹ️Note

A comprehension is an expression, not a statement. It can appear inside a function call, a return value or another comprehension, and it always produces a value. A for statement is the right tool when the body performs several actions rather than producing one value.

2. Scope, Readability and Side Effects

A comprehension runs in its own scope. The iteration variable does not leak into the surrounding function, which is a deliberate change from Python 2 and a frequent source of surprise for people reading old code.²

python

x = "outer"
squares = [x * x for x in range(4)]
print(squares)
print(x)

The output is [0, 1, 4, 9] and then outer. The comprehension used x as its own local name and restored the outer binding afterwards. That isolation is what allows a comprehension to reuse a short name without clobbering a variable that the rest of the function still needs.

An assignment expression is the exception. A walrus inside a comprehension binds in the containing scope, because the whole point of the operator is to keep a value that the surrounding code will use.²

python

doubled = [y := value * 2 for value in range(3)]
print(doubled)
print(y)

The output is [0, 2, 4] and then 4. The name y survives the comprehension, while value does not. This is useful for a value that is expensive to compute and needed after the loop, and confusing when it is not intended.

Closures created inside a comprehension follow the ordinary late binding rule. The loop variable is looked up when the function is called, not when it is created, so every closure sees the final value.

python

functions = [lambda: index for index in range(3)]
print([function() for function in functions])

The output is [2, 2, 2], not [0, 1, 2]. The fix is to bind the current value as a default argument, as in lambda index=index: index. The same trap exists in a for loop, so it is not a property of comprehensions, but comprehensions make it easier to write the closure on one line and miss it.

Comprehensions stop being readable when they contain more than two for clauses, a nested conditional expression, or a call that mutates state. At that point a named function or a plain loop is shorter for the next reader, even if it is longer for the writer.

python

# Readable: one loop, one filter, one expression.
names = [user.name for user in users if user.active]

# Hard to read: nested loops, two filters, conditional expression.
labels = [
    f"{role}:{name.upper()}" if role else name
    for team in teams
    for role, name in team.members
    if name and not name.startswith("_")
]

The second example is not wrong. It is simply doing enough that the reader has to simulate it, and a function with a docstring would carry the same logic with less effort.

⚠️Warning

Do not put a side effect such as print, append or a network call inside a comprehension. The expression is meant to be a pure mapping from input to output, and a hidden side effect makes the evaluation order matter in ways that readers will not expect.

The matching exercise below connects each construct to the result it produces.

Matching Question

How to answer

Matching exercise with 5 pairs. Match each item on the left with its answer on the right.

Match each construct to the container it produces.

[x * 2 for x in range(3)]

Drop here

{x * 2 for x in range(3)}

Drop here

{x: x * 2 for x in range(3)}

Drop here

(x * 2 for x in range(3))

Drop here

sum(x * 2 for x in range(3))

Drop here

Drag answers from here:

A list with three values

A set with three values

A dict with three keys

A generator that yields on demand

A single number

3. Generator Functions: Yield as a Pause

A function that contains yield does not run when it is called. It returns a generator object, and the body runs only when the caller asks for the first value. Each yield pauses the function and hands one value out, and the next request resumes immediately after that yield.³

python

def count_up_to(limit: int):
    number = 0
    while number < limit:
        yield number
        number += 1


counter = count_up_to(3)
print("created")
print(next(counter))
print(next(counter))
print(list(counter))

The output is created, then 0, then 1, then [2]. The string created proves that the call did not run the body. The three values arrive one at a time, and the final list consumes the remainder of the sequence.

The video below builds the same function step by step and shows the memory argument with a timed comparison against a list.

Corey SchaferWatch on YouTube

Description

A video lesson that defines a generator function, traces its execution through next calls, and compares the memory footprint of a generator against a list built from the same range.

The state diagram below shows the three states a generator can occupy. The arrows are the calls that move it from one state to another.

💡 Tip: Use Ctrl + Mouse Wheel to zoom, or click the fullscreen icon

Description

A state diagram of a generator. Calling the function creates it. next moves it from created to suspended at the first yield, and each further next moves it between suspended states. Reaching a return or calling close moves it to the closed state.

A for loop performs the same next calls and stops when StopIteration is raised. That is why a generator can be iterated exactly once: the loop consumes the sequence and leaves the object exhausted.

💡 Tip: Use Ctrl + Mouse Wheel to zoom, or click the fullscreen icon

Description

A sequence diagram of a caller consuming a generator. The first next runs the body to the first yield and returns a value. The second next resumes after the yield and returns the next value. The third next reaches the end and raises StopIteration.

A generator can also return a value. That value is not yielded; it becomes the value attribute of the StopIteration that ends the iteration, and yield from can capture it.⁴

python

def inner():
    yield 1
    return "done"


def outer():
    result = yield from inner()
    print("inner returned", result)


print(list(outer()))

The output is inner returned done, then [1]. The return value travels through the delegation, not through the iteration, which is why list never shows it.

📌Important

A generator function is not a collection. It has no length, no indexing and no reuse. If a caller needs to iterate twice, the caller must either materialise the values with list or call the generator function again.

4. Generator Expressions and Memory

A generator expression looks like a list comprehension with parentheses. It produces the same values, but it computes them on demand and stores only the current position. The difference in memory is large and easy to measure.

python

import sys

values = [number ** 2 for number in range(100_000)]
stream = (number ** 2 for number in range(100_000))
print(sys.getsizeof(values))
print(sys.getsizeof(stream))

On this interpreter the list object is about 800,984 bytes and the generator object is about 208 bytes. The list holds a pointer for every element, and the integer objects themselves are additional. The generator holds a frame and a code reference, and its size does not grow with the number of values.

That measurement is the reason to prefer a generator in three situations: the input is large, the input is unbounded, or only a prefix of the result is needed. A list remains the better choice when the result is small, when it must be indexed or sorted repeatedly, or when it crosses a function boundary that expects a sequence.

The quadrant below places the common result types by memory cost and reuse.

💡 Tip: Use Ctrl + Mouse Wheel to zoom, or click the fullscreen icon

Description

A quadrant chart of result types. A list and a set sit in the reusable but costly corner, a tuple is also reusable and costly, and a generator and an iterator sit in the single use but cheap corner.

The positions are a judgement, not a measurement of any particular program. The important axis is reuse: a list can be traversed as often as needed, while a generator is spent after one pass.

python

stream = (number * 2 for number in range(3))
print(list(stream))
print(list(stream))

The first line prints [0, 2, 4], and the second prints []. The generator is exhausted, and the second list has nothing left to collect. This is the most common generator bug, and it is silent unless the caller checks the result.

🚨Danger

Never wrap an infinite generator in list, tuple or sorted. The call will run until memory or patience runs out. Use itertools.islice to take a bounded prefix, or stop the loop with a condition inside the consumer.

The fill in the blank exercise checks the vocabulary of laziness.

Fill in the Blanks

How to answer

Fill-in-the-blank exercise with 4 blanks. Type the missing words, then check your answer.

Complete the description of generator evaluation.

Calling a generator function returns a blank without running the body. Each call to blank runs until the next yield and returns one value. When the function returns, the iterator raises blank. A second pass over the same object returns blank, because the generator is exhausted.

Word Bank, drag words into the blanks above:

generator object

next()

StopIteration

nothing

list

a new generator

IndexError

the first value again

5. Pipelines and yield from

A pipeline is a chain of generators where each stage consumes the previous one and yields a transformed item. Nothing runs until the final consumer asks for a value, and then one item flows through the whole chain before the next item starts.

python

def read_lines(paths):
    for path in paths:
        with open(path, encoding="utf-8") as handle:
            yield from handle


def errors(lines):
    for line in lines:
        if " ERROR " in line:
            yield line


def words(lines):
    for line in lines:
        yield len(line.split())


total = sum(words(errors(read_lines(["app.log"]))))
print(total)

The chain reads one line, tests it, counts its words and adds the count, then moves to the next line. Peak memory is one line plus the buffers of the open files, regardless of how many lines the files contain.

💡 Tip: Use Ctrl + Mouse Wheel to zoom, or click the fullscreen icon

Description

A left to right flowchart of a lazy pipeline. read_lines yields lines from files, errors keeps lines containing the word ERROR, words yields a word count per line, and sum consumes the counts into one total.

The order of the stages is part of the program. words(errors(lines)) counts words only in error lines, while errors(words(lines)) compares integers with a string and fails immediately. The drag and drop exercise asks you to order a pipeline that keeps only long error messages.

Ordering Question

How to answer

Ordering exercise with 5 items. Move them into the correct order, then check your answer.

Order the pipeline stages so that only long error lines are counted, from the source to the final consumer.

read_lines(paths)

errors(lines)

long_lines(lines)

count_lines(lines)

sum(counts)

yield from is the short form for a delegation loop. It forwards next, send, throw and close to the sub-iterator, and it evaluates to the sub-generator's return value.⁴

python

def chain(*iterables):
    for iterable in iterables:
        yield from iterable


print(list(chain([1, 2], "ab", range(3, 5))))

The output is [1, 2, 'a', 'b', 3, 4]. Writing the equivalent loop by hand would be longer and would not forward send and throw correctly.

Recursive flattening is the other classic use. The function calls itself with yield from, and the recursion stops when the item is not a list or tuple.

python

def flatten(items):
    for item in items:
        if isinstance(item, (list, tuple)):
            yield from flatten(item)
        else:
            yield item


print(list(flatten([1, [2, [3, 4], 5], 6])))

The output is [1, 2, 3, 4, 5, 6]. A deeply nested input would create a deep call chain, so the recursive form is best for data that is shallow and known, not for arbitrary user input with no depth limit.

💡Tip

Prefer yield from over a manual for item in iterable: yield item loop. The manual loop works for plain iteration, but it drops the return value and the protocol calls that a delegating generator is expected to forward.

6. The Generator Protocol: send, throw and close

send resumes a generator and makes its argument the value of the paused yield expression. The first call must be next or send(None), because the generator is not yet paused at a yield that could receive a value.⁵

python

def accumulator():
    total = 0
    while True:
        value = yield total
        if value is None:
            break
        total += value


running = accumulator()
print(next(running))
print(running.send(10))
print(running.send(5))

The output is 0, 10, 15. The first next starts the body and pauses at the yield, which produces the initial total. Each send resumes at that yield, assigns the argument to value, updates the total and pauses again at the next iteration.

This style is how generator based coroutines worked before async and await existed. It is still useful for stateful consumers such as parsers, running aggregates and test doubles, but new asynchronous code should use async def and await rather than this protocol directly.

python

running = accumulator()
try:
    running.send(10)
except TypeError as exc:
    print(exc)

The output is can't send non-None value to a just-started generator. The error is a useful reminder that the first step is always a plain next or a send(None).

close raises GeneratorExit at the suspension point. A finally block inside the generator runs, which is how a pipeline releases a file or a connection when the consumer stops early.

python

def resource():
    try:
        yield 1
        yield 2
    finally:
        print("cleanup")


handle = resource()
print(next(handle))
handle.close()

The output is 1, then cleanup. The second value is never produced, and the cleanup code still runs. A generator that is garbage collected without being closed may run the same finally block at an unpredictable time, so explicit close or a with block is the safer pattern.

⚠️Warning

A generator that catches GeneratorExit and yields another value raises RuntimeError. The close signal is not a request to continue; it is the last chance to release resources.

7. itertools and Bounded Consumption

The itertools module supplies the building blocks that a pipeline needs without a custom generator function. islice takes a prefix, chain joins iterables, takewhile stops at the first failure, accumulate yields running totals, groupby groups adjacent equal keys, and batched splits an iterable into fixed size tuples.⁶

python

import itertools


def naturals():
    number = 0
    while True:
        yield number
        number += 1


print(list(itertools.islice(naturals(), 5)))
print(list(itertools.batched(range(7), 3)))

The output is [0, 1, 2, 3, 4], then [(0, 1, 2), (3, 4, 5), (6,)]. The infinite generator is safe because islice stops after five values. batched is the standard way to send a bounded number of records to a database or an API.

python

import itertools


def save_batches(rows, size=500):
    for batch in itertools.batched(rows, size):
        print(f"saving {len(batch)} rows")

The function consumes any iterable, including a generator, and never holds more than one batch at a time. That is the shape that keeps a bulk import inside a fixed memory budget.

zip has a related safety feature. Since Python 3.10 it accepts strict=True, which raises ValueError when the inputs have different lengths instead of silently truncating to the shortest one.²

python

try:
    print(list(zip([1, 2], [3], strict=True)))
except ValueError as exc:
    print(exc)

The output is zip() argument 2 is shorter than argument 1. Without strict, the result would be [(1, 3)], and the missing value would disappear without a warning.

ℹ️Note

itertools.groupby groups only adjacent equal keys. Sort the input by the same key first when equal keys may be separated, or the result will contain several groups with the same name.

8. Real-World Case Studies

Two production shapes show the difference between a comprehension and a generator, and why the choice is about the consumer rather than the syntax.

Streaming a log directory

A support tool must find the slowest requests across a directory of rotated logs. The files are large, the directory is larger, and the report only needs the top twenty durations.

python

import heapq
from pathlib import Path


def lines(paths):
    for path in paths:
        with open(path, encoding="utf-8", errors="replace") as handle:
            yield from handle


def durations(lines):
    for line in lines:
        parts = line.split()
        if len(parts) != 3 or parts[1] != "GET":
            continue
        try:
            yield int(parts[2])
        except ValueError:
            continue


paths = Path("logs").rglob("*.log")
slowest = heapq.nlargest(20, durations(lines(paths)))
print(slowest)

The pipeline never builds a list of all lines or all durations. nlargest keeps a heap of twenty entries, so memory stays bounded by the heap and the current line. The errors="replace" argument keeps one corrupt byte from aborting the whole scan, and the try around int skips malformed rows without hiding a programming error.

The comprehension version of the same code would be [int(line.split()[2]) for line in ...], which is shorter and wrong for two reasons: it loads every duration, and it raises on the first malformed line. The generator version makes both decisions explicit.

Paginating an API without loading every page

A client must process active users from a paginated endpoint. The endpoint returns pages of one hundred records and signals the end with an empty list. The consumer only needs the first fifty active users, and the account may have millions.

python

import itertools


def pages(fetch, size=100):
    page = 1
    while True:
        items = fetch(page, size)
        if not items:
            break
        yield from items
        page += 1


def active(users):
    for user in users:
        if user["status"] == "active":
            yield user


first_fifty = list(itertools.islice(active(pages(fetch)), 50))

The fetch function is injected, which makes the client testable without a network. yield from items forwards the records of one page, and islice stops the whole chain after fifty active users. Later pages are never requested, which is the behaviour that keeps a report fast and polite.

💡Tip

Design a pipeline stage as a function that takes an iterable and returns an iterable. That signature composes with files, API pages, database cursors and test lists without knowing which one it received.

9. Pitfalls and a Review Checklist

Generators fail quietly more often than other constructs, because an exhausted generator looks exactly like an empty one. The table below collects the failures that appear most often in review.

Pitfall	Symptom	Fix
Reusing an exhausted generator	Second loop does nothing	Call the function again or materialise a list
Wrapping an infinite generator in `list`	Process hangs or runs out of memory	Use `islice` or a stopping condition
Closure over the loop variable	Every closure returns the final value	Bind with a default argument
Generator never closed	File or connection stays open	Use `with` or call `close`
`return` value ignored	Cleanup result disappears	Capture it through `yield from` or catch `StopIteration`
Side effect inside a comprehension	Order dependent bugs	Use a loop or a named function
`groupby` on unsorted input	Duplicate groups	Sort by the same key first

The checklist below is short enough to apply while reading a diff.

Does the comprehension fit on two or three lines without hiding a branch?
Is the result reused, and if so is it a list rather than a generator?
Does every generator that opens a resource close it on early exit?
Is every infinite source bounded by islice, takewhile or a consumer break?
Does yield from replace every manual delegation loop?
Are zip inputs checked with strict=True where a length mismatch matters?
Is groupby fed sorted input?

ℹ️Note

A generator function is also a natural place to put a try/finally around a resource. The cleanup runs when the generator is closed, including when a consumer stops early, which is difficult to arrange with a plain iterator class.

2026 Updates (as of November 2026)

Python 3.15.2 is the current stable documentation set.⁷
Python 3.15 supports unpacking inside comprehensions, so [*row for row in rows] flattens a list of lists and {**mapping for mapping in mappings} merges a sequence of dicts.⁸⁹
Comprehensions have been inlined by the compiler since Python 3.12, which removed the hidden function call and its frame while preserving the isolated scope. A traceback now points at the comprehension line in the enclosing function instead of a separate <listcomp> frame.¹⁰
itertools.batched has been available since Python 3.12 and is the standard way to split a stream into fixed size tuples.⁶
zip(strict=True) has been available since Python 3.10 and turns a silent length mismatch into a ValueError.²

📚 Did you know?

A generator's return value is stored in StopIteration.value, which is why yield from can capture it but a for loop cannot.
Since Python 3.7, a StopIteration that escapes a generator body is converted into a RuntimeError, because the generator protocol uses that exception to mean "finished" and cannot distinguish the two.¹¹
A comprehension variable has its own scope, but a walrus assignment inside the comprehension binds in the containing scope, which is a deliberate exception to the isolation rule.

📌Important

Comparative Verdict. Use a list comprehension for a small, reusable result that the next line will index or iterate more than once. Use a set or dict comprehension when uniqueness or keyed lookup is the point. Use a generator expression for a single pass over a large or unbounded source, and a generator function when the stage needs state, cleanup or a name. Reach for itertools before writing a custom iterator, and reach for a plain loop when the comprehension stops being readable.