Generators

A generator function is the cheapest way to write an iterator: calling it runs nothing and returns a generator object, and each next() runs the body up to the next yield. Interviewers use the topic to find out whether you know what is suspended between yields, what laziness actually buys, and the two ways generators go wrong: one that is silently exhausted, and one that quietly holds memory or a resource. Outputs below are from Python 3.14.5.

The iterator protocol underneath is in Iterables and iterators.

What yield actually does

def gen():
    print("start")
    x = yield 1            # yield is an expression: send() delivers x
    print("got", x)
    yield 2
    return "done"          # the return value rides on StopIteration

g = gen()                  # nothing printed yet
print(next(g))             # start / 1
print(g.send("hi"))        # got hi / 2
try:
    next(g)
except StopIteration as e:
    print(e.value)         # done

Calling gen() builds a generator object that holds a suspended frame: the local variables, the position in the bytecode, the evaluation stack. Since 3.11 that frame lives inside the generator object itself rather than as a separate frame object; g.gi_frame is created only when something asks for it, and is None once the generator has finished. The object is small, sys.getsizeof(g) is 184 bytes, however large the sequence it produces.

Its states

inspect.getgeneratorstate(g) reports GEN_CREATED, GEN_RUNNING, GEN_SUSPENDED or GEN_CLOSED, and the protocol enforces them:

• g.send("x") before the first next() raises TypeError: can't send non-None value to a just-started generator; there is no yield waiting for the value yet.
• Calling next() on a generator that is already running raises ValueError: generator already executing: two threads pulling one generator, or a generator that ends up calling next() on itself.

What laziness buys, with numbers

sum(x for x in range(10**6))      # 36.8 ms, peak memory ~0 MB
sum([x for x in range(10**6)])    # 45.2 ms, peak memory 40.4 MB
sum(range(10**6))                 # 12.3 ms: no Python-level loop at all

• Memory. The generator expression never holds more than one item; the list comprehension builds all million first. Memory, not speed, is the argument: per item a generator can even be slower than a comprehension for small inputs.
• Early exit. any(is_bad(r) for r in rows) stops at the first match; the list form computes every element first.
• Infinite and streaming sources. A generator over a socket, a log tail or itertools.count() is fine; a list of one is not.
• Pipelines. Chained generators process one item end to end through every stage, and no stage materialises a list.

Pipelines and yield from

def read_pages(paths):
    for p in paths:
        yield from read_lines(p)   # delegates; not: for line in ...: yield line

def numbered(lines):
    for i, line in enumerate(lines, 1):
        yield f"{i}: {line}"

def non_empty(lines):
    return (l for l in lines if l.strip())   # a generator expression

for out in numbered(non_empty(read_pages(paths))):
    ...

yield from delegates to a sub-iterator, forwarding send() and throw() and returning the sub-generator’s return value; the loop form drops all three (yield from semantics). itertools supplies the standard stages (itertools — efficient iterator combinators).

Ending a generator: close(), GeneratorExit, finally

g.close() throws GeneratorExit into the generator at the paused yield, so finally blocks and with exits inside it run. Two rules:

def stubborn():
    try:
        yield 1
    except GeneratorExit:
        yield 2                    # RuntimeError: generator ignored GeneratorExit

def tidy():
    try:
        yield 1
    except GeneratorExit:
        return "flushed 1 row"     # close() returns this on 3.13+; None on 3.12

A generator must not yield after GeneratorExit. And since 3.13, close() returns the value the generator returns while handling it, which lets a consumer collect a final result from a generator it stops early.

When cleanup actually runs

In CPython a generator is closed the moment its reference count reaches zero: drop the last reference to a half-consumed generator and its finally runs immediately. Cleanup is delayed only while something still references it:

• A reference cycle. The generator sits in a cycle and nothing runs until the cycle collector does.
• A stored reference. An attribute, a list of iterators, a traceback frame that captured it.
• Another interpreter. PyPy has no reference counting, so cleanup waits for its garbage collector.

When cleanup must happen at a known point (a database cursor, an open file), do not rely on any of that: with contextlib.closing(gen()) as g: closes it on exit.

The frame keeps everything alive

def report():
    rows = load_everything()       # 100 MB, bound before the first yield
    for r in summarize(rows):
        yield r

g = report()
next(g)                            # 100 MB stays alive while g is suspended
g.close()                          # ...and is freed here (or when g is dropped)

Everything a suspended generator’s frame references stays referenced: large objects, open files, locks. A generator created, partly consumed and then kept around (in a cache, on an object, in a list of pending work) keeps all of it. Drop references you no longer need before the first yield, or close the generator when you are done with it.

StopIteration inside a generator (PEP 479)

def first_words(lines):
    for line in lines:
        yield next(iter(line.split()))     # an empty line raises StopIteration

list(first_words(["a b", ""]))
# RuntimeError: generator raised StopIteration

Since 3.7, a StopIteration that escapes a generator’s body is turned into a RuntimeError, because otherwise it would silently end the generator and whatever loop is consuming it. Use next(it, default) inside generators, or return to end one deliberately.

Generators you use without writing them

• @contextmanager drives a generator with one yield: next() on enter, and on an error throw() into the paused yield (Context managers: the with protocol and contextlib).
• Async generators (async def with yield) are consumed with async for and closed with aclose(). An abandoned one is finalized later by the event loop (asyncio.run calls loop.shutdown_asyncgens()), so its finally can run late; contextlib.aclosing() makes it deterministic (Coroutines and async def).

When a generator is the wrong choice

• You need len(), indexing or several passes. A generator gives none of them; you needed a list.
• The data is small. A list is simpler to debug and print, and the memory saving is nothing.
• Errors far from their cause. In a pipeline, an exception surfaces at the consumer’s next(), several stages away from the line that failed, and the traceback runs through every stage.
• Two consumers. itertools.tee gives each its own iterator, but buffers everything one has seen and the other has not; if they diverge, it is a list in disguise.

Where it breaks in production

• Exhausted and reused. A generator passed to two consumers gives the second one nothing, silently (Generator exhaustion).
• The half-consumed generator holding a resource. Kept alive by a reference, it keeps its file or cursor open. Use contextlib.closing.
• A large object pinned by a suspended frame. See above; it looks like a memory leak in a profiler and disappears when the generator is closed.
• return inside a generator does not return a value to the caller of next(); it rides on StopIteration.value, which only yield from and hand-written drivers read.
• A generator shared between threads. The second thread’s next() raises ValueError: generator already executing; give each thread its own generator, or put a queue between them.

Related

• Iterables and iterators - the iterator protocol a generator implements
• Generator exhaustion - the single-use trap, in full
• yield from semantics - delegation in detail
• itertools — efficient iterator combinators - the standard pipeline stages
• Context managers: the with protocol and contextlib - @contextmanager, closing and aclosing
• Coroutines and async def - where send()-driven generators went