Backend / Python core / Tricky questions / 27_yield_from_semantics.md

yield from semantics

Updated 3 min read source
On this page8
  1. The gotcha
  2. Minimal repro
  3. What yield from actually does
  4. Capturing the return value
  5. Composing generators
  6. Difference from async await
  7. Common pitfalls
  8. Interview angle

yield from semantics

The gotcha

yield from gen looks like a shortcut for for x in gen: yield x. It’s not — it delegates the full generator protocol (send, throw, return). The naive for ... yield form silently breaks send() and discards the subgenerator’s return value.

Minimal repro

python 
def sub():
    x = yield 1
    y = yield 2
    return x + y          # subgenerator's return value

# Naive delegation — broken
def parent_naive():
    for v in sub():
        yield v

# Correct delegation
def parent():
    result = yield from sub()
    print("sub returned:", result)
    yield result
python 
g = parent()
print(next(g))            # 1
print(g.send(10))         # 2     — sends 10 into sub's first yield
print(g.send(20))         # sub returned: 30 \n 30

With parent_naive, g.send(10) is just next(g) — the value is discarded because the for loop only consumes via __next__. The naive form is a one-way data flow.

What yield from actually does

python 
def parent():
    yield from sub()

is roughly equivalent to:

python 
def parent():
    _i = iter(sub())
    try:
        _y = next(_i)
    except StopIteration as _e:
        _r = _e.value
    else:
        while True:
            try:
                # forward sent value
                _s = yield _y
            except GeneratorExit:
                _i.close(); raise        # forward close
            except BaseException as _e:
                # forward exception
                _i.throw(_e)
            else:
                try:
                    _y = _i.send(_s) if _s is not None else next(_i)
                except StopIteration as _e:
                    # capture return value
                    _r = _e.value
                    break
    # _r is what `yield from` evaluates to

It forwards next, send, throw, close to the subgenerator and captures the return value when the subgenerator ends.

Capturing the return value

python 
def reader():
    line = yield "ready?"
    return f"got: {line}"

def driver():
    result = yield from reader()
    yield result

g = driver()
print(next(g))          # "ready?"
print(g.send("hello"))  # "got: hello"

A generator’s return value doesn’t yield — it sets StopIteration.value. The yield from expression evaluates to that value, giving you a clean way to compose generators that produce a final result.

Composing generators

yield from is what made coroutines possible in pre-async Python:

python 
def chunks(lines):
    chunk = []
    for line in lines:
        chunk.append(line)
        if len(chunk) == 100:
            yield chunk
            chunk = []
    if chunk:
        yield chunk

def process(source):
    yield from chunks(source)   # transparent delegation

Tree traversal becomes elegant:

python 
def walk(node):
    yield node.value
    for child in node.children:
        yield from walk(child)

Difference from async await

yield from was the prototype for await. Both delegate work to a sub-coroutine. In modern Python:

python 
# Generator-based coroutine (legacy, pre-3.5)
@asyncio.coroutine
def fetch_old():
    data = yield from request()   # yield from
    return data

# Native coroutine (3.5+)
async def fetch_new():
    # await replaces yield from for coroutines
    data = await request()

For ordinary generators producing values, yield from is still the right tool. For async, use await.

Common pitfalls

  • Forgetting yield from and using for x in sub(): yield x — works for read-only delegation, breaks send/throw.
  • Trying yield from on a non-iterable — TypeError. yield from requires an iterable; for a single value, just yield value.
  • yield from of a generator that already started — works, but you only get the rest, not from the beginning.
python 
g = sub()
next(g)                # consumes first yield

def parent():
    yield from g       # picks up where g left off

Interview angle 4

  • Q: “What does yield from do that a for loop doesn’t?” — full protocol delegation: forwards send, throw, close, captures return value.
  • Q: “Why is yield from needed for coroutines?” — bidirectional data flow; values can be sent into the subgenerator, return values bubble out.
  • Follow-up: “How do you capture a generator’s return value?” — result = yield from gen() — return value becomes StopIteration.value, surfaced as the expression value.
  • Follow-up: “What replaced yield from for async?” — await (PEP 492, 3.5+).

See Generator exhaustion, Generators and Iterators in Python, async def returns a coroutine, doesn't run.

Previous
nonlocal vs global