Iterables and iterators

Every for loop in Python is the iterator protocol. Interviewers use the topic to find out whether you know the protocol underneath the syntax: the two roles, what __next__ must guarantee, and the iterator that is silently exhausted. Generators, the cheapest way to write an iterator, have their own note: Generators.

The protocol under the for loop

for x in xs:
    body(x)

# behaves like:
it = iter(xs)              # xs.__iter__()  - an iterable gives an iterator
while True:
    try:
        x = next(it)       # it.__next__()  - the iterator gives values
    except StopIteration:
        break
    body(x)

“Behaves like”, not “is”. The compiled loop never builds that try:

GET_ITER                 iter(xs)
FOR_ITER   (to END_FOR)  next(it), or jump out when exhausted
STORE_FAST  x
...body...
JUMP_BACKWARD            to FOR_ITER
END_FOR
POP_ITER                 drop the iterator (new in 3.14)

FOR_ITER calls the iterator type’s C-level next slot. A built-in iterator signals the end by returning nothing, with no StopIteration object allocated at all; only a Python-level __next__ has to raise one. That is part of why built-in iterators are faster than hand-written ones, and why StopIteration is a protocol signal rather than an error.

Iterable vs iterator

Two roles, and the distinction is the one that gets probed:

A list is an iterable: every for calls iter() and gets a new iterator, so you can loop it twice. A generator is an iterator — iter(gen) is gen — and once it has raised StopIteration it is done forever. That asymmetry is the source of most iterator bugs.

What counts as iterable

An object with no __iter__ can still be looped if it has __getitem__: the legacy sequence protocol calls it with 0, 1, 2, … until IndexError. The abstract base class does not know about it:

class Seq:
    def __getitem__(self, i):
        if i >= 3:
            raise IndexError
        return i * 10

list(Seq())                                   # [0, 10, 20]
isinstance(Seq(), collections.abc.Iterable)   # False

So the only reliable test for “can I iterate this?” is to call iter(x) and catch TypeError, which is what the docs recommend. isinstance(x, Iterable) answers “does it define __iter__?”.

Single-pass objects you meet every day

• Files. A file object is its own iterator: iter(f) is f. A second for line in f prints nothing, because the position is at the end; f.seek(0) or reopening starts again.
• map, filter, zip, enumerate, reversed, csv.reader. All iterators, all spent after one pass.
• Dict views are not. d.keys(), d.values() and d.items() are iterables that reflect the live dict and hand out a fresh iterator on every loop.

Tools for consuming iterators

next(it, None)                          # a default instead of StopIteration
iter(lambda: f.read(65536), b"")        # call until the sentinel comes back
operator.length_hint(it)                # an estimate that consumes nothing

• next(it, default). A bare next() in helper code can leak a StopIteration up into a caller’s loop or generator, where it ends things silently or becomes a RuntimeError. Give it a default.
• iter(callable, sentinel). The two-argument form calls the function until it returns the sentinel: the idiomatic loop over fixed-size chunks of a file or socket, with no while True and no break.
• operator.length_hint() (PEP 424) asks __length_hint__ for an estimate without consuming anything; list() uses it to presize. It is a hint: a list iterator reports what is left, while map and generators report 0 because they cannot know.

Changing a collection while iterating it

The list case is the dangerous one: for x in lst: lst.remove(x) on [1, 2, 3, 4] leaves [2, 4], because the iterator is just an index and the items shifted under it. Iterate over a copy (for x in lst[:]), or build a new list (Mutating a dict during iteration).

Writing one by hand, and the trap in it

class Countdown:
    def __init__(self, start):
        self.n = start

    def __iter__(self):
        return self            # makes this an iterator: single use

    def __next__(self):
        if self.n <= 0:
            raise StopIteration
        self.n -= 1
        return self.n + 1

c = Countdown(3)
print(list(c))    # [3, 2, 1]
print(list(c))    # []  - the state lives on the instance, and it is spent

Returning self from __iter__ couples the iteration state to the object. If you want something that can be looped repeatedly, __iter__ should return a new iterator each time — and the simplest way to do that is to make __iter__ a generator.

The fix: a generator __iter__

class Countdown:
    def __init__(self, start):
        self.start = start

    def __iter__(self):        # a generator function: fresh state per call
        n = self.start
        while n > 0:
            yield n
            n -= 1

c = Countdown(3)
print(list(c), list(c))       # [3, 2, 1] [3, 2, 1]

This is the answer to “write an iterator class”: don’t hand-roll __next__ unless you need external control of the state.

The __next__ contract

A hand-written __next__ has four rules to keep, and each one is easy to break by accident.

Once exhausted, stay exhausted

After __next__ raises StopIteration, every later call must raise it again. The docs call an iterator that resumes “broken”, and the standard tools assume it never happens:

class Flaky:
    def __init__(self):
        self.items = [1, 2]
    def __iter__(self):
        return self
    def __next__(self):
        if self.items:
            return self.items.pop(0)
        raise StopIteration

f = Flaky()
list(f)              # [1, 2]
f.items.append(3)
list(f)              # [3]: it came back to life

A consumer that stopped at the first StopIteration (zip, chain, a for that broke out) has moved on, and the late item is either lost or picked up by whichever consumer asks next. If the source can grow later, it is a queue, not an iterator: model it as one.

Special methods are looked up on the type

c = Counter(3)
c.__next__ = lambda: 99     # set on the instance
next(c)                     # 1: next() looks at type(c).__next__, not at c

next(), iter() and for all go through the type, so patching an instance changes nothing. Setting __iter__ = None on a class opts it out of iteration entirely: iter(x) raises TypeError even if __getitem__ exists, instead of falling back to the indexing protocol.

__iter__ returns self

An iterator’s __iter__ must return self, so the iterator can be handed to anything that calls iter() on its argument, for included. Without it the object works with next() and fails in a loop.

StopIteration stays inside the iterator

StopIteration is a signal to whoever is driving the iterator. Raised from code that is not an iterator’s __next__ (a helper’s bare next() inside a generator, say), it ends the wrong loop or, inside a generator, becomes RuntimeError: generator raised StopIteration (PEP 479). Use next(it, default) in helpers.

What a Python __next__ costs

The built-in iterator is a C function that signals the end by returning nothing. A Python __next__ is a full function call per item, plus a real StopIteration object at the end; a generator sits between the two, because resuming a suspended frame is cheaper than a call. For hot loops over large data, lean on built-ins and itertools; for everything else the difference does not matter.

The async protocol

The same contract has an async twin: __aiter__ returns the iterator, async def __anext__ returns the next item, and StopAsyncIteration ends it. async for drives it, and aiter() and anext() (3.10) are the built-ins, anext(it, default) included:

class Ticks:
    def __init__(self, n):
        self.n = n
    def __aiter__(self):
        return self
    async def __anext__(self):
        if self.n == 0:
            raise StopAsyncIteration
        self.n -= 1
        await asyncio.sleep(0)
        return self.n

[x async for x in Ticks(3)]        # [2, 1, 0]

Async generators (async def with yield) are the easy way to write one, just as generators are for the sync protocol (Coroutines and async def).

Where it breaks in production

• Exhausted and reused. A generator passed to two consumers gives the second one nothing, silently. Detail and the fix in Generator exhaustion.
• len() and indexing. Neither works on an iterator; sum(1 for _ in g) consumes it, and operator.length_hint() is only an estimate. If you need the exact length, you needed a list.
• Generator-specific failures (a resource held by a half-consumed generator, a pinned frame, return values) are in Generators.
• Mutating the underlying collection while iterating a dict or set raises; a list silently skips. See Mutating a dict during iteration.

Related

• Generators - generators: the cheapest iterator
• Generator exhaustion - the single-use trap, in full.
• itertools — efficient iterator combinators - the standard pipeline stages.
• Context managers: the with protocol and contextlib - @contextmanager is a generator with one yield.
• Coroutines and async def - where send()-driven generators went.
• Behavioral Design Patterns (GoF) - the Iterator pattern, named.