Iterators and generators in Python — theory with the library shelf, a yield that pauses and a report that came out empty the second time

I was preparing the overdue-loan notices for my faculty library: a list of who had gone past the 15 days. First I printed the notices for the front desk and then, with the same variable, the ones for email. The first report was perfect. The second one, empty. No error, no traceback, nothing: an empty list and a very confused me.

# The overdue-loans report that came out empty the second time
MAX_DAYS = 15   # loan days (made-up figure for the example)
loans = [('Laura', 'Python in Depth', 3), ('Pablo', 'Computer Networks', 19),
             ('Marta', 'Data Structures', 22), ('Iván', 'Operating Systems', 9),
             ('Sara', 'Python in Depth', 16)]

# Looks like a list comprehension, but it has brackets: it is a GENERATOR EXPRESSION
overdue = (member for member, title, days in loans if days > MAX_DAYS)

print('Notices for the front desk:', list(overdue))
print('Notices for email:         ', list(overdue))   # ...and here it comes out empty

Output:

Notices for the front desk: ['Pablo', 'Marta', 'Sara']
Notices for email:          []

What I thought was a list was a generator expression (round brackets, not square ones), and a generator can only be walked once. To understand why, we need to look at what Python does every time you write a for. Spoiler: you have been using iterators since day one of FP1 without knowing it.

This topic comes right after Python operator overloading, where the library Shelf learnt to answer len(), in and +=. Today we teach it to be walked with for, which is what we promised there. After this comes the practice with three programs.

Before you start — what makes an iterable different from an iterator

Two almost identical words for two different things, and FP2 quizzes love that:

• Iterable: any object you can walk with a for. Lists, tuples, strings, dictionaries, sets, range, files… What makes it iterable is the __iter__() method, which returns an iterator.
• Iterator: the object that actually does the walking, item by item. It has __next__(), which hands over the next item, and __iter__(), which returns itself. When nothing is left, __next__() raises StopIteration.

The picture that worked for me: the list is the shelf, and the iterator is the finger pointing at the book you are on. The shelf has no idea where you are; the finger does.

titles = ['Computer Networks', 'Data Structures', 'Python in Depth']
print(type(titles).__name__)     # list: can be walked (iterable), but is NOT an iterator

it = iter(titles)                # we ask the list for an iterator
print(type(it).__name__)         # list_iterator: this one knows where it is
print(next(it))                  # each next() moves one position
print(next(it))
print(next(it))
try:
    next(it)                     # no books left
except StopIteration:
    print('StopIteration: nothing left')

other = iter(titles)             # a NEW iterator starts again from the beginning
print(next(other))
next(titles)                     # a list has no __next__: TypeError

Output:

list
list_iterator
Computer Networks
Data Structures
Python in Depth
StopIteration: nothing left
Computer Networks
Traceback (most recent call last):
  File "iterable.py", line 16, in <module>
    next(titles)                     # a list has no __next__: TypeError
    ~~~~^^^^^^^^
TypeError: 'list' object is not an iterator

Notice four things in the output:

1. type(titles) is list and type(it) is list_iterator: they are different objects. iter() does not convert the list; it builds an iterator that walks it.
2. Each next(it) moves one slot. Three books, three successful next() calls.
3. The fourth next() raises StopIteration. It is not a bug in the program: it is the iterator’s way of saying “that’s all”.
4. other = iter(titles) starts again at “Computer Networks”. The list has not been used up; the old iterator has.

And the last line is the classic exam trap: next(titles) gives TypeError: 'list' object is not an iterator. A list is iterable, but it is not an iterator: it has no __next__.

In the drawing, the violet boxes are the list and the arrows are what each next() does. The fourth arrow points past the list: that is where StopIteration appears.

Question	Iterable (list, string, range…)	Iterator (iter(lst), a generator…)
Can it go in a for?	Yes	Yes
Does it have __iter__?	Yes, and it returns a new iterator	Yes, and it returns itself
Does it have __next__?	No	Yes
Does it know where it is?	No	Yes
Does it run out?	No: walk it as often as you like	Yes: at the end it gives nothing more

The iter() and next() functions — the for loop under the hood

A for has no magic in it. When you write for t in titles:, Python does exactly this:

titles = ['Computer Networks', 'Data Structures', 'Python in Depth']

print('With for:')
for t in titles:
    print(' ', t)

print('The same, by hand:')
it = iter(titles)             # 1. the for asks the iterable for an iterator
while True:
    try:
        t = next(it)          # 2. each lap asks for the next item
    except StopIteration:     # 3. when the iterator runs out, the loop ends
        break
    print(' ', t)             # 4. the body of the for

Output:

With for:
  Computer Networks
  Data Structures
  Python in Depth
The same, by hand:
  Computer Networks
  Data Structures
  Python in Depth

Both blocks print the same because they are the same. The for takes care of four steps you do not see:

1. It calls iter(titles), which in turn calls titles.__iter__().
2. On each lap it calls next(it), which calls it.__next__().
3. When StopIteration appears, it catches it by itself and leaves the loop without showing you any error.
4. In between, it runs the body with the value it received.

That is why things as different as a list, an open file or a dictionary work with for: it can ask any of them for an iterator. And that is why, if you write your own class with __iter__, it will also work with for, list(), sum(), max(), in and friends: all of them use the same protocol inside.

What if I call next() when nothing is left and I am not in a for? Then nobody catches the StopIteration and the program stops with a traceback. If you are not sure whether items remain, use the second argument: next(it, 'nothing') returns 'nothing' instead of raising the exception (you will see it in the generators section).

Building your own iterator — the iter and next methods

Let’s take the library Shelf. The most direct way to make it walkable is to turn it into its own iterator: __iter__ puts the cursor at zero and returns self, and __next__ hands over the book at that position.

class Shelf:
    """Iterable and iterator at once: __iter__ returns self."""

    def __init__(self, titles):
        self.titles = list(titles)

    def __iter__(self):
        self.pos = 0              # the cursor lives INSIDE the shelf...
        return self               # ...and every walk shares the same one

    def __next__(self):
        if self.pos >= len(self.titles):
            raise StopIteration   # "nothing left" signal
        title = self.titles[self.pos]
        self.pos += 1
        return title


e = Shelf(['Networks', 'Data', 'Python'])
print(list(e))                    # a single walk: works

pairs = 0
for a in e:                       # outer walk
    for b in e:                   # inner walk: sets pos = 0 again!
        pairs += 1
print('Pairs compared:', pairs, '(expected 9)')

Output:

['Networks', 'Data', 'Python']
Pairs compared: 3 (expected 9)

With a single walk (list(e)) it works perfectly. But as soon as we nest two for loops over the same shelf, to compare every book with every book, we get 3 pairs instead of 3 × 3 = 9. What happened? Follow the cursor lap by lap:

Step	Who asks	self.pos before	What happens	self.pos after
1	for a calls iter(e)	—	sets the cursor to 0	0
2	for a asks for the next one	0	a = 'Networks'	1
3	for b calls iter(e)	1	sets the cursor to 0 again	0
4-6	for b asks three times	0, 1, 2	b = Networks, Data, Python (3 pairs)	3
7	for b asks again	3	StopIteration: the inner loop ends	3
8	for a asks for the next one	3	StopIteration: the outer loop ends too!	3

The problem is that both loops share a single cursor, because the cursor (self.pos) lives inside the shelf.

The problem with your own iterator — iterable vs iterator

The solution is the same one Python lists use (and the one in the FP2 notes with Java’s Range): separate the iterable (the shelf, which stores the books) from the iterator (the cursor, which remembers where it is). Every time someone asks for iter(e), the shelf builds a new cursor.

class ShelfIterator:
    """The cursor: a new one for each walk."""

    def __init__(self, titles):
        self.titles = titles
        self.pos = 0              # each iterator keeps its own position

    def __iter__(self):
        return self               # an iterator is iterable too (so it works in a for)

    def __next__(self):
        if self.pos >= len(self.titles):
            raise StopIteration
        title = self.titles[self.pos]
        self.pos += 1
        return title


class Shelf:
    """Only iterable: stores the books and makes cursors."""

    def __init__(self, titles):
        self.titles = list(titles)

    def __iter__(self):
        return ShelfIterator(self.titles)   # each for gets a NEW cursor


e = Shelf(['Networks', 'Data', 'Python'])
pairs = 0
for a in e:
    for b in e:                   # another cursor: does not disturb the outer one
        pairs += 1
print('Pairs compared:', pairs)
print(list(e), list(e))           # can be walked as many times as you like

it = iter(e)
print(type(it).__name__, iter(it) is it)

Output:

Pairs compared: 9
['Networks', 'Data', 'Python'] ['Networks', 'Data', 'Python']
ShelfIterator True

Now there are 9 pairs, and list(e) can be repeated as often as you like. The last line confirms the rule from the previous section: iter(e) returns a ShelfIterator, and asking that iterator for iter() returns the same object (True).

On the left, a single red cursor for both loops; on the right, one cursor per walk. The shelf on the right stores no position at all: that is why nobody can “use it up”.

So when is it fine for __iter__ to return self? When the object is the walk: a list iterator, a generator, an open file. Nobody expects to walk the same list_iterator twice.

Generators — the Pythonic way to build iterators

Writing an iterator class with its __next__, its cursor and its raise StopIteration works, but it is a lot of text for something so common. Python has a shortcut: the generator function, a normal function that uses yield instead of return. Calling it does not run its body: it returns a generator, an iterator built by Python with the whole protocol already in place.

Why yield pauses the function instead of ending it

This is the part that is hardest to picture, so let’s go slowly and with prints inside:

def turns():
    print('  [gen] opens the desk')
    yield 'Laura'                 # stops HERE and hands over 'Laura'
    print('  [gen] carries on after Laura')
    yield 'Pablo'                 # stops here the second time
    print('  [gen] closes the desk')
    # at the end, the generator raises StopIteration by itself


g = turns()                       # creating the generator runs NOTHING yet
print('Created:', type(g).__name__)
print('I ask 1 ->', next(g))
print('I ask 2 ->', next(g))
print('I ask 3 ->', next(g, 'nobody else'))   # with a default value, next() does not raise StopIteration

Output:

Created: generator
  [gen] opens the desk
I ask 1 -> Laura
  [gen] carries on after Laura
I ask 2 -> Pablo
  [gen] closes the desk
I ask 3 -> nobody else

Read the output line by line, because each one tells you something:

• Created: generator appears before “opens the desk”. Calling turns() did not run a single line of the function: it only created the object.
• The first next(g) starts the function, prints “opens the desk”, reaches yield 'Laura' and stops there, handing over 'Laura'. Only then is I ask 1 -> Laura printed.
• The second next(g) continues right after the first yield: it prints “carries on after Laura” and stops at yield 'Pablo'.
• The third one prints “closes the desk” and reaches the end of the function. A generator that finishes raises StopIteration by itself; since we passed a default value (next(g, 'nobody else')), next returns that value instead of the exception.

The difference from return in one sentence: return ends the function and its variables disappear; yield pauses it with all its local variables and its execution point saved, and the next next() wakes it up right there. It is a bookmark: you close the book at the yield and, when you open it, you carry on from that line, not from the first one.

Generators with loops — the most common use

Usually you don’t write one yield after another but a yield inside a loop. And here we keep the promise from the previous topic: the __iter__ of the Shelf can be a generator function. Python is perfectly happy with __iter__ containing yield: calling it returns a new generator, which is exactly what we asked for in the cursors section.

class Book:
    def __init__(self, title, author, on_loan=False):
        self.title = title
        self.author = author
        self.on_loan = on_loan


class Shelf:
    def __init__(self, name):
        self.name = name
        self._books = []

    def __iadd__(self, book):        # as in the previous topic: e += book
        self._books.append(book)
        return self

    def __len__(self):
        return len(self._books)

    def __iter__(self):              # GENERATOR: each for gets a new one
        for book in self._books:
            yield book

    def available(self):             # another generator: only the books not on loan
        for book in self:            # uses the __iter__ above
            if not book.on_loan:
                yield book


e = Shelf('Computing')
e += Book('Computer Networks', 'Tanenbaum', on_loan=True)
e += Book('Data Structures', 'Weiss')
e += Book('Python in Depth', 'Ramírez', on_loan=True)
e += Book('Operating Systems', 'Stallings')

print(len(e), 'books in', e.name)
for book in e:
    print(f' - {book.title}' + (' (on loan)' if book.on_loan else ''))
print('Available:', [l.title for l in e.available()])
print('Authors:', ', '.join(l.author for l in e))   # generator expression without brackets
print('Type of iter(e):', type(iter(e)).__name__)

Output:

4 books in Computing
 - Computer Networks (on loan)
 - Data Structures
 - Python in Depth (on loan)
 - Operating Systems
Available: ['Data Structures', 'Operating Systems']
Authors: Tanenbaum, Weiss, Ramírez, Stallings
Type of iter(e): generator

Look how little it takes. These are the important lines:

def __iter__(self):              # GENERATOR: each for gets a new one
    for book in self._books:
        yield book

def available(self):             # another generator: only the books not on loan
    for book in self:            # uses the __iter__ above
        if not book.on_loan:
            yield book

• __iter__ walks the internal list and hands over each book with yield. No cursor, no __next__, no raise StopIteration: the generator keeps its own position and raises StopIteration when the inner for finishes.
• Since every call to __iter__ creates a new generator, the shelf can be walked in nested loops and as many times as you want (it is the good version from the previous section, in three lines).
• available() is another generator that uses the first one (for book in self) and only lets through the books that are not on loan. That is why “Data Structures” and “Operating Systems” come out.
• type(iter(e)).__name__ says generator: what the for receives is a generator.

Generator vs list — when to use each

A generator does not store its items: it computes them when asked. That has a huge advantage (memory) and a price (it can only be walked once and has no indexes). Look at it with real numbers:

import sys

lst = [n * 2 for n in range(1_000_000)]    # square brackets: the million numbers, NOW
gen = (n * 2 for n in range(1_000_000))    # round brackets: none of them yet

print('List:     ', sys.getsizeof(lst), 'bytes')
print('Generator:', sys.getsizeof(gen), 'bytes')
print('Same sum:', sum(lst) == sum(gen))
print('Summing the generator again:', sum(gen))   # exhausted: nothing left to add

Output:

List:      8448728 bytes
Generator: 200 bytes
Same sum: True
Summing the generator again: 0

The list takes more than 8 million bytes (8 MB, just the pointers to the numbers, not counting the numbers themselves) and the generator 200 bytes, whether it has a million items or a billion. Both add up to the same… but the last line shows the opening story again: the second sum of the generator gives 0, because the first sum(gen) already walked all of it. The exact size depends on the Python version (this output is from 64-bit Python 3.13).

If you need…	Use	Why
To walk the result several times	List	The generator runs out after the first walk
To access by position (x[3]) or know len()	List	A generator has no indexes and no length
To walk a lot of data once (a huge file, a million numbers)	Generator	Only one item in memory at a time
An infinite sequence (card numbers, Fibonacci)	Generator	An infinite list does not fit anywhere
Only the first ones that meet a condition	Generator	It stops computing as soon as you stop

Generator expressions

Like a list comprehension, but with round brackets: (member for member, title, days in loans if days > MAX_DAYS). It is the generator from the opening story, and it is the sneakiest trap in the topic, because it looks very much like [...].

• [x * 2 for x in data] builds the whole list right now. You can walk it as often as you like.
• (x * 2 for x in data) builds a generator. It computes nothing until you walk it, and only once.
• Inside a call with a single argument the brackets can go: sum(x * 2 for x in data) or ', '.join(b.author for b in e), as in the shelf program. That way you add or join without creating any intermediate list.

An infinite generator — impossible with a list

Since a generator computes on demand, it can go on forever. Membership card numbers are a good example: there is always a next one.

from itertools import islice


def cards(first=1):
    """Membership card numbers: the sequence never ends."""
    n = first
    while True:                  # no exit condition: infinite generator
        yield f'S-{n:04d}'
        n += 1


print(list(islice(cards(), 3)))  # islice takes 3 and stops asking
g = cards(1205)
print(next(g), next(g))

for card in cards(18):
    if card == 'S-0021':         # without this break the for would never end
        break
    print(card)

Output:

['S-0001', 'S-0002', 'S-0003']
S-1205 S-1206
S-0018
S-0019
S-0020

There are three ways to use it without hanging: cut it with itertools.islice(gen, n), which takes n and stops asking; ask by hand with next(); or walk it with a for that has a break. Notice the f'S-{n:04d}' format too: it pads with zeros up to four digits (S-0018, S-1205).

yield from and return inside a generator

Two more pieces that show up in quizzes and in the practice:

• yield from other hands over, one by one, every item of other (a generator, a list…). It is the short form of for x in other: yield x, and it is how you chain generators.
• return inside a generator does not give a value to whoever runs the for: it ends the generator (inside, it raises StopIteration). If you write return 'something', that value travels inside the exception, in .value; a normal for ignores it.

def from_shelf(name, titles):
    for title in titles:
        yield f'{name}: {title}'


def catalogue():
    yield from from_shelf('Computing', ['Networks', 'Python'])   # hands over everything from that generator
    yield from from_shelf('Mathematics', ['Calculus'])
    return 'end of catalogue'     # in a generator, return ENDS it (raises StopIteration)


for line in catalogue():
    print(line)

g = catalogue()
for _ in range(3):
    next(g)
try:
    next(g)
except StopIteration as end:
    print('StopIteration with value:', end.value)

Output:

Computing: Redes
Computing: Python
Mathematics: Calculus
StopIteration with value: end of catalogue

The for at the top walks the catalogue without noticing that there are two chained generators or that there is a return: it simply ends. Below, asking by hand, we see that the fourth next() raises StopIteration and that the exception carries 'end of catalogue' inside. In practice, a return with no value is what you will use to cut a generator early (you will see it in primes_up_to).

Common mistakes with iterators and generators

Watch it step by step in Python Tutor

Python Tutor runs your code line by line and draws the variables and objects; with generators you can really see the function “frozen”. Choose Python 3, paste this program and step through it with Next:

def available(books):
    for title, on_loan in books:
        if not on_loan:
            yield title


books = [('Networks', True), ('Data', False), ('Python', False)]
g = available(books)
first = next(g)
second = next(g)
rest = list(g)
print(first, second, rest)

Output:

Datos Python []

What to watch:

1. At g = available(books) a generator object appears, but the arrow does not enter the function: nothing has run yet.
2. At first = next(g) the available frame appears on the right, goes through ('Networks', True) without handing it over (it is on loan) and stops at the yield with title = 'Data'.
3. When it comes back, the frame does not disappear as with a normal function: it stays saved with title and the loop half done. That is the paused state.
4. second = next(g) resumes at the yield and hands over 'Python'. Then rest = list(g) finishes the loop with nothing else: rest is [].

Quick summary

Concept	What it is	How to recognise it
Iterable	Something you can walk	Has __iter__, which returns an iterator
Iterator	The cursor that walks	Has __next__, and __iter__ returns self
StopIteration	The “that’s all” signal	The for catches it by itself
Generator function	A function with yield	Calling it returns a generator and runs nothing
Generator expression	(expr for x in data)	Round brackets instead of square ones
yield from	Hand over everything from another iterable	Chains generators
return in a generator	Ends it	Raises StopIteration

Three ideas to keep:

1. A for is iter() + next() + catching StopIteration. Anything with __iter__ works with for, list(), sum() and in.
2. yield pauses, it does not end: the function keeps its variables and carries on from there on the next next(). Calling a generator function runs nothing.
3. Generators run out. If you will walk something more than once, keep a list, or make __iter__ build a new walk every time.

In the iterators and generators practice you will use all of this in three programs: an infinite Fibonacci, a pipeline of prime number generators and the library’s booking queue built with linked nodes. And once you have all this under control, the next topic teaches you how to check that your code does what it says: Python testing with unittest.

Next step: practiceIterators and generators in Python — practice with endless Fibonacci, a prime pipeline and the library booking queue (and the laptop that kept thinking)

Did this article help you? Here is how you can support SergioLearns →