Iterators and generators in Python — how the for loop works under the hood
Iterators and generators in Python are the mechanism behind every for loop you’ve ever written. When you write for item in collection: Python is doing several things automatically — and understanding what those things are turns a beginner Python programmer into someone who can design efficient, memory-friendly code. This article covers everything: the iterable/iterator distinction, iter(), next(), __iter__, __next__, StopIteration and yield.
Table of Contents
What happens when you write a for loop
Every time you write this:
for item in [1, 2, 3]:
print(item)
Python is actually doing this behind the scenes:
_iterator = iter([1, 2, 3]) # step 1: get an iterator
while True:
try:
item = next(_iterator) # step 2: get the next value
print(item)
except StopIteration: # step 3: stop when exhausted
break
Two separate concepts are at work here — iterable and iterator — and understanding the difference between them is the key to everything in this article.
Iterable vs Iterator — the crucial distinction
An iterable is anything you can iterate over — a list, a string, a tuple, a dictionary, a set. To be iterable, an object must have an __iter__ method that returns an iterator.
An iterator is the object that actually does the iteration — it keeps track of position and produces one value at a time. An iterator must have both __iter__ (returns itself) and __next__ (returns the next value or raises StopIteration).
my_list = [1, 2, 3] # iterable — has __iter__ # Step 1: get the iterator from the iterable my_iterator = iter(my_list) # calls my_list.__iter__() # Step 2: get values one by one print(next(my_iterator)) # → 1 (calls my_iterator.__next__()) print(next(my_iterator)) # → 2 print(next(my_iterator)) # → 3 print(next(my_iterator)) # → StopIteration raised
The list is an iterable — you can get multiple independent iterators from it, each starting from the beginning. The iterator is exhaustible — once it raises StopIteration, it’s done.
my_list = [1, 2, 3] # Two independent iterators from the same list it1 = iter(my_list) it2 = iter(my_list) next(it1) # → 1 next(it1) # → 2 next(it2) # → 1 — it2 starts fresh, it1's position doesn't affect it2
This is why you can loop over a list multiple times — each for loop gets a new iterator. If the list itself were the iterator, you could only loop over it once.
iter() and next() — the built-in functions
iter(obj) calls obj.__iter__() and returns the iterator. next(iterator) calls iterator.__next__() and returns the next value.
Both accept an optional default argument for next() — if provided, it returns this value instead of raising StopIteration when exhausted:
it = iter([1, 2]) print(next(it, 'exhausted')) # → 1 print(next(it, 'exhausted')) # → 2 print(next(it, 'exhausted')) # → 'exhausted' (no StopIteration) print(next(it, 'exhausted')) # → 'exhausted'
Implementing your own iterator
Any class with __iter__ and __next__ is a proper iterator. Here’s the classic countdown example:
class Countdown:
def __init__(self, start):
self.current = start
def __iter__(self):
return self # an iterator returns itself
def __next__(self):
if self.current <= 0:
raise StopIteration
value = self.current
self.current -= 1
return value
countdown = Countdown(5)
for n in countdown:
print(n, end=' ')
# → 5 4 3 2 1
# Manual iteration
it = iter(Countdown(3))
print(next(it)) # → 3
print(next(it)) # → 2
print(next(it)) # → 1
# next(it) → StopIteration
Notice that __iter__ returns self — an iterator is its own iterator. The for loop calls iter(countdown) which calls countdown.__iter__() which returns countdown itself. Then it calls next(countdown) repeatedly until StopIteration.
Separating the iterable from the iterator
The countdown above is both iterable and iterator in one class. A cleaner design separates them — the iterable creates fresh iterators each time:
class NumberRange:
"""Iterable — like range(). Can be iterated multiple times."""
def __init__(self, start, stop, step=1):
self.start = start
self.stop = stop
self.step = step
def __iter__(self):
return NumberRangeIterator(self.start, self.stop, self.step)
class NumberRangeIterator:
"""Iterator — does the actual iteration. Single-use."""
def __init__(self, start, stop, step):
self.current = start
self.stop = stop
self.step = step
def __iter__(self):
return self # iterator always returns itself
def __next__(self):
if self.current >= self.stop:
raise StopIteration
value = self.current
self.current += self.step
return value
# Can iterate multiple times because each for loop gets a new iterator
r = NumberRange(1, 6)
print(list(r)) # → [1, 2, 3, 4, 5]
print(list(r)) # → [1, 2, 3, 4, 5] — fresh iterator each time
for n in NumberRange(0, 10, 2):
print(n, end=' ')
# → 0 2 4 6 8
Generators — the lazy shortcut
Writing a full iterator class with __iter__ and __next__ is verbose for simple cases. Python’s yield keyword lets you write a generator function that behaves as an iterator automatically:
# This generator function replaces the entire NumberRange + NumberRangeIterator pair
def number_range(start, stop, step=1):
current = start
while current < stop:
yield current # pause here and give back current
current += step # resume here on the next next() call
for n in number_range(1, 6):
print(n, end=' ')
# → 1 2 3 4 5
gen = number_range(0, 10, 2)
print(next(gen)) # → 0
print(next(gen)) # → 2
print(next(gen)) # → 4
How yield works
yield is the key mechanism. When a function contains yield it becomes a generator function. Calling it doesn’t execute the body — it returns a generator object. The body only runs when you call next() on the generator:
def show_steps():
print('Step 1')
yield 'result 1' # pause here
print('Step 2')
yield 'result 2' # pause here
print('Step 3')
# function ends → StopIteration
gen = show_steps() # nothing executes yet
print('Before first next()')
value = next(gen) # executes until first yield
print(f'Got: {value}') # → result 1
value = next(gen) # resumes from first yield, executes until second yield
print(f'Got: {value}') # → result 2
next(gen) # resumes, prints Step 3, function ends → StopIteration
Before first next() Step 1 Got: result 1 Step 2 Got: result 2 Step 3 StopIteration
The generator function is lazy — it only computes the next value when asked. The local variables (current, step, etc.) are preserved between next() calls — the function is literally paused at the yield line and resumes from there.
Why generators matter — memory efficiency
The most important practical advantage of generators is memory. A list stores all its values at once. A generator produces them one at a time:
import sys
# List — all values in memory at once
my_list = [n ** 2 for n in range(1_000_000)]
print(f'List size: {sys.getsizeof(my_list):,} bytes') # ~8.7 MB
# Generator — produces one value at a time
def squares(n):
for i in range(n):
yield i ** 2
gen = squares(1_000_000)
print(f'Generator size: {sys.getsizeof(gen)} bytes') # ~104 bytes
# Both give the same values when iterated
print(next(gen)) # → 0
print(next(gen)) # → 1
print(next(gen)) # → 4
For a million values the list uses ~8.7 MB. The generator uses 104 bytes regardless of size. For truly large sequences (processing log files, reading large datasets line by line) this difference is not just performance — it’s the difference between a program that runs and one that crashes with a memory error.
Generator expressions — inline generators
Just as list comprehensions are compact versions of list-building loops, generator expressions are compact versions of simple generator functions:
# List comprehension — creates all values immediately
squares_list = [n ** 2 for n in range(10)] # list
# Generator expression — creates them lazily
squares_gen = (n ** 2 for n in range(10)) # generator
# Syntax: () instead of []
print(type(squares_list)) # → <class 'list'>
print(type(squares_gen)) # → <class 'generator'>
# Both iterate the same way
for s in squares_gen:
print(s, end=' ')
# → 0 1 4 9 16 25 36 49 64 81
Generator expressions are particularly useful as arguments to functions that consume iterables — they avoid creating an intermediate list:
# sum() with generator expression — never creates a list total = sum(n ** 2 for n in range(1_000_000)) # memory efficient # Equivalent but less efficient total = sum([n ** 2 for n in range(1_000_000)]) # creates full list first
Common generator patterns in FP2
# 1. Infinite sequence
def natural_numbers(start=1):
n = start
while True: # infinite — caller decides when to stop
yield n
n += 1
gen = natural_numbers()
print([next(gen) for _ in range(5)]) # → [1, 2, 3, 4, 5]
# 2. Filter and transform
def even_squares(limit):
for n in range(limit):
if n % 2 == 0:
yield n ** 2
print(list(even_squares(10))) # → [0, 4, 16, 36, 64]
# 3. Reading a file lazily
def read_lines(filename):
with open(filename) as f:
for line in f:
yield line.strip()
# Processes one line at a time — works even on huge files
# for line in read_lines('large_file.txt'):
# process(line)
# 4. Sliding window
def sliding_window(iterable, size):
items = list(iterable)
for i in range(len(items) - size + 1):
yield tuple(items[i:i+size])
for window in sliding_window([1, 2, 3, 4, 5], 3):
print(window)
# → (1, 2, 3), (2, 3, 4), (3, 4, 5)
# 5. Fibonacci — classic generator
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
fib = fibonacci()
print([next(fib) for _ in range(10)])
# → [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
yield from — delegating to another iterable
yield from is shorthand for yielding each item from a sub-iterable. It’s used to compose generators:
def chain(*iterables):
for it in iterables:
yield from it # equivalent to: for item in it: yield item
for item in chain([1, 2], [3, 4], [5]):
print(item, end=' ')
# → 1 2 3 4 5
# Also works with generators
def first_n(gen, n):
for _ in range(n):
yield next(gen)
def flatten(nested):
for item in nested:
if isinstance(item, (list, tuple)):
yield from flatten(item) # recurse into nested
else:
yield item
print(list(flatten([1, [2, [3, 4]], 5, [6]])))
# → [1, 2, 3, 4, 5, 6]
The full iterator protocol summary
# FOR LOOP — what Python actually does
for item in obj:
body
# Is equivalent to:
_it = iter(obj) # obj.__iter__()
while True:
try:
item = _it.__next__()
body
except StopIteration:
break
# ITERABLE — has __iter__, returns a fresh iterator each time
class MyIterable:
def __iter__(self):
return MyIterator(self) # returns a separate iterator object
# ITERATOR — has both __iter__ (returns self) and __next__
class MyIterator:
def __iter__(self):
return self # always returns self
def __next__(self):
if done:
raise StopIteration
return next_value
# GENERATOR FUNCTION — easiest way to create an iterator
def my_generator():
yield value1
yield value2
# StopIteration raised automatically when function ends
Visualise with Python Tutor
Copy this code into pythontutor.com and step through it:
def countdown(n):
while n > 0:
yield n
n -= 1
# Manual iteration
gen = countdown(3)
print(next(gen)) # 3
print(next(gen)) # 2
print(next(gen)) # 1
# for loop uses the same mechanism
for n in countdown(3):
print(n)
Step through and observe four key moments. When countdown(3) is called it doesn’t execute the function body — it returns a generator object. Only when next(gen) is called does execution begin. On the first next(), the function runs until yield n — it pauses with n = 3 and returns 3. On the second next(), execution resumes from just after the yield — n -= 1 runs, n becomes 2, the while checks, and yield n pauses again with 2. When the while condition fails (n becomes 0), the function reaches its end and StopIteration is raised automatically. The for loop catches StopIteration and stops — you never see it.
Quick summary
# ITERABLE vs ITERATOR
# Iterable: has __iter__, returns a fresh iterator each call
# examples: list, tuple, str, dict, set, range
# Iterator: has __iter__ (returns self) AND __next__
# remembers position, raises StopIteration when done
# BUILT-IN FUNCTIONS
iter(obj) # calls obj.__iter__()
next(it) # calls it.__next__()
next(it, default) # returns default instead of raising StopIteration
# THE FOR LOOP INTERNALLY
# _it = iter(obj)
# while True:
# try: item = next(_it)
# except StopIteration: break
# IMPLEMENTING AN ITERATOR
class Counter:
def __init__(self, start, stop):
self.current = start
self.stop = stop
def __iter__(self):
return self # iterator returns itself
def __next__(self):
if self.current >= self.stop:
raise StopIteration
value = self.current
self.current += 1
return value
# GENERATOR FUNCTION — easiest iterator
def counter(start, stop):
while start < stop:
yield start # pause and return value
start += 1 # resume here on next next() call
# GENERATOR EXPRESSION
gen = (x ** 2 for x in range(10)) # lazy, no list created
# yield from — delegate to sub-iterable
def chain(*iters):
for it in iters:
yield from it
# MEMORY: generator vs list
# list = all values at once → lots of memory
# generator = one value at a time → ~104 bytes always
# COMMON PATTERNS
# Infinite sequence: while True: yield value; advance
# Fibonacci: a, b = 0, 1; while True: yield a; a,b = b,a+b
# File reading: for line in file: yield line.strip()
# Sliding window: for i in range(len-size+1): yield items[i:i+size]
# GENERATOR EXHAUSTION
gen = (x for x in range(3))
list(gen) # → [0, 1, 2]
list(gen) # → [] — generator exhausted!
# Unlike list: iter(list) always creates fresh iterator
In the next article we practice iterators and generators with real programs — a lazy file processor, a custom range and an infinite sequence of primes.
2 Comments