yield instead of return
def countdown(n):
while n > 0:
yield n
n -= 1
return ends a function and discards its local state. yield pauses the function and preserves every local variable exactly as it was. The function resumes from that line when the next value is requested.
Calling countdown(3) does not run any of the body. It returns a generator object. The code inside starts executing only when something asks for a value: next(), a for loop, list(), or sum().
That lazy behavior is the whole point. A function that returns a list builds the entire list before the caller sees a single item. A generator produces one item at a time, only when asked. For small sequences the difference is invisible. For large ones it can mean the difference between a program that runs and one that runs out of memory.
Think about reading a file with a billion lines. A function that returns a list of all lines must load a billion strings into memory. A generator that yields one line per call holds only one string at a time. The caller processes each line and moves on. Total memory usage stays constant regardless of file size.
Generators also simplify code that produces sequences incrementally. A function that computes Fibonacci numbers, pages through an API, or walks a directory tree can yield each result as it finds it. The caller controls how many results to consume. It can stop after ten, after a hundred, or after finding a match. The generator never computes more than what the caller asks for.
The yield keyword can also appear in an expression. value = yield item sends item out and receives a value from the caller via .send(). That two-way communication is an advanced use you will rarely need as a beginner, but knowing it exists explains why generators are sometimes described as coroutines.
Generator expressions are the inline form: (x * x for x in range(10)) produces a generator without defining a function. The syntax looks like a list comprehension but uses parentheses instead of brackets. It is useful when you need a lazy sequence and the logic fits on one line.
This page covers yield versus return, lazy evaluation, memory savings, generator expressions, and the common patterns for using generators in everyday code.
Why bother
Memory. A list comprehension over a million items builds a million items. A gen holds a spot and the local variables. This Is a fixed few hundred bytes what the size. The page prints both.
That is what makes infinite sequences possible:
def naturals():
n = 1
while True:
yield n
n += 1
Nothing is built up front. So There's nothing to run out of. Take what you need with itertools.islice.
Exhausted after one pass
g = countdown(2)
list(g) # [2, 1]
list(g) # []
A gen walks forward once and doesn't rewind. If you need the values twice, either store them in a list or call the gen function again to get a fresh one. This catches everyone once, often as a mysteriously empty second loop.
Generator expressions
(n * n for n in nums)
Round brackets instead of square. The same syntax to a comprehension, no list built. Inside a call you can drop the extra brackets: sum(n * n for n in nums).
What "lazy" buys you in practice
Three things, and they're easy to state concretely.
Memory that doesn't scale with the input: a gen over a ten-million-line file holds one line, not ten million. Work that is never done: if the consumer stops after ten items, the gen found ten, not the whole sequence. And the ability to represent a thing that has no end. Nothing is built up front.
The cost is that you get one pass and no length. If you need len(), indexing, or a second traversal, you need a list, and calling list() on the gen is the clear way to say so.
Reading files, the canonical example
def lines_matching(path, needle):
with open(path, encoding="utf-8") as f:
for line in f:
if needle in line:
yield line.rstrip()
The file object is already a gen of lines, and this wraps it in another that filters. Nothing is read until the caller starts iterating, and only one line is in memory at a time even so of file size.
Note the with inside the gen: the file stays open across yields and closes when the gen is exhausted or garbage-collected. That is often what you want and is worth being conscious of, because a gen abandoned halfway holds the file open until it's collected.
yield from
Delegating to another iterable is one line:
def all_items(groups):
for group in groups:
yield from group
yield from group yields each item of group in turn. This Is both shorter and faster than the clear inner loop, and it turns truly valuable with recursive structures - walking a tree of nested lists is four lines with yield from and a lot more without.
Generators and pipelines
Because each stage pulls from the previous one, a chain of gens processes one item end-to-end before starting the next. Memory stays flat no matter how many stages there are, and no mid lists exist.
This is the shape behind most stream processing in Python: read, filter, transform, aggregate - each a small gen, composed at the end. The option, building a list at each stage, works fine until the data doesn't fit, at which point the pipeline version is the only one that still runs.
When a list is simply better
Small collections, results you need twice, any of it you want to index, and any of it you want to print for debugging. A gen that you right away wrap in list() has bought nothing, and a comprehension says what you meant more right.
Watching a pipeline run
The claim that a gen pipeline handles one item at a time end to end is easier to believe when you can see the order:
def numbers():
for n in [1, 2, 3]:
print(" produced", n)
yield n
def doubled(source):
for n in source:
print(" doubled", n * 2)
yield n * 2
for value in doubled(numbers()):
print("got", value)
produced 1
doubled 2
got 2
produced 2
doubled 4
got 4
produced 3
doubled 6
got 6
Nothing is batched. numbers makes one value, doubled transforms that one value, the loop receives it, and only then does any of it ask for the second. The list-building equal would have printed all three "made" lines first, then all three "doubled" lines.
This is the whole arg for gens in six lines of output. Memory stays flat because only one item is in flight, and if the loop stopped after the first value, numbers would never give the third.
A generator is an iterator, and what that means
for isn't special syntax for lists. It calls iter() on what it's given to get an iterator, then calls next() on that again and again until StopIteration is raised, and that exception is what ends the loop.
A gen function returns an object building just that protocol, which is why it works all over a list does. In a for, in sum, in list(), in unpacking. It's also why the same object can only be walked once: an iterator holds a spot, and there's no way to move it backwards.
Seeing the protocol right makes the action concrete:
def two():
yield 1
yield 2
g = two()
print(next(g), next(g))
try:
next(g)
except StopIteration:
print("exhausted")
1 2
exhausted
Two details follow from this. iter() on a list returns a *new* iterator each time, which is why a list can be looped over again and again. iter() on a gen returns the gen itself. This Is why it can't. And a return inside a gen doesn't return a value to the caller — it raises StopIteration, and any value goes on the exception rather than to the loop.
Where generators surprise people
Three actions account for most of the confusion, and all three follow from laziness.
Nothing runs until you ask. Calling a gen function executes none of its body. If the first line validates an arg and raises, the exception arrives at the first next(), not at the call. Possibly in a fully different function. Where that matters, split it: a normal function that validates and then returns the gen.
The values are found late. A gen formula that closes over a variable reads that variable when it runs, not when it was written. Rebind the variable in between and the gen sees the new value. This Is the same late-binding action as the lambda-in-a-loop trap.
Exhaustion is silent. A second pass over a used gen makes nothing and raises nothing. The symptom is an empty result far from the cause, and the diagnosis is almost always that a thing already consumed it — often a sum() or a len(list(...)) written for a debugging print.
The habit that avoids all three: decide early whether a thing is a stream or a collection. If it's a collection, call list() on it once and pass the list around.
send, throw and close
A gen can also receive values. This Is the feature that made Python's coroutines possible before async existed.
g.send(value) resumes the gen and makes the paused yield formula test to value, rather than to None as it does with next(). That turns the gen from a producer into a thing that can be fed — the classic example is a running mean that accepts numbers and yields the mean so far.
g.throw(exc) raises an exception at the point of the yield, letting the gen handle it or clean up. g.close() raises GeneratorExit there, which is what lets a finally or a with inside a gen run its cleanup when the gen is discarded.
In everyday code you'll use none of these right. They're worth knowing because they explain things you'll see: why contextlib.contextmanager can turn a gen into a with block, why a with inside a gen is safe, and why async def looks so much like a gen. It grew out of one.
Generators against the class you would otherwise write
Before gens, making a sequence lazily meant writing a class with the iterator protocol on it by hand. Checking the two is the clearest way to see what yield really saves.
The class version has to store its own state as fields, decide in __next__ what to do next based on those fields, and raise StopIteration itself when it's finished. any of it with a loop in it turns a state machine: what was a for and an if turns into a counter, a flag, and a chain of if-tests rebuilding where it had got to.
The gen version stores the state silently. The local variables, the spot in the loop, the depth of the nesting. All of it's the paused frame, and yield is where it pauses. That is why a gen for walking a nested structure is four lines and the equal class is thirty: recursion and loops carry the state for you, and a class has to make it clear.
There's still a case for the class. It can hold methods other than iteration, it can be reset, it can expose its spot, and it can run __len__ so that len() works. A gen can do none of those. When the object is a collection with an order, a class is right. When it's a process that makes values, yield is right.
The hands-on version of that rule: if you find yourself writing self.index, self.buffer and a __next__ full of branches, the code you meant to write is a gen function with a loop in it.
Questions people ask about Generators and yield
What is the difference between a gen and a list comprehension? Brackets. [x for x in y] builds a list; (x for x in y) makes values on demand.
Can I get the length of a gen? Not without consuming it. sum(1 for _ in g) counts it, and leaves it exhausted.
Can a gen have a return? Yes, and it ends the gen. The returned value is attached to StopIteration rather than handed to the loop.
Are gens faster than lists? Not per item. They're faster to start, use far less memory, and avoid work the consumer never asks for.
Can I index a gen? No. Use itertools.islice to skip and take, or build a list.
What is the difference between yield and yield from? yield makes one value. yield from makes each value of another iterable, and passes send and throw through to it.
Why does my gen not raise until later? Because the body doesn't run until the first value is requested.
Can I nest gen expressions? Yes, and the inner one is consumed lazily too. Readability is often the limit rather than any technical one.
Do gens work with in? Yes, and the test consumes the gen up to the match, leaving the rest. Testing twice won't behave the way you expect.
Is a file object a gen? Not just, but it's an iterator over lines, which is why for line in f works and why it too can only be walked once without seeking back.
Can two loops share one gen? They can consume it between them, each taking what the other hasn't. That is at times handy and much more often a bug.
What happens to a gen I stop halfway? It stays paused until it's garbage-collected, at which point GeneratorExit is raised inside it so finally blocks and with statements can clean up.
The one-line summary worth carrying
A gen is a function that makes a sequence instead of returning a value, and it does it one item at a time, on demand, remembering just where it stopped.
All else on this page is a consequence of that sentence. The memory saving is because only one item exists at a time. The single pass is because a paused function can't be rewound. The infinite sequences are because nothing is built in advance. The late exceptions are because the body hasn't run yet. And the pipelines work because a gen consuming another gen is still only asking for one item.
Generators and yield: recap in one screen
- A gen function returns an object. The body runs only when values are asked for.
- One pass, no length, no indexing. Call
list() when you need any of those. - Memory is flat even so of the size of the sequence. This Is what makes infinite sequences and huge files workable.
- A pipeline of gens moves one item end to end at a time, doing no work the consumer doesn't ask for.
for works by calling next() until StopIteration. A gen simply runs that protocol.
The same pause, one level up
A gen pauses at yield and hands control back to whoever called next. A coroutine pauses at await and hands control back to the event loop. That is very nearly the same machinery. A frame kept alive between resumptions — and the resemblance isn't a coincidence: async def grew out of gens, and for a few releases you wrote coroutines *as* gens, with yield from standing where await now goes.
Two pages take it from here:
Laziness is also the whole point of an interview question that looks like it's about heaps: merging k sorted sequences. heapq.merge returns a gen, and if you're going to materialise the entire result it's *slower* than concatenating and sorting — but it makes the first value in a fraction of a millisecond, which is the reason to reach for it when the sequences are files you'd rather not hold in memory all at once.