Py
Python/Iterators & Generators/Generators & yield

Generators & yield

A generator uses yield to produce values lazily, suspending its frame between values so each item is computed only when asked — an iterator you get almost for free.

Advanced40mInterview Production

Learning objectives

  • › Explain how yield suspends and resumes a function frame, preserving locals
  • › Write generator functions and generator expressions
  • › Use lazy evaluation to process large or infinite streams in constant memory
  • › Compose generator pipelines and delegate to sub-generators with yield from

Prerequisites

FunctionsIterators & the iterator protocolfor loops

Version notes

  • · `yield from` (delegation to a sub-generator) arrived in Python 3.3 (PEP 380).
  • · Generator `.send()`, `.throw()` and `.close()` come from PEP 342 (Python 2.5), which turned generators into coroutines.
  • · Per PEP 479 (default in 3.7), a StopIteration that bubbles out of a generator becomes a RuntimeError — end a generator with `return`, never `raise StopIteration`.
  • · Async generators (`async def` with `yield`) were added in Python 3.6 (PEP 525).

Simple explanation

What it is. A generator is a function that contains at least one `yield`. Calling it does not run the body — it hands back a generator object (which is an iterator). Each time you call next() on it, the body runs until it hits a yield, gives you that value, then *freezes* — local variables, instruction pointer and all — until the next call resumes it right where it paused.

Why it exists. Building a list computes and stores every element up front, costing memory proportional to its size and forcing you to finish before you use anything. A generator produces one value at a time on demand, so it uses roughly constant memory, lets you start consuming immediately, and can even model infinite sequences a list could never hold.

The problem it solves. Lazy, memory-cheap iteration: streaming a file bigger than RAM line by line, chaining transformation stages into a pipeline, or generating an unbounded sequence — all with tiny, constant memory and none of the boilerplate of a custom iterator class.

How it works. `yield value` returns value to whoever is driving the generator and suspends the frame, saving its locals on the heap. The next next() resumes just after that yield. Falling off the end (or a bare `return`) raises StopIteration to signal completion. `yield from sub` delegates to another generator, re-yielding all of its values and forwarding send/throw.

Analogy: A vending machine, not a delivery truck. Ask a generator for one item and it makes exactly one, then pauses with the motor still warm, ready to make the next only when you press the button again. A list is the whole truckload dumped on your doorstep whether or not you ever open the boxes.

Common misconceptions

  • "Calling a generator function runs it." — It only builds the generator object. Not a line of the body executes until you iterate (next/for/list).
  • "A generator can be looped over multiple times." — It is single-use. Once exhausted it yields nothing; call the function again for a fresh generator.
  • "yield is just a return that gives back a value." — return ends the function; yield *pauses* it. One generator can yield thousands of times, resuming after each.
  • "Generators are only a stylistic nicety." — They give O(1) memory for streaming and can represent infinite data, which a list fundamentally cannot.

Common alternatives

  • A hand-written iterator class (__iter__/__next__) — same protocol, much more code
  • Building a list eagerly when the data is small and you need indexing or repeated iteration
  • itertools for ready-made lazy operators (islice, chain, count, groupby)
  • async generators (`async def` + `yield`) for asynchronous, awaitable streams

Interactive visualization

Step 1 / 5
1def countdown(n):
2    while n > 0:
3        yield n
4        n -= 1
5 
6gen = countdown(3)
7print(next(gen))   # 3
8print(next(gen))   # 2

Heap / objects

ggenerator

suspended @ start, n=3

References

gen → g

Calling a generator runs no code

countdown(3) does NOT execute the body. It builds a generator object with the frame paused at the top.

Code examples

Beginneryield and generator expressions
python
Loading editor…

Note: squares(5) returns a generator without running any of the body; each next() runs to the next yield and then freezes the frame.

Behind the scenes

  • ▸Calling a generator function returns a generator object at once; the body runs only when driven by next()/for. `gen.gi_frame` is the suspended stack frame and `gen.gi_code` its compiled code.
  • ▸`yield` compiles to a YIELD_VALUE bytecode; the frame's instruction pointer and locals are preserved on the heap so execution resumes exactly where it paused.
  • ▸A generator is its own iterator: it implements __next__ and returns self from __iter__, so it drops straight into for loops, list(), sum(), unpacking and every other iterator consumer.
  • ▸`.send(x)` resumes the generator with the paused `yield` expression evaluating to x; `.throw(exc)` raises exc at the yield point; `.close()` raises GeneratorExit so `finally` blocks can clean up.
  • ▸`yield from sub` forwards values, sent values and exceptions to the sub-generator and evaluates to sub's return value — the exact machinery the original generator-based coroutines (and early async) were built on.
  • ▸Per PEP 479 (3.7+), a StopIteration raised *inside* a generator is converted to RuntimeError, so a stray StopIteration can't masquerade as normal termination — end a generator with `return`.

Interview insights

Key definitions

  • · Generator function: a function containing `yield`; calling it returns a generator object without executing the body.
  • · Generator expression: the lazy `(expr for x in it)` form — like a list comprehension but it yields on demand instead of building a list.
  • · Lazy evaluation: producing each value only when it is requested, rather than computing them all up front.
What is the difference between a list comprehension and a generator expression?

Concise answer: `[x for x in it]` builds the whole list in memory immediately (eager). `(x for x in it)` returns a generator that produces one value at a time on demand (lazy) — constant memory, single-pass, and usable for infinite streams.

Detailed answer: A list comprehension evaluates every element up front and stores them, so it costs O(n) memory and you can index and re-iterate it. A generator expression evaluates nothing until iterated, then yields one value per next() call — O(1) memory, single-pass, and able to feed a pipeline or stream data larger than RAM. Prefer the generator when you pass over the data once, when it is large or infinite, or when you feed another consumer like sum() or a for loop.

Common mistake: Wrapping a genexp in list() 'just in case', discarding the memory benefit; or reusing a genexp after it has been consumed.

Follow-ups:

  • · Which uses less memory?
  • · When would you still prefer a list?
What does `yield` actually do to a function?

Concise answer: It turns the function into a generator. yield hands a value back to the caller and suspends the function's frame — locals and position intact — so the next next() resumes right after the yield instead of restarting.

Detailed answer: The presence of yield makes calling the function return a generator object rather than running it. When driven, the body runs to the first yield, produces that value, and freezes; CPython saves the frame (instruction pointer + locals) on the heap. The following next() unfreezes it and continues after the yield. Reaching the end or a `return` raises StopIteration. This suspend/resume is exactly the mechanism async/await later reused for coroutines.

Common mistake: Expecting the body (and any prints/side effects at the top) to run when you *call* the generator function rather than when you iterate it.

Follow-ups:

  • · Is a generator single-use?
  • · How is this related to async/await?
What does `yield from` do and when would you use it?

Concise answer: `yield from iterable` delegates to a sub-iterator, re-yielding all its values (and forwarding .send()/.throw()). Use it to flatten or compose generators without a manual for-yield loop.

Detailed answer: `yield from sub` is equivalent to `for v in sub: yield v` for plain iteration, but it also transparently forwards sent values and thrown exceptions and evaluates to sub's return value. It is the clean way to build generator pipelines, flatten nested iterables, and split a big generator into helpers. Introduced in PEP 380 (3.3), it underpinned generator-based coroutines before native async/await.

Common mistake: Writing `yield sub` (which yields the sub-generator object itself) when you meant `yield from sub`.

Follow-ups:

  • · How does yield from differ from a plain for/yield?
  • · What value does it return?

Knowledge check

What's the output?Question 1 of 3

What does this print?

python
def g():
    yield 1
    yield 2
gen = g()
print(list(gen))
print(list(gen))
Multiple choiceQuestion 2 of 3

What happens when you *call* a generator function?

ScenarioQuestion 3 of 3

Why choose a generator over building a list?

Score: 0/3