Generators and the `yield` Keyword
Python · Core Language · Run with python3
Explanation
A generator function uses `yield` instead of `return` — calling it returns a generator object (an iterator). Execution pauses at each `yield` and resumes on the next `next()` call. Generators are memory-efficient for large sequences: instead of building a full list in memory, values are produced on demand. `yield from` delegates to a sub-generator, enabling generator pipelines and recursive generators.
Real-World Scenario
Reading a 10 GB log file line by line: `for line in open("huge.log")` already uses a generator under the hood. A custom pipeline `parse(decompress(read_chunks("data.gz")))` where each step is a generator processes terabytes without loading the file into RAM. ```python import itertools def fibonacci(): """Infinite Fibonacci generator.""" a, b = 0, 1 while True: yield a a, b = b, a + b # Take first 10 Fibonacci numbers fibs = list(itertools.islice(fibonacci(), 10)) print(fibs) # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34] def read_in_chunks(file_obj, chunk_size=1024): """Generator that yields file content in fixed-size chunks.""" while chunk := file_obj.read(chunk_size): yield chunk ```
Quick Snippet
import itertools
def fibonacci():
"""Infinite Fibonacci generator."""
a, b = 0, 1
while True:
yield a
a, b = b, a + b
# Take first 10 Fibonacci numbers
fibs = list(itertools.islice(fibonacci(), 10))
print(fibs) # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
def read_in_chunks(file_obj, chunk_size=1024):
"""Generator that yields file content in fixed-size chunks."""
while chunk := file_obj.read(chunk_size):
yield chunkFull Example
# Compile: python3 generators.py
# Output:
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
# 55
import itertools
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
fibs = list(itertools.islice(fibonacci(), 10))
print(fibs)
# Sum the next Fibonacci number
gen = fibonacci()
for _ in range(10):
next(gen)
print(next(gen)) # 55
# Generator pipeline — lazy processing
def read_lines(text):
for line in text.splitlines():
yield line.strip()
def filter_nonempty(lines):
for line in lines:
if line:
yield line
def to_upper(lines):
for line in lines:
yield line.upper()
data = "\n hello \n\n world \n"
pipeline = to_upper(filter_nonempty(read_lines(data)))
print(list(pipeline)) # ['HELLO', 'WORLD']
python3 generators.py