Generators, Iterators & itertools

This lesson on Generators, Iterators & itertools is hands-on and example-driven. You will master Python's Iterator Protocol to traverse sequences on demand, control manual iteration loops, and build memory-efficient data pipelines. You will also use sentinel-based iterators and abstract sequence typing to decouple business logic from underlying data containers.

What You'll Be Able To Do

• Implement the iterator protocol manually using iter() and next() alongside StopIteration exception handling.
• Distinguish between multi-pass iterables and stateful single-pass iterators in runtime memory.
• Process streaming data sources using the two-argument iter(callable, sentinel) pattern.
• Type-hint functions with Iterable to support polymorphic sequence inputs without container coupling.
• Enforce immutability and hashability on custom objects using dataclasses(frozen=True) for set-based sequence processing.

Detailed Concept Walkthrough

1. Iterator Protocol Fundamentals

The iterator protocol is Python's universal mechanism for sequential data access, defined by the __iter__ and __next__ dunder methods. Any construct that traverses data—including for loops, list comprehensions, and generators—relies on this contract under the hood.

• Mechanism: An object implements __iter__() to return an iterator instance, and __next__() to return subsequent elements one at a time. When no items remain, __next__() raises a StopIteration exception to signal completion.
• Under the Hood: High-level constructs like for item in collection: translate directly to acquiring an iterator via iter(collection) and repeatedly calling next() within an implicit try/except StopIteration block.
• Best Practice: Rely on implicit iteration like for loops for standard traversal, but understand the protocol so you can implement custom streamable objects or debug consumption issues.

numbers = [10, 20, 30]
# Explicitly requesting an iterator
it = iter(numbers)

print(next(it))  # 10
print(next(it))  # 20
print(next(it))  # 30
# next(it) now raises StopIteration

Key Takeaway: Python iteration is driven entirely by __iter__() returning an object whose __next__() method yields items until raising StopIteration.

2. Manual Iteration and Error Handling

Manual iteration provides programmatic control over sequence traversal by pairing iter() and next() inside explicit error-handling loops.

• Execution Flow: Calling iter(sequence) initializes the iterator pointer before a while True loop repeatedly retrieves items with next(iterator). Traversal halts cleanly when the StopIteration exception is caught.
• Under the Hood: Python exceptions are lightweight control flow mechanisms; StopIteration carries no stack trace penalty when raised internally during loop termination.
• Syntax Rule: When using next(), you can optionally provide a default fallback argument (e.g., next(it, None)) to avoid raising StopIteration when the iterator is exhausted.

items = ["alpha", "beta", "gamma"]
iterator = iter(items)

while True:
    try:
        val = next(iterator)
        print(f"Processing: {val}")
    except StopIteration:
        break  # Sequence fully consumed

Key Takeaway: A standard for loop is syntactic sugar for a while True loop that invokes next() and catches StopIteration.

3. Iterable vs. Iterator Distinction

An iterable is a container capable of generating new iterators, whereas an iterator is an active state machine that tracks traversal progress and yields values.

• Mechanism: Calling iter() on an iterable (like a list or tuple) creates an independent, fresh iterator starting at index 0. Calling iter() on an iterator returns the iterator itself (self), preserving its current position.
• Under the Hood: Because an iterator references its own running state, multiple consumers reading from the same iterator will advance the single shared cursor, making iterators single-pass consumers.
• Nuance: Iterables can be traversed multiple times by spawning separate iterators; exhausted iterators cannot be reset and will continuously raise StopIteration.

data = [1, 2, 3]  # Iterable
it1 = iter(data)   # Fresh iterator
it2 = iter(data)   # Independent fresh iterator

print(next(it1))   # 1
print(next(it2))   # 1 (it2 has independent state)

it3 = iter(it1)    # iter() on iterator returns it1 itself
print(it3 is it1)  # True (shares identical state)

Key Takeaway: Iterables create new iterators; iterators return themselves and maintain single-pass consumption state.

4. Sentinel-Based Iteration

The two-argument form of iter() creates an iterator that continuously invokes a callable until the callable returns a designated sentinel value.

• Mechanism: Invoking iter(callable, sentinel) produces a callable iterator. Each call to next() executes callable() with zero arguments, yielding the result until it matches sentinel, which raises StopIteration.
• Under the Hood: This eliminates the need for manual while True boilerplate and intermediate state flags when consuming streaming I/O or polling network sockets.
• Best Practice: Use iter(f.readline, '') to read text files line by line or iter(partial(f.read, chunk_size), b'') for binary streams without loading full contents into memory.

# Reading file lines until hitting the empty string sentinel
with open("sample.txt", "w") as f:
    f.write("line 1\nline 2\n")

with open("sample.txt", "r") as f:
    # iter(callable, sentinel) stops when readline returns ""
    for line in iter(f.readline, ""):
        print(line.strip())

Key Takeaway: iter(callable, sentinel) turns any zero-argument function into a standard iterator that terminates upon returning the sentinel.

5. Iterators for Abstraction and Typing

Treating function arguments as general iterables decouples business logic from specific container types, enabling flexible polymorphism across lists, tuples, sets, and streams.

• Mechanism: Type-hinting parameters with collections.abc.Iterable[T] signals that the function only requires sequence traversal, accepting any concrete iterable object.
• Under the Hood: If downstream processing adds items to sets or uses them as dictionary keys, elements must implement __hash__ and __eq__. Mutable objects fail hashability checks by default.
• Best Practice: Use @dataclass(frozen=True) to create immutable, auto-hashable domain models that can be safely processed inside iterable pipelines and set operations.

from collections.abc import Iterable
from dataclasses import dataclass

@dataclass(frozen=True)
class Record:
    id: int
    name: str

def process_records(records: Iterable[Record]) -> set[Record]:
    # Works with list, tuple, generator, or set
    return {r for r in records if r.id > 0}

result = process_records([Record(1, "Alice"), Record(2, "Bob")])
print(result)

Key Takeaway: Accepting Iterable[T] maximizes function reusability, while frozen=True ensures custom objects remain hashable across set operations.

Topics Covered in Generators, Iterators & itertools

1. Iterator Protocol Fundamentals (02:01 - 02:40) — Covers the roles of the iter and next dunder methods across Python loops and comprehensions.
2. Manual Iteration & StopIteration (02:40 - 04:46) — Demonstrates manual traversal using iter and next inside try-except blocks handling StopIteration.
3. Iterable vs. Iterator (04:46 - 07:38) — Contrasts multi-pass iterables with single-pass stateful iterators and explores self-referential iter returns.
4. Sentinel-based Iteration (07:38 - 09:16) — Shows how the two-argument iter callable pattern reads streams until hitting a sentinel value.
5. Iterators for Abstraction (09:16 - 11:47) — Explains polymorphic sequence typing with Iterable and requirements for frozen hashable dataclasses.

Python Cheat Sheet

• iter(iterable) — Returns an iterator object from an iterable container

  it = iter([1, 2, 3])
  

• next(iterator, [default]) — Retrieves the next element or returns default if exhausted

  item = next(it, None)
  

• iter(callable, sentinel) — Iterates calling zero-arg function until output equals sentinel

  for line in iter(f.readline, ""):
      print(line)
  

• class __iter__() / __next__() — Dunder methods required to implement the custom iterator protocol

  def __iter__(self):
      return self
  

• raise StopIteration — Signals the termination of manual sequence iteration

  raise StopIteration
  

• @dataclass(frozen=True) — Generates immutable, hashable dataclass for set/dict iterable pipelines

  @dataclass(frozen=True)
  class Item:
      val: int
  

Comparison Table

Common Pitfalls

• Mistake: Expecting an iterator to restart from the beginning after a loop. Avoid: Re-create the iterator from the original iterable or pass a collection instead.
• Mistake: Calling iter(callable, sentinel) with a function call instead of a callable reference. Avoid: Pass the function name without parentheses, like f.readline instead of f.readline().
• Mistake: Inserting mutable custom dataclass instances into sets during iterable pipelines. Avoid: Add frozen=True to the dataclass decorator to ensure automatic hash generation.

FAQs

• Why does calling iter() on an iterator return the iterator itself? This ensures iterators are directly usable in for loops and functions expecting an iterable without resetting their state.
• What happens if next() is called on an already exhausted iterator? It continuously raises StopIteration on every subsequent call unless a default fallback value is explicitly provided.
• Can I convert an iterator back into a reusable iterable? No, once consumed, you must reconstruct the data into a container like a list or recreate the original iterable source.