Python for Data Engineers
Back to Async, Concurrency & Performance

Generators, Iterators & itertools

Lazy evaluation. Process huge datasets with O(1) memory; compose pipelines elegantly. FIND_VIDEO: search 'python generators iterators itertools tutorial' — recommended channel: mCoding / Raymond Hettinger. Aim for 10 min or under.

21 minutesVideo LessonPDF notes
🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.
Loading progress...

Key moments

  1. Iterator Protocol Basics — Introduces the iterator protocol along with the iter and next dunder methods.
  2. StopIteration and For Loops — Demonstrates manual next traversal, StopIteration handling, and automatic for-loop execution.
  3. Iterables vs. Iterators — Contrasts repeatable iterable sequence creation against self-returning stateful iterators.
  4. Sentinel-Based Iteration — Applies the two-argument iter function with sentinel values for streamlined file reading.
  5. Type Abstraction and Hashability — Uses Iterable type hints for polymorphism and explains why set iterables require frozen dataclasses.
  6. Custom Iterator Implementations — Builds custom infinite and finite iterator classes using Self annotations and StopIteration guards.
  7. Streaming with itertools — Constructs functional pipelines using itertools count, repeat, accumulate, permutations, and combinations.
PDF notes

Frequently asked questions

Why must custom iterators implement __iter__ in addition to __next__?

Implementing `__iter__` to return `self` allows the iterator to be passed directly to `for` loops and functions expecting generic iterables.

What happens when you pass an already exhausted iterator into a for loop?

The loop calls `next()` immediately, receives `StopIteration`, and terminates without executing the loop body or throwing an error.

When is permutations preferred over combinations in itertools?

Use `permutations` when element ordering matters (`('a', 'b')` != `('b', 'a')`), and use `combinations` when groups with identical members are considered equal.

How was this lesson?

Your feedback helps us refine explanations and catch bugs.