OOP for Pipeline Builders: Classes Where They Earn Their Keep

# OOP for Pipeline Builders Data pipelines are mostly **functions**: extract, transform, load. Functions are easy to test and easy to parallelise. So when does a data engineer reach for a `class`? Exactly two situations: **state** and **boundaries**. ## 1. Boundaries: wrap the outside world Every pipeline talks to things that misbehave: REST APIs that rate-limit, databases that drop connections, S3 buckets with their own opinions about slashes. A class makes a superb **boundary object** — one place that owns the connection, the credentials, the retries, and the logging: ```python class RestApiSource: def __init__(self, base_url, api_key, max_retries=3): self.base_url = base_url.rstrip("/") self.session = requests.Session() self.session.headers["Authorization"] = f"Bearer {api_key}" self.max_retries = max_retries def fetch_page(self, endpoint, params=None): for attempt in range(1, self.max_retries + 1): try: resp = self.session.get( f"{self.base_url}/{endpoint}", params=params, timeout=30 ) resp.raise_for_status() return resp.json() except requests.RequestException as exc: log.warning("fetch failed (attempt %s/%s): %s", attempt, self.max_retries, exc) time.sleep(2 ** attempt) raise PipelineError(f"{endpoint} failed after {self.max_retries} tries") ``` Notice what the class buys: the retry policy, the session reuse, and the auth header live in **one** place. Every pipeline stage that needs that API shares the behaviour instead of copy-pasting `try/except` blocks. This is the exact shape you will build in the Week 4 ingestion lab. ## 2. State: carry lineage with the record Transforms should stay pure functions (`def clean(row) -> row`). But the *machinery around them* — watermark trackers for incremental loads, schema registries, dead-letter queues — is stateful by nature. A small class with `load_state()` / `save_state()` beats a dict passed through six functions. ## 3. Dataclass vs class vs NamedTuple You met all three in this module. The rule of thumb: - **NamedTuple**: a fixed record with no behaviour (a row schema). - **Dataclass**: a record with validation/defaults and tiny helpers (`is_valid()`, `to_dict()`). - **Class**: a boundary or a stateful machine (sources, sinks, trackers). If your dataclass grows locks, threads, or network calls, promote it to a class. If your class holds only data, demote it to a dataclass. ## 4. When NOT to use classes - Pure row-level transforms — keep them functions so Spark, pandas, and `multiprocessing` can serialise them without drama. - One-off glue scripts — a `main()` with functions is finished faster and reviewed faster. - Configuration — use Pydantic settings (this module), not a hand-rolled `Config` class. ## The interview answer "The pipeline stages are functions; the boundaries are classes." Say that, then walk through a source class like the one above — constructor takes config, methods do one I/O thing each, retries and logging inside — and you have answered the seniority question before it is asked.