OOP for Pipeline Builders: Classes Where They Earn Their Keep

Data pipelines are mostly functions: extract, transform, load. Functions are easy to test and easy to parallelise. So when does a data engineer reach for a class? Exactly two situations: state and boundaries.

1. Boundaries: wrap the outside world

Every pipeline talks to things that misbehave: REST APIs that rate-limit, databases that drop connections, S3 buckets with their own opinions about slashes. A class makes a superb boundary object — one place that owns the connection, the credentials, the retries, and the logging:

class RestApiSource:
    def __init__(self, base_url, api_key, max_retries=3):
        self.base_url = base_url.rstrip("/")
        self.session = requests.Session()
        self.session.headers["Authorization"] = f"Bearer {api_key}"
        self.max_retries = max_retries

    def fetch_page(self, endpoint, params=None):
        for attempt in range(1, self.max_retries + 1):
            try:
                resp = self.session.get(
                    f"{self.base_url}/{endpoint}", params=params, timeout=30
                )
                resp.raise_for_status()
                return resp.json()
            except requests.RequestException as exc:
                log.warning("fetch failed (attempt %s/%s): %s",
                            attempt, self.max_retries, exc)
                time.sleep(2 ** attempt)
        raise PipelineError(f"{endpoint} failed after {self.max_retries} tries")

Notice what the class buys: the retry policy, the session reuse, and the auth header live in one place. Every pipeline stage that needs that API shares the behaviour instead of copy-pasting try/except blocks. This is the exact shape you will build in the Week 4 ingestion lab.

2. State: carry lineage with the record

Transforms should stay pure functions (def clean(row) -> row). But the machinery around them — watermark trackers for incremental loads, schema registries, dead-letter queues — is stateful by nature. A small class with load_state() / save_state() beats a dict passed through six functions.

3. Dataclass vs class vs NamedTuple

You met all three in this module. The rule of thumb:

• NamedTuple: a fixed record with no behaviour (a row schema).
• Dataclass: a record with validation/defaults and tiny helpers (is_valid(), to_dict()).
• Class: a boundary or a stateful machine (sources, sinks, trackers).

If your dataclass grows locks, threads, or network calls, promote it to a class. If your class holds only data, demote it to a dataclass.

4. When NOT to use classes

• Pure row-level transforms — keep them functions so Spark, pandas, and multiprocessing can serialise them without drama.
• One-off glue scripts — a main() with functions is finished faster and reviewed faster.
• Configuration — use Pydantic settings (this module), not a hand-rolled Config class.

The interview answer

"The pipeline stages are functions; the boundaries are classes." Say that, then walk through a source class like the one above — constructor takes config, methods do one I/O thing each, retries and logging inside — and you have answered the seniority question before it is asked.