Recap — When to Build vs Buy Ingestion

# When to Build vs Buy Ingestion For each data source you need to load, you choose: managed tool, self-hosted tool, or custom pipeline. Most teams use a mix. ## Managed (Fivetran) Pros: - Click-config connectors for 400+ sources. - High reliability; schemas managed automatically. - Zero ops burden. - Good monitoring built-in. Cons: - Expensive at scale (per-row pricing). - Limited customization. - Vendor lock-in (your pipelines depend on Fivetran's infrastructure). When to choose: - Small/medium teams without dedicated DE bandwidth. - Critical sources where reliability matters more than cost. - Sources Fivetran covers well (SaaS apps especially). Typical pricing: $200-2000/month per source connector at moderate volume. At 1B+ monthly events: hundreds of thousands per year. ## Self-hosted managed (Airbyte OSS, Meltano) Pros: - Same connector model as Fivetran. - No per-row fees. - Open source, modifiable. - You control deployment. Cons: - You operate it (Kubernetes/Docker, monitoring, upgrades). - Connector quality varies (community-maintained ones can be flaky). - Schema management more manual. When to choose: - High-volume pipelines where Fivetran cost is prohibitive. - Teams with platform engineering capability. - Regulated environments where self-hosting is required. Total cost: compute infrastructure ($200-1000/month) + engineering time to operate (variable). Often 5-10x cheaper than equivalent Fivetran at scale. ## Custom Python pipelines Pros: - Full control. - No connector to install/configure. - Works for unusual sources. Cons: - You build and maintain. - Schema evolution is on you. - Reliability is on you. When to choose: - Sources without managed connectors. - One-off integrations (data dump from a vendor's S3). - When existing connectors don't fit your needs. ## The hybrid stack Most production data teams in 2026: ``` - Fivetran for 3-5 critical SaaS sources (Stripe, Salesforce, Zendesk). - Airbyte OSS for high-volume sources (Postgres CDC, Shopify). - Custom Python for one-off integrations (vendor file drops, internal APIs). ``` Three tools, each chosen per source. No one-size-fits-all. ## Build vs buy framework For each source ask: 1. **Volume**: rows per month? - 100M: self-host or custom. 2. **Criticality**: how bad is a 1-day outage? - High: Fivetran (SLA + support). - Low: self-host or custom. 3. **Customization needs**: standard connector configuration suffice? - Yes: Fivetran or Airbyte. - No (need custom logic, source-specific): custom. 4. **Team capability**: do you have engineers to operate ingestion infra? - Yes: self-host or custom. - No: Fivetran. 5. **Source coverage**: does Fivetran have this connector? - Yes: probably use it. - No: Airbyte OSS or custom. ## Cost example For a SaaS startup with these sources: | Source | Volume | Choice | Monthly cost | |---|---|---|---| | Stripe | 100K events/month | Fivetran | $200 | | Salesforce | 10K records/month | Fivetran | $200 | | Postgres (app DB) | 10M events/month | Fivetran | $1500 | | Shopify | 200K orders/month | Fivetran | $400 | | Vendor file drops | 100K rows/month | Custom Python | $0 | Total: ~$2300/month Fivetran + custom code. At 100x scale (1B rows/month from Postgres): - Fivetran cost: ~$15K/month. - Airbyte OSS: ~$500/month infra + 1 engineer's part-time attention. Switching saves ~$14K/month. Pays for the engineering quickly. ## The custom Python pipeline pattern For unusual sources, you write: ```python # pipelines/custom_source.py import requests from datetime import datetime from sqlalchemy import create_engine def fetch_from_source(since): response = requests.get( "https://api.weird-vendor.com/data", params={"since": since.isoformat()}, headers={"Authorization": f"Bearer {API_KEY}"}, ) return response.json() def load_to_warehouse(records): engine = create_engine(DATABASE_URL) df = pd.DataFrame(records) df["_loaded_at"] = datetime.utcnow() df.to_sql("raw_weird_vendor_data", engine, schema="raw", if_exists="append") def run_pipeline(last_run_at): records = fetch_from_source(since=last_run_at) load_to_warehouse(records) log_completion(rows=len(records)) # Schedule with Airflow / cron / GitHub Actions ``` Patterns to follow: - Incremental (use `since` parameter). - Error handling. - `_loaded_at` audit column. - Logging. - Schema validation. Orchestrate via Airflow (next lesson) so retries and monitoring come for free. ## What about Singer / Stitch / Meltano specifically Singer is an open spec for data extraction taps/targets. Meltano is a framework using Singer taps. ```bash meltano add extractor tap-shopify meltano add loader target-snowflake meltano run tap-shopify target-snowflake ``` Pros: open ecosystem, large catalog of taps. Cons: connector quality varies wildly; some taps abandoned. In 2026, Airbyt