Why dbt Exists — The Analytics Engineering Wave

Before dbt, analytics SQL lived in three places: a Looker LookML file, a cron-scheduled .sql file someone wrote in 2019, and someone's Notion. There was no lineage, no tests, no docs, no version control discipline. Production data pipelines were "person X knows how this works".

dbt's central insight: treat SQL like application code. Version control. Modular. Testable. Documented. Compiles into a dependency graph. CI/CD ready.

What dbt actually is

dbt = a CLI (and now also a cloud service) that:

1. Reads your models/*.sql files (each one a single SELECT statement).
2. Resolves dependencies between them via ref() calls.
3. Compiles them into the right order.
4. Executes them against your warehouse (Snowflake, BigQuery, Redshift, Databricks, Postgres).
5. Runs tests and generates documentation as part of the run.

That's it. No new query language. No new database. No new BI tool. Just SQL files + Jinja templating + a compiler + a runner.

What it solves

• Lineage — ref('stg_orders') tells dbt "this model depends on stg_orders". The DAG is built automatically. You can answer "what breaks if I change this column" in seconds.
• Idempotency — every model is a SELECT statement materialized by dbt. Rerunning is safe; the model is rebuilt from source.
• Tests — not_null, unique, accepted_values, relationships declared in YAML, plus custom SQL tests. Tests run with every dbt build.
• Documentation — descriptions in YAML, lineage diagrams generated automatically, browsable via dbt docs serve.
• Environment management — dev, staging, prod configs with separate target schemas / databases. Same code, different output destinations.
• Reproducibility — git-controlled, code-reviewed, testable. The end of "Bob's mystery SQL file in cron".

What it doesn't solve

• Data extraction — dbt doesn't pull data from sources. Use Fivetran, Airbyte, Meltano, or homegrown CDC for that. dbt starts where data has already landed in the warehouse.
• Orchestration beyond dbt run — dbt knows how to run models in the right order. It doesn't schedule itself. Use Airflow, Dagster, Prefect, GitHub Actions, or dbt Cloud's scheduler.
• Real-time — dbt is batch-oriented. Streaming/realtime processing needs Materialize, RisingWave, or a streaming framework.
• Python-heavy work — dbt is SQL-first. dbt-python models (running Python in the warehouse) exist but are limited. For heavy ML/feature engineering, use a separate tool.
• BI dashboards — dbt builds the tables. Looker/Tableau/Metabase build the dashboards on top.

The split: dbt-core vs dbt Cloud

• dbt-core — the open-source CLI. Free. Self-hosted. You set up orchestration yourself.
• dbt Cloud — managed service with IDE, scheduler, CI/CD, hosted docs. Paid per developer seat.

Most teams start with dbt-core + GitHub Actions (or Airflow) and graduate to Cloud when team size or governance needs it.

The mental model

[ Warehouse raw schema ]
        ↓
[ dbt SQL files: stg_*, int_*, fct_*, dim_* ]
        ↓ (dbt run)
[ Warehouse mart schema ]
        ↓
[ BI tools ]

dbt sits squarely between "data has landed in the warehouse" and "BI tools query the warehouse". It's the modeling layer, and only the modeling layer.

What changes about your workflow

Before dbt: open SQL Workbench, write a SELECT, CREATE TABLE AS, hope it works, paste into Looker.

After dbt:

1. Create a .sql file in your dbt project.
2. Write the SELECT statement using ref() to point at upstream models.
3. Run dbt run --select <model> to materialize.
4. Add tests in schema.yml.
5. Commit, push, code review.
6. Merge → CI runs dbt build → production updates.

The discipline is the change. The SQL itself looks almost the same.

Takeaway

dbt is a workflow upgrade, not a new language. It takes SQL scripts and turns them into a maintainable, tested, documented, version-controlled, dependency-aware system. Most modern analytics teams have either adopted it or actively regret not adopting it.