Documentation — Descriptions, dbt docs, and Why It Matters

This lesson on Documentation — Descriptions, dbt docs, and Why It Matters is hands-on and example-driven. You will document your dbt models and columns directly alongside your transformation logic using schema.yml files and reusable Markdown doc blocks. You will generate warehouse metadata catalogs and host an interactive, lineage-aware documentation web portal using the dbt CLI.

What You'll Be Able To Do

• Define model-level and column-level descriptions within a co-located schema.yml file.
• Compile project metadata and database catalog artifacts using dbt docs generate.
• Host and inspect the local dbt documentation web interface using dbt docs serve.
• Author external Markdown documentation blocks to manage long-form, reusable text.
• Inject external Markdown doc blocks into schema.yml configurations using the doc Jinja function.

Detailed Concept Walkthrough

1. Schema Metadata Definition

Model documentation in dbt lives alongside transformation SQL files in declarative schema.yml configurations. This establishes version-controlled data dictionaries directly within your development repository.

• Structure & Location: Create a schema.yml file in the exact same directory as your model SQL files. This YAML file declares model metadata, high-level table descriptions, and nested column-level properties.
• Downstream Clarity: Explicitly documenting column names, descriptions, and data types provides clarity for downstream analytics consumers. Without this file, materialized database tables retain only raw technical names without contextual business meaning.
• Best Practice: Update model descriptions and column specifications in the same commit as your SQL transformation changes. Co-locating documentation with code ensures that documentation remains accurate as business logic evolves.

version: 2

models:
  - name: fct_events
    description: "Cleaned event log tracking user interaction over time."
    columns:
      - name: event_id
        description: "Primary key unique identifier for an event."
      - name: event_timestamp
        description: "UTC timestamp when the event was recorded."

Key Takeaway: Store metadata in schema.yml next to your model files to maintain synchronized, version-controlled documentation.

2. Documentation Generation and Serving

dbt converts YAML metadata, project lineage, and warehouse information schemas into a static documentation web interface via two dedicated CLI commands.

• Catalog Generation: Run dbt docs generate to inspect your dbt project files and query the warehouse information schema. This compiles your Jinja code and creates the metadata manifest and catalog.json artifact.
• Local Web Hosting: Run dbt docs serve to start a local web server hosting the interactive documentation portal. This command serves static HTML/JS assets that read the generated JSON metadata files.
• Execution Flow: Execute generate before serve whenever YAML descriptions or SQL definitions are modified. The serve command only hosts existing static build outputs and does not recompile updated code on the fly.

# Step 1: Generate catalog.json and manifest files from project and database
dbt docs generate

# Step 2: Start local web server (defaults to http://localhost:8080)
dbt docs serve

Key Takeaway: Run dbt docs generate to build metadata artifacts before hosting them locally with dbt docs serve.

3. Navigating the Documentation Interface

The dbt Docs portal combines user-written schema descriptions with automatic dependency tracking, compiled SQL artifacts, and column-level warehouse attributes.

• Lineage & Dependencies: The interface automatically parses ref and source macros to display interactive dependency graphs. It maps relationships between raw inputs like seed files, staging models, and downstream analytics marts.
• Metadata Inspection: The web portal renders model descriptions, column lists, and warehouse data types side-by-side. Analysts can inspect technical schemas without needing direct query access to the database catalog.
• SQL Verification: Users can toggle between source Jinja template code and compiled warehouse SQL. This allows developers to verify how references and variables resolve into raw database queries.

-- Example model relying on an upstream seed dependency
-- The Docs UI maps this seed dependency automatically
SELECT
    event_id,
    event_name,
    event_timestamp
FROM {{ ref('seed_raw_events') }}

Key Takeaway: The Docs UI surfaces column definitions, warehouse data types, compiled SQL, and dependency lineage in a single interface.

4. Markdown Doc Blocks Injection

Complex or lengthy descriptions can be maintained in standalone Markdown files and injected into YAML metadata via Jinja doc blocks.

• Doc Block Definition: Create a Markdown file (.md) within your dbt project and wrap your formatted text in a docs tag block. This allows you to write detailed, multi-paragraph markdown without cluttering YAML syntax.
• Jinja Injection: Reference the doc block inside schema.yml using the {{ doc('block_name') }} macro in place of a raw string. When generating docs, dbt replaces the Jinja call with the full Markdown body.
• Syntax Rule: Doc block names must be unique across the entire dbt project namespace. Any naming mismatch between the markdown definition and the schema.yml doc tag will trigger a compilation failure.

<!-- File: models/events.md -->
{% docs events_description %}
### Events Table
This table captures all frontend user interactions.
- **Grain:** One row per user event.
- **Latency:** Updated hourly.
{% enddocs %}

# File: models/schema.yml
version: 2
models:
  - name: fct_events
    description: "{{ doc('events_description') }}"

Key Takeaway: Use doc blocks in Markdown files to inject long-form, reusable documentation into schema.yml metadata.

Topics Covered in Documentation — Descriptions, dbt docs, and Why It Matters

1. Context of Data Documentation (0:00 - 0:54) — Reviews historical documentation pain points and the necessity of documenting software and data pipelines.
2. Unenriched Warehouse Models (0:54 - 1:34) — Demonstrates that physical database tables lack inherent contextual metadata without dbt documentation layers.
3. Creating schema.yml Metadata (1:34 - 3:07) — Constructs a schema.yml file declaring multi-line model descriptions and column-level data type metadata.
4. Generating and Serving Docs (3:07 - 3:36) — Executes dbt docs generate to compile catalog artifacts and dbt docs serve to launch the local web server.
5. Exploring the Docs UI (3:36 - 4:59) — Navigates model lineage graphs, column definitions, seed dependencies, and compiled SQL in the browser.
6. Injecting External Doc Blocks (4:59 - 5:51) — Creates standalone Markdown doc blocks and injects them into schema.yml using the doc Jinja function.

dbt Cheat Sheet

• schema.yml — Defines model and column metadata in YAML format

  version: 2
  models:
    - name: my_model
      description: "A model"
  

• dbt docs generate — Compiles project metadata and generates catalog.json

  dbt docs generate
  

• dbt docs serve — Hosts generated documentation on a local web server

  dbt docs serve --port 8080
  

• {% docs <name> %} ... {% enddocs %} — Defines a reusable Markdown documentation block

  {% docs model_info %}
  # Description
  {% enddocs %}
  

• {{ doc('<name>') }} — Injects an external doc block into YAML metadata

  description: "{{ doc('model_info') }}"
  

Comparison Table

Common Pitfalls

• Mistake: Serving documentation without regenerating artifacts after editing files. Avoid: Run dbt docs generate before dbt docs serve to update catalog.json.
• Mistake: Misspelling the doc block name within schema.yml. Avoid: Ensure the string inside doc function exactly matches the docs block identifier.
• Mistake: Placing schema.yml in a distant, non-model directory. Avoid: Co-locate schema.yml directly in the same folder as corresponding model SQL files.

FAQs

• Where does dbt docs generate output its compiled metadata? It outputs static documentation files, including catalog.json and manifest.json, directly into your project's target/ directory.
• Can I use Markdown formatting directly inside schema.yml strings? Yes, basic inline markdown works in schema.yml, but extensive markdown is cleaner when managed inside dedicated .md doc blocks.
• Why does dbt connect to the warehouse during dbt docs generate? It queries warehouse system tables to extract column data types, table statistics, and physical object sizes into catalog.json.
• Can a single Markdown file contain multiple doc blocks? Yes, you can define multiple distinct {% docs name %} ... {% enddocs %} blocks within a single .md file.