What is dbt

dbt (Data Build Tool) - dbt turns modular SQL into tested, documented data models inside our data warehouse. It brings software-engineering best practices (version control, tests, documentation, lineage) to analytics workflows and enables incremental builds that speed refreshes and improve reliability.

Why dbt

• Faster, cheaper refreshes — incremental models and smart materializations process only new/changed rows instead of re-processing entire tables, reducing runtime and compute cost so our marts refresh faster.
• Higher data reliability — schema tests (not_null, unique, relationships) catch issues early so BI shows fewer data incidents.
• Clear lineage & docs — DAGs make it easy to trace dashboard numbers back to sources, speeding debugging and onboarding.
• Maintainable, modular SQL — composable models (ref()), configs, and macros let us structure transformations like software projects.

Key concepts

• Project / models — each SQL file is a model that becomes a table/view in the warehouse; models are composable via ref().
• Materializations — how a model is stored: view, table, incremental, ephemeral, materialized_view; choose based on performance/cost trade-offs.
• Incremental models — only process new/changed rows on subsequent runs — ideal for large historic tables and frequent refreshes.
• Sources & tests — declare raw tables as sources and add schema tests to enforce quality before data reaches BI.

Architecture Design

├── /opt/airflow/repo/dags/dashboard/
│   └── dag_dbt_dashboard_daily.py → *main dag to run all dbt models*
│
├── /opt/airflow/repo/dags/dbt_project/ → *dbt related files*
│   └── models/
│				├── intermediate/*
│				├── marts/*
│				└── staging/*
│   ├── profiles.yml
│   └── dbt_project.yml

Example dbt file

models/staging/marts/dm_marketing_trial_expiration.sql

{{
config(
    materialized='table',
    schema='marts'
)
}}
SELECT
    company_id,
    company_name,
    contact_phone,
    contact_email,
    trial_start_date,
    expiration_date,
    days_until_expiration AS days_left,
    expiration_segment,
    refresh_date
FROM {{ref('int_marketing_trial_expiration')}}
WHERE days_until_expiration BETWEEN 0 and 5
ORDER BY days_left, company_name

How dbt sped up our dashboard

• Create small staging models (stg_*) to clean/standardize raw sources.
• Build marts (facts & dimensions) materialized view or table