Modern Data Stack

• The modern data stack is a layered architecture that separates orchestration, transformation, and compute into distinct tools, each doing one thing well.

1
2
3

Airflow            ← when to run  (orchestration)
  └─ dbt           ← what SQL to run  (transformation middleware)
       └─ Snowflake / BigQuery / Spark   ← actual computation

Airflow runs on a lightweight scheduler pod. dbt runs in a Python process. All heavy computation stays inside the warehouse or Spark cluster.

dbt uses standard SQL — there is no proprietary dialect. Jinja templating is a preprocessing layer compiled away before the warehouse sees any query.

Key Jinja functions:

• {{ ref('model_name') }} — resolves to the fully-qualified table/view name; builds the dependency DAG
• {{ source('schema', 'table') }} — references raw source tables declared in schema.yml
• {% if is_incremental() %} — conditional logic for incremental materialization

1
2
3
4
5
6
7

-- what you write
SELECT * FROM {{ ref('stg_orders') }}
WHERE {% if is_incremental() %} created_at > (SELECT MAX(created_at) FROM {{ this }}) {% endif %}

-- what the warehouse receives (compiled)
SELECT * FROM analytics.stg_orders
WHERE created_at > (SELECT MAX(created_at) FROM analytics.fct_orders)

Adapters translate compiled SQL to the target dialect:

• dbt-snowflake, dbt-bigquery, dbt-redshift
• dbt-databricks / dbt-spark — same dbt model code runs on Spark SQL
• dbt-duckdb — popular for local development

The result: one set of dbt models runs against different engines with minimal config change.

Both Airflow and dbt have a concept of a DAG, but they operate at different scopes:

With astronomer-cosmos: dbt’s model DAG is unrolled into individual Airflow tasks (one task per dbt model), giving fine-grained retry and observability at the model level inside Airflow.

Before Airflow — pipelines were cron jobs chained with &&:

• No retry on failure
• No visibility into what ran or failed
• No dependency tracking between jobs

Before dbt — transformations were 300 SQL scripts in a folder:

• No lineage (which table feeds which)
• No tests or data quality checks
• No modularity or reuse

Spark solves a different problem: data that doesn’t fit on one machine. It’s a distributed compute engine, not a warehouse. Airflow can orchestrate Spark jobs via SparkSubmitOperator or Kubernetes pod operators. dbt can target Spark via dbt-spark / dbt-databricks.