Data Pipelines

• What is a data pipeline?

A data pipeline is a system that ingests raw data from multiple data sources, transforms it and then loads it into a data store such as a  data lake or  data warehouse for analysis and operational use.

• Pipelines handle ingestion, transformation, validation, and loading

• Airflow
  • DAG-based workflow orchestrator; schedules and monitors pipeline steps
• Medallion Architecture
  • Layered data quality pattern: Bronze → Silver → Gold
• ETL vs ELT
  • Comparison of the two dominant pipeline patterns and when to use each