The challenge

Data teams operate blind. They know when a pipeline job fails, but they do not know when data arrives late, volumes drift silently, or a schema change upstream breaks a downstream model three days later. By the time someone notices, the damage has already reached business reports.

How we solved it

• - Collect pipeline metadata from Airflow DAG runs, dbt model executions, and database statistics into a centralized observability store
• - Detect anomalies in data freshness, row volume, null rates, and schema drift using statistical baselines
• - Track end-to-end lineage from source ingestion through transformation to final dashboard consumption
• - Alert on SLA breaches with contextual information that helps engineers diagnose root cause in minutes instead of hours

Execution story

Every pipeline stage emits metadata into a PostgreSQL observability store. A Python service runs anomaly detection on freshness and volume baselines. dbt tests add quality signals at the transformation layer. Streamlit provides a unified dashboard where engineers can see pipeline health, trace lineage, and drill into anomalies without switching between Airflow UI, dbt docs, and database queries.