AI & Data
Retail Data Migration Platform
Incremental migration of retail transactions from legacy SQL Server into a layered Snowflake warehouse, orchestrated with Airflow and modelled with dbt.
- Client
- Engineering case study
- Stack
- Python, Airflow, dbt, Snowflake
The problem
Legacy on-prem SQL Server data was locked away from modern analytics, and full reloads were slow and error-prone.
The approach
- 01Modular Python ETL with watermark-based incremental loading and ROW_NUMBER deduplication.
- 02Snowflake RAW → STAGING → MARTS architecture with dbt transformations.
- 03Airflow DAG orchestration, automated data-quality tests, audit logging and failure alerting.
The outcome
A repeatable, observable pipeline pattern that moves legacy data to the cloud safely, and the data foundation AI work depends on.
Stack
- Python
- Airflow
- dbt
- Snowflake
- SQL Server
- Docker
- PostgreSQL

