Build a data warehouse from scratch, including full load, daily incremental load, design schema, SCD Type 1 and 2.
-
Updated
Feb 1, 2023 - Python
Build a data warehouse from scratch, including full load, daily incremental load, design schema, SCD Type 1 and 2.
A Data Warehousing project for retail sales using dimension modelling best practices with SCD type 2 on AWS Redshift. Utilizing AWS Lambda, Glue Workflows and Python Shell jobs to create and automate an ELT pipeline where batch data coming into S3 is loaded onto Redshift and necessary transformations are performed to meet requirements.
Production-grade ELT pipeline classifying VIX volatility regimes. Features incremental models, SCD-2 snapshots, recursive CTEs, and Slim CI/CD. Stack: dbt, DuckDB/MotherDuck, Python.
Enterprise Data Warehouse with Star Schema, SCD Type 2, ETL pipelines, and multi-currency analytics (SQL Server + Python)
Working with SCD Type (Change Data Capture) and need a Data Vault model to test Azure Data Factory v2? - This Code with Help!
In this project we'll create real time healthcare patient data pipeline as data source and use arious services and tools like Azure Eventhubs, Azure Databricks, Delta lake and synapse analytics. also, implement medallion architecture, schema evolution and create facts and dimension tables and connect the cleaned and transformed data to PowerBI.
A complete data integration solution that migrates data from multiple source systems into a modern data warehouse, with full documentation, quality validation, and analytical reporting.
End-to-End Retail Lakehouse on Microsoft Fabric | Medallion Architecture | PySpark | SCD Type-2 | DirectLake
Design and implement a full ELT data pipeline using Snowflake and S3, featuring star schema modelling, SCD Type 1 & 2 handling, and incremental load automation
A production-grade Modern Data Stack (MDS) implementation featuring automated ELT, SCD Type 2 history tracking, and CI/CD quality guardrails using Dagster, dbt Core, DuckDB, and Soda.
Production-style Slowly Changing Dimension (SCD Type 2) pipeline built with Snowflake, dbt, and AWS S3. Demonstrates secure S3 ingestion, layered bronze/silver/gold modeling, dbt snapshots for historical tracking, and analytics-ready views identifying active vs historical records.
End-to-end data pipeline: CSV → Bronze → Silver → Gold on AWS S3 with PySpark, Airflow, Great Expectations, SCD Type 2, and Streamlit dashboard
End-to-end Azure Databricks retail data engineering project using Medallion Architecture (Bronze, Silver, Gold). Implements Auto Loader, Unity Catalog, Delta Lake, SCD Type 1 & 2 dimensions, and Fact Orders for analytics-ready star schema modeling.
Enterprise-grade Microsoft Fabric Lakehouse project implementing Medallion architecture (Bronze, Silver, Gold) for financial transactio
End-to-end cloud retail intelligence platform integrating data engineering, predictive analytics, and executive BI dashboards.
Add a description, image, and links to the scd-type-2 topic page so that developers can more easily learn about it.
To associate your repository with the scd-type-2 topic, visit your repo's landing page and select "manage topics."