awesome-open-source-data-engineering
A curated list of open source tools used in analytics platforms and data engineering ecosystem
- 633
- 21 mo ago
A curated list of open source tools used in analytics platforms and data engineering ecosystem
A data lakehouse on Databricks with the medallion architecture: PySpark, Spark SQL, Delta Lake and Unity Catalog. The project of the free Databricks bootcamp.
Hybrid recommender system + RAG chatbot for IT courses (FastAPI + Express/TS + PostgreSQL), with an independent Data Lakehouse module (Spark/Kafka/dbt/Trino) and full CI/CD
This repository is a place for the Data Warehousing course at the Information Systems & Analytics department, Santa Clara University.
Data quality gates for an Apache Iceberg lakehouse: Soda Core v4 contracts per medallion layer, write-audit-publish on Iceberg branches with Spark, gates on DuckDB or Trino, Airflow 3 orchestration, and reproducible benchmarks.
Apache Iceberg feature store and data lakehouse on Backblaze B2 — versioned feature tables, schema evolution, snapshot rollback, and DuckDB time-travel queries via PyIceberg, with B2 as the warehouse (no data warehouse to run)
Repair Silver from retained Bronze history and reconcile Gold without reloading the source. A runnable Python and Delta Lake demo with scoped repairs, change data feed, and rebuild validation.