prefect
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
- 24k
- Python
- Apache-2.0
- 5 h ago
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
Examples of using Prefect for background tasks in web applications
Automated pipeline that daily fetches, stores, and indexes arXiv research papers in MinIO.
Luigi, Airflow, Prefect Examples for PyCon RU 2019 Presentation
Minimal containerised examples of ETL scripts/flows/components in Python
Fast, drop-in Rust replacement for Python's croniter with PyO3 abi3 wheels. 17x–41x faster recurrence calculation, 100% upstream test parity (267/267 tests).
A Prefect pipeline that periodically scrapes one or more GitHub repositories, generates embeddings, and indexes them in ChromaDB.