luigi
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
- 19k
- Python
- Apache-2.0
- 5 wk ago
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Saisoku is a Python module that helps you build complex pipelines of batch file/directory transfer/sync jobs.
Материалы для курса Введение в Data Engineering: дата пайплайны
Luigi, Airflow, Prefect Examples for PyCon RU 2019 Presentation