distributed_systems_for_ml
A 6-week masterclass in Distributed Deep Learning Infrastructure from scratch. Covers the $\alpha-\beta$ communication model, custom Ring All-Reduce primitives, Parameter Servers (Hogwild!), 1F1B Pipeline Parallelism, ZeRO/FSDP memory sharding, gradient compression with error feedback, and fault-tolerant elastic training simulators.
- 0
- Jupyter Notebook
- MIT
- 6 mo ago