OpenEmbedding
OpenEmbedding is an open source framework for Tensorflow distributed training acceleration.
- 33
- C++
- Apache-2.0
- 4 y ago
OpenEmbedding is an open source framework for Tensorflow distributed training acceleration.
A 6-week masterclass in Distributed Deep Learning Infrastructure from scratch. Covers the $\alpha-\beta$ communication model, custom Ring All-Reduce primitives, Parameter Servers (Hogwild!), 1F1B Pipeline Parallelism, ZeRO/FSDP memory sharding, gradient compression with error feedback, and fault-tolerant elastic training simulators.