Microsoft DeepSpeed achieves the fastest BERT training time

ZeRO-2 & DeepSpeed: Shattering Barriers of Deep Learning Speed & Scale Permalink

An Order-of-Magnitude Larger and Faster Training with ZeRO-2

The Fastest and Most Efficient BERT Training through Optimized Transformer Kernels

Turing-NLG: A 17-billion-parameter language model by Microsoft Permalink

DeepSpeed was used to train the world’s largest language model.

ZeRO & DeepSpeed: New system optimizations enable training models with over 100 billion parameters Permalink

Developed by Microsoft AI & Research.