DeepSpeed Mixture-of-Quantization (MoQ)
DeepSpeed introduces new support for model compression using quantization, called Mixture-of-Quantization (MoQ). MoQ is designed on top of QAT (Quantization...
Tutorials
Installation Details
The quickest way to get started with DeepSpeed is via pip, this will install the latest release of DeepSpeed which is not tied to specific PyTorch or CUDA ve...
Autotuning
Automatically discover the optimal DeepSpeed configuration that delivers good training speed
Getting Started with DeepSpeed on Azure
This tutorial will help you get started running DeepSpeed on Azure virtual machines. Looking forward, we will be integrating these techniques and additional ...
CIFAR-10 Tutorial
Train your first model with DeepSpeed!
Curriculum Learning: A Regularization Method for Efficient and Stable Billion-Scale GPT Model Pre-Training
Note: This tutorial was updated on 10/29/2021. Changes include: 1) A more detailed tuning strategy. 2) Pipeline parallelism support. 3) Token-based learning ...
Flops Profiler
Measure the parameters, latency, and floating-point operations of your model
DCGAN Tutorial
Train your first GAN model with DeepSpeed!
Getting Started
First steps with DeepSpeed
Getting Started with DeepSpeed for Inferencing Transformer based Models
DeepSpeed-Inference introduces several features to efficiently serve transformer-based PyTorch models. It supports model parallelism (MP) to fit large models...
Learning Rate Range Test
This tutorial shows how to use to perform Learning Rate range tests in PyTorch.
Megatron-LM GPT2
If you haven’t already, we advise you to first read through the Getting Started guide before stepping through this tutorial.
Getting Started with DeepSpeed-MoE for Inferencing Large-Scale MoE Models
DeepSpeed-MoE Inference introduces several important features on top of the inference optimization for dense models (DeepSpeed-Inference blog post). It embra...
Mixture of Experts for NLG models
In this tutorial, we introduce how to apply DeepSpeed Mixture of Experts (MoE) to NLG models, which reduces the training cost by 5 times and reduce the MoE m...
Mixture of Experts
DeepSpeed v0.5 introduces new support for training Mixture of Experts (MoE) models. MoE models are an emerging class of sparsely activated models that have s...
1-Cycle Schedule
This tutorial shows how to implement 1Cycle schedules for learning rate and momentum in PyTorch.
1-bit Adam: Up to 5x less communication volume and up to 3.4x faster training
Note: On 03/07/2022 we released 0/1 Adam, which is a new communication-efficient Adam optimizer partially following the 1-bit Adam’s design. Compared to the ...
1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB’s Convergence Speed
Watch out! 1) The NCCL-based implementation requires PyTorch >= 1.8 (and NCCL >= 2.8.3 when you have 64 or more GPUs). See details below. 2) Although 1...
Pipeline Parallelism
DeepSpeed v0.3 includes new support for pipeline parallelism! Pipeline parallelism improves both the memory and compute efficiency of deep learning training ...
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
In this tutorial, we are going to introduce the progressive layer dropping (PLD) in DeepSpeed and provide examples on how to use PLD. PLD allows to train Tra...
Using PyTorch Profiler with DeepSpeed for performance debugging
This tutorial describes how to use PyTorch Profiler with DeepSpeed.
DeepSpeed Sparse Attention
In this tutorial we describe how to use DeepSpeed Sparse Attention (SA) and its building-block kernels. The easiest way to use SA is through DeepSpeed launch...
DeepSpeed Transformer Kernel
This tutorial shows how to enable the DeepSpeed transformer kernel and set its different configuration parameters.
ZeRO-Offload
ZeRO-3 Offload consists of a subset of features in our newly released ZeRO-Infinity. Read our ZeRO-Infinity blog to learn more!
Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam
Watch out! 1) The NCCL-based implementation requires PyTorch >= 1.8 (and NCCL >= 2.8.3 when you have 64 or more GPUs). See details below. 2) Although 0...