Getting Started with DeepSpeed on Azure
This tutorial will help you get started running DeepSpeed on Azure virtual machines. Looking forward, we will be integrating these techniques and additional ...
Tutorials
CIFAR-10 Tutorial
Train your first model with DeepSpeed!
Flops Profiler
Measure the parameters, latency, and floating point operations of your model
DCGAN Tutorial
Train your first GAN model with DeepSpeed!
Getting Started
First steps with DeepSpeed
Installation Details
The quickest way to get started with DeepSpeed is via pip, this will install the latest release of DeepSpeed which is not tied to specific PyTorch or CUDA ve...
Learning Rate Range Test
This tutorial shows how to use to perform Learning Rate range tests in PyTorch.
Megatron-LM GPT2
If you haven’t already, we advise you to first read through the Getting Started guide before stepping through this tutorial.
1-Cycle Schedule
This tutorial shows how to implement 1Cycle schedules for learning rate and momentum in PyTorch.
1-bit Adam: Up to 5x less communication volume and up to 3.4x faster training
Note: This tutorial is updated on 03/04/2021 to reflect the 1-bit Adam v2. Changes include: 1) NCCL-based implementation which provides better performance an...
Pipeline Parallelism
DeepSpeed v0.3 includes new support for pipeline parallelism! Pipeline parallelism improves both the memory and compute efficiency of deep learning training ...
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
In this tutorial, we are going to introduce the progressive layer dropping (PLD) in DeepSpeed and provide examples on how to use PLD. PLD allows to train Tra...
DeepSpeed Sparse Attention
In this tutorial we describe how to use DeepSpeed Sparse Attention (SA) and its building-block kernels. The easiest way to use SA is through DeepSpeed launch...
DeepSpeed Transformer Kernel
This tutorial shows how to enable the DeepSpeed transformer kernel and set its different configuration parameters.
ZeRO-Offload
We recommend that you read the tutorials on Getting Started and ZeRO before stepping through this tutorial. ZeRO-Offload is a ZeRO optimization that offloa...
Zero Redundancy Optimizer (ZeRO)
If you have not done so already, we advise that you read the DeepSpeed tutorials on Getting Started and Megatron-LM GPT-2 before stepping through this tutori...