Tutorials

1-Cycle Schedule

This tutorial shows how to implement 1Cycle schedules for learning rate and momentum in PyTorch.

Getting Started with DeepSpeed on Azure

This tutorial will help you get started running DeepSpeed on Azure virtual machines. Looking forward, we will be integrating these techniques and additional ...

BingBertSQuAD Fine-tuning

BERT Pre-training

CIFAR-10 Tutorial

Train your first model with DeepSpeed!

Getting Started

First steps with DeepSpeed

Learning Rate Range Test

This tutorial shows how to use to perform Learning Rate range tests in PyTorch.

Megatron-LM GPT2

If you haven’t already, we advise you to first read through the Getting Started guide before stepping through this tutorial.

DeepSpeed Transformer Kernel

This tutorial shows how to enable the DeepSpeed transformer kernel and set its different configuration parameters.