An Overview of Distributed Training

2022.07.15

�Sponsored by 系统设计开荒小分队/东哥IT笔记

Why do we need distributed training?

• Deep learning is training data heavy
• Complex model needs to store intermediate results

​

Type of Parallelism

• Data Parallelism
• Model Parallelism
• Graph Parallelism
• Task Parallelism
• Hybrid Parallelism/ Mixed Parallelism

Data Parallelism

Data Parallelism

• Synchronized training
• Asynchronous training

Synchronized training

1. Forward pass begins at the same
2. After computing gradients, start communicating, aggregating (all-reduce)
3. After all the gradients are combined, the copy of these updated gradients is sent to all the workers. 
4. Now after getting the updated gradients using the all-reduce algorithm, each worker continues with the backward pass and updates the local copy of the weights normally.

All-Reduce Algorithm

• Instead of having a single machine to perform the aggregation task, we can distribute the aggregation task on all machines.

​

• All machines share the load of storing and maintaining global parameters

Ring All-Reduce

Tree All-Reduce

Asynchronized training

In synchronized training, worker has to wait for other workers in order to move ahead, thus whole process is only as fast as the slowest worker in the cluster.

• Asynchronized training:
• Workers train independently
• Tolerance towards machine computation power difference

Parameter Server

• Workers that act as parameter servers�- maintain the globally shared parameters

​

• Workers that train the model�- Perform training tasks

​

​

​

Parameter Server

Steps:

1. Replicate the model in all of our workers and each worker uses a subset of data for training.
2. Each training worker fetches the parameters from the parameter servers.
3. Each training worker performs a training loop and sends the gradients back to all the parameter servers which then update the model parameters.

​

Parameter Server

Parameter Server

• Disadvantages:
• Using stale version of the model during training
• Parameter server as a bottleneck �- Single point of failure�- Bottleneck for communication�Solution: introducing multiple parallel servers

​

Parameter Server

• Disadvantages:
• Using stale version of the model during training
• Parameter server as a bottleneck �- Single point of failure�- Bottleneck for communication�Solution: introducing multiple parallel servers

​

Parameter Server

• Advantages:

1. Efficient communication

2. Flexible consistency models

3. Elastic Scalability

4. Fault Tolerance and Durability

5. Ease of Use

​

Model Parallelism

• When model is too large to fit on a single worker node�
• Depends on specific model architecture

Model Parallelism

Model Parallelism

Centralized and Decentralized Training

*Distributed Training of Deep Learning Models: A Taxonomic Perspective

Benefits of Distributed Training

• Efficiency
• Scalability 
• Cost effectiveness
• Fault tolerance and reliability

​

​

Distributed Training Frameworks

• Pytorch
• Tensorflow
• Amazon Sagemaker
• Keras
• Horovod

​

Distributed Training Frameworks

• Pytorch
• Provided torch.distributed package API
• Tensorflow
• Built-in support for distributed training, provided  tf.distribute.Strategy API
• Amazon Sagemaker
• Both data parallelism and model parallelism are supported
• Keras
• Simple and easy to use
• Horovod
• Distributed deep learning training framework for TensorFlow, Keras, and PyTorch.

​

References

• https://arxiv.org/pdf/2007.03970.pdf
• https://neptune.ai/blog/distributed-training
• http://www.juyang.co/distributed-model-training-ii-parameter-server-and-allreduce/
• https://www.run.ai/guides/gpu-deep-learning/distributed-training
• https://medium.com/intel-student-ambassadors/distributed-training-of-deep-learning-models-with-pytorch-1123fa538848
• https://frankdenneman.nl/2020/02/19/multi-gpu-and-distributed-deep-learning/
• https://pytorch.org/tutorials/beginner/dist_overview.html
• https://aws.amazon.com/sagemaker/distributed-training/
• https://www.cnblogs.com/yihaha/p/7265280.html
• https://www.modb.pro/db/32182

​

​