Using Multi-Resolution Data to Accelerate Neural Network Training Time
With ECP ExaLearn Project
K. Wang, S. Lee, J. Balewski, A. Sim, P. Nugent, A. Agrawal, A. Choudhary, K. Wu, and W. Liao. “Using Multi-Resolution Data to Accelerate Neural Network Training in Scientific Applications,” in the 22nd International Symposium on Cluster, Cloud and Internet Computing (CCGrid), 2022.
Scientific Achievement
Training deep networks could be extremely computationally intensive, especially when the training sample size is large. This project presents a new multi-resolution training strategy (MRT) that reduced the end-to-end training time up to 30% and 23% for CosmoFlow and DeepCAM on two DOE leadership supercomputers, Summit at OLCF and Cori at NERSC.
Significance and Impact
MRT reduces the training time for large-scale scientific neural network models on HPC systems, while maintaining the same model accuracy. The idea of using multiple resolution training data is applicable to all training methods using data sets with spatial property.
Technical Approach
MRT trains the CNN model using a reduced-resolution data at the early stage and automatically switches to the original-resolution data when loss flattens.
Training with lower-resolution data takes much less time.
MRT transfers knowledge from the coarse version problem to accelerate the training process on the original problem.
Validation loss curves of CosmoFlow of baseline and MRT. The training switches from low to original resolution at epoch 55. The original training time is 1073.6s and MRT reduces it to 752.8s, using 32 GPUs on Cori.
Validation intersection over union (IoU) curves of DeepCAM of baseline and MRT. The training switches from low to original resolution at iteration 1890. The original training time is 4924.8s and MRT reduces it to 3787.6s, using 64 GPUs on Cori.
End-to-end training time (till converged) between baseline and MRT on Cori and Summit