1 of 5

Model Parallelism for Inference

2 of 5

Aim

Creating a parallelizable model architecture that trains normally but can be split into different sub-models with minimal loss of accuracy and minimal network overload for distributed systems inference. Here in this research we touch upon structure level optimization of CNN using Decoupled CNN architecture (DeCNN), introduced in a 2021 IEEE paper.

3 of 5

Details

We implemented DeCNN using Resnet-34 and compared its performance and inference speed with non-decoupled architecture. We found that there is tradeoff between performance and inference speed for the DeCNN architecture, which was balanced with some modifications based on the paper. Also, we just implemented DeCNN (Scheme 1) and ignored additional optimizations mentioned in the paper, which would result in much better results. Overall, the project showed promise, but is best suited to be explored further only when the specific requirement arises, due to the complexity (OpenMPI and C++) of the different optimisation schemes (Scheme 2 & 3).

4 of 5

Result Summary

Size of Test set for Inference: 10,000

Inference done in CPU with

No of workers: 2

Batch size: 256

​

Base CNN model: Renet34

DeCNN Model: Resnet34-

shuffle-1.5x

DeCNN Model: Resnet34-

shuffle-1x

Number of epochs

90

90

90

Dataset

Tiny imagenet

Tiny imagenet

Tiny imagenet

Train top1 accuracy

71.71

70.63

63.94

Train top5 accuracy

88.47

87.70

84.11

Val top1 accuracy

58.93

58.61

56.28

Val top5 accuracy

80.99

81.07

80.20

The number of kernels in Decoupled CNN are increased by 1.5 times the base CNN to tackle with accuracy losses

Model

Time (secs)

Resnet34

964

Resnet34_x1(decoupled CNN with same number of kernels as base Resnet34)

466

Resnet34_x15(decoupled CNN with increased number of kernels by 1.5 times the base CNN)

838

5 of 5

Future Scope

Moving forward, Scheme 2 and 3 which were left in the study can be implemented. Also, can explore the possibility of extending this method for other types of NN architectures. A research project can be carrying both research on technical aspects (as this field has not been explored much) and business value (specifically for IoT use cases).