1 of 10

ADMM in PyTorch

Alternating Direction Method of Multipliers

​

​

​

​

Nishant Borude

Bhushan Sonawane

Sri Haindavi

Mihir Chakradeo

2 of 10

Motivation

  • Decentralized optimizations
    • Superfast when run in parallel
  • Minimal assumptions on objective function and constraints
    • Loss function doesn’t need to be differentiable

​

​

3 of 10

State of the art

  • Available Implementation
    • CVX group from Stanford
      • Cumbersome to setup and use
      • Not parallel
  • No ready to use implementation in ML/DL frameworks
    • Tensorflow, PyTorch

4 of 10

ADMM Algorithm

Augmented Lagrangian :

​

​

Update Rules :

​

​

5 of 10

Contribution

  • Implemented in PyTorch framework
    • Lasso
      • Serial
      • Parallel
    • Ridge Regression
      • Serial
      • Parallel

​

https://github.com/bhushan23/pytorch/tree/admm/torch/optim

​

6 of 10

  • ADMM for Lasso

Lasso Problem :

​

Distributed ADMM Form:

​

​

Update Rules:

7 of 10

Results

  • Compared the results for Lasso with sklearn implementation on Diabetes Dataset
  • MSE: 2553.13 2769.15

8 of 10

2. ADMM for Ridge Regression

Ridge Regression :

​

ADMM Form :

​

Update Rules:

​

9 of 10

Results

  • Compared the results for Lasso with sklearn implementation on Diabetes Dataset
  • MSE: 2559.32 2799.13

10 of 10

References