Neural Tangent Kernel:

Convergence and Generalization in Neural Networks

​

Arthur Jacot, Franck Gabriel, Clement Hongler

NeuraIPS 2018

CSG@UEF Reading Club

​

Xuechen Liu

2022.01.28

​

(Equations are largely adopted from CMSC828W@Uni. Of Maryland College Park)

Background: Tangent Kernel

• Kernel - A close-form method to reduce the cost for methods that depends only on the inner product between pairs of input samples
• “Similarity measurements”
• Lots of regression algorithms can be ‘kernelized’, e.g. SVM, PCA
• For speech processing, almost every module in acoustic frontend (including DFT) can be ‘kernelized’
• Tangent kernel: “Kernel of gradient”

Neural Tangent Kernel

• Main focus:
• Formulate learning dynamics of fully-connected DNN, with infinite width limit
• Understand the generalization capability of DNN
• We equalize kernel methods as a 2-layer network
• NTK is then a simple induction via Taylor expansion of a network

Neural Tangent Kernel

• We consider fully-connected network as an overparameterized model
• Of course, gradient updates works well
• But if we observe the weights, we can see that as the width increases, the weights are pretty static. This is called lazy training
• So…..that doesn’t give us many intuitions if m goes large enough, not to say if it goes to infinity.

Neural Tangent Kernel

• We look for help by induction using the well-known Taylor expansion
• ……and ignore some terms.
• The approximated gradient term only on the initial value of weights and input variable, we can equate it to a kernel
• Yes, we have NTK now
• In the case of DNN, as the width goes to infinity, the kernel will converge to expectation values w.r.t weights

This only depends on w and x

Gradients of NTK

• Core application of NTK: describe the training dynamics (here, under gradient descent)
• Note that our model, DNN, is (almost always) an over-parameterized model
• NTK converges exponentially

Chain Rule

Derivatives

Gradients of NTK

• Core application of NTK: describe the training dynamics (here, under gradient descent)
• Note that our model, DNN, is (almost always) an over-parameterized model
• NTK converges exponentially
• with the rate of convergence decided by the eigenvalues of K(w0)

ODE

Solution of ODE

Performance of NTK

• NTK simplifies the infinite-width DNN to a linear model
• However, its performance still needs to be evaluated
• Original paper (2018) does not provide any large-scale experiment
• Some early results on MLP does not outperform SoTA DNN models with SGD training
• Of course, later variants show difference and early bad results were mainly on images

Hayou, Soufiane et al. “Mean-field Behaviour of Neural Tangent Kernel for Deep Neural Networks.” arXiv: Machine Learning (2019): n. Pag.

Shuxiao Chen et al. “Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity”, NeuraIPS 2020.

My Takeaway: Interpretability of Speech Systems?

• NTK has been mainly used as a tool for learning the dynamics for overparameterized models
• In practice, its prototype seems only works for very wide networks (close to be approx. ed)
• However, its close connection between DNN and kernel methods may provide potentials for learnable/parameterized systems
• Learnable MFCC/PCEN/PNCC can be good subjects, for example (yeah I have no shame, bite me)
• It also can be used for taking advantage of generalization capabilities e.g. attack
• Finally, Google has a good toolkit with Colabs:
• https://github.com/google/neural-tangents

​

Next Paper