Data Geometry and DL - Lecture 10

Equivariant NNs, Geometric Machine Learning

Source (lecture notes on GML): https://arxiv.org/abs/2104.13478

A youtube minicourse: Link

Equivarince in Machine Learning: motivations

• Symmetries are part of nature, but coordinates do not respect them
• Respecting symmetries: why?
• Diminish number of degrees of freedom – better accuracy
• lower dimension of loss function minima – improved convergence
• Examples: chemistry, images/3D, physics

Symmetry actions on NN weights

• We desire that a symmetry group action is propagated throughout the network in a “suitable way”

​

• Basic classification of what “suitable” can mean:

​

​

​

​

“Steerable” = equivariant (1st paper, SCNNs)

• Natural mathematical framework – equivariant function spaces:
• Representation Theory
• Nonabelian Fourier Transform
• Homogeneous spaces + Harmonic Analysis

​

Example: spherical harmonics

Abelian and Nonabelian Harmonic Analysis

• Abelian groups – examples: Translations in R^d, Z^d, T^d.
• Convolution (over an abelian group) is equivariant under translations
• Any linear equivariant functional is a convolution
• Fourier transform: Convolution → Product
• Natural basis of functions: Fourier modes

​

• Nonabelian compact groups: Extension of Fourier theory, Relation to L^2

​

• Homogeneous spaces G/H: invariant functions expressed with Fourier too

​

• Clebsch-Gordan allows to change reps

​

• Computations are ad-hoc for each group, but a large theory exists.

​

• Example: S^2 = SO(3)/SO(2) → spherical harmonics basis for L^2

​

Nonabelian harmonic analysis – one-slide refresher (cpt. groups)

Plancherel, Peter-Weyl

Convolution and Fourier

Homogeneous space case – example

Geometric Deep Learning: symmetries and manifolds

• Example: Spherical CNNs (Group = SO(3) in this case) — paper

​

• We consider convolution on the sphere (invariant under SO(3))

​

• Convolution via nonabelian group theory

​

• Convolution → Product in SO(3)-Fourier coordinates

​

​

​

​

• Strategy for SO(3)-CNNs →

​

• “Clebsch-Gordan nets” extend it

and work with general tensor products

of representations (paper)

• Nontransitive cpt group actions (paper)
• Groups on homog. sp. survey (paper)

Geometric Deep Learning: symmetries and manifolds

Equivariance can be implemented in 3 general types of ways (survey on GNN case)

• Use Fourier/Clebsch-Gordan coefficients – a.k.a. the “Irrep method”
• Equivariance is automatic if we use functions of the distances/angles for SO, SE groups, or we study other expressions of invariant functions (paper) – Scalarization
• Approximate convolutions directly via sampling or other methods–regular rep. method

paper

E(n) GNN paper

paper

ENN use examples

paper

(extension to a product of SE(3) in progress)

paper

ENN use examples

• Example: SE(3)-transformers (paper)
• Idea: transformer for equivariant GNN

Switching equivariance on and off – possibilities

• Sparse convolutions: add anchor points at random positions; do the same on Lie groups (paper)

​

​

​

​

​

​

​

• Approximate equivariance (paper – theoretical study missing)

Switching equivariance on and off – possibilities

• Partial equivariances (paper)

​

​

​

​

​

​

​

​

• Several works try to “learn symmetries”
• Learning physics gauge theory, using ENNs

References (S. Trivedi)

• Theoretical aspects (basic) of basic steerable networks https://arxiv.org/pdf/2004.05154.pdf
• More theoretical aspects (slightly more advanced) https://arxiv.org/abs/2105.13926 and https://arxiv.org/abs/2105.05400
• A very comprehensive survey on gauge equivariant models https://arxiv.org/pdf/2106.06020.pdf
• A short survey on their use in molecular representations https://arxiv.org/pdf/2107.12375.pdf

​

Course

webpage

Register here

to mailing list