2 of 105

Tess Smidt

2018 Alvarez Fellow

in Computing Sciences

Unintended features of Euclidean symmetry equivariant neural networks

Workshop on Equivariance

and Data Augmentation

2020.09.04

All I wanted was 3D rotation equivariance and I got...

....geometric tensors, space groups, point groups, selection rules, normal modes, degeneracy, 2nd order phase transitions, and a much better understanding of physics.

3 of 105

Symmetry emerges when different ways of representing something “mean” the same thing.

Symmetry of representation vs. objects

4 of 105

Euclidean symmetry, E(3):

Symmetry of 3D space

The freedom to choose your coordinate system

Symmetry emerges when different ways of representing something “mean” the same thing.

Symmetry of representation vs. objects

5 of 105

Euclidean symmetry, E(3):

Symmetry of 3D space

The freedom to choose your coordinate system

3D Translation

3D Rotation

3D Inversion

Mirrors

We transform between coordinate systems with...

Symmetry emerges when different ways of representing something “mean” the same thing.

Symmetry of representation vs. objects

6 of 105

Euclidean symmetry, E(3):

Symmetry of 3D space

The freedom to choose your coordinate system

Symmetry of geometric objects

Looks the same under specific rotations, translations, and inversion (includes mirrors).

Symmetry emerges when different ways of representing something “mean” the same thing.

Symmetry of representation vs. objects

7 of 105

Euclidean symmetry, E(3):

Symmetry of 3D space

The freedom to choose your coordinate system

Symmetry of geometric objects

Looks the same under specific rotations, translations, and inversion (includes mirrors).

Symmetry emerges when different ways of representing something “mean” the same thing.

Symmetry of representation vs. objects

8 of 105

Invariance vs. Equivariance in 3D space

Does NOT change ⇨ Invariant

Changes deterministically ⇨ Equivariant

Properties of a vector under E(3)

Translation

Rotation

Inversion

3D vector

9 of 105

H -0.21463 0.97837 0.33136

C -0.38325 0.66317 -0.70334

C -1.57552 0.03829 -1.05450

H -2.34514 -0.13834 -0.29630

C -1.78983 -0.36233 -2.36935

H -2.72799 -0.85413 -2.64566

C -0.81200 -0.13809 -3.33310

H -0.98066 -0.45335 -4.36774

C 0.38026 0.48673 -2.98192

H 1.14976 0.66307 -3.74025

C 0.59460 0.88737 -1.66708

H 1.53276 1.37906 -1.39070

Coordinates are most general, but sensitive to translations, rotations, and inversion.

Three ways to make models “symmetry-aware” for 3D data

e.g. How to make a model that “understands” the symmetry of atomic structures?

10 of 105

Approach 1:

Data Augmentation

Throw data at the problem and see what you get!

Approach 3:

Invariant models

Equivariant models

If there’s no model that naturally handles coordinates,

we will make one.

Approach 2:

Invariant Inputs

Convert your data to invariant representations so the neural network can’t possibly mess it up.

H -0.21463 0.97837 0.33136

C -0.38325 0.66317 -0.70334

C -1.57552 0.03829 -1.05450

H -2.34514 -0.13834 -0.29630

C -1.78983 -0.36233 -2.36935

H -2.72799 -0.85413 -2.64566

C -0.81200 -0.13809 -3.33310

H -0.98066 -0.45335 -4.36774

C 0.38026 0.48673 -2.98192

H 1.14976 0.66307 -3.74025

C 0.59460 0.88737 -1.66708

H 1.53276 1.37906 -1.39070

Coordinates are most general, but sensitive to translations, rotations, and inversion.

Three ways to make models “symmetry-aware” for 3D data

e.g. How to make a model that “understands” the symmetry of atomic structures?

11 of 105

Approach 1:

Data Augmentation

Throw data at the problem and see what you get!

Approach 3:

Invariant models

Equivariant models

If there’s no model that naturally handles coordinates,

we will make one.

Approach 2:

Invariant Inputs

Convert your data to invariant representations so the neural network can’t possibly mess it up.

👌

😭

😍

H -0.21463 0.97837 0.33136

C -0.38325 0.66317 -0.70334

C -1.57552 0.03829 -1.05450

H -2.34514 -0.13834 -0.29630

C -1.78983 -0.36233 -2.36935

H -2.72799 -0.85413 -2.64566

C -0.81200 -0.13809 -3.33310

H -0.98066 -0.45335 -4.36774

C 0.38026 0.48673 -2.98192

H 1.14976 0.66307 -3.74025

C 0.59460 0.88737 -1.66708

H 1.53276 1.37906 -1.39070

Coordinates are most general, but sensitive to translations, rotations, and inversion.

Three ways to make models “symmetry-aware” for 3D data

e.g. How to make a model that “understands” the symmetry of atomic structures?

12 of 105

For 3D data, data augmentation is expensive, ~500 fold augmentation

and you still don’t get the guarantee of equivariance (it’s only emulated).

training without rotational symmetry

training with symmetry

13 of 105

How Euclidean Neural Networks achieve equivariance to Euclidean symmetry

(high level)

Euclidean Neural Networks encompass

Tensor Field Networks (arXiv:1802.08219)

Clebsch-Gordon Nets (arXiv:1806.09231)

3D Steerable CNNs (arXiv:1807.02547)

Cormorant (arXiv:1906.04015)

SE(3)-Transformers (arXiv:2006.10503)

e3nn (github.com/e3nn/e3nn)

(Technically, e3nn is the only one that implements inversion)

Some relevant folks… Mario Geiger, Ben Miller, Risi Kondor, Taco Cohen, Maurice Weiler, Daniel E. Worrall, Fabian B. Fuchs, Max Welling, Nathaniel Thomas, Shubhendu Trivedi,...

14 of 105

Convolutional filters based on learned radial functions and spherical harmonics.

Euclidean Neural Networks are similar to convolutional neural networks,

EXCEPT with special filters and (geometric) tensor algebra!

15 of 105

Let g be a 3d rotation matrix.

a_-1

+a₀

+a₁

D is the Wigner-D matrix.

It has shape

and is a function of g.

Spherical harmonics of a given L transform together under rotation.

b_-1

+b₀

+b₁

16 of 105

Feature 1:

Everything in the network is a geometric tensor!

Scalar multiplication gets replaced with the more general geometric tensor product.

Contract two indices to one with Clebsch-Gordan Coefficients.

Dot product

Cross

product

Outer product

Example: How do you “multiply” two vectors?

Scalar,

Rank-0

Vector, Rank-1

Matrix,

Rank-2

Euclidean Neural Networks are similar to convolutional neural networks,

EXCEPT with special filters and (geometric) tensor algebra!

17 of 105

The input to our network is geometry and features on that geometry.

18 of 105

The input to our network is geometry and features on that geometry.

We categorize our features by how they transform under rotation.

Features have “angular frequency” L

where L is a positive integer.

Scalars

Vectors

3x3 Matrices

Frequency

Doesn’t change with rotation

Changes with same frequency as rotation

19 of 105

The input to our network is geometry and features on that geometry.

We categorize our features by how they transform under rotation.

Features have “angular frequency” L

where L is a positive integer.

Scalars

Vectors

3x3 Matrices

Frequency

Doesn’t change with rotation

Changes with same frequency as rotation

20 of 105

Given a molecule and a rotated copy,

predicted forces are the same up to rotation.

(Predicted forces are equivariant to rotation.)

Additionally, networks generalize to molecules with similar motifs.

21 of 105

Primitive unit cells, conventional unit cells, and supercells of the same crystal produce the same output (assuming periodic boundary conditions).

22 of 105

1s 2s 2s 2p 2p 3d

1s 2s 2p

Networks can predict molecular Hamiltonians in any orientation

from seeing a single example.

23 of 105

Feature 1: All data (input, intermediates, output) in E(3)NNs are geometric tensors.

Geometric tensors are the “data types” of 3D space and have many forms.

Vector

Pseudo-

vector

Double-

Headed

Ray

Spiral

Rotation 丄

Reflection ‖

Inversion

3x3 matrix expressed as linear combination of spherical harmonics

24 of 105

Feature 2: The outputs have equal or higher symmetry than the inputs.

Curie’s principle (1894):

input

random model 1

random model 2

random model 3

Tetrahedron

Octahedron

“When effects show certain asymmetry, this asymmetry must be found

in the causes that gave rise to them.”

25 of 105

“When effects show certain asymmetry, this asymmetry must be found

in the causes that gave rise to them.”

Feature 2: The outputs have equal or higher symmetry than the inputs.

Curie’s principle (1894):

input

random model 1

random model 2

random model 3

Tetrahedron

Octahedron

Implement group equivariance and get all subgroups for FREE!

e.g. space groups, point groups

26 of 105

Feature 2: The outputs have equal or higher symmetry than the inputs.

Symmetry compiler -- can’t fit a model that does symmetrically make sense

T. E. Smidt, M. Geiger, B. K. Miller. https://arxiv.org/abs/2007.02005 (2020)

27 of 105

Symmetrically degenerate rectangle!

✓

✗

D_2h￫ D_4h

D_4h￫ D_2h

Feature 2: The outputs have equal or higher symmetry than the inputs.

Symmetry compiler -- can’t fit a model that does symmetrically make sense

T. E. Smidt, M. Geiger, B. K. Miller. https://arxiv.org/abs/2007.02005 (2020)

28 of 105

✓

✗

D_2h￫ D_4h

D_4h￫ D_2h

Feature 2: The outputs have equal or higher symmetry than the inputs.

Symmetry compiler -- can’t fit a model that does symmetrically make sense

T. E. Smidt, M. Geiger, B. K. Miller. https://arxiv.org/abs/2007.02005 (2020)

Network predicts degenerate outcomes!

29 of 105

✓

✗

D_2h￫ D_4h

D_4h￫ D_2h

Feature 2: The outputs have equal or higher symmetry than the inputs.

Symmetry compiler -- can’t fit a model that does symmetrically make sense

T. E. Smidt, M. Geiger, B. K. Miller. https://arxiv.org/abs/2007.02005 (2020)

Network predicts degenerate outcomes!

30 of 105

✓

D_2h￫ D_4h

✓

✗

D_2h￫ D_4h

→ Learns anisotropic inputs. → Model can fit.

Input

Output

L = 0 + 2 + 4

L = 0

T. E. Smidt, M. Geiger, B. K. Miller. https://arxiv.org/abs/2007.02005 (2020)

Use gradients to “find” what’s missing.

D_4h￫ D_2h

Feature 3: We can find data that is implied by symmetry.

Using gradients of loss wrt input we can find symmetry breaking “order parameters”

Even L > 0 break degeneracy between x and y directions.

31 of 105

developers and collaborators of e3nn

(and atomic architects)

Mario Geiger

Ben Miller

Tess Smidt

Koctiantyn Lapchevskyi

Boris

Kozinsky

Simon

Batzner

Josh Rackers

Thomas

Hardin

Tahnee

Gehm

32 of 105

tensor field networks

Google Accelerated Science Team

Stanford

Patrick Riley

Steve Kearnes

Nate Thomas

Lusann Yang

Kai Kohlhoff

33 of 105

Feel free to reach out if you have any questions!

Tess Smidt

tsmidt@lbl.gov

A Quick Recap!

3D Euclidean symmetry:

rotations, translation, inversion

Different coordinate systems

⇨ same physical system

Euclidean Neural Networks are equivariant to E(3)

Convolutional filters

⇨ learned radial functions

and spherical harmonics

Geometric tensor algebra

Equivariant nonlinearities (did not discuss)

Equivariance can have unintended features.

1) Symmetry specific data types

2) Output symmetry equal to inputs

Implement group equivariance and get all subgroups for FREE!
Symmetry compilers

3) Grad loss wrt input can break symmetry

34 of 105

Resources on Euclidean neural networks:

e3nn Code (PyTorch):

http://github.com/e3nn/e3nn

e3nn_tutorial:

http://blondegeek.github.io/e3nn_tutorial/

Papers:

Tensor Field Networks (arXiv:1802.08219)

Clebsch-Gordon Nets (arXiv:1806.09231)

3D Steerable CNNs (arXiv:1807.02547)

Cormorant (arXiv:1906.04015)

SE(3)-Transformers (arXiv:2006.10503)

tfnns on proteins (arXiv:2006.09275)

e3nn on QM9 (arXiv:2008.08461)

e3nn for symm breaking (arXiv:2008.08461)

My past talks (look for video / slide links):

https://blondegeek.github.io/talks

Books on group theory of E(3)

Group Theory: Application to the Physics of Condensed Matter by Dresselhaus(x2) and Jorio
Math. Theory of Symm. in Solids, Bradley and Cracknell

Feel free to reach out if you have any questions!

Tess Smidt

tsmidt@lbl.gov

A Quick Recap!

3D Euclidean symmetry:

rotations, translation, inversion

Different coordinate systems

⇨ same physical system

Euclidean Neural Networks are equivariant to E(3)

Convolutional filters

⇨ learned radial functions

and spherical harmonics

Geometric tensor algebra

Equivariant nonlinearities (did not discuss)

Equivariance can have unintended features.

1) Symmetry specific data types

2) Output symmetry equal to inputs

Implement group equivariance and get all subgroups for FREE!
Symmetry compilers

3) Grad loss wrt input can break symmetry

35 of 105

Calling in backup (slides)!

36 of 105

Applications so far...

Finding order parameters of 2nd order structural phase transitions
Molecular dynamics (Harvard)
Molecule and crystal property prediction (FU Berlin)
Inverting invariant representations of atomic geometries (Sandia)
Autoencoding Geometry
Predicting molecular Hamiltonians (TU Berlin)
Long range interactions (FU Berlin, TU Berlin)
Electron density prediction for large molecules (Sandia)
Predicting chemical shifts for NMR (Merck, MIT)
Conditional protein design (UW)
Inverse design of optical properties of nanoparticle assemblies (LBL and UW)
Phonon properties of crystal structures (MIT)
Anharmonic elastic properties of crystal structures (UTEP)
...

37 of 105

Predict ab initio forces for molecular dynamics

Preliminary results originally presented at

APS March Meeting 2019.

Paper in progress.

Testing on liquid water, Euclidean neural networks (Tensor-Field Molecular Dynamics) require less data to train than traditional networks to get state of the art results.

Data set from: [1]

Zhang, L. et al. E. (2018).

PRL, 120(14), 143001.

Boris

Kozinsky

Simon

Batzner

38 of 105

Euclidean neural networks can manipulate geometry,

which means they can be used for generative models such as autoencoders.

39 of 105

geometry

features

To encode/decode, we have to be able to convert geometry into features and vice versa.

We do this via spherical harmonic projections.

Euclidean neural networks can manipulate geometry,

which means they can be used for generative models such as autoencoders.

40 of 105

Equivariant neural networks can learn to invert invariant representations.

Which can be used to recover geometry.

Network can predict spherical harmonic projection...

Invariant features + coordinate frame

ENN

Peak finding

Josh Rackers

Thomas

Hardin

41 of 105

Pooling

Unpooling

We can also build an autoencoder for geometry: e.g. Autoencoder on 3D Tetris

Centers deleted

42 of 105

Pooling

Unpooling

We can also build an autoencoder for geometry: e.g. Autoencoder on 3D Tetris

43 of 105

Other atoms

Convolution center

We encode the symmetries of 3D Euclidean space (3D translation- and 3D rotation-equivariance).

We use points. Images of atomic systems are sparse and imprecise.

vs.

We use continuous convolutions with atoms as convolution centers.

Euclidean Neural Networks are similar to convolutional neural networks...

44 of 105

We encode the symmetries of 3D Euclidean space (3D translation- and 3D rotation-equivariance).

We use points. Images of atomic systems are sparse and imprecise.

vs.

Other atoms

Convolution center

We use continuous convolutions with atoms as convolution centers.

Euclidean Neural Networks are similar to convolutional neural networks...

45 of 105

We encode the symmetries of 3D Euclidean space (3D translation- and 3D rotation-equivariance).

We use continuous convolutions with atoms as convolution centers.

We use points. Images of atomic systems are sparse and imprecise.

vs.

Euclidean Neural Networks are similar to convolutional neural networks...

Other atoms

Convolution center

46 of 105

Translation equivariance

Convolutional neural network ✓

Rotation equivariance

Data augmentation

Radial functions (invariant)

Want a network that both preserves geometry and exploits symmetry.

47 of 105

Invariant featurizations can be very expressive if well-crafted

Many invariant featurizations use equivariant operations

e.g. a (simplified) SOAP kernel for ethane molecule C₂H₆

Project neighbors of given atom onto spherical harmonics (equivariant quantity).

Interact signals from different atoms via tensor dot product (equivariant operation) to produce scalars (invariant quantity).

Give scalars to model.

(1)

(2)

(3)

(Favored for kernel methods)

48 of 105

For a function to be equivariant means that we can act on our inputs with g

OR act our outputs with g and we get the same answer (for every operation).

For a function to be invariant means g is the identity (no change).

Layer

out

Layer

out

49 of 105

Why limit yourself to equivariant functions?

You can substantially shrink the space of functions you need to optimize over.

This means you need less data to constrain your function.

All learnable functions

All learnable equivariant functions

All learnable functions constrained by your data.

Functions you actually wanted to learn.

50 of 105

Why not limit yourself to invariant functions?

You have to guarantee that your input features already

contain any necessary equivariant interactions (e.g. cross-products).

All learnable equivariant functions

Functions you actually wanted to learn.

All learnable invariant functions.

All invariant functions constrained by your data.

51 of 105

Neural networks are specially designed for different data types.

Assumptions about the data type are built into how the network operates.

Arrays ⇨ Dense NN

2D images

⇨ Convolutional NN

Text ⇨ Recurrent NN

Components are independent.

The same features can be found anywhere in an image. Locality.

Sequential data. Next input/output depends on input/output that has come before.

Graph ⇨ Graph (Conv.) NN

3D physical data

⇨ Euclidean NN

Data in 3D Euclidean space. Freedom to choose coordinate system.

Topological data. Nodes have features and network passes messages between nodes connected via edges.

52 of 105

Neural networks are specially designed for different data types.

Assumptions about the data type are built into how the network operates.

Symmetries emerge from these assumptions.

Arrays ⇨ Dense NN

2D images

⇨ Convolutional NN

Text ⇨ Recurrent NN

Components are independent.

The same features can be found anywhere in an image. Locality.

Sequential data. Next input/output depends on input/output that has come before.

Graph ⇨ Graph (Conv.) NN

3D physical data

⇨ Euclidean NN

Data in 3D Euclidean space. Equivariant to choice of coordinate system.

No symmetry!

2D-translation symmetry

(forward) time-translation symm.

permutation symmetry

3D Euclidean symmetry E(3): 3D rotations translations and inversion

Topological data. Nodes have features and network passes messages between nodes connected via edges.

53 of 105

✓

If you can craft a good representation -- great!

But deep learning’s specialty is feature learning.

So, maybe use a different machine learning approach (e.g. kernel methods).

Neural networks can’t mess up invariant representations.

You can use ANY neural network with an invariant representation.

Invariant representations can be used for other machine learning algorithms

(e.g. kernel methods).

54 of 105

Analogous to... the laws of (non-relativistic) physics have Euclidean symmetry,

even if systems do not.

The network is our model of “physics”. The input to the network is our system.

55 of 105

A Euclidean symmetry preserving network produces outputs that preserve

the subset of symmetries induced by the input.

O(3)

O_h

Pm-3m

(221)

SO(2) + mirrors

(C_∞v)

3D rotations and inversions

2D rotation and mirrors along cone axis

Discrete rotations and mirrors

Discrete rotations, mirrors, and translations

56 of 105

Properties of a system must be compatible with symmetry.

Which of these situations (inputs / outputs) are symmetrically allowed / forbidden?

57 of 105

✓

✗

Properties of a system must be compatible with symmetry.

Which of these situations (inputs / outputs) are symmetrically allowed / forbidden?

58 of 105

✓

✗

Properties of a system must be compatible with symmetry.

Which of these situations (inputs / outputs) are symmetrically allowed / forbidden?

59 of 105

✓

✗

Properties of a system must be compatible with symmetry.

Which of these situations (inputs / outputs) are symmetrically allowed / forbidden?

60 of 105

Equivariance can have unintuitive consequences.

Partition graph with permutation equivariant function into two sets using ordered labels.

Predict node labels

[0, 1] vs. [1, 0]

61 of 105

Equivariance can have unintuitive consequences.

Partition graph with permutation equivariant function into two sets using ordered labels.

You can’t due to degeneracy.

[0, 1]

[1, 0]

[0, 1]

[1, 0]

There’s nothing to distinguish one partition to be “first” vs. “second”.

Predict node labels

[0, 1] vs. [1, 0]

62 of 105

Convolutions: Local vs. Global Symmetry

Convolutions capture local symmetry. Interaction of features in later layers yields global symmetry.

e.g. Coordination environments in crystals

Atomic systems form geometric motifs that can appear at multiple locations and orientations.

(Local symmetry)

Space group:

Symmetry of unit cell

(Global symmetry)

63 of 105

Translation symmetry in 2D:

Features “mean” the same thing in any location.

Symmetry emerges when different ways of representing something “mean” the same thing.

Representation can have symmetry, operations can preserve symmetry, and objects can have symmetry.

✓

✗

✓

64 of 105

Translation symmetry in 2D:

Features “mean” the same thing in any location.

Symmetry emerges when different ways of representing something “mean” the same thing.

Representation can have symmetry, operations can preserve symmetry, and objects can have symmetry.

Symmetry of 2D objects

Boundaries “break” global translation symmetry.

Periodic boundary conditions preserve

discrete translation symmetry.

✓

✗

✓

65 of 105

Permutation symmetry, S_N:

Symmetry of sets

The freedom to list things in any order

Symmetry emerges when different ways of representing something “mean” the same thing.

Representation can have symmetry, operations can preserve symmetry, and objects can have symmetry.

66 of 105

Permutation symmetry, S_N:

Symmetry of sets

The freedom to list things in any order

Symmetry of elements of a graph

Graph automorphism, specific nodes are indistinguishable (same global connectivity)

Symmetry emerges when different ways of representing something “mean” the same thing.

Representation can have symmetry, operations can preserve symmetry, and objects can have symmetry.

67 of 105

A bit of group theory! Don’t worry just a bit!

Formally, what are invariant vs. equivariant functions

function (neural network)...

vector in vector space

inputs

outputs

weights

...which is equivalent to writing.

68 of 105

A bit of group theory! Don’t worry just a bit!

Formally, what are invariant vs. equivariant functions

function (neural network)...

element of group

representation of g acting on vector space

vector in vector space

inputs

outputs

weights

...which is equivalent to writing.

69 of 105

A bit of group theory! Don’t worry just a bit!

Formally, what are invariant vs. equivariant functions

function (neural network)...

element of group

representation of g acting on vector space

vector in vector space

inputs

outputs

weights

...which is equivalent to writing.

equivariant to x if

70 of 105

A bit of group theory! Don’t worry just a bit!

Formally, what are invariant vs. equivariant functions

function (neural network)...

element of group

representation of g acting on vector space

vector in vector space

inputs

outputs

weights

...which is equivalent to writing.

If we want to be equivariant to x, this has to be the case…

weights must be “scalars”

equivariant to x if

71 of 105

A bit of group theory! Don’t worry just a bit!

Formally, what are invariant vs. equivariant functions

function (neural network)...

element of group

representation of g acting on vector space

vector in vector space

inputs

outputs

weights

...which is equivalent to writing.

If we want to be equivariant to x, this has to be the case…

weights must be “scalars”

equivariant to x if

72 of 105

A bit of group theory! Don’t worry just a bit!

Formally, what are invariant vs. equivariant functions

function (neural network)...

element of group

representation of g acting on vector space

vector in vector space

inputs

outputs

weights

...which is equivalent to writing.

If we want to be equivariant to x, this has to be the case…

weights must be “scalars”

equivariant to x if

(special case) invariant to x if

73 of 105

M. Zaheer et al, Deep Sets, NeurIPS 2017

74 of 105

Convolutional neural networks can “cheat” by being sensitive to “boundaries”.

(e.g. Predict geodesics on projected maps with and without periodic boundary conditions)

User: Stebe

https://en.wikipedia.org/wiki/Gall-Peters_projection

✓

✗

✓

Nodes can be distinguished due to differing topology by latitude (e.g. poles)!

Boundaries break symmetry.

Pixels cannot be distinguished due to translation equivariance.

75 of 105

In the physical sciences...

What our our data types?

3D geometry and geometric tensors...

...which transform predictably under 3D rotation, translation, and inversion.

These data types assume Euclidean symmetry.

⇨ Thus, we need neural networks that preserve Euclidean symmetry.

76 of 105

Scalars

Energy
Mass
Isotropic *

Vectors

Force
Velocity
Acceleration
Polarization

Pseudovectors

Angular momentum
Magnetic fields

Matrices, Tensors, …

Moment of Inertia
Polarizability
Interaction of multipoles
Elasticity tensor (rank 4)

Atomic orbitals

Output of Angular Fourier Transforms

Vector fields on spheres

(e.g. B-modes of the Cosmic Microwave Background)

Geometric tensors take many forms. They are a general data type beyond materials.

77 of 105

Our unit test: Trained on 3D Tetris shapes in one orientation,

these network can perfectly identify these shapes in any orientation.

TRAIN

TEST

Chiral

78 of 105

Several groups converged on similar ideas around the same time.

Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds

(arXiv:1802.08219)

Tess Smidt*, Nathaniel Thomas*, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, Patrick Riley

Points, nonlinearity on norm of tensors

Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network

(arXiv:1806.09231)

Risi Kondor, Zhen Lin, Shubhendu Trivedi

Only use tensor product as nonlinearity, no radial function

3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data

(arXiv:1807.02547)

Mario Geiger*, Maurice Weiler*, Max Welling, Wouter Boomsma, Taco Cohen

Efficient framework for voxels, gated nonlinearity

*denotes equal contribution

79 of 105

Several groups converged on similar ideas around the same time.

Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds

(arXiv:1802.08219)

Tess Smidt*, Nathaniel Thomas*, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, Patrick Riley

Points, nonlinearity on norm of tensors

Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network

(arXiv:1806.09231)

Risi Kondor, Zhen Lin, Shubhendu Trivedi

Only use tensor product as nonlinearity, no radial function

3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data

(arXiv:1807.02547)

Mario Geiger*, Maurice Weiler*, Max Welling, Wouter Boomsma, Taco Cohen

Efficient framework for voxels, gated nonlinearity

*denotes equal contribution

Tensor field networks + 3D steerable CNNs

= Euclidean neural networks (e3nn)

80 of 105

Let g be a 3d rotation matrix.

a_-1

+a₀

+a₁

D is the Wigner-D matrix.

It has shape

and is a function of g.

Spherical harmonics of a given L transform together under rotation.

b_-1

+b₀

+b₁

81 of 105

Convolve

Bloom

Make points to cluster

Symmetric Cluster

Cluster bloomed points

Combine

Convolve with point origins of cluster members

Geometry

New Geometry

How to encode (Pooling layer). Recursively convert geometry to features.

82 of 105

1^st

2^nd

Convolve

Bloom

Make new points

Cluster

Merge duplicate points

Combine

Convolve with origin point

of new points

Geometry

New Geometry

How to decode (Unpooling layer). Recursively convert features to geometry.

83 of 105

Discrete geometry

Reduce geometry to single point.

Create geometry from single point.

We want to convert geometric information (3D coordinates of atomic positions)

into features on a trivial geometry (a single point)

and back again.

Single point with continuous

latent representation

(N dimensional vector)

84 of 105

Reduce geometry to single point.

Create geometry from single point.

Atomic structures are hierarchical and can be constructed from recurring geometric motifs.

We want to convert geometric information (3D coordinates of atomic positions)

into features on a trivial geometry (a single point)

and back again.

Discrete geometry

Single point with continuous

latent representation

(N dimensional vector)

85 of 105

Reduce geometry to single point.

Create geometry from single point.

Encode geometry
Encode hierarchy

(Need to do this in a recursive manner)

We want to convert geometric information (3D coordinates of atomic positions)

into features on a trivial geometry (a single point)

and back again.

Discrete geometry

Single point with continuous

latent representation

(N dimensional vector)

Atomic structures are hierarchical and can be constructed from recurring geometric motifs.

Decode geometry
Decode hierarchy

86 of 105

To autoencode, we have to be able to convert geometry into features and vice versa.

We do this via spherical harmonic projections.

87 of 105

...where the electrons are...

Given an atomic structure,

Energy (eV)

Momentum

...and what the electrons are doing.

...use quantum theory and supercomputers to determine...

What a computational materials physicist does:

Structure

Properties

88 of 105

Quantum Theory / Molecular dynamics

+ Supercomputers

Properties

Hypothesize

Inverse Design

Zooooom!

Map

Structure

We want to use deep learning to speed up calculations, hypothesize new structures, perform inverse design, and organize these relations.

Primer?

89 of 105

Quantum Theory / Molecular dynamics

+ Supercomputers

Properties

Hypothesize

Inverse Design

Zooooom!

Map

Structure

We want to use deep learning to speed up calculations, hypothesize new structures, perform inverse design, and organize these relations.

The problems start here

90 of 105

Given a single example of a degenerate solution,

it knows what other solutions are possible by symmetry.

(Useful for ensuring you’re not biasing your sampling.)

91 of 105

To be rotation-equivariant means that we can rotate our inputs

OR rotate our outputs and we get the same answer (for every operation).

Layer

out

Rot

Layer

out

Rot

92 of 105

For L=1 ⇨ L=1, the filters will be a learned, radially-dependent linear combinations of the L = 0, 1, and 2 spherical harmonics.

L=2

Random filters for

L=1 ⇨ L=1…

(3 in L=1 channels by

3 out L=1 channels)

… as a function of increasing r.

Time showing filter for varying r, where

0 ≤ r ≤ r_max^.

(+ / –)

Radial distance is magnitude

as a function of angle

94 of 105

Predictions for O_h symmetry

Ground Truth

Prediction of network trained with symmetry breaking input and given symmetry breaking input along z.

Prediction of network trained with symmetry breaking input but given trivial input

(single scalar).

Superposition of 6 rotationally degenerate solutions.

95 of 105

A brief primer on deep learning

deep learning ⊂ machine learning ⊂ artificial intelligence

Skip?

96 of 105

model (“neural network”):

Function with learnable parameters.

A brief primer on deep learning

97 of 105

model (“neural network”):

Function with learnable parameters.

Linear transformation

Element-wise nonlinear function

Learned

Parameters

Ex: "Fully-connected" network

A brief primer on deep learning

98 of 105

model (“neural network”):

Function with learnable parameters.

Neural networks with multiple layers can learn more complicated functions.

Learned

Parameters

Ex: "Fully-connected" network

A brief primer on deep learning

99 of 105

model (“neural network”):

Function with learnable parameters.

Neural networks with multiple layers can learn more complicated functions.

Learned

Parameters

Ex: "Fully-connected" network

A brief primer on deep learning

100 of 105

100

deep learning:

Add more layers.

A brief primer on deep learning

101 of 105

101

data:

Want lots of it. Model has many parameters. Don't want to easily overfit.

https://en.wikipedia.org/wiki/Overfitting

A brief primer on deep learning

102 of 105

102

cost function:

A metric to assess how well the model is performing.

The cost function is evaluated on the output of the model.

Also called the loss or error.

A brief primer on deep learning

103 of 105

103

way to update parameters:

Construct a model that is differentiable

Easiest to do with differentiable programming frameworks: e.g. Torch, TensorFlow, JAX, ...

Take derivatives of the cost function (loss or error) wrt to learnable parameters.

This is called backpropogation (aka the chain rule).

error

A brief primer on deep learning

104 of 105

104

http://deeplearning.stanford.edu/wiki/index.php/Feature_extraction_using_convolution

convolutional neural networks:

Used for images. In each layer, scan over image with learned filters.

A brief primer on deep learning

105 of 105

105

http://cs.nyu.edu/~fergus/tutorials/deep_learning_cvpr12/

convolutional neural networks:

Used for images. In each layer, scan over image with learned filters.

A brief primer on deep learning

Back