1 of 54

Machine-Learning Interatomic Potentials

Quantum accuracy on-the-fly

Seminar for the course

“Computing Methods for Physics”,

14 Dec. 2023, Sapienza - University of Rome.

Image credits: [1]

Flavio Giuliani

2^nd-year PhD student in

Theoretical Condensed Matter Physics.

2 of 54

Outline

Motivation: modelling disordered materials
Ab-Initio Molecular Dynamics
Machine-Learning Interatomic Potentials
Applications

3 of 54

Detailed outline

Motivation

Disordered materials in technology applications: Phase-Change Materials
How to model disordered materials at atomic level?

Molecular Dynamics with DFT

Adiabatic approximation
Classical MD for the nuclei
The bottleneck of molecular simulations.

Machine-Learning Interatomic Potentials

What is Machine Learning?
How to construct a MLIP
Artificial Neural Network method
How to validate a MLIP

Applications

Behler-Parrinello method for Silicon.
Examples for Silicon, Water, PCMs.

4 of 54

Motivation: modelling disordered materials
Ab-Initio Molecular Dynamics
Machine-Learning Interatomic Potentials
Applications

5 of 54

Disordered materials in applications: PCMs

Our group focuses on the theoretical study of Phase-change materials (PCMs) made of chalcogenide elements (Ge,Sb,Te,...).

Applications:

Rewritable optical data storage media (CD, DVD, Blu-ray).
Fast switches for radio-frequency communication.
Energy storage in solar cells.
Fast & Non-volatile memory storage.
Tunable resistor (memristor) for neuromorphic computing.
and more …

6 of 54

Phase-change principle

Legend:

Top figures: Ge₂Sb₂Te₅ (GST). Ge, Sb, and Te atoms are rendered as white, yellow, and blue balls, respectively.

Bottom graph: resistance vs. SET/RESET cycle for Sc_0.2Sb₂Te₃ (SST).

7 of 54

Phase-change principle

SET/RESET through an intense light or current pulse.

READ through a weak light or current pulse.

8 of 54

Advantages of PCM storage memory

PCM has an ideal combination of:

Fast writing speed
Non-volatility
Large storage density
Medium fabrication cost

9 of 54

Crystallization/Amorphization dynamics

determines PCM features

Blue: crystal phase

Yellow: amorphous phase

Interaction between experiments and theoretical modelling is needed to improve device performance.

10 of 54

Atomistic simulations: space & time scales

Image modified from:

Ozboyaci et al., Quart. Rev. of Biophys. 49. (2016).

DFT applications are limited to:

〜 100 ps,

〜 10 nm

〜 100 atoms/cell

Beyond these scales, approximations are needed.

11 of 54

Atomistic simulation of disordered systems

💎 Crystal symmetries ⇒ reduction to unit cell.

~ 1 - 100 atoms

Simple cubic crystal.

~ 0.1 - 10 nm

12 of 54

Atomistic simulation of disordered systems

💎 Crystal symmetries ⇒ reduction to unit cell.

Crystal growth in disordered bulk antimony.

Dragoni et al., Nanoscale (2021).

🌊 Amorphous/disordered systems

(liquids, glasses, biomolecules, complex systems, …)

Long-wavelength vibrational modes.

Large relaxation time near criticality.

Study of collective phenomena (⟶).

Free energy computations.

Reduction of finite-size effects.

⇒ need of large space- and time-scales.

10-10² nm, 10³-10⁶atoms

13 of 54

Motivation: modelling disordered materials
Ab-Initio Molecular Dynamics
Machine-Learning Interatomic Potentials
Applications

14 of 54

Ab-Initio Molecular Dynamics (AIMD)

Born-Oppenheimer adiabatic approximation

Electronic+nuclear (e+n) Schrödinger equation:

with:

Coulomb interaction between systems a,b.

Kinetic energy of system a.

Legend

Number of electrons/nuclei.

Spatial coordinates.

separation of energy/time scales

separate the wavefunction:

15 of 54

Electron problem & Nuclear problem

2) Solve the nuclear problem, using the ground state electronic energy as an effective interatomic potential:

Solve the electron problem at fixed nuclei:

16 of 54

Electron problem at fixed nuclei

You just learned how to solve it, using DFT software:

INPUT:

Chemical composition
Crystal structure

Unit cell
Basis vectors

Pseudopotentials
Calculation details

OUTPUT:

Total energy U
Wavefunctions φ_E
Electronic density
Electronic bands
Density of states

17 of 54

2) Nuclear problem

The ground state electronic energy U acts as an effective interatomic potential:

U is also known as Potential Energy Surface (PES) of the nuclei.

Trivial example:

isolated diatomic molecule

18 of 54

2) Nuclear problem

The ground state electronic energy U acts as an effective interatomic potential:

U is also known as Potential Energy Surface (PES) of the nuclei.

Trivial example:

isolated diatomic molecule

System with many atoms

Configuration r^N

19 of 54

Classical dynamics of the nuclei

Classical dynamics is described by Hamilton (Newton) equations:

Effective Hamiltonian for the nuclei:

Classical approximation:

As a rule of thumb, if λ_DeBroglie<< d_typical

then quantum effects are negligible

(e.g. high temperature T>T_Debye, heavy nuclei, ….)

20 of 54

Ergodic hypothesis

Central idea of equilibrium statistical mechanics.

For an ergodic system at equilibrium:

Average of observables A over a trajectory {r^N(t),p^N(t)} which solves the equations of motion

Average of observables A over an ensemble probability distribution f(r^N,p^N)

Molecular Dynamics approach for sampling observables.

Monte Carlo approach for sampling observables.

21 of 54

Classical Molecular Dynamics algorithm

Solving classical MD on a computer:

Start by selecting:

Initial configuration r^N and velocities v^N.
Boundary conditions (cell vectors and periodicity).
Integration algorithm (Velocity Verlet).
Integration timestep (~fs).
Other parameters (thermostats,...).

Then, for each iteration:

Compute the forces between atoms.
Integrate Newton’s equations of motion.
Sample statistical averages.

Frenkel, Smit, Understanding Molecular Simulations. (1996 textbook).

22 of 54

Classical Molecular Dynamics algorithm

Solving classical MD on a computer:

Bottleneck:

Computation of DFT forces at each step:

Self-consistent calculation of the electronic charge density n(r). Requires multiple iterations and scales poorly with system size.

Hellman-Feynman theorem: integration of -dH/dR_i along n(r) gives the force on atom i. Easy to compute.

23 of 54

Motivation: modelling disordered materials
Ab-Initio Molecular Dynamics
Machine-Learning Interatomic Potentials
Applications

24 of 54

Methods for computing the Interatomic Potential

or Potential Energy Surface (PES)

Image adapted from:

Unke et al., Chem. Rev., 121, 16, 10142–10186 (2021).

Configuration r^N

❌ Slow implementation.

✅ Accurate PES.

✅ Dynamical chemical bonds.

Examples: DFT, Coupled Cluster method, other quantum-based methods.

25 of 54

Methods for computing the Interaction Potential

❌ Slow implementation.

✅ Accurate PES.

✅ Dynamical chemical bonds.

Examples: DFT, Coupled Cluster method, other quantum-based methods.

✅ Fast implementation.

❌ Inaccurate PES.

❌ Static chemical bonds.

Examples: harmonic bonds, rigid bonds, Lennard-Jones, other empirical parametrizations.

Image adapted from:

Unke et al., Chem. Rev., 121, 16, 10142–10186 (2021).

or Potential Energy Surface (PES)

Configuration r^N

26 of 54

Methods for computing the Interaction Potential

❌ Slow implementation.

✅ Accurate PES.

✅ Dynamical chemical bonds.

✅ Fast implementation.

❌ Inaccurate PES.

❌ Static chemical bonds.

Image adapted from:

Unke et al., Chem. Rev., 121, 16, 10142–10186 (2021).

or Potential Energy Surface (PES)

Configuration r^N

27 of 54

Machine Learning

ML is the development of statistical algorithms that, after being “trained” on a set of tasks, can generalize to solve new tasks without being explicitly programmed to solve them.

Side note: these categories often overlap.

Image source:

http://machinelearningintro.uwesterr.de , Chapter 8

28 of 54

Machine Learning

ML is the development of statistical algorithms that, after being “trained” on a set of tasks, can generalize to solve new tasks without being explicitly programmed to solve them.

Side note: these categories often overlap.

The task of ML Interatomic Potentials:

PES U(r^N),

forces {F_i}

Given an atomic configuration, compute the DFT interaction potential and its first derivatives w.r.t. positions (forces).

29 of 54

Machine Learning

ML is the development of statistical algorithms that, after being “trained” on a set of tasks, can generalize to solve new tasks without being explicitly programmed to solve them.

Side note: these categories often overlap.

The task of ML Interatomic Potentials:

PES U(r^N),

forces {F_i}

Given an atomic configuration, compute the DFT interaction potential and its first derivatives w.r.t. positions (forces).

⇒ Supervised learning

30 of 54

How to construct a MLIP

Define the task

31 of 54

How to construct a MLIP

Define the task

32 of 54

0) Task definition

A universal MLIP is not possible with current methods.

One must keep in mind the tasks for which the MLIP will be used and the key material’s features that it has to capture to do so:

Which material(s)? Which system size?
Under which conditions? In which thermodynamic phase(s)?
Which dynamical features? E.g. crystallization kinetics.
How accurate?

33 of 54

Reference database

r^N = {r₁,...,r_N}

The database should:

Be representative of the relevant material’s conditions.
Contain uncorrelated configurations.
Be accurate enough for the task.

AIMD sampling: save uncorrelated configurations along trajectories under different conditions.

Crystal structure prediction: explore diversified PES minima at different pressures.

Random small distortions: sample small energy deviations above known PES points.

34 of 54

2) Representation of atomic structure

Variety of MLIP representations

Selection criteria:

Physics informed.
Symmetry-preserving.
Agnostic (as much as possible).
Differentiable.
Computationally efficient.

Image adapted from:

Musil et al., Chemical Reviews 121 (16), 9759-9815 (2021).

35 of 54

3) Regression (nonlinear fitting)

Linear regression based on a similarity kernel

Compare the new structure to the ones in the dataset, and interpolate linearly between their energies.

Artificial Neural Network (NN)

Universal approximator based on consecutive matrix-vector multiplications and nonlinear activation functions; efficient GPU implementation.

Other nonlinear methods …

How to find the optimal fitting parameters w in the nonlinear case?

Define a loss function L(w) between the target y and the prediction ŷ = f(x|w).
Use a minimization algorithm on L(w). (usually based on Stochastic gradient descent).

36 of 54

Simple models for a neuron

Basic introduction to Artificial NNs

Biological neuron

37 of 54

Simple models for a neuron

Basic introduction to Artificial NNs

Biological neuron

McCulloch & Pitts (1943) model

Binary input/output with a sum threshold θ

38 of 54

Simple models for a neuron

Basic introduction to Artificial NNs

Biological neuron

Rosenblatt's Perceptron (1958) model

Key ingredients:

Linear mapping & non-linear threshold Θ.

(Smoother thresholds: tanh, sigmoid, …)

39 of 54

Perceptron

non-linear activation function σ

(e.g. tanh, sigmoid, …)

linear mapping with weights (w,b)

The decision boundary is a hyperplane:

Supervised learning task:

“Find the optimal parameters (w, b) to linearly classify x”

has two possible interpretations:

Linear classification:

Find the hyperplane which creates the best separation between the two classes.

Linear regression:

Find the hyperplane which fits best to the boundary between the two classes.

40 of 54

Non-linear Regression with Multi-Layer Perceptrons

Single Layer

More dimensions:

x,y,b are vectors, W is a matrix.

41 of 54

Non-linear Regression with Multi-Layer Perceptrons

Single Layer

Multi-Layer

Universal approximation theorem:

A Multi-Layer Perceptron is a universal approximator, if deep enough.

42 of 54

Neural-Network Interatomic Potential

Summary

N nuclear positions

N local structures

N represen-

tations

N local energies

Total energy U = sum of local energies

U is invariant to particle permutations and independent of system size.

43 of 54

How to validate a MLIP

Test the predictions on both training data and independent data.

44 of 54

How to validate a MLIP

Example: benchmark for Silicon on an independent melt-quench MD simulation

45 of 54

Motivation: modelling disordered materials
Ab-Initio Molecular Dynamics
Machine-Learning Interatomic Potentials
Applications

46 of 54

Behler-Parrinello method (2007)

2) Representation of atomic structure:

2-body (distances) and 3-body (distances+angles) symmetry functions with manually-tuned parameters:

with

3) A Neural Network regressor for each species, each with 2 hidden layers of ~40 nodes.

Example architecture for a 3-species system.

47 of 54

Behler-Parrinello method (2007)

Example architecture for a 3-species system.

Good results on disordered Silicon

using 48 symmetry functions

Predicted Radial pair correlation (red)

compares well with the DFT one (black).

Energy error: ~ 5 meV/atom

Force error: ~ 0.2 eV/Å

2) Representation of atomic structure:

2-body (distances) and 3-body (distances+angles) symmetry functions with manually-tuned parameters.

3) A Neural Network regressor for each species, each with 2 hidden layers of ~40 nodes.

48 of 54

512 atoms

10⁵ atoms

Structural transitions in dense disordered silicon

Deringer et al., Nature 589, 59–64 (2021).

“GAP” method.

49 of 54

Phase diagram of water

Red: MLIP (“DeepMD” method trained on SCAN DFT data).

Grey: experiment.

Blue: TIP4P/2005 model.

Zhang, Wang, Car, Weinan, Phys. Rev. Lett. 126, 23 (2021).

“DeepMD” method.

50 of 54

MLIP for phase-change materials: GeTe

Sosso et al., Phys. Rev. B 85, 17 (2012).

“Behler-Parrinello” method.

Good description of both crystal and liquid phase

Radial distribution

Angular distribution

51 of 54

Device-scale modelling of PCMs

Crystallization from the disordered bulk phase.

Dragoni et al., Nanoscale (2021).

“Behler-Parrinello” method for Sb.

Zhou et al., Nat Electron (2023).

“GAP” method for Ge₂Sb₂Te₅.

52 of 54

Conclusion

The bottleneck of large-scale quantum-accurate molecular dynamics simulations is the computation of forces.

Machine-Learning Interatomic Potentials (in particular neural-network implementations) allow to compute forces both accurately and efficiently.

This allows for the study of disordered materials on larger time and space scales, relevant for technological applications and for fundamental study of their phase transitions.

53 of 54

Main references

Behler; First Principles Neural Network Potentials for Reactive Simulations of Large Molecular and Condensed Systems. Angew. Chem. Int. Ed. (2017).
Morrow, Gardner, Deringer; How to validate machine-learned interatomic potentials. J. Chem. Phys.; 158 (12): 121501 (2023).
Behler and Parrinello, Generalized Neural-Network Representation of High-Dimensional Potential-Energy Surfaces. Phys. Rev. Lett. (2007).
Zhang, Mazzarello, Wuttig et al. Designing crystallization in phase-change materials for universal memory and neuro-inspired computing. Nat Rev Mater 4, 150–168 (2019).