1 of 52

Physics-informed Neural Networks

2 of 52

Deep Learning

  • Advantages of data-driven approaches
    • If enough data is available, a legitimate level of prediction performance can be achieved without domain knowledge.

​

​

​

​

​

2

3 of 52

Deep Learning

  • Advantages of data-driven approaches
    • If enough data is available, a legitimate level of prediction performance can be achieved without domain knowledge.

​

​

​

​

​

  • Drawbacks
    • Interpretability (how)
    • Generalizability (cross domain or extrapolation)

​

  • To overcome
    • Physics-informed AI (data + physics to build a model)

3

4 of 52

Direction of Deep Learning in ME

  • Artificial Intelligence + Mechanical Engineering

4

Data-driven AI

5 of 52

Direction of Deep Learning in ME

  • Artificial Intelligence + Mechanical Engineering

5

Physics-informed AI

6 of 52

Physics-informed AI

6

7 of 52

Physics-informed AI

7

8 of 52

Physics-informed AI

8

9 of 52

Physics-informed AI

9

10 of 52

Physics-informed AI

10

B.C./I.C. or more data

11 of 52

Physics-informed AI

11

B.C./I.C. or more data

12 of 52

Why Deep Learning Needs Physics?

  • Why do data-driven ‘black-box’ methods fail?
    • May output result that is physically inconsistent
    • Easy to learn spurious relationships that look good only on training and test data
      • Can lead to poor generalization outside the available data (out-of-sample prediction tasks)
    • Interpretability is absent
      • Discovering the mechanism of an underlying process is crucial for scientific advancements

​

​

12

extrapolation

interpolation

For training

For testing

13 of 52

Why Deep Learning Needs Physics?

  •  

13

14 of 52

Taxonomy of Informed Deep Learning

14

Differential Equation

Algebraic Equation

Knowledge Graph

Simulation Result

Human Feedback

Knowledge Representation

Knowledge Integration

Feature Engineering

Designing

Regularizing

Deep Neural Networks

ANN

CNN

RNN

GNN

Generative Model

Sung Wook Kim, "Recent Advances of Artificial Intelligence in Manufacturing Industrial Sectors: A Review," IJPE

15 of 52

Taxonomy of Informed Deep Learning

15

Differential Equation

Algebraic Equation

Knowledge Graph

Simulation Result

Human Feedback

Knowledge Representation

Knowledge Integration

Feature Engineering

Designing

Regularizing

Deep Neural Networks

ANN

CNN

RNN

GNN

Generative Model

Sung Wook Kim, "Recent Advances of Artificial Intelligence in Manufacturing Industrial Sectors: A Review," IJPE

16 of 52

Deep Learning for Solving Differential Equations

16

17 of 52

Multilayer Feedforward Networks are Universal Approximators

  • The Universal Approximation Theorem
    • Neural Networks are capable of approximating any Borel measurable function
    • Neural Networks (1989)

17

18 of 52

Differential Equations

  • Types of ODEs
    • Linear

​

    • Nonlinear

​

​

  • Partial Differential Equation

​

​

​

  • Types of ODE / PDE problems
    • Initial Value Problem (IVP)
    • Boundary Value Problem (BVP)

​

​

18

19 of 52

Neural Networks for Solving Differential Equations

  • Neural Algorithm for Solving Differential Equations
    • Journal of Computational Physics (1990)
    • Neural minimization for finite difference equation

​

  • ANN for ODE and PDE
    • IEEE on Neural Networks (1998)

19

20 of 52

Journal of Computational Physics (2019)

  • M. Raissi, P. Perdikaris, G.E. Karniadakis

20

Navier-Stokes equation

Burgers' equation

21 of 52

Architecture of Physics-informed Neural Networks (PINN)

  • NN as an universal function approximator

​

  • Given
    • ODE or PDE
    • Some measured data from initial and boundary conditions

​

​

​

​

  • Adding constraints for regularization
    • Regularized by physics, but matched with sparse data

​

​

21

22 of 52

Nature Reviews Physics (2021)

  • Physics-informed machine learning

​

  • How to embed physics in ML

22

  • Observation bias
  • Inductive bias
  • Learning bias

23 of 52

Examples

  •  

23

Raissi et al. Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations, 2017

24 of 52

Example: Vortex Shedding

  • Navier-Stocks equation (momentum)
  • PDE

​

​

​

​

​

​

  • Boundary conditions
  • Initial conditions (5000 points used)

​

24

CFD simulation

Deep Learning

25 of 52

Displacement Field Reconstruction

  • Reconstruct full displacement field from the sparse measurements and embedding the governing equations into loss functions

25

Yuan et al. Machine learning for structural health monitoring: challenges and opportunities, 2020

# of sensors

Error

26 of 52

Summary of Physics-informed AI

  • Leverage inductive bias and sparsity

26

27 of 52

Lab 1: Simple Example

​

​

​

​

​

​

​

​

  • Let's look at the ODE

​

  • Initial condition

​

  • The exact solution

​

​

​

27

28 of 52

Lab 1: Simple Example

28

29 of 52

Lab 1: Simple Example

29

30 of 52

Lab 1: Simple Example

30

31 of 52

Lab 1: Simple Example

31

32 of 52

Lab 2: Solve Lab 1 Again using DeepXDE

  • Embed a PDE into the loss of the neural network using automatic differentiation
  • Compare the PINN algorithm to a standard finite element method (FEM)

​

  • DeepXDE: A Deep Learning Library for Solving Differential Equations
    • DeepXDE — DeepXDE 0.14.0 documentation
    • Library for solving forward and inverse partial differential equations (PDEs) via physics-informed neural network (PINN)

​

32

33 of 52

Lab 2: Solve Lab 1 Again using DeepXDE

33

DeepXDE

34 of 52

Lab 2: Solve Lab 1 Again using DeepXDE

34

DeepXDE

35 of 52

Lab 2: Solve Lab 1 Again using DeepXDE

35

DeepXDE

36 of 52

Lab 3: Euler Beam (Solid Mechanics)

​

​

  • Partial differential equations & boundary conditions

​

​

​

  • One Dirichlet boundary condition on the left boundary:
  • One Neumann boundary condition on the left boundary:
  • Two boundary conditions on the right boundary:

​

  • The exact solution is

​

36

37 of 52

Lab 3: Euler Beam (Solid Mechanics)

  • Make a neural network and loss functions

37

38 of 52

Lab 3: Euler Beam (Solid Mechanics)

38

39 of 52

Lab 3: Euler Beam (Solid Mechanics)

39

 

 

 

 

40 of 52

Lab 4: Navier-Stokes Equations (Fluid Mechanics)

  •  

40

41 of 52

Lab 4: Navier-Stokes Equations (Fluid Mechanics)

​

​

  • Hydraulic diameter is

​

  • The Reynolds number of this system is

​

  • 2D Navier-Stokes Equations & boundary conditions (for steady state)

​

​

41

42 of 52

Lab 4: Navier-Stokes Equations (Fluid Mechanics)

  • Two Dirichlet boundary conditions on the plate boundary (no-slip condition)

​

​

​

  • Two Dirichlet boundary conditions at the inlet boundary

​

​

​

  • Two Dirichlet boundary conditions at the outlet boundary

​

42

43 of 52

CFD Solution

  •  

43

44 of 52

Lab 4: Navier-Stokes Equations (Fluid Mechanics)

  • Make a neural network and loss functions

44

 

 

 

 

45 of 52

Lab 4: Navier-Stokes Equations (Fluid Mechanics)

45

 

 

 

 

46 of 52

Lab 4: Navier-Stokes Equations (Fluid Mechanics)

46

47 of 52

Lab 4: Navier-Stokes Equations (Fluid Mechanics)

47

48 of 52

Plot Results (Adam Optimizer)

48

49 of 52

Train More (L-BFGS Optimizer)

  • Limited-memory BFGS (L-BFGS or LM-BFGS) is an optimization algorithm in the family of quasi-Newton methods that approximates the Broyden–Fletcher–Goldfarb–Shanno algorithm (BFGS) using a limited amount of computer memory

49

50 of 52

Plot Results (Adam + L-BFGS)

50

51 of 52

PINN vs. CFD

51

PINN

CFD

52 of 52

Validation: Velocity Profile at the End of the Plate

  • Fully developed velocity profile at the infinite parallel plates flow are known as

​

52