1 of 52

Embedding Physical Knowledge �into Neural Networks

Prof. Seungchul Lee

1

2 of 52

Potentials of Data-driven Approach

  • Deep learning as a data-driven approach

​

​

2

3 of 52

Deep Learning

  • Advantages of data-driven approaches
    • If enough data is available, a legitimate level of prediction performance can be achieved without domain knowledge

​

​

​

​

​

3

4 of 52

Limitations of Pure Data-driven Approaches

  •  
  • Interpretability: Physically interpretable output

​

4

Interpolation

Extrapolation

Extrapolation

5 of 52

Prior Knowledge (or Physical Knowledge)

  • Knowledge available before model training
    • Governing laws & physical conditions
    • Valid ranges & monotonic trends
    • Conservation laws
    • Symmetry & invariance

5

6 of 52

Direction of Deep Learning in ME

  • Artificial Intelligence + Mechanical Engineering

6

Data-driven AI

7 of 52

Direction of Deep Learning in ME

  • Artificial Intelligence + Mechanical Engineering

7

Physics-informed AI

…

8 of 52

Physics-informed AI

8

9 of 52

Physics-informed AI

9

10 of 52

Physics-informed AI

10

11 of 52

Physics-informed AI

11

12 of 52

Physics-informed AI

12

More Robust and Efficient AI Model with Data + Physics

13 of 52

Knowledge Integration

  • Integrating external knowledge into DNNs is a critical step to ensure that models not only rely on large datasets but also leverage existing domain knowledge.

​

    • Feature Engineering: Tailoring input features to better align with domain-specific knowledge

​

    • Designing: Modifying model architectures to incorporate external knowledge sources

​

    • Regularizing: Adding constraints, priors, or penalties to the loss function ensures that the model adheres to known properties of the domain. This prevents overfitting and encourages the model to find solutions consistent with real-world knowledge

​

13

14 of 52

Neural Networks: Loss Function

  • Measures error between target values and predictions

​

​

​

  • Examples
    • Squared loss (for regression):

​

​

​

    • Cross entropy (for classification):�

14

15 of 52

How to Enforce Constraints

15

16 of 52

Constraints

  • Constraints on neural network outputs arise from physics, mathematics, conservation principles, and symmetry

​

  • The key question: How strictly must the constraint be enforced?

​

  • Two guiding factors: certainty of prior knowledge + cost of violation
    • Hard enforcement: Exact, certain constraints
    • Soft enforcement: Approximate or uncertain constraints

16

17 of 52

1) Soft Constraints: Penalty-Based Enforcement

  •  

17

18 of 52

2) Hard Constraints: Architectural Enforcement

  •  

18

19 of 52

Positivity Constraints

19

20 of 52

Positivity Constraints

  • Require that the network output is always non-negative:

​

​

​

  • Arises naturally in engineering and physics:
    • Stress intensity factors
    • Energy densities
    • Material stiffness values

20

21 of 52

Enforcing Positivity

  • Soft Enforcement
    • Add a penalty term that grows when outputs go negative:
  •  

21

22 of 52

Monotonicity Constraints

22

23 of 52

Monotonicity Constraints

  • Monotone Increasing:

​

​

​

​

​

​

​

​

​

​

​

  • e.g., Beam deflection increases monotonically with applied load

​

​

​

​

​

  • Monotone Decreasing:

​

​

​

​

​

​

​

​

​

​

​

  • e.g., Fatigue life decreases monotonically with stress amplitude

23

 

 

 

 

24 of 52

Enforcing Monotonicity: (1) Non-Negative Derivative

  •  

24

25 of 52

Enforcing Monotonicity: (2) Soft Enforcement

  •  

25

26 of 52

Boundary and Initial Conditions

26

27 of 52

Boundary and Initial Conditions

  • PDE problems require outputs to satisfy prescribed boundary or initial conditions
  • Penalizing violations (soft)

​

  • I.C.

​

​

​

  • B.C.

27

28 of 52

Boundary and Initial Conditions

  •  

28

29 of 52

Examples: Masking Functions

  •  

29

30 of 52

Enforcing Initial Conditions

  •  

30

31 of 52

Conservation Laws or Invariants

31

32 of 52

Conservation Laws or Invariants

  •  

32

Same net mass flow out

Net mass flow in

Control volume

Steady state: Net flow in = Net flow out

33 of 52

Enforcing Conservation Laws: Soft vs. Hard

  • Soft Enforcement
    • Add the violation as a squared penalty term:

​

​

​

​

  • Hard Enforcement
    • Correct the raw output by an additive shift:

​

33

34 of 52

Invariance

34

35 of 52

Invariance

  • Invariance is the special case where the output is unchanged by the transformation:

​

​

​

  • Examples:
    • Total energy of a molecule: same value regardless of rotation or translation
    • Image classifier ("cat / not cat"): same label regardless of small pixel shifts

35

36 of 52

Enforcing Invariance: Three Strategies

  1. Data Augmentation

​

  • Invariant Input Features

​

  • Pooling Over Transformations

​

36

37 of 52

1) Data Augmentation

  •  

37

 

 

 

Rotation / Translation

 

 

 

Rotation / Translation

38 of 52

2) Invariant Input Features

  •  

38

 

 

 

Rotation / Translation

 

39 of 52

3) Pooling Over Transformations

  •  

39

or

40 of 52

Example: Invariance of Total Strain Energy

  •  

40

 

41 of 52

Enforcing Invariance

  •  

41

42 of 52

Enforcing Invariance

  •  

42

43 of 52

Equivariance

43

44 of 52

Equivariance

  •  

44

45 of 52

Common Transformations

45

Translation

Rotation

Reflection

Permutation

2

1

3

46 of 52

1) Data Augmentation

  •  

46

47 of 52

2) Output Symmetrization

  •  

47

48 of 52

Example: Symmetry of a Clamped-Clamped Beam

  •  

48

 

49 of 52

Clamped-Clamped Beam

  • Original input

​

​

​

​

​

​

​

​

​

​

​

​

  • Original response

​

​

​

​

​

​

​

​

​

​

49

 

 

 

50 of 52

Clamped-Clamped Beam

  • Reflected response

​

​

​

​

​

​

​

​

​

​

  • If you mirror the load, the deflection mirrors with it
  •  

50

 

 

 

A network trained without symmetry constraints may not satisfy this automatically

51 of 52

Example of Output Symmetrization

  •  

51

 

 

 

 

 

 

 

 

52 of 52

Example of Output Symmetrization

  •  

52