PINN: �Challenges and Overcoming Strategies

Prof. Seungchul Lee

Physics-informed Neural Networks (PINNs)

• PINN can be used as a physics solver to address physics problems
• PINN can operate without data by utilizing physics loss
• Both physics and data can also be used to calculate the loss function

2

Major Limitations of PINN

3

4) Temporal Causality

2) Sampling Strategy

1) Loss Imbalance

3) Complex Solution Structures

(1) Loss Imbalance

4

Loss Imbalance

• It has been covered in the previous lecture.

5

(2) Sampling Strategy

6

Redistribute Collocation Points

• Mesh generation in CFD
• Fine mesh is required in areas with radical shifts.

7

Localized Solution Variations

• Sharp local variations can occur within the solution domain
• These regions are more difficult to resolve accurately

8

Uniform Collocation Sampling

• Collocation points are distributed uniformly across the domain
• High-gradient regions may be under-resolved

9

Sampling Strategy: Kirsch Problem

• Problem setup
• Kirsch problem: infinite plate with a circular hole under uniaxial tension
• Strong stress concentration near the hole

10

Sampling Strategy: Kirsch Problem

11

Solution from Naïve PINN

• Not bad, but it could be better

12

Need for Local Refinement

• Allocate more collocation points where the solution varies rapidly
• Use the point budget more efficiently

13

Focused Collocation Sampling

• Keep the total number of collocation points unchanged
• Increase point density near the high-gradient region
• Reduce point density in smoother regions

​

14

Solution from Focused Collocation PINN

15

Remarks

• Focused Collocation PINN delivers high-fidelity solutions in steep-gradient regions while maintaining sampling efficiency across smooth domains

​

• When the hard region is not known in advance 
• Here the stress concentration was known to sit at the hole, so the refinement could be placed by hand.
• When the location of the difficulty is not known, the residual itself is the guide
• Evaluate it on a dense candidate set, and add or move collocation points to where it is largest

16

(3) Complex Solution Structures

17

Limitations of PINN for Complex Solution Structures

• Degraded learning performance for complex PDE solutions

18

Limitations of PINN for Complex Solution Structures

• Degraded learning performance for complex PDE solutions

​

​

​

​

​

​

​

​

​

​

​

• A standard PINN uses a single fully connected network, with smooth activations, as the solution over the entire computational domain

19

Spectral Bias in PINNs

• Definition: Neural networks learn low-frequency components before high-frequency ones
• Limitation: Fails to capture fine, rapid spatial/temporal oscillations
• Core Impact: Severe accuracy drops for highly oscillatory PDE solutions

20

Frequency

Fourier amplitude

Spectral Bias in PINNs

• Progressive learning from low to high frequencies
• Slow formation of high-frequency structure

21

Frequency-wise Convergence

• Fast convergence of low-frequency components
• Each higher component converges later, and the error in the highest one is still large when the lowest has long been exact.

22

(3-1) Fourier Feature Embedding

23

Fourier Feature Embedding

• Lab 1: High-Frequency Solution

​

​

​

​

​

​

• The exact solution is

24

Naïve PINN

• Failure to recover high-frequency solution

25

Fourier Feature Embedding

26

Fourier Feature Embedding

27

PINN with Fourier Feature Embedding

• Naïve PINN: Poor recovery of high-frequency oscillations
• Fourier-feature PINN: Accurate recovery of oscillatory structure

​

28

Frequency-Domain Analysis

• FFT-based comparison of learned frequency components
• Improved recovery of target high-frequency modes

29

Training Process

30

(3-2) Enriched PINN

31

Singularity/Sharp Gradient

• Most of the domain is easy and a small region is not
• The small region is usually the one the engineer cares about
• A single smooth network must represent both with one set of parameters
• The smooth part mostly wins, because it is most of the domain and most of the loss.

32

Lab: Shear Force Discontinuity

33

Naïve PINN

• Single smooth NN over the entire domain
• Difficulty in representing the discontinuous shear force
• Smoothed prediction near the point-load location

34

Enriched PINN

• The enriched model does not ask the network to produce the singularity
• It adds the known singular stress, with a coefficient to be learned, to the network output

35

Enriched PINN

36

Naïve PINN vs. Enriched PINN

• Naïve PINN fails to capture the discontinuous jump in shear force
• Enriched PINN accurately reproduces the discontinuity using the known basis
• Significant error reduction through the learned enrichment coefficient

37

Decomposition of the Enriched Solution

38

(3-3) Domain Decomposition

39

Heterogeneous or Multi-Material Media

40

Gradient Discontinuity

• Different local physics across the domain or
• Different material properties
• Sharp change across an interface
• Interface coupling between local solutions

41

Lab 3. Material Interface with Gradient Discontinuity

42

Naïve PINN

• Single global NN for both materials
• Smoothing of the gradient change near the interface
• Large interface-region error

43

Domain Decomposition (1/2)

44

Domain Decomposition (2/2)

45

Naïve PINN vs. Multi-Domain PINN

46

47

(4) Temporal Causality

48

Transient Problems (or Time-dependent Problems)

49

Transient Problems (or Time-dependent Problems)

• The solution evolves from earlier states to later states
• Initial-condition (IC) information must propagate through time

50

Transient Problems

• Naive PINNs train all time points simultaneously
• Late-time PDE residuals could be minimized before early-time states are resolved

​

• Lab: one-dimensional Allen–Cahn equation

51

Transient Problems

• Small PDE residuals do not guarantee a correct time trajectory
• Without IC propagation, the model can converge to a wrong solution

​

​

​

​

​

​

​

​

​

• Enforce temporal causality during training
1. Causal PINN
2. Time-Marching

52

(1) Causal PINN

• Weights later-time residuals based on earlier-time convergence

53

Causal PINN: Demo

54

Causal PINN: Demo

55

(2) Time-Marching

• Decompose a long transient problem into short time windows,
• Train one time window at a time

56

Time-Marching

• Decompose a long transient problem into short time windows,
• Train one time window at a time
• Solving them one after another

​

57

Time-Marching

58

Time-Marching

• Decompose a long transient problem into short time windows,
• Train one time window at a time
• Use the previous prediction as the next initial condition:

59

Time-Marching

• Decompose a long transient problem into short time windows,
• Train one time window at a time
• Use the previous prediction as the next initial condition:

60

Time-Marching: Demo

61

Time-Marching: Demo

62

Causal PINN + Time Marching

• Time marching sets the global order of time windows
• Causal weighting sets the local order inside each window

63