1 of 24

Derivatives with AD and FDM

1

2 of 24

PINN Workflow

  • Standard data-only NN

2

 

 

 

 

Total loss

Derivatives

​

​

​

​

​

​

​

​

​

​

 

 

 

 

 

3 of 24

PINN Workflow (Simplified)

  •  

3

 

 

 

 

 

 

Total loss

Derivatives

​

​

​

​

​

​

​

​

​

​

 

 

 

4 of 24

PINN Workflow

  • Two approaches to evaluate the derivatives required by the physics residual
    • Automatic Differentiation (AD)
    • Finite Difference Method (FDM)

4

 

 

 

 

Total loss

Derivatives

​

​

​

​

​

​

​

​

​

AD or FDM

 

 

 

 

 

5 of 24

Automatic Differentiation (AD)

  • Computation is decomposed into elementary operations
  • AD records the sequence of operations in a computation graph
  • Each operation has a known local derivative

5

 

 

 

 

 

Elementary

operations

Local derivatives

 

 

×

σ

+

 

 

 

 

×

 

 

 

 

Computation graph

6 of 24

AD

  •  

6

 

 

 

×

σ

+

 

 

 

 

×

 

 

 

 

 

 

 

7 of 24

AD

  •  

7

 

×

σ

+

 

 

 

 

×

 

 

 

 

σ’

×

 

 

 

 

 

 

8 of 24

Finite Difference Method (FDM)

  • Numerical approximation of derivatives
  • Approximates derivatives using predictions at point of interest and neighboring points
  • Standard finite-difference stencils are easiest on structured grids

8

 

 

 

 

 

 

 

9 of 24

FDM

  • Convenient when output is image/field
  • 2D 5-point stencil
  • Equivalent to applying kernel to full field:

9

 

 

 

10 of 24

AD vs FDM in PINN

  • AD – differentiate the NN output through its computation graph
  • FDM – approximate the derivative using NN output at neighboring points
  • Same PDE loss, different derivative evaluation

10

 

 

 

 

 

 

 

 

AD:

FDM:

 

 

 

Autograd

11 of 24

AD vs FDM in PINN

  • Same PDE loss, only differ in derivative evaluation

11

AD

FDM

12 of 24

AD: Strengths and Weaknesses

  • Strengths
    • Accurate derivatives
    • Arbitrary/scattered collocation points
    • Convenient for coordinate-based PINNs
  • Weaknesses
    • High memory cost
    • Expensive for higher-order derivatives
    • Requires retaining computation graphs

12

Arbitrary collocation points

 

 

 

 

 

13 of 24

FDM: Strengths and Weaknesses

  • Strengths
    • Simple derivative evaluation
    • Lower memory cost
    • Natural for image/field outputs
  • Weaknesses
    • Discretization error
    • Neighboring points required (structured grid)
    • Multiple NN evaluations needed

13

Derivative from local neighboring predictions

FDM error due to finite h

 

14 of 24

Effect of Step Size h

  • Discretization error decreases as h gets smaller

14

15 of 24

Code Example

  •  

15

 

16 of 24

Heat Source: Gaussian Heater

  • Gaussian heat source at plate center

16

 

17 of 24

Collocation Points

17

18 of 24

Neural Network and Loss

18

19 of 24

PDE: AD

19

20 of 24

PDE: FDM

20

21 of 24

Train

21

22 of 24

Results

22

Method

Relative L2 Error

No. of epochs

Maximum T (°C)

Training time (s)

Peak GPU Memory (MB)

AD-PINN

1.932e-03

20000

21.627

168.14

22.25

FDM-PINN

3.903e-03

20000

21.640

138.44

19.30

23 of 24

AD vs FDM: Selection

23

Prefer AD when…

Prefer FDM when…

Collocation points are arbitrary/scattered

Structured field/image output

Coordinates are explicit NN inputs

High-order AD is computationally expensive

Discretization error matters

Memory constraints

24 of 24

Summary

  • AD
    • Continuous coordinate derivates
    • No finite-difference discretization error
    • But higher-order graph cost
  • FDM
    • Simple local stencil operations
    • Natural on structured fields
    • Introduces discretization error and requires neighbors
  • Both construct the same physics residual

24