PINN: �Multiple Loss Terms and Weighting

Prof. Seungchul Lee

Overfitting: Start with Linear Regression

2

Recap: Nonlinear Regression

• Polynomial (here, quad is used as an example)

3

Nonlinear Regression

4

10 input points with degree 9 (or 10)

Polynomial Fitting with Different Degrees

5

Low error on input data points,

but high error nearby

• Underfit
• Good fit
• Overfit

​

Important to find the right balance between model complexity and generalization

Loss

• Loss: Residual Sum of Squares (RSS)
• decreases as the polynomial degree increases
• a lower error does not necessarily indicate a better model

6

Low error on input data points,

but high error nearby

Issue with Rich Representation

• Low error on input data points, but high error nearby
• Low error on training data, but high error on testing data

7

Overfitting

• One of the most common problem data science professionals face

​

• Overfitting occurs when a machine learning model performs well on the training data but poorly on unseen or test data.

​

• The model essentially "memorizes" the noise and details in the training set rather than learning the general patterns that can be applied to new data.

8

Signs of Overfitting

• Large gap between training and test loss:

​

• Training loss continues to decrease, but test loss increases.

9

Causes of Overfitting

• Model Complexity
• A model with too many parameters (e.g., deep neural networks) can overfit the training data.

​

• Insufficient Training Data
• If the training dataset is small, the model may fit every point precisely, capturing noise rather than general patterns.

​

• Noisy Data
• If the data has a lot of noise or irrelevant features, the model may try to learn this noise.

​

• Too Many Training Iterations
• Training for too long can lead to overfitting, where the model starts fitting noise instead of actual patterns.

​

10

Generalization Error

• Difference between a machine learning model's performance on the training data and its performance on unseen data (test or validation data)

11

Regularization to Reduce Overfitting

12

Generalization Error

13

Representational Difficulties

14

With Less Basis Functions: Fewer RBF Centers

15

With Less Basis Functions: Fewer RBF Centers

• Least-squares fits for different numbers of RBFs

​

16

Representational Difficulties

17

Regularization (Shrinkage Methods)

• Often, overfitting associated with very large estimated parameters
• We want to balance
• how well function fits data
• magnitude of coefficients 

​

​

​

​

​

​

​

• multi-objective optimization
• 𝜆 is a tuning parameter

18

Regularization (Shrinkage Methods)

19

RBF: Start from Rich Representation

20

RBF with Regularization

21

RBF with Regularization

22

23

How L2 Regularization Works

​

​

​

​

• Effect on Weights:
• L2 regularization forces the model to keep weights small.
• Interpretation:
• L2 regularization distributes the penalty uniformly across all weights, shrinking their magnitude.
• Overfitting Reduction:
• By limiting the model's ability to learn large weights, the model becomes less likely to overfit the noise in the training data.

​

​

24

Probabilistic Linear Regression

25

Probabilistic Linear Regression

​

​

​

​

​

​

​

• Inference idea

​

​

​

• Change your viewpoint of data
• Generative model

26

data = underlying pattern + independent noise

Generative Model: Regression

27

Likelihood and Log-Likelihood

• The likelihood of the parameters given the dataset is:

​

​

​

​

• Taking the logarithm (log-likelihood):

28

Maximum Likelihood Estimation (MLE)

29

Maximum Likelihood Estimation (MLE)

• In a matrix form

​

​

​

​

​

​

• Taking gradients of the log-likelihood and setting them to zero:

30

Maximum Likelihood Estimation (MLE)

31

Linear Regression: A Probabilistic View

32

Linear Regression: A Probabilistic View

• Demonstrate

33

Linear Regression: A Probabilistic View

• Demonstrate
• samples are independent

​

​

​

​

​

​

​

​

​

​

​

• If a linear model is correctly specified and well estimated, then the residual errors should approximately follow a Gaussian distribution:

34

Bayesian View of Linear Regression

35

Generative Model: Regression

36

37

38

Images from Prof. Philipp Hennig at University of Tubingen

39

Images from Prof. Philipp Hennig at University of Tubingen

Maximum-a-Posteriori (MAP)

40

Images from David S. Rosenberg at Bloomberg ML EDU

Posterior

• Posterior probability
• Bayes rule

​

​

​

​

• Log posterior probability

​

​

​

​

​

​

• Maximize log posterior probability�

41

Maximum-a-Posteriori (MAP)

42

Summary: MLE vs MAP

43

From Statistical Priors to Physical Priors

44

From Statistical Priors to Physical Priors

• Consider a governing equation written with all terms on one side,

​

​

• The residual

​

​

• Measures how badly the physics is violated over collocation points

​

​

​

• A MAP estimate whose prior is a physical law rather than a statement about parameter magnitudes

45

Strategies for Choosing the Weights

46

PINN Architecture

47

PINN Architecture

• PINNs train multiple physical conditions in one objective
• The optimizer sees only the combined scalar loss

48

Poisson Equation

• Poisson equation

​

​

• with homogeneous Dirichlet boundary conditions

​

​

​

• Exact solution

49

Naive PINN

• Poisson equation

​

​

​

​

​

​

​

​

​

• Most natural thing to do is to add them

50

Loss Imbalance

• Equal weights do not imply equal gradient contributions
• Parameters are updated by the sum of gradients from each loss term

51

Loss Imbalance

• Equal weights do not imply equal gradient contributions
• Parameters are updated by the sum of gradients from each loss term

52

Loss Imbalance

• Equal weights do not imply equal gradient contributions
• Parameters are updated by the sum of gradients from each loss term

53

“A single loss can dominate the parameter update”

Mitigating Loss Imbalance

• Apply weights to individual loss terms

54

Mitigating Loss Imbalance

• Apply weights to individual loss terms
1. Fixed weighting
2. Adaptive weighting

55

Fixed Weighting

• A Well-Chosen Weight

• A Badly Chosen Weight

​

56

Two Limitations of a Fixed Weight

1. The right value is not known in advance

​

• Even a right value does not stay right

57

Fixed Weighting

• Weights are set before training and kept constant
• Simple, but suitable weights are problem dependent

58

Fixed Weighting

• Cannot adapt when gradient scales change during training

59

Adaptive Weighting

• Automatically adjusts weights from training signals

60

The Optimizer Sees Gradients, Not Losses

• The parameters are updated by gradient descent on the total loss

​

​

​

• Because the objective is a sum, so is its gradient

​

​

​

• What decides the direction of the step is the gradient contribution of each term,
• the length and direction of its vector, and
• not the scalar value of the loss

61

Gradient-Based Adaptive Weighting

62

Gradient-Based Adaptive Weighting

​

​

​

​

​

​

​

​

​

​

• The weights are recomputed at every iteration from the current gradients

63

Moving-Average Adaptive Weighting

• Instantaneous weights are too noisy to use directly
• Move the weight part of the way toward the measurement, not all the way

​

​

​

• An exponential moving average of the instantaneous weights

64

Moving-Average Adaptive Weighting

65

Hard Constraint

• Embed mandatory conditions directly into the network output
• Remove selected constraints from loss-gradient competition

66

Hard Constraint by a Masking Function

• Build the boundary condition into the output instead of penalizing it

​

​

​

​

​

​

​

​

​

​

​

• One term left in the objective, and no weight to choose

67

Hard Constraint by a Masking Function

68