Machine learning for Physics

​

Mehdi Bennani

Jan 27th 2026

Confidential - Google DeepMind

Agenda

• Introduction to Physics Informed Deep Learning

​

• Discovery of unstable singularities with Physics Informed Neural Networks

Confidential - Google DeepMind

Introducing Physics Informed Neural Networks

Confidential - Google DeepMind

Let’s start with a toy physics problem

Confidential - Google DeepMind

A falling tennis ball

A governing equation :

This equation can be solved analytically :

Confidential - Google DeepMind

The experimental data

Confidential - Google DeepMind

A naive model may not generalize beyond the training domain

​

Confidential - Google DeepMind

Enforcing the governing equation during training may lead to better OOD generalization.

​

Confidential - Google DeepMind

How ?

Confidential - Google DeepMind

Physics Informed Neural Networks are optimized wrt the governing equations

​

Neural Network

Input

Model

Output

Loss

The governing equation is written as a differentiable loss function

M. Raissi, P. Perdikaris, G.E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.

Journal of Computational Physics, Volume 378, 2019

Confidential - Google DeepMind

Discovery of Unstable Singularities

Confidential - Google DeepMind

Joint work with

T. Buckmaster

G. Cao-Labora

​

J. Gomez Serrano

Y. Lai

​

T. Leger

Y. Wang

​

Confidential - Google DeepMind

Google DeepMind Team

Confidential - Google DeepMind

The Clay Millennium Problem

Open problem

Importance

Do the incompressible 3D Navier-Stokes’ equations develop singularities in finite time?

- 200 year old open problem

- One of the seven Clay Millennium Problems

Impactful related problems

- Incompressible 3D Euler equations

- Incompressible Porous Media equations

- Boussinesq equations

- Córdoba-Córdoba-Fontelos equation

Confidential - Google DeepMind

Singularities

Situations, such as when quantities like velocity or pressure become infinite, are called ‘singularities’ or ‘blow ups’.

​

They help mathematicians identify fundamental limitations in the equations of fluid dynamics, and help improve our understanding of how the physical world functions.

Confidential - Google DeepMind

Unstable

Stable

Unstable states are much more difficult to converge to

Higher unstable

Confidential - Google DeepMind

Stable vs Unstable Solutions

Finding unstable solutions is very challenging

​

• Classical methods are not applicable

​

​

Blow-up solutions for 3D Euler and Navier-Stokes are expected to be unstable

​

Confidential - Google DeepMind

Stable vs Unstable Solutions

Let us take the linearization of the equation at a stationary solution:

​

​

​

​

where

• r is the perturbation to the stationary solution
• L is a linear operator.

​

​

A self-similar solution is unstable if L has a portion of its spectrum in the positive real half plane.

Confidential - Google DeepMind

Simpler

Complex

Our recent results

Confidential - Google DeepMind

Quick Sketch of one of the problems

Confidential - Google DeepMind

The Incompressible Porous Media (IPM) Equations

where the 2D vector is the velocity and the scalar is the density.

Confidential - Google DeepMind

Evaluation

Confidential - Google DeepMind

Metrics

We track the maximum residuals and their derivatives over the domain

​

For example, for the following equation :

​

​

​

​

The residuals are defined as :

​

​

​

​

Then, we track the following derivatives :

Confidential - Google DeepMind

Maximum residuals over a fine eval grid below 10^-13

Success Criteria

Confidential - Google DeepMind

Methods

Confidential - Google DeepMind

• Coordinates transformation

​

• Optimizing wrt the derivatives of the residuals

​

• Second order optimization

​

• Sampling training points adaptively wrt the errors

​

• Multi-stage training

Some Important Ingredients

Confidential - Google DeepMind

Methods

​

Coordinates transformation

Confidential - Google DeepMind

The limitations of unbounded domains

In the cartesian coordinates, x₁ and x₂ are defined over the R²

​

The cartesian coordinates introduce the following limitations :

• Intractable for sampling
• Non normalized inputs to the neural network

Confidential - Google DeepMind

We design a coordinate transformation that maps the infinite domain to a finite one

​

​

​

​

​

​

• These new (q, β) coordinates lie in the [0, 1]² domain

Confidential - Google DeepMind

Methods

​

Higher order residuals optimization

Confidential - Google DeepMind

The solutions may have subtle features in their higher derivatives

Wang, Yongji, et al. "Asymptotic Self-Similar Blow-up Profile for Three-Dimensional Axisymmetric Euler Equations Using Neural Networks." arXiv, 2023,

Confidential - Google DeepMind

We add higher order derivatives of the residuals to the optimization objective

Recall that, for one of the equations, the residuals are defined as :

​

​

​

​

​

Then, we optimize the following objective for the given equation:

Czarnecki, Wojciech M. et al. “Sobolev Training for Neural Networks.” Neural Information Processing Systems (2017).

Confidential - Google DeepMind

Methods

​

Second order optimization

Confidential - Google DeepMind

Second-order Optimization : Newton’s method

​

• Approximate h(θ) by its 2nd-order Taylor series around current θ :

​

​

​

• Minimize this local approximation to obtain:

​

​

​

• Update current iterate with this:

Confidential - Google DeepMind

Second-order Optimization : Gauss-Newton

​

Nice properties:

• always positive semi-definite (stable curvature)

Following J. Martens, JMLR, 2020:

• The Generalized Gauss-Newton matrix (GGN) is defined as

​

​

where

H_l is the Hessian of l w.r.t. θ

and J_f is the Jacobian f w.r.t. θ.

• For the square loss H_l is equal to the identity matrix

Confidential - Google DeepMind

Second order optimization converges to significantly lower errors

Confidential - Google DeepMind

Methods

​

Adaptive sampling

Confidential - Google DeepMind

Adaptive sampling to capture the error spikes

In order to capture fine error spikes, we sample the collocation points wrt the a distribution, defined as the absolute max residuals across equations.

Confidential - Google DeepMind

Methods

​

Multi-stage training

Confidential - Google DeepMind

Optimization steps

Wang et al, Multi-stage neural networks: Function approximator of machine precision

Journal of Computational Physics, 2024

• Second-order methods improve errors up to O(1e-8)

​

​

• Then, multi-staging methods lead to O(1e-13) errors.

Confidential - Google DeepMind

Second stage training enables 10^-13 errors

Confidential - Google DeepMind

Multi-stage training algorithm (simplified)

Wang et al, Multi-stage neural networks: Function approximator of machine precision

Journal of Computational Physics, 2024

1. Train the first neural network u₀(x) with weight initialization
2. Calculate the residual r₁(x, u₀)
3. Estimate the dominant frequency and scale of r₁
4. Define u₁(x) = u₀(x) + ε₁ u₁(x, k₁)
5. Plug u₁(x) into the governing equations
6. Train the neural network to fit u₁(x, k₁).

Confidential - Google DeepMind

Several other methods on the preprint

“Discovery of Unstable Singularities”

Preprint

Blog

Confidential - Google DeepMind

Thank you !

Confidential - Google DeepMind