1 of 29

Scientific Machine Learning

Prof. Seungchul Lee

Industrial AI Lab.

2 of 29

Introduction

2

3 of 29

Course Information for AX50011 (1/2)

  • Course title: Scientific Machine Learning

​

  • Instructor: Prof. Seungchul Lee
    • Office: N7-4, 6102
    • Email: seunglee@kaist.ac.kr

​

  • This course introduces the methodological transition from classical numerical analysis to AI-based scientific computing. Focusing on PDE-based problems, it covers the core principles, training frameworks, limitations, and advanced extensions of Neural ODEs, Physics-Informed Neural Networks, and Operator Learning, together with emerging topics such as hybrid workflows and equation discovery. The course aims to develop AX-oriented AI modeling capabilities for the analysis, prediction, surrogate modeling, and model discovery of complex physical systems.

​

3

4 of 29

Course Information for AX50011 (2/2)

  • Course details
    • Tuesday and Thursday 14:30 ~ 16:00 at N7 2110

​

  • Teaching assistants: TBD

​

  • Office hours: by appointment

​

4

5 of 29

Instructor

  • 2023.09 – present: KAIST

​

  • 2018 – 2023.08: POSTECH

​

  • 2013 – 2017: UNIST

​

  • 2010, Ph.D. from the University of Michigan, Ann Arbor
  • 2008, M.S. from the University of Michigan, Ann Arbor

​

  • 2001, B.S. of Mechanical Engineering from Seoul National University

5

6 of 29

Lecture Materials

6

7 of 29

Course Schedule

7

Week

Contents

Assignments

Week 1

Foundations of Scientific Machine Learning

​

Week 2

Numerical Analysis Fundamentals

HW#01

Week 3

Neural ODEs

HW#02

Week 4

PINN: Introduction

HW#03

Week 5

PINN: Limitations and Extensions

HW#04

Week 6

Operator Learning: DeepONet and Extensions

HW#05

Week 7

Operator Learning: FNO and Extensions

HW#06

Week 8

Midterm Exam

​

Week

Contents

Assignments

Week 9

Introduction to Hybrid Workflows

​

Week 10

Equation Discovery: SINDy

HW#07

Week 11

Transfer Learning and Few-Shot Learning

HW#08

Week 12

Domain Adaptation and Contrastive Learning

HW#09

Week 13

Generative AI: Diffusion Models

HW#10

Week 14

Transformers in Scientific Machine Learning

HW#11

Week 15

Basics of Large Language Models

​

Week 16

Term Project

​

8 of 29

Homework/Exams/Projects

  • Homework
    • Hand-written and programming exercises
    • Due one week after posted
    • No late homework will be accepted

​

  • Exams
    • Will be only one part: hand-written exam

​

  • Projects
    • Provides an opportunity to apply scientific machine learning methods to a physical or engineering problem
    • Formulate a problem, develop an appropriate AI based approach, and analyze the results
    • Aims to deepen students’ understanding of scientific machine learning through practical problem-solving experience

8

9 of 29

Grading

  • Homework 20 %

​

  • Midterm 30 %

​

  • Final 30%

​

  • Project 20%

​

9

10 of 29

Introduction to Scientific Machine Learning

10

11 of 29

A Rapidly Growing Field

  • The number of publications each year

11

Source: Scopus keyword search

12 of 29

Potentials of Data-driven Approach (1/2)

  • Deep learning as a data-driven approach

12

13 of 29

Potentials of Data-driven Approach (2/2)

  • Advantages of data-driven approaches
    • If enough data is available, a legitimate level of prediction performance can be achieved without domain knowledge

​

13

14 of 29

Limitations of Pure Data-driven Approaches

  • Generalizability: out-of-distributed data

​

​

​

​

​

​

​

​

  • Leveraging scientific knowledge for big data analysis increases:
    • Reliability
    • Interpretability
    • Robustness on out-of-distributed data

​

  • Interpretability: scientifically interpretable output

​

14

Interpolation

Extrapolation

Extrapolation

15 of 29

Learning Without Science (1/2)

  • Artificial intelligence only

15

Data-driven AI

16 of 29

Learning Without Science (2/2)

16

17 of 29

Learning With Science (1/2)

  • Artificial intelligence + Science

17

Ex: Physics-informed AI

18 of 29

Learning With Science (2/2)

18

19 of 29

Scientific Machine Learning

  • Major problem
    • Naively using deep learning for scientific tasks usually leads to:
      • Lack of interpretability
      • Poor generalization
      • Lots of training data required

​

​

  • Do neural networks really “understand” the scientific tasks they are being applied to?
  • Solution

19

Source: Mishra, S. & Moseley, B. (2024), AI in the Sciences and Engineering, ETH Zürich, Course 401-4656-21L

Machine Learning

SciML

more powerful, robust,

interpretable models

Scientific Knowledge

20 of 29

Topics

20

21 of 29

Numerical Analysis

  • Analytical solution
    • Simple geometry
    • Linear equation
    • Simple BC/IC
    • Time-independent input
  • Real engineering problem
    • Complex geometry
    • Nonlinearity
    • Complex BC/IC
    • Time-dependent input

21

Numerical approximation 🡪 Approximate solution

21

22 of 29

Neural ODEs

  • Neural ODE defines continuously transforming vector field
    • Estimate the ODE function at a finer step

​

​

​

​

  • Neural ODE operates as

22

 

23 of 29

PINN

​

​

23

  • NN as an universal function approximator

​

  • Given
    • ODE or PDE
    • Initial and boundary conditions

​

  • Aim to approximate the solution of PDEs (target function) by training neural networks

​

​

​

​

[

24 of 29

Operator

  • Function: map numbers (vector) to numbers (vector)
  • Operator: map functions to functions

​

24

25 of 29

Neural Operator

  • Neural networks are capable of learning function-to-function mappings

​

25

26 of 29

DeepONet

  •  

26

27 of 29

FNO

  • The Global Path: captures non-local, long-range dependencies and global periodic patterns in the frequency domain
  • The Local Path: a local linear transformation that preserves and passes local spatial information directly

27

28 of 29

How to Study

28

29 of 29

ML/DL Lecture Materials

  • For those who are unfamiliar with ML/DL:

​

29

Machine Learning

Deep Learning