SEMINAR WEB ENGINEERING (WS 2020/2021)�DATA AUGMENTATION

Data Augmentation

​

Model Checkpointing

• What are model checkpoints?
• Saved snapshots of model state during training to resume or restore later.
• Basic concept: Like saving your game progress - stores everything about your model at a point in time

Introduction

Source: Medium article

• Real-world analogy: Think of checkpoints like taking snapshots on your phone - you can always go back to that exact moment.
• Why we need them: Training can take days - imagine losing progress due to power failure!

What is a Checkpoint?

Model weights??

• The actual numbers your model has learned.
• Think of them as your model's "memory".
• Stored as large arrays of floating-point numbers.

What's Inside a Checkpoint?

Source: Medium

• Stores momentum and velocity for optimizers like Adam
• Important because it helps your model learn smoothly
• Example: Adam keeps track of past gradients to decide future updates

Optimizer state

• Current epoch number (which training round you're on)
• Current learning rate
• Best accuracy achieved so far
• Like keeping track of your high score in a game

Training progress

How to Save Checkpoints?

• After every epoch (might waste storage)
• Every N epochs (e.g., every 5 epochs)
• When you hit a new "high score" (best validation accuracy)

When to save?

• Each checkpoint might be 500MB or larger!
• Solution: Keep only last 3 checkpoints + best one!

​

Naming convention:

• Bad: 'model.pth'
• Good: 'model_epoch10_acc85.3.pth'

Storage problems

• Always verify your checkpoint after saving
• Keep backup checkpoints

Corrupted saves

Demo