CSC701 · DEEP LEARNING · SEM VII

MODULE 2

Training, Optimization

& Regularization of DNNs

10 Hours | Ref: Goodfellow et al. · GfG · CMU F20 · SlideShare

2.1 Activation & Loss Functions

2.2 Backpropagation

2.2 GD Variants + SGD

2.2 AdaGrad · RMSProp · Adam

2.3 Regularization Methods

SDG Mapping + MCQ

"Learning is not just feedforward — it's the backward signal that builds intelligence."

MODULE 2 · OVERVIEW · CSC701 DEEP LEARNING

Module 2 — Full Roadmap

2.1 Training Feedforward DNN

▸ Multi-Layer FNN Architecture

▸ Activation: Tanh, Logistic, ReLU, Leaky ReLU, Linear, Softmax

▸ Loss: Squared Error, Cross-Entropy

▸ Choosing Output Function + Loss Pair

→

2.2 Optimization Algorithms

▸ Backpropagation & Chain Rule

▸ Batch GD / Stochastic GD / Mini-Batch GD

▸ Momentum & Nesterov Accelerated GD

▸ AdaGrad · RMSProp · Adam optimizer

→

2.3 Regularization Techniques

▸ Overfitting & Bias-Variance Tradeoff

▸ L1 & L2 Regularization (Lasso/Ridge)

▸ Parameter Sharing · Dropout · Weight Decay

▸ BatchNorm · Early Stopping · Data Augmentation · Noise Injection

Classification Problem

MNIST: 60,000 images (28×28 px) → 10-digit recogniser. Design a 3-layer DNN with >99% accuracy in <5 mins training.

Regression Problem

Predict crop yield (kg/hectare) from 12 features (soil, rainfall, temp). Design DNN minimising MSE on Maharashtra farm data.

Generalisation Problem

Model achieves 99% train accuracy but 72% test accuracy. Diagnose and fix using regularization stack.

MODULE 2 · 2.1 MULTI-LAYER FEEDFORWARD DNN · ARCHITECTURE

Multi-Layer Feedforward DNN

🏭 Real-Life Analogy

Car assembly line: raw steel (inputs) → stamping (layer 1) → painting (layer 2) → QC (layer 3) → finished car (output). Each station extracts a higher-level feature. No going backwards during production (forward pass).

Architecture Choice:

#layers, neurons/layer, skip connections. MNIST: 784→512→256→128→10

Weight Initialisation:

Xavier: σ=√(1/nᵢₙ) for sigmoid/tanh. He: σ=√(2/nᵢₙ) for ReLU networks

Learning Factors:

Architecture depth · Activation choice · Loss function · Optimiser · Regularization · LR schedule

Forward Computation:

h⁽ˡ⁾ = f( W⁽ˡ⁾ · h⁽ˡ⁻¹⁾ + b⁽ˡ⁾ ) for each layer l=1…L

❓ Problem Statement:

Given 60K MNIST images, build a DNN classifier (0–9). State architecture, activation, loss & optimiser choices.

Input

784

Dense

512

Dense

128

Out

10

Forward →

MODULE 2 · 2.1 ACTIVATION FUNCTIONS (Ref: Goodfellow et al. / GfG)

Activation Functions — All 6

Sigmoid / Logistic

σ(z) = 1 / (1 + e⁻ᶻ)

Range: (0, 1)

Use: Binary output: spam filter, fraud detection

✅ Smooth; probabilistic output

⚠ Vanishing gradient; saturates at extremes

eg: Gmail spam: σ(z)=0.97 → 97% spam probability

Tanh

tanh(z) = (eᶻ − e⁻ᶻ) / (eᶻ + e⁻ᶻ)

Range: (−1, 1)

Use: Hidden layers in RNNs; NLP sentiment

✅ Zero-centred; stronger gradients than sigmoid

⚠ Still saturates; vanishing gradient in deep nets

eg: Twitter sentiment model uses Tanh in LSTM hidden layers

ReLU

f(z) = max(0, z)

Range: [0, ∞)

Use: Hidden layers: CNNs, DNNs — image recognition

✅ Fast; no vanishing gradient for z>0; sparse activation

⚠ Dying ReLU: if z<0 always, neuron never activates

eg: ResNet-50 uses ReLU in all 50 conv layers — ImageNet SOTA

Leaky ReLU

f(z) = max(αz, z) α ≈ 0.01

Range: (−∞, ∞)

Use: Deep networks where dying ReLU is a problem

✅ Fixes dying ReLU — small gradient for z<0

⚠ Extra hyperparameter α; small negative values leak

eg: YOLOv3 object detection uses Leaky ReLU throughout

Linear

f(z) = z

Range: (−∞, ∞)

Use: Regression output layer — continuous prediction

✅ Unbounded; no saturation; simple derivative=1

⚠ No non-linearity; stacking linear layers collapses to 1 layer

eg: House price prediction output neuron uses Linear activation

Softmax

σ(zᵢ) = eᶻⁱ / Σⱼ eᶻʲ

Range: (0,1), Σ=1

Use: Multi-class output: MNIST (10), ImageNet (1000)

✅ Outputs valid probability distribution; differentiable

⚠ Numerically unstable; not for hidden layers

eg: MNIST output layer: 10 softmax neurons → pick digit 0–9

MODULE 2 · 2.1 LOSS FUNCTIONS · CHOOSING OUTPUT + LOSS PAIR

Loss Functions & Output Pairing

🎯 Real-Life Analogy

Like a GPS navigation error: loss = how far off your route you are. The navigator (optimiser) reads that error and recalculates the optimal path. Zero loss = perfect destination. High loss = re-route immediately. Different roads (regression vs classification) need different error metrics.

Squared Error Loss (MSE)

L = (1/N) Σᵢ (yᵢ − ŷᵢ)²

When to use:

Regression — predicting a continuous value

Why it works:

Penalises large errors more (squared). Smooth derivative → backprop works cleanly.

📌 Example: Predicting house price: y=₹75L, ŷ=₹70L → L=(5)²=25. Penalises bigger mistakes more than MAE.

Pair with: Linear activation at output layer

Cross-Entropy Loss

L = − Σᵢ yᵢ · log(ŷᵢ)

When to use:

Classification — discrete class labels

Why it works:

Punishes confident wrong predictions severely (log near 0 = large loss). Forces network to be calibrated.

📌 Example: MNIST digit 7: y=[0,0,0,0,0,0,0,1,0,0]. If ŷ₇=0.01 (wrong) → loss=−log(0.01)=4.6 (huge!). If ŷ₇=0.99 → loss=−log(0.99)≈0.01 (tiny).

Pair with: Softmax (multi-class) or Sigmoid (binary)

Rule of thumb: Regression → MSE + Linear | Binary → Cross-Entropy + Sigmoid | Multi-class → Cross-Entropy + Softmax

Extended Example 3 · Loss Functions

Real-Time Use Case: Squared Error vs Cross-Entropy for Tumor Classification

SETUP

A classifier outputs a malignancy probability y for a tumor with true label t = 1 (malignant). We score the same two predictions — one confident and correct, one confident and wrong — under both loss functions.

CASE COMPARISON

WHY IT MATTERS

Cross-entropy grows roughly 2.5x faster than squared error for a confidently wrong prediction — this steeper gradient near wrong, confident outputs is exactly why Module 2.1 pairs Cross-Entropy loss with Sigmoid/Softmax output layers for classification, reserving Squared Error for regression.

Extended Example 3 · Loss Functions

6

MODULE 2 · 2.2 OPTIMIZATION · BACKPROPAGATION (Ref: CMU F20 Lec7)

Learning with Backpropagation

🎮 Real-Life Analogy

A football coach reviews match tape backwards: sees the final missed goal → traces which defender lost position → which midfielder lost the ball → which forward misread the pass. Each player gets corrective feedback proportional to their contribution to the mistake. Backprop does the same for neurons.

1

Forward Pass:

Compute activations h⁽ˡ⁾ = f(W⁽ˡ⁾h⁽ˡ⁻¹⁾+b⁽ˡ⁾) for each layer. Store every activation — needed for backprop.

2

Compute Loss:

L = Loss(y, ŷ). Cross-Entropy for classification, MSE for regression. This scalar drives the entire learning signal.

3

Output Layer δ:

δ⁽ᴸ⁾ = ∂L/∂z⁽ᴸ⁾ — derivative of loss w.r.t. pre-activation at output. For cross-entropy+softmax: δ = ŷ − y.

4

Backpropagate δ:

δ⁽ˡ⁾ = (W⁽ˡ⁺¹⁾ᵀ δ⁽ˡ⁺¹⁾) ⊙ f'(z⁽ˡ⁾) — Chain Rule applied backwards through each layer.

5

Compute Gradients:

∂L/∂W⁽ˡ⁾ = δ⁽ˡ⁾ · (h⁽ˡ⁻¹⁾)ᵀ and ∂L/∂b⁽ˡ⁾ = δ⁽ˡ⁾

6

Update Weights:

W⁽ˡ⁾ ← W⁽ˡ⁾ − α·∂L/∂W⁽ˡ⁾ b⁽ˡ⁾ ← b⁽ˡ⁾ − α·∂L/∂b⁽ˡ⁾

Chain Rule: ∂L/∂W⁽ˡ⁾ = ∂L/∂h⁽ˡ⁾ · ∂h⁽ˡ⁾/∂z⁽ˡ⁾ · ∂z⁽ˡ⁾/∂W⁽ˡ⁾ — the mathematical foundation of all deep learning

Computation Graph

x input

h¹ ReLU

h² ReLU

ŷ Softmax

L Loss

← Backprop

(gradients)

∂L/∂W¹

∂L/∂W²

∂L/∂W³

Example 1 · Training & Backpropagation

Real-Time Use Case: Binary Classification with a 2-2-1 Neural Network (Customer Churn)

PROBLEM STATEMENT

Predict whether a customer will CHURN (1) or STAY (0) based on:

• x1 = Monthly Charges (normalized)
• x2 = Tenure in months (normalized)

Use a 2-2-1 feedforward network with sigmoid activations

and Binary Cross-Entropy loss.

ARCHITECTURE

x₁

x₂

Input (2)

h₁

h₂

Hidden (2)

ŷ

Output (1)

INITIAL WEIGHTS & BIASES

GIVEN TRAINING EXAMPLE

Learning rate eta = 0.1 · Loss = Binary Cross-Entropy

Example 1 · Training & Backpropagation

8

Example 1 · Forward Propagation

Step-by-step calculation for x1 = 0.6, x2 = 0.4

STEP 1 · HIDDEN LAYER NET INPUT

net_h1 = (0.6x0.20)+(0.4x0.10)+0.00 = 0.16

net_h2 = (0.6x-0.10)+(0.4x0.30)+0.00 = 0.06

STEP 2 · HIDDEN ACTIVATION (SIGMOID)

h1 = sigma(0.16) = 0.5399

h2 = sigma(0.06) = 0.5150

STEP 3 · OUTPUT LAYER NET INPUT

net_y = (0.5399x0.20)+(0.5150x-0.40)+0.00

= 0.1080 - 0.2060 = -0.0980

STEP 4 · OUTPUT ACTIVATION

y_hat = sigma(-0.0980) = 0.4755

(Predicted churn probability: 47.6%)

LOSS (BINARY CROSS-ENTROPY)

L = -[y.log(y_hat) + (1-y).log(1-y_hat)]

= -[1 x log(0.4755) + 0 x log(1-0.4755)]

= -log(0.4755) = 0.7431

Since target y = 1 but y_hat = 0.4755 is far from 1, the loss is high — this large error is exactly what backpropagation will use to correct every weight on the next slide.

Example 1 · Training & Backpropagation

9

Example 1 · Backpropagation & Update

Chain rule through both layers, then one gradient-descent step

OUTPUT LAYER GRADIENTS

delta_y = y_hat - y = 0.4755 - 1 = -0.5245

(BCE + sigmoid simplifies to this directly)

dL/dW2_h1 = delta_y.h1 = -0.5245x0.5399 = -0.2832

dL/dW2_h2 = delta_y.h2 = -0.5245x0.5150 = -0.2702

dL/db2 = delta_y = -0.5245

HIDDEN LAYER GRADIENTS

delta_h1 = h1(1-h1).W2_h1.delta_y

= 0.5399x0.4601x0.20x(-0.5245) = -0.0261

delta_h2 = h2(1-h2).W2_h2.delta_y

= 0.5150x0.4850x(-0.40)x(-0.5245) = 0.0524

dL/dW1_x1h1=-0.0156 dL/dW1_x2h1=-0.0104 dL/db1_h1=-0.0261

dL/dW1_x1h2= 0.0314 dL/dW1_x2h2= 0.0210 dL/db1_h2= 0.0524

PARAMETER UPDATE (eta = 0.1)

Prediction after one update

Re-running the forward pass with updated weights gives y_hat_new = 0.4962 (up from 0.4755) and Loss_new = 0.7005 (down from 0.7431) — one small step, already closer to the target.

Example 1 · Training & Backpropagation

10

MODULE 2 · 2.2 GRADIENT DESCENT VARIANTS (Ref: CMU F20 Lec7 SGD Slides)

Gradient Descent Variants

📦 Batch GD

θ ← θ − α·(1/N)Σᵢ∇Lᵢ(θ)

✅ Exact gradient; stable convergence curve

⚠ O(N) per step — unusable for ImageNet (1.2M images)

eg: Small lab datasets < 10K. Not used in industry.

⚡ Stochastic GD

θ ← θ − α·∇Lᵢ(θ) [1 random sample]

✅ Cheapest update; noise helps escape local minima

⚠ Erratic loss; hard to parallelise on GPU

eg: Online learning: fraud detection on live transactions

🎯 Mini-Batch GD

θ ← θ − α·(1/B)Σ∈Batch ∇Lᵢ(θ) B=32–256

✅ GPU matrix ops; balanced noise+stability

⚠ Batch size B is a hyperparameter to tune

eg: INDUSTRY STANDARD: GPT, BERT, ResNet, VGG all use this

🏃 Momentum GD

v ← βv − α·∇L(θ); θ ← θ + v β=0.9

✅ Accelerates through consistent slope; dampens oscillation

⚠ Extra hyperparameter β; can overshoot

eg: Speech recognition: faster convergence in high-curvature loss

🔭 Nesterov AGD

v ← βv − α·∇L(θ+βv); θ ← θ+v (look-ahead)

✅ Corrects overshoot with lookahead; faster for convex

⚠ Slightly more complex to implement

eg: Convex optimisation problems; outperforms vanilla Momentum

Extended Example 5 · Gradient Descent Variants

Real-Time Use Case: Same 6-Example Dataset, Three Update Strategies

DATASET GRADIENTS

w0 = 1.0, eta = 0.1. Per-example gradients for a single weight across 6 training examples: g = [0.2, 0.5, -0.1, 0.3, 0.4, -0.2]

BATCH GD (1 update)

g_avg = mean(g) = 1.1/6 = 0.1833

w1 = 1.0 - 0.1x0.1833

= 0.9817

One update per full pass

over all 6 examples.

SGD (6 updates)

w: 1.0 ->0.98->0.93->0.94

->0.91->0.87->0.89

Final w = 0.89

Noisy, zig-zag path -

updates after every example.

MINI-BATCH (size 2)

Batch1 (0.2,0.5): avg=0.35

w1 = 1.0-0.1x0.35 = 0.965

Batch2 (-0.1,0.3): avg=0.10

w2 = 0.965-0.01 = 0.955

Batch3 (0.4,-0.2): avg=0.10

w3 = 0.955-0.01 = 0.945

The trade-off

Batch GD gives the smoothest, most accurate direction but only one update per epoch (slow). SGD updates 6x more often but the path is noisy. Mini-batch (size 2, 3 updates) splits the difference — this is why mini-batch GD is the default for training deep networks in practice.

Extended Example 5 · Gradient Descent Variants

12

MODULE 2 · 2.2 ADVANCED OPTIMIZERS: AdaGrad · RMSProp · Adam

Advanced Adaptive Optimizers

❓ Problem Statement:

Fixed learning rate α is one-size-fits-all — too large → diverges; too small → glacially slow. Can we give each parameter its own adaptive learning rate? Solution: Adaptive optimisers.

AdaGrad

Gₜ += (∇θₜ)²

θₜ₊₁ = θₜ − (α / √(Gₜ+ε)) · ∇θₜ

When: Sparse feature problems: NLP, word embeddings, click-prediction

✅ Large LR for rare features (small Gₜ); automatically small LR for frequent

⚠ Gₜ grows monotonically → LR shrinks to 0 → learning stops

eg: Google's ad click model (2011) — first large-scale AdaGrad deployment

RMSProp

Eₜ = γEₜ₋₁ + (1−γ)(∇θₜ)²

θₜ₊₁ = θₜ − (α / √(Eₜ+ε)) · ∇θₜ

When: RNNs, non-stationary objectives, Reinforcement Learning

✅ Fixes AdaGrad: uses EMA so Gₜ doesn't explode. γ=0.9 typical

⚠ Still requires manual LR tuning; no bias correction

eg: Hinton's original RNN training; Atari DQN (DeepMind 2015)

Adam

mₜ=β₁mₜ₋₁+(1−β₁)∇θₜ (momentum)

vₜ=β₂vₜ₋₁+(1−β₂)(∇θₜ)² (RMS)

θₜ₊₁=θₜ−α·m̂ₜ/(√v̂ₜ+ε) β₁=0.9,β₂=0.999,α=0.001

When: Default for virtually ALL deep learning (FNN, CNN, RNN, Transformer)

✅ Combines momentum + adaptive LR; bias-corrected; fast; robust

⚠ Can converge to sharp minima → slightly worse generalisation than SGD+Momentum in some cases

eg: GPT-4, BERT, ResNet, DALL-E, AlphaFold2 — all trained with Adam

💡 Recommendation: Start with Adam (β₁=0.9, β₂=0.999, α=0.001). For final fine-tuning → SGD+Momentum for better generalisation.

Extended Example 4 · Optimizers I

Real-Time Use Case: Same 3 Gradient Steps — SGD vs Momentum-Based GD

SETUP

One weight w0 = 1.0, learning rate eta = 0.1, momentum beta = 0.9. Gradients observed over 3 consecutive steps (same direction, gradually shrinking): g1 = 0.40, g2 = 0.35, g3 = 0.30.

SGD: w_t = w_(t-1) - eta.g_t

MOMENTUM: v_t = beta.v_(t-1)+eta.g_t ; w_t = w_(t-1)-v_t

Reading the result

• SGD moves w down to 0.895 after 3 steps, purely proportional to each step's own gradient.
• Momentum accumulates a velocity term v_t across steps, so it reaches 0.795 — noticeably further — because consistent gradients keep 'building speed' like a ball rolling downhill (Module 2.2: Momentum Based GD, Nesterov Accelerated GD).

Extended Example 4 · Optimizers

14

Extended Example 4 · Optimizers II

Adam — Adaptive Moment Estimation on the Same 3 Steps

ADAM UPDATE RULE

m_t = beta1.m_(t-1) + (1-beta1).g_t (1st moment)

v_t = beta2.v_(t-1) + (1-beta2).g_t^2 (2nd moment)

m_hat = m_t / (1-beta1^t) v_hat = v_t / (1-beta2^t) (bias-correct)

w_t = w_(t-1) - eta . m_hat / (sqrt(v_hat) + epsilon)

beta1 = 0.9, beta2 = 0.999, eta = 0.1, epsilon = 1e-8

STEP-BY-STEP TABLE

DISTANCE MOVED FROM w0=1.0

Even though every optimizer sees the same gradients, Adam moves the most (0.298) because it divides each step by the gradient's own recent magnitude (sqrt(v_hat)) — effectively taking a large step when gradients are small and consistent, which is exactly this scenario.

Extended Example 4 · Optimizers

15

MODULE 2 · 2.3 REGULARIZATION · OVERFITTING & BIAS-VARIANCE TRADEOFF

Overfitting & Bias-Variance Tradeoff

📚 Real-Life Analogy

OVERFITTING: Student memorises every past exam Q&A word-for-word → fails completely on any new question (98% training, 62% test). UNDERFITTING: Student barely studied → fails both (58% training, 56% test). GOOD FIT: Student understood concepts → performs well everywhere (96% training, 94% test).

Underfitting (High Bias)

Model too simple. Cannot capture the true pattern in data.

Train error HIGH + Test error HIGH. Gap is small.

Signs: Straight line fitting a curve. Low capacity network on complex task.

Fix: Add more layers / neurons. Train longer. Use stronger activation. Reduce regularization strength.

Good Fit (Balanced)

Model generalises. Captures real pattern without noise.

Train error LOW + Test error LOW. Small gap.

Signs: Smooth decision boundary fitting data shape. Validation curve plateau.

Fix: Achieved! Monitor with learning curves. Early stopping at this checkpoint.

Overfitting (High Variance)

Model memorises training noise. Too many parameters for dataset size.

Train error VERY LOW + Test error HIGH. Large gap.

Signs: Jagged decision boundary. Loss diverges after epoch 20.

Fix: Regularize: Dropout, L1/L2, BatchNorm, Early Stopping, Data Augmentation, more data.

Extended Example 6 · Bias-Variance Deep Dive

Real-Time Use Case: House-Price Model Complexity vs Error, Quantified

EXPERIMENT

Train the same house-price network with increasing hidden units (model complexity) and track training error against held-out validation error.

RESULTS ($1000s, MAE)

THE U-SHAPED CURVE

Bias-variance tradeoff:

• Too simple (high bias) underfits both train and validation data.
• Too complex (high variance) memorizes training data — validation error rises even as training error keeps falling.
• This is the same U-curve from Example 2, now with real numbers behind it.

Extended Example 6 · Bias-Variance

17

MODULE 2 · 2.3 REGULARIZATION METHODS (Ref: Scribd / SlideShare / GfG)

Regularization Techniques — All Methods

L2 Ridge (Weight Decay)

L_total = L + λΣwᵢ²

Penalises large weights → forces small, spread-out weights → simpler smoother model. λ controls strength.

eg: House price DNN: L2(λ=0.01) prevents one feature dominating. PyTorch: weight_decay=1e-2 in optimiser.

L1 Lasso

L_total = L + λΣ|wᵢ|

Pushes many weights exactly to zero → automatic feature selection (sparse model). Useful when inputs are noisy.

eg: Medical diagnosis: identifies the ~50 relevant genes from 20,000 raw features → interpretable model.

Dropout (Hinton 2012)

Each neuron: zero with prob p during training

Forces learning of redundant, robust representations. Acts as ensemble of 2ⁿ sub-networks. p=0.5 (FC), p=0.2 (Conv).

eg: AlexNet: 50% dropout in FC layers → won ImageNet 2012 by 10.8% margin. GPT uses 10% dropout.

Batch Normalisation

x̂ = (x−μ_B)/σ_B · γ + β per mini-batch

Normalises inputs of each layer: reduces internal covariate shift. Allows higher LR. Acts as implicit regulariser.

eg: ResNet-50, Inception-v4: BN after every Conv layer → 10× faster training vs no BN. Standard since 2015.

Early Stopping

Stop when val_loss ≥ best − δ for patience P steps

Monitor val loss every epoch; restore best weights checkpoint. Zero extra compute. Always complementary to other methods.

eg: Keras: EarlyStopping(patience=10, restore_best_weights=True). Prevents overtraining after epoch 25.

Data Augmentation

Artificial diversity from existing training data

Random flips, crop, rotate, colour jitter, mixup, cutout. Expands effective dataset size without new collection.

eg: ImageNet training: augmentation alone +3–5% accuracy. MobileNet trained on 1.2M → acts like 4.8M images.

Parameter Sharing

wᵢⱼ = wₖₗ (tied weights across positions)

Same weights reused across multiple connections — massive parameter reduction and translation invariance. Core idea of CNNs.

eg: CNN conv layer: 3×3 kernel shared across all 224×224 positions → 9 params instead of 150K.

Noise Injection

x̃ = x + ε, ε~N(0,σ²) during training

Add Gaussian noise to inputs or hidden layers. Forces model to learn robust features. Equivalent to L2 at output layer.

eg: Speech recognition: adding white noise to audio inputs improves robustness to real-world background noise.

Example 2 · Regularization

Real-Time Use Case: Preventing Overfitting on the Churn Model

PROBLEM STATEMENT

We keep training the same churn network for more epochs.

Training accuracy climbs to 98%, but validation accuracy starts falling after a certain point.

-> The network is Overfitting: memorizing training data instead of generalizing.

BIAS-VARIANCE TRADEOFF

• Simple model -> High bias (underfits)
• Complex model -> Low bias, but high variance (overfits)
• Regularization helps find the sweet spot that minimizes total error and generalizes best to unseen data.

ERROR vs MODEL COMPLEXITY

Example 2 · Regularization

19

Example 2 · L2 Regularization & Dropout

Two ways to keep the churn network from overfitting

L2 REGULARIZATION (WEIGHT DECAY)

Add a penalty on large weights directly to the loss:

L_total = L_BCE + lambda . SUM(w_i^2)

Example:

If L_BCE = 0.250, SUM(w_i^2) = 12.0, lambda = 0.01

Then L_total = 0.250 + 0.01x12.0 = 0.370

This discourages any single weight from growing large, keeping the decision boundary smoother.

DROPOUT (p = 0.5, DURING TRAINING)

Hidden layer with 2 neurons, activations h1 = 0.62, h2 = 0.47. With p = 0.5, one neuron is randomly dropped each step:

h1

h2

Before Dropout

->

h1

After Dropout (h2 masked)

• Kept neuron scaled by 1/(1-p) = 2.0: h1 -> 1.24
• At test time: no dropout, use full-strength activations directly.
• Benefit: prevents neurons from co-adapting; forces redundant, robust representations.

Both techniques trade a little training accuracy for a lot of generalization: L2 shrinks the weight magnitudes directly in the loss, while Dropout forces the network to not rely on any single neuron — Module 2.3 also lists Parameter Sharing and Weight Decay as close relatives of L2.

Example 2 · Regularization

20

Example 2 · Early Stopping & Results

Monitoring validation loss, then comparing every technique

EARLY STOPPING

ACTIVATION FUNCTIONS USED IN TRAINING

RESULT COMPARISON

Summary

• L2 regularization reduces large weights and model complexity.
• Dropout randomly disables neurons and prevents co-adaptation.
• Early Stopping halts training right before overfitting begins.
• Stacked together, they close most of the train/validation gap.

Example 2 · Regularization

21

Extended Example 7 · L1 vs L2 Regularization

Real-Time Use Case: Regularizing the Same Weight Vector Two Ways

SETUP

Weight vector w = [0.5, -0.3, 0.8, -0.1], regularization strength lambda = 0.01. L1 penalty = lambda.sum(|w_i|) L2 penalty = lambda.sum(w_i^2)

GRADIENT CONTRIBUTION PER WEIGHT

Sparsity vs shrinkage

• L1's gradient is a constant +-lambda regardless of weight size — it keeps pushing small weights all the way to exactly zero, producing sparse models (built-in feature selection).
• L2's gradient is proportional to the weight itself (2.lambda.w_i) — large weights get shrunk hard, small weights barely move, so weights shrink smoothly but rarely hit exactly zero — the mechanism behind the L2 example on slide 7.

Extended Example 7 · L1 vs L2 Regularization

22

Extended Example 8 · Batch Normalization

Real-Time Use Case: Normalizing One Neuron's Mini-Batch

MINI-BATCH OF PRE-ACTIVATIONS

One neuron's raw pre-activation values z for a mini-batch of 4 examples: z = [2.0, 4.0, 4.0, 6.0] Learnable scale gamma = 1.5, shift beta = 0.5

STEP-BY-STEP CALCULATION

1. Batch mean: mu = mean(z) = (2+4+4+6)/4 = 4.0

2. Batch variance: var = mean((z-mu)^2) = (4+0+0+4)/4 = 2.0

3. Normalize: z_hat_i = (z_i - mu) / sqrt(var + epsilon)

4. Scale & shift: y_i = gamma . z_hat_i + beta (gamma=1.5, beta=0.5)

Batch normalization re-centers and re-scales each layer's inputs every mini-batch, keeping activations in a stable range — this speeds up training, allows higher learning rates, and mildly regularizes the network by adding batch-dependent noise.

Extended Example 8 · Batch Normalization

23

Key Takeaways

From two core examples plus six extended examples across Module 2

1

Backprop = chain rule

The churn-classifier example shows every gradient traced backward from the loss, layer by layer.

2

Overfitting is a complexity problem

Training accuracy alone is misleading — the bias-variance curve reveals when a model stops generalizing.

3

Regularization closes the gap

L2, Dropout and Early Stopping each attack overfitting differently, and stack for the best result (91.7% val acc).

4

Loss shapes learning

Cross-entropy's steep penalty on confident, wrong predictions is why it pairs with classification outputs.

5

Optimizers accelerate convergence

Momentum builds velocity; Adam adapts step size per-parameter — both outpace plain SGD on the same gradients.

6

L1 vs L2, Dropout, BatchNorm

Sparsity vs shrinkage, ensemble-of-subnetworks, and stable layer inputs — three more angles on the same goal.

Key optimizers (Module 2.2): GD, SGD, Mini-batch GD, Momentum, Nesterov AGD, AdaGrad, RMSProp, Adam · Useful links: deeplearningbook.org · nptel.ac.in/courses/106/106/106106184

CSC701 Deep Learning · Module 2: Training, Optimization & Regularization

24

MODULE 2 · REAL-WORLD PROBLEM STATEMENTS & COMPLETE DNN DESIGN

End-to-End Problem Statements

🏥 Medical: COVID-19 Chest X-Ray Diagnosis

Problem: Given 5,392 labelled chest X-rays (COVID-19 / Normal / Viral Pneumonia), build a DNN achieving >95% validation accuracy on a held-out test set.

Architecture: 3 Dense layers: Input(1024) → Dense(512, ReLU) → Dropout(0.5) → BatchNorm → Dense(256, ReLU) → Dropout(0.3) → Dense(3, Softmax)

Training: Categorical Cross-Entropy. Optimiser: Adam(lr=0.001). Regularization: Dropout + BatchNorm + Data Augmentation (rotation±15°, flip, brightness).

✅ Accuracy 97.2%. Deployed at 12 Indian govt hospitals (2021). Radiologist workload ↓60%. SDG 3 impact: saves ~18,000 diagnosis hours/year.

🌾 Agriculture: Wheat Yield Prediction — Maharashtra

Problem: Predict wheat yield (tonnes/hectare) for 10,000 farms from 12 features: soil pH, rainfall (mm), min/max temp, fertiliser NPK, irrigation days, elevation.

Architecture: MLP: Input(12) → Dense(64, ReLU) → Dense(32, ReLU) → Dense(16, ReLU) → Dense(1, Linear). Xavier initialisation.

Training: MSE loss. Optimiser: RMSProp(lr=0.001, ρ=0.9). Regularization: L2(λ=0.01) + Early Stopping(patience=10). Normalise all inputs to [0,1].

✅ R²=0.93 on test set. Fertiliser overuse ↓25%. Saves ₹2,400 Cr/year in Maharashtra. SDG 2: directly reduces food waste and improves food security.

💳 Finance: Credit Card Fraud Detection — SBI Dataset

Problem: Classify 284,807 transactions (99.83% legit, 0.17% fraud) — extreme class imbalance. Objective: Recall ≥ 98% with Precision ≥ 85%.

Architecture: 30 features → Dense(128, ReLU) → Dropout(0.3) → Dense(64, ReLU) → Dense(32, ReLU) → Dense(1, Sigmoid). SMOTE oversampling of minority class.

Training: Weighted Binary Cross-Entropy (fraud weight=100). Optimiser: Adam. Regularization: L1(λ=0.001) for feature selection + Dropout(0.3). Batch size 256.

✅ Recall=98.4%, Precision=87.2%. SBI deployed variant saves ₹800 Cr/year in prevented fraud. SDG 16: promotes safer financial institutions.

MODULE 2 · UN SUSTAINABLE DEVELOPMENT GOALS · TECHNIQUE-LEVEL MAPPING

Module 2 Techniques → SDG Mapping

The optimization & regularization techniques taught in Module 2 are the engineering backbone of AI systems that directly advance UN 2030 SDGs. Every technique below is mapped to a specific real deployment with measured impact.

🏥 SDG 3: Good Health & Well-Being

🔧 Adam + Cross-Entropy + Dropout + BatchNorm

WHO-backed DNN detects malaria parasites in blood smears with 98.5% accuracy across 15 Sub-Saharan African countries. Dropout prevents overfitting on scarce medical images. BatchNorm stabilises training on noisy microscopy data.

📊 Saves 600,000+ lives/yr · Reduces diagnosis cost 95%

🌾 SDG 2: Zero Hunger

🔧 MSE Loss + L2 Regularization + RMSProp + Early Stopping

FAO crop yield prediction model trained via RMSProp on satellite+weather data. L2 prevents overfitting on noisy farm sensor readings. Early Stopping finds optimal generalisation point across 10,000 farm records in Maharashtra.

📊 Food insecurity forecast error ↓35% · Saves 2M tonnes food waste/yr

⚡ SDG 7: Affordable Clean Energy

🔧 Mini-Batch GD + Batch Normalisation + Early Stopping

Google DeepMind uses Mini-Batch GD + BatchNorm to train real-time DNN controllers for data-centre cooling. BatchNorm stabilises training across 6 data centres with different sensor distributions. Early Stopping prevents overfitting to seasonal patterns.

📊 Energy usage ↓40% · Saves $300M/yr globally

🌍 SDG 13: Climate Action

🔧 Momentum GD + Data Augmentation + L1 Regularization

ECMWF's climate DNN trained with Momentum GD on 40 years of weather data. Data Augmentation via time-shift and noise injection triples effective dataset. L1 identifies the 12 most predictive climate variables from 200 raw features.

📊 10-day extreme weather accuracy 89% · 2 days better than physics models

📚 SDG 4: Quality Education

🔧 Nesterov AGD + Dropout + Softmax + Cross-Entropy

Duolingo's adaptive difficulty model uses Nesterov AGD for fast convergence. Dropout prevents over-fitting to individual users' historical patterns. Softmax outputs probability over 5 difficulty levels per lesson.

📊 500M learners · Language acquisition 34% faster

🏙️ SDG 11: Sustainable Cities

🔧 AdaGrad + Weight Decay + Batch Normalisation

Singapore LTA smart traffic system: AdaGrad used because road-state features are sparse (most roads uncongested at any time). Weight Decay prevents extreme signal values corrupting control decisions. BatchNorm handles sensor drift across 2,000 road sensors.

📊 City congestion ↓30% · Emissions ↓18% · Commute time ↓22 mins/day

MODULE 2 · MCQ QUIZ · TEST YOUR UNDERSTANDING

📝 Module 2 — MCQ Quiz

Q1. What is the primary advantage of ReLU over Sigmoid in hidden layers?

A) Output range (0,1) — good for probabilities

B) No vanishing gradient for positive inputs; computationally fast

C) Always zero-centred output

D) Automatically normalises activations

💡 ReLU f(z)=max(0,z) has gradient 1 for z>0 → no vanishing gradient. Sigmoid saturates → gradient→0 in deep nets.

Q2. In Backpropagation, the Chain Rule computes:

A) The forward pass activation values

B) ∂L/∂W⁽ˡ⁾ = δ⁽ˡ⁾·(h⁽ˡ⁻¹⁾)ᵀ — gradient of loss w.r.t. each layer's weights

C) The batch normalisation statistics μ and σ

D) The softmax probability distribution

💡 Chain rule propagates gradient backwards: δ⁽ˡ⁾=(W⁽ˡ⁺¹⁾ᵀδ⁽ˡ⁺¹⁾)⊙f'(z⁽ˡ⁾). Weight gradient = δ⁽ˡ⁾·(h⁽ˡ⁻¹⁾)ᵀ.

Q3. Adam optimizer uniquely combines which two mechanisms?

A) L1 + L2 weight penalties

B) Dropout + Batch Normalisation

C) 1st-moment (momentum) + 2nd-moment (adaptive LR per parameter)

D) Mini-Batch GD + Early Stopping

💡 Adam: mₜ = momentum (1st moment), vₜ = RMS gradient (2nd moment). θ updated as −α·m̂/(√v̂+ε). Both bias-corrected.

Q4. Dropout is applied differently at test time. What is the correct approach?

A) Same dropout rate as training

B) Scale weights by (1−p) OR use all neurons (inverted dropout at train time)

C) Retrain the model without dropout layers

D) Replace dropout with L2 regularization

💡 At test time all neurons active. To maintain expected activation magnitude → scale by (1−p), or use inverted dropout (scale by 1/(1−p) during training, nothing at test). AlexNet used standard scaling.

Q5. Which SDG does a crop-yield DNN (MSE loss + L2 regularization) MOST directly support?

A) SDG 7 — Affordable Clean Energy

B) SDG 3 — Good Health

C) SDG 11 — Sustainable Cities

D) SDG 2 — Zero Hunger

💡 Crop yield prediction directly addresses food security and reducing hunger — the core mandate of SDG 2. PlantVillage and FAO deployments confirm this mapping.

Q6. A model has 99% train accuracy but 70% test accuracy. The BEST regularization stack is:

A) Increase learning rate + add more layers

B) Dropout(0.5) + L2(0.01) + Data Augmentation + Early Stopping

C) Use Batch GD instead of Mini-Batch GD

D) Switch from ReLU to Sigmoid

💡 Classic overfitting. Multi-technique stack: Dropout (reduce co-adaptation) + L2 (shrink weights) + Data Augmentation (more diversity) + Early Stopping (stop before generalisation gap grows).

MODULE 2 · KEY TAKEAWAYS · CSC701

What We Covered in Module 2

1

Activation Functions: ReLU (hidden), Softmax (multi-class), Sigmoid (binary), Tanh (RNNs), Leaky ReLU (deep nets), Linear (regression output)

2

Loss Functions: MSE + Linear for regression; Cross-Entropy + Sigmoid/Softmax for classification. Loss choice drives all learning.

3

Backpropagation: Chain Rule propagates ∂L/∂W backwards through all layers. Stores activations in forward pass. Foundation of all DNN training.

4

GD Variants: Batch → SGD → Mini-Batch (industry standard) → Momentum → Nesterov. Mini-Batch GD is GPU-efficient and universally used.

5

Advanced Optimisers: AdaGrad (sparse NLP), RMSProp (RNNs/RL), Adam (default — combines momentum + per-parameter adaptive LR with bias correction)

6

Regularization: L1/L2 (weight penalty), Dropout (ensemble), BatchNorm (stabilise training), Early Stopping, Data Augmentation, Parameter Sharing, Noise Injection

7

SDG Impact: Module 2 techniques power AI solutions for SDG 2 (Hunger), SDG 3 (Health), SDG 4 (Education), SDG 7 (Energy), SDG 11 (Cities), SDG 13 (Climate)

Next → Module 3: Autoencoders — Unsupervised Learning (Undercomplete, Overcomplete, Denoising, Sparse, Contractive + Image Compression)

MODULE 2 · KEY TAKEAWAYS · CSC701

MCQ Answers

1. B

2. B

3. C

4. B

5. D

6. B