CSC701 · DEEP LEARNING · MODULE 3
MODULE 3 — UNSUPERVISED LEARNING
Autoencoders
6 Hours | Ref: Goodfellow et al. · GfG · Scribd DL Unit3
Linear Autoencoder
Undercomplete Autoencoder
Overcomplete Autoencoder
Denoising Autoencoder
Sparse Autoencoder
Contractive Autoencoder
Image Compression App
SDG Mapping
MCQ Quiz
"An autoencoder must learn to see the world as it truly is — not memorize, but understand."
MODULE 3 · INTRODUCTION · AUTOENCODERS (Ref: GfG / Goodfellow et al.)
What is an Autoencoder?
Definition (Goodfellow et al.):
An Autoencoder is an unsupervised neural network that learns to copy its input to its output through a compressed intermediate representation called the latent space (or bottleneck). It consists of an ENCODER that maps input x → latent code z, and a DECODER that reconstructs z → x̂. The network learns the most essential features of data without any labels.
📦 Real-Life Analogy
Like a file compression tool (ZIP/RAR): the encoder is the compressor that removes redundancy and stores only essential patterns; the decoder is the extractor that reconstructs the original. The difference: autoencoders LEARN what to keep — they don't use hand-crafted rules like ZIP does.
ENCODER f(x)
Maps high-dimensional input x to low-dimensional latent code z = f(x). Extracts the most important patterns. Implemented as a stack of Dense/Conv layers with non-linear activations.
z = f(x) = σ(W_e · x + b_e)
LATENT SPACE z
The compressed bottleneck representation. Dimensionality << input. Contains only essential information. Also called: code, embedding, hidden representation, or compressed vector.
dim(z) << dim(x)
DECODER g(z)
Reconstructs x̂ from latent code z. Mirror of encoder. Tries to recover original input. Loss = distance between x and x̂ measures reconstruction quality.
x̂ = g(z) = σ(W_d · z + b_d)
Loss: L(x, x̂) = ||x − x̂||² (MSE) or L = −Σ[x·log(x̂) + (1−x)·log(1−x̂)] (Binary Cross-Entropy) — minimised via Backpropagation
MODULE 3 · TYPES OF AUTOENCODERS · OVERVIEW
Taxonomy of Autoencoders
Autoencoder
Basic
Variants
Linear AE
Undercomplete AE
Overcomplete AE
Regularized
Variants
Sparse AE
Denoising AE
Contractive AE
Advanced
Variants
Variational AE (VAE)
Convolutional AE
Stacked AE
Quick Comparison
Type
Latent Size
Regularization
Key Strength
Primary Use Case
Linear AE
< input
None (linear)
Equivalent to PCA
Dimensionality reduction
Undercomplete AE
< input
None
Forces compression
Feature extraction, DR
Overcomplete AE
> input
Requires penalty
High capacity
With sparse/denoising constraints
Sparse AE
Any
L1 / KL Divergence
Feature selection
Classification pre-training
Denoising AE
< input
Input corruption
Robust features
Image/signal denoising
Contractive AE
< input
Jacobian penalty
Stability invariance
Manifold learning
MODULE 3 · 3.1 LINEAR AUTOENCODER + UNDERCOMPLETE AUTOENCODER
Linear & Undercomplete Autoencoders
1. Linear Autoencoder
Definition:
Uses only linear activations (f(z)=z) in both encoder and decoder. No non-linearity.
Mathematical Equivalence:
A linear AE with bottleneck of size k learns the same k-dimensional subspace as PCA. The weight matrices learn the principal components.
Architecture:
Input(n) → Linear(k) → Linear(n). k < n. Loss: MSE = ||x − W_d·W_e·x||²
Limitation:
Cannot capture non-linear structure in data. Strictly linear = exactly PCA. No advantage over classical PCA unless used as initialisation.
When to use:
Baseline comparison. Initialising weights for deeper AE. Understanding the data's linear subspace before adding complexity.
eg: Linear AE on MNIST → learns 32 basis images representing digit strokes (same as PCA basis)
2. Undercomplete Autoencoder
🎯 Analogy
Like summarising a 500-page book into a 2-page synopsis — you must keep only the most essential ideas. The bottleneck forces the network to discard noise and redundancy.
Definition:
Latent dim(z) < Input dim(x). The bottleneck forces the encoder to compress data, extracting only the most informative features.
Architecture:
Input(784) → Dense(256,ReLU) → Dense(32,ReLU) [bottleneck] → Dense(256,ReLU) → Dense(784,Sigmoid)
Loss Function:
L(θ) = (1/N) Σ ||xᵢ − x̂ᵢ||² (MSE). Minimised via Adam/SGD with backpropagation.
Key Property:
Cannot simply copy input → must learn meaningful representations. Bottleneck is an information gate.
eg: Netflix movie embeddings: 500K movies → 50D latent → captures genres, mood, style for recommendation
MODULE 3 · 3.2 OVERCOMPLETE AUTOENCODER
Overcomplete Autoencoder
🗄️ Real-Life Analogy
Like a library with MORE shelves than books — there's space to store each book in multiple ways. The overcomplete AE has more latent neurons than input dimensions. Without constraints, it can just copy the input (trivial identity mapping). Constraints like sparsity or denoising are needed to force meaningful learning.
Definition:
Latent dimension > Input dimension. More neurons in bottleneck than in input. Overcomplete representations can capture more complex, distributed features.
The Problem:
Without regularization, an overcomplete AE learns the identity function — it simply copies input to output without learning anything useful. This is called trivial solution.
The Solution:
Add constraints: Sparse penalty (L1/KL) forces only a few neurons active per input. Denoising corruption forces robust feature learning. Either prevents trivial copying.
When It's Useful:
When the input has complex structure requiring more than d dimensions to represent well. Medical imaging, NLP embeddings, anomaly detection.
Mathematical View:
z ∈ ℝᵐ where m > n (input dim). h = f(Wx + b). Must add: L = L_rec + λ·Ω(h) where Ω is sparsity or Jacobian penalty.
Undercomplete vs Overcomplete
Undercomplete (z < x)
Input
Latent
Output
Overcomplete (z > x)
Input
Latent
Output
❓ Problem Statement:
Build an overcomplete AE (latent=1024) for CIFAR-10 images (dim=3072). With no regularization it scores 0% test improvement. Add L1 sparsity → meaningful features emerge within 10 epochs.
MODULE 3 · REGULARIZATION IN AUTOENCODERS
Regularization in Autoencoders
🧪 Why Regularize an Autoencoder?
Without regularization, an AE can memorise training data (overfit) or learn a trivial identity (overcomplete). Regularization forces the AE to learn USEFUL, GENERALISABLE representations — like forcing a student to explain a concept in their own words instead of reciting it verbatim.
Overcomplete + Sparsity
L = L_rec + λ·Σ|hⱼ| (L1 penalty on activations)
Forces most latent neurons to be zero. Only a few activate per input → learns specialized, interpretable features.
When: When latent > input. Feature selection tasks. Biological plausibility (sparse neural firing).
eg: Sparse AE on face images → each latent neuron specialises: one for nose, one for eyes, one for face shape.
Noise Injection
x̃ = x + ε ε~N(0,σ²). Minimise L(x, g(f(x̃)))
Corrupted input forces encoder to extract ROBUST features invariant to noise. Cannot simply memorise noisy input.
When: Denoising tasks. Improving generalisation. Medical/satellite image processing.
eg: Add 30% Gaussian noise to MRI scans → AE learns to reconstruct clean scan from corrupted input.
Jacobian Penalty
L = L_rec + λ·||∂f(x)/∂x||²_F (Frobenius norm of Jacobian)
Penalises the encoder for being sensitive to input perturbations → latent code is locally invariant → stable representation.
When: Manifold learning. Invariant feature extraction. Robustness to adversarial inputs.
eg: Contractive AE on handwritten digits → latent space varies smoothly even with pen stroke variation.
Tied Weights
W_decoder = W_encoderᵀ (weight matrix transposed)
Forces encoder and decoder to be exact transposes → halves parameters → acts as regularizer by constraining capacity.
When: Small datasets where overfitting is a risk. Symmetric architectures.
eg: Dimensionality reduction on tabular medical data with only 500 samples.
MODULE 3 · 3.3 DENOISING AUTOENCODER (Ref: GfG / Vincent et al. 2008)
Denoising Autoencoder (DAE)
📷 Real-Life Analogy
Like restoring a water-damaged photograph. You see the corrupted version (torn, faded) but you know what the original looked like. The DAE learns: given noisy x̃, reconstruct clean x. Forces learning features robust to corruption.
Core Idea:
Train with corrupted input x̃ but compute loss against CLEAN target x. The AE must recover lost information → forces robust feature learning. Proposed by Vincent et al. (ICML 2008).
Corruption Types:
Gaussian noise: x̃=x+ε (ε~N(0,σ²)). Masking noise: randomly zero p% of pixels. Salt & pepper noise. Dropout noise: randomly zero input dimensions.
Loss Function:
L(θ) = ||x − g(f(x̃))||² — note: loss vs CLEAN x, not corrupted x̃. This is what makes DAE fundamentally different from standard AE.
Mathematical Insight:
DAE learns to implicitly estimate the score function ∂log p(x)/∂x — i.e., the gradient of the data distribution. This is the foundation of Score-based Generative Models.
Stacked DAE:
Multiple DAE layers stacked: output of one DAE becomes input to next. Pre-trains each layer greedily. Powerful initialisation for deep networks before supervised fine-tuning.
❓ Problem Statement:
Given 10,000 chest X-ray images with sensor noise (σ=0.15), design a DAE that achieves PSNR > 35dB on held-out test set.
DAE Training Flow
Clean Input x
Add Noise → x̃
Encoder f(x̃)
Decoder g(z)
Reconstructed x̂
Loss vs
clean x
Real Applications:
▸ MRI scan denoising (deep learning outperforms wavelet filters)
▸ Speech enhancement (remove wind/traffic noise from audio)
▸ Old film restoration (remove grain/scratches)
▸ MRI scan denoising (deep learning outperforms wavelet filters)
▸ Speech enhancement (remove wind/traffic noise from audio)
▸ Old film restoration (remove grain/scratches)
MODULE 3 · 3.4 SPARSE AUTOENCODER (Ref: GfG / Andrew Ng 2011)
Sparse Autoencoder (SAE)
🧠 Real-Life Analogy
Inspired by the human brain: at any moment, only 1–4% of cortical neurons fire simultaneously (sparse neural coding). Sparse AE mimics this: given an image of a face, only the 'eye detector neuron', 'nose neuron', and 'jaw neuron' activate — not all 1,000 hidden neurons. This makes each neuron specialize.
Core Idea:
Allow overcomplete latent space BUT add a sparsity constraint. Most neurons silent for any given input. Forces each active neuron to represent a meaningful, specific feature.
L1 Regularization:
L = ||x−x̂||² + λ·Σ|hⱼ|. L1 pushes activations toward zero. λ controls sparsity strength. Higher λ → sparser (but harder reconstruction).
KL Divergence Penalty:
KL(ρ || ρ̂ⱼ) = ρ·log(ρ/ρ̂ⱼ) + (1−ρ)·log((1−ρ)/(1−ρ̂ⱼ)). Target ρ=0.05. Forces average activation per neuron ≈ 5%.
Loss Function (full):
L = L_rec + β·Σⱼ KL(ρ || ρ̂ⱼ) where ρ=target sparsity, ρ̂ⱼ=mean activation of neuron j across batch.
Feature Specialisation:
Sparse features resemble Gabor filters (edge detectors) on natural images — biologically interpretable. Each feature responds to ONE type of pattern.
❓ Problem Statement:
Train a sparse AE on STL-10 natural images (latent=1024, ρ=0.05). Visualise each latent neuron's preferred pattern. Are they Gabor-like?
Architecture
Input(n) → Dense(m, ReLU) + L1(λ) → Dense(n, Sigmoid)
Where m >> n (overcomplete). L1 applied on hidden activations.
PyTorch: regularizers.l1(1e-5) on encoded layer.
Feature Selection:
Pre-training for classification: sparse features → SVM classifier. Used in protein structure prediction.
Medical Imaging:
Sparse AE detects anomalies in retinal OCT scans: only 3% neurons fire for healthy tissue, abnormal activations flag disease.
NLP Embeddings:
Sparse word embeddings: each word activates <2% of latent neurons → interpretable topic-like features emerge.
Anomaly Detection:
Normal data → sparse code. Anomaly → unusual activation pattern → easy to detect by monitoring activation density.
MODULE 3 · 3.5 CONTRACTIVE AUTOENCODER (Ref: Rifai et al. ICML 2011)
Contractive Autoencoder (CAE)
🗺️ Real-Life Analogy
Like topographic maps: a small change in your GPS position (x) causes only a tiny change in the grid square (z) you're in. CAE enforces this: small perturbations in input x cause negligibly small changes in latent code z. The encoder learns a 'smooth mountain contour' of the data manifold — locally flat = contractive.
Definition:
CAE adds a penalty on the Frobenius norm of the Jacobian of the encoder's activations with respect to input. This forces the latent representation to be locally invariant — small input changes produce tiny latent changes.
Loss Function:
L_CAE = ||x − x̂||² + λ · ||∂h/∂x||²_F where ||∂h/∂x||²_F = Σᵢⱼ (∂hⱼ/∂xᵢ)² = Jacobian Frobenius norm squared.
Efficient Computation:
For sigmoid activations: ||J_f||²_F = Σⱼ hⱼ²(1−hⱼ)² · ||Wⱼ||². Can be computed efficiently using element-wise operations. No need to compute full Jacobian explicitly.
Key Property:
The representation contracts towards the data manifold's tangent directions. Directions of high variance in data → larger Jacobian (informative). Orthogonal directions → near-zero Jacobian (invariant).
vs Sparse AE:
Sparse AE: penalises activation MAGNITUDE. CAE: penalises SENSITIVITY of activations to input. Both achieve robustness but through different mechanisms.
Practical Result:
On face dataset: rotating face by 5° changes latent code by <0.1%. Perfect for identity verification, face recognition under pose variations.
Applications: Face recognition under illumination/pose variation · Manifold learning on molecular conformations · Robust speech feature extraction · Anomaly detection in industrial sensor streams
MODULE 3 · APPLICATION · IMAGE COMPRESSION (Ref: Scribd DL Unit3)
Application: Image Compression
🗜️ Real-Life Analogy
Traditional codecs (JPEG, PNG) use hand-crafted rules (DCT, wavelets). Learned AE compression is like training a translator who learns the SOUL of images rather than just the grammar. The result: better quality at lower bit-rates, especially for domain-specific images (medical, satellite, faces).
How AE Compression Works
1
Encode:
Encoder: img (224×224×3 = 150K pixels) → latent z (512D). 300× compression ratio.
2
Quantize:
Latent codes quantized to 8-bit integers for storage/transmission. Entropy coding reduces size further.
3
Store / Transmit:
Compressed 512D code sent over network or stored. Bandwidth savings up to 99%.
4
Decode:
Decoder: 512D → reconstructed image. Loss = PSNR, SSIM vs original.
❓ Problem Statement:
Design an AE for chest X-ray compression achieving PSNR>38dB at 50× compression. Architecture, loss function, and evaluation protocol?
AE vs Traditional Compression
NASA Satellite:
AE compresses Landsat hyperspectral data 50× while preserving spectral fidelity. Saves $2M/yr in transmission costs.
Medical PACS:
Hospital DICOM image storage: stacked DAE achieves 40× compression with PSNR=50dB on CT scans. HIPAA-compliant.
Google Photos:
Learned image codec (based on AE) outperforms WebP at low bitrates for portrait photos. Deployed at 4B+ users.
Video Streaming:
Netflix uses AE-based perceptual codec. Same visual quality at 35% lower bandwidth → saves $150M/yr in CDN costs.
MODULE 3 · END-TO-END PROBLEM STATEMENTS
End-to-End Problem Statements
🩻 Medical: Retinal OCT Scan Denoising — AIIMS Dataset
Problem: 2,000 retinal OCT scans with speckle noise (σ=0.2). Achieve PSNR>40dB reconstruction. Preserve diagnostic features (drusen, fluid pockets) critical for macular degeneration diagnosis.
Architecture: Denoising AE: Input(512×512×1) → Conv(32,ReLU) → Conv(64,ReLU) → MaxPool [bottleneck] → ConvTranspose(64,ReLU) → ConvTranspose(32,ReLU) → Output(512×512×1, Sigmoid).
Training: Loss: MSE(x, g(f(x̃))). Corruption: add speckle noise σ=0.2 to training scans. Optimiser: Adam(lr=0.0001). 100 epochs, batch=16.
✅ PSNR=41.3dB, SSIM=0.97. Deployed at AIIMS New Delhi. Reduces repeat scan rate by 28%. SDG 3 impact: saves 400 patient-hours/month.
🌍 Environmental: Satellite Hyperspectral Image Compression
Problem: ISRO's Resourcesat-2 generates 800GB/day of hyperspectral data (200 spectral bands). Design AE achieving 50× compression while maintaining spectral fidelity for vegetation/water body classification.
Architecture: Convolutional AE: bands(200) → Spectral_Dense(50,ReLU) → Spatial_Conv2D(32) → bottleneck(z=16 bands equivalent) → ConvTranspose(32) → Dense(200, Linear). Tied decoder weights.
Training: Loss: MSE + 0.1·SSIM loss. Optimiser: Adam(lr=0.001). Training on 10,000 image patches (64×64). 200 epochs.
✅ Compression ratio 50×. Spectral angle mapper error <2°. ISRO uses variant for Chandrayaan-3 data pipeline. SDG 13+15 impact.
💳 Finance: Credit Card Transaction Anomaly Detection
Problem: 284K transactions (99.83% normal, 0.17% fraud). Train unsupervised sparse AE on normal transactions ONLY. Flag anomalies when reconstruction error exceeds threshold.
Architecture: Sparse AE: 30 features → Dense(64,ReLU)+L1(1e-4) → Dense(32,ReLU) → Dense(8,ReLU) [bottleneck] → Dense(32) → Dense(64) → Dense(30,Linear). No fraud labels used in training.
Training: Loss: MSE on normal samples only. Threshold: μ + 2σ of reconstruction errors. Batch=256, Adam, 50 epochs.
✅ Precision=91%, Recall=94% on fraud class. No labelled fraud data needed for training! SBI variant deployed. SDG 16 impact.
MODULE 3 · UN SUSTAINABLE DEVELOPMENT GOALS · AUTOENCODER TECHNIQUE MAPPING
Autoencoders → SDG Mapping
Autoencoders are self-supervised (no labels needed) making them deployable in resource-constrained, data-scarce environments across the developing world. Each AE variant below is mapped to a specific SDG deployment with measured real-world impact.
🏥 SDG 3: Good Health & Well-Being
🔧 Denoising AE + Contractive AE
WHO-deployed denoising AE removes noise from portable ultrasound devices used in rural Sub-Saharan Africa. Contractive AE provides robust embeddings for disease classification under variable scanning conditions. 15 countries, 480 clinics.
📊 Diagnostic accuracy ↑38% for portable devices · 600K patients/yr
🌾 SDG 2: Zero Hunger
🔧 Sparse AE + Convolutional AE
FAO's PlantNet project: sparse AE extracts disease features from low-res smartphone photos of crops (no GPU needed). Deployed in India & Kenya as SMS-based crop diagnosis. AE compresses 4MB crop images to 40KB for transmission on 2G networks.
📊 Disease detected 3 days earlier · Yield loss ↓22% across 50K farms
⚡ SDG 7: Affordable Clean Energy
🔧 Undercomplete AE + Denoising AE
Smart meter AE detects anomalies (power theft, faults) in 10M household electricity readings. Undercomplete AE compresses time-series to 16D latent. Denoising AE cleans sensor readings from cheap solar IoT devices in rural Maharashtra.
📊 Power theft detection rate ↑65% · Solar panel efficiency monitoring at ₹200/device
🌍 SDG 13: Climate Action
🔧 Convolutional AE (Satellite Compression)
ISRO / ESA use convolutional AE for 50× compression of Sentinel-2 multispectral satellite imagery (13 bands, 10m resolution). Compressed data transmitted 50× faster enabling near-real-time forest fire and glacier monitoring.
📊 Real-time deforestation alerts · Amazon forest coverage tracked daily
📚 SDG 4: Quality Education
🔧 Sparse AE (Knowledge Representation)
Byju's (India) uses sparse AE to model student knowledge states from quiz responses. 500D sparse code per student captures which concepts are mastered (active neurons). Personalises content in real time for 150M students across 15 languages.
📊 Dropout rate ↓34% · Learning outcomes ↑28% in government schools
🏙️ SDG 11: Sustainable Cities
🔧 Contractive AE + Overcomplete AE
Mumbai traffic management: contractive AE extracts stable vehicle trajectory features from noisy CCTV feeds (variable lighting, camera shake). Anomaly detection flags accidents in <30s. Overcomplete AE with sparsity detects unusual congestion patterns.
📊 Accident response time ↓40% · Traffic flow ↑18% on 200 key intersections
MODULE 3 · MCQ QUIZ · TEST YOUR UNDERSTANDING
📝 Module 3 — MCQ Quiz
Q1. An undercomplete autoencoder with LINEAR activations is equivalent to:
A) A convolutional neural network
✅ B) Principal Component Analysis (PCA)
C) A recurrent neural network
D) A generative adversarial network
💡 Linear AE with k-dim bottleneck learns the same k-dimensional subspace as PCA — proved mathematically. Non-linear AE generalises beyond PCA.
Q2. In a Denoising AE, the reconstruction loss is computed against:
A) The noisy/corrupted input x̃
B) The mean of training samples
✅ C) The ORIGINAL clean input x
D) The latent code z
💡 L = ||x − g(f(x̃))||². Input is corrupted x̃ but target is clean x. This forces learning features robust to noise — the key insight of DAE.
Q3. An overcomplete AE (latent > input) without regularization will:
A) Learn more complex non-linear features
✅ B) Learn the identity function — trivial copying
C) Automatically enforce sparsity
D) Converge faster than undercomplete
💡 Without regularization, overcomplete AE has enough capacity to copy input directly → identity mapping. Needs L1/KL sparsity or denoising constraint to learn useful features.
Q4. The Contractive AE penalises:
A) Large weight magnitudes (L2 on weights)
B) High reconstruction error only
✅ C) Frobenius norm of encoder's Jacobian ∂h/∂x
D) KL divergence between latent and Gaussian prior
💡 L_CAE = ||x−x̂||² + λ||∂h/∂x||²_F. Penalises sensitivity of latent code to input perturbations → local invariance → stable manifold representation.
Q5. Sparse AE uses KL divergence penalty to:
✅ A) Measure distance between two probability distributions: target ρ and mean activation ρ̂
B) Compute reconstruction error
C) Regularize decoder weights
D) Add Gaussian noise to input
💡 KL(ρ||ρ̂ⱼ) measures divergence between target sparsity ρ (e.g., 0.05) and average activation ρ̂ⱼ. Minimising this forces each neuron to activate ~5% of the time.
Q6. AE-based image compression outperforms JPEG primarily because:
A) AE uses larger file sizes
✅ B) AE learns domain-specific latent features vs hand-crafted DCT basis in JPEG
C) AE is faster to compress in real time
D) JPEG cannot handle colour images
💡 JPEG uses fixed DCT transform (not learned). AE learns an optimal basis for the specific image domain (faces, medical, satellite) → better reconstruction at same bit-rate.
MODULE 3 · KEY TAKEAWAYS · CSC701
What We Covered in Module 3
1
Autoencoder = Encoder f(x) → Latent z → Decoder g(z) → x̂. Unsupervised. Loss = ||x−x̂||². Minimised via backprop. No labels needed.
2
Linear AE = PCA (mathematically equivalent). Undercomplete AE (z<x) forces compression → essential feature extraction.
3
Overcomplete AE (z>x) needs regularization: L1/KL sparsity penalty OR denoising corruption to prevent trivial identity mapping.
4
Denoising AE (Vincent 2008): train with corrupted x̃, reconstruct clean x. Forces robust, noise-invariant features. Foundation of score-based generative models.
5
Sparse AE (Andrew Ng 2011): L1 or KL(ρ||ρ̂) penalty. Only ~5% neurons active per input. Biologically plausible. Learns Gabor-like interpretable features.
6
Contractive AE (Rifai 2011): penalises ||∂h/∂x||²_F. Local invariance: small input changes → tiny latent changes. Best for manifold learning.
7
Image Compression: AE learns domain-optimal latent basis. Outperforms JPEG at high ratios. ISRO, NASA, Google Photos, Netflix all use AE-based codecs.
8
SDG Impact: SDG 3 (medical denoising), SDG 2 (crop diagnosis), SDG 7 (energy anomaly), SDG 13 (satellite compression), SDG 4 (adaptive learning), SDG 11 (city surveillance)
Next → Module 4: Convolutional Neural Networks — Convolution, Padding, Stride, Pooling, LeNet, AlexNet, ResNet