10315 Recitation 8

PyTorch Basics

What is PyTorch?

• Open Source Machine Learning Framework
• Used to … ML and deep learning models
• train
• predict with
• research
• deploy
• Python Package

Why PyTorch?

• Very Popular
• Has both low and high level APIs
• Flexible and easier to use for building neural networks
• Others are great for their own reasons

Why is important?

Research

thousands of papers

CVPR

ICML

NeurIPS

…

AI Jobs

machine learning engineer

perceptions engineer

(list goes on …)

​

Applications

object recognition

text classification

deep reinforcement learning

image generation

….

Coursework

11485

16385

10403

10417

…

Why is important?

Research

thousands of papers

CVPR

ICML

NeurIPS

…

AI Jobs

machine learning engineer

perceptions engineer

(list goes on …)

​

Applications

object recognition

text classification

deep reinforcement learning

image generation

….

Coursework

11485

16385

10403

10417

…

HW7

What does PyTorch Do?

gradient descent

backpropagation

directed acyclic graphs

GPU computing

distributed processing

transform impls.

Abstracts Away Complexities of Deep Learning

Layers

Models

Datasets

Data Transforms

Optimizers

Loss Functions

How to Use PyTorch

Tensors

The basic building block of PyTorch

• Tensors keep track of their computation graph
• Used to construct gradients via backpropagation for gradient descent

​

PyTorch Tensors are a lot like NumPy arrays

• broadcasting
• reshape
• multi-dimensional
• very similar functions
• can convert between NumPy Arrays and PyTorch Tensors too

DataLoaders

Used for inputs to machine learning models and allows us to iterate through the data

​

Just instantiate object: requires dataset as input

​

Optional parameters

• batch size
• random shuffling
• collating (combining multiple samples)
• multiprocessing
• sampling

Docs

Datasets

Dataset Class used by subclassing

• used to define how to access data
• must implement following functions
• __init__ (self, …) = initializes dataset
• __getitem__(self, i) = return ith item in dataset
• __len__(self) = returns length of dataset
• great if you can’t store all data in RAM
• load data as you need it
• Pytorch has several datasets

Docs

Modules and Layers

Use Modules to define neural networks and Layers to define internal layers of the Neural Network

​

Module: the neural network base class

• must choose subclass to use (like Dataset)
• must implement following functions when subclassing
• __init__(self, ...) = initialize model, layers, and weights
• forward(self, x) = predict on input x
• Note: we don’t need to define a backward propagation function because Pytorch automatically keeps track of the gradients!

​

Layers: instantiate in subclassed module

• because layer information persists, you must define the layer in the __init__(self,...) part of the Module AND use it in forward(self, x) for the gradients to work appropriately
• many layers in PyTorch: Linear, Sigmoid, ReLU, Convolutions,...

​

​

Losses and Optimizers

Use to train neural network and update parameters

​

Losses: compute the loss between model output and label

• returns a PyTorch tensor
• use .backward() to perform backpropagation
• computes all gradients w.r.t. loss

​

Optimizer: performs gradient descent updates on model weights

• initialize with model parameters and other hyperparameters (learning rate, etc.)
• use .zero_grad() to re-initialize gradients
• MUST be called at start of every step of gradient descent
• use .step() to do 1 step of gradient descent (update weights)

Training

Using the basic building blocks from before to train model

​

​

Pseudocode Training Loop

​

for each epoch:

for each (x_batch, y_batch) of data:

reset gradients to 0

preds = model(batch)

loss = loss_fn(preds, y_batch)

​

compute gradients

update weights

Training Loop

GPU Training

If we have access to a GPU, we can move our data and model over to the GPU so that we can train using a faster processor.

​

Need to move 2 objects over

• our data
• our classifier

Use the .to(...) method to move objects to the GPU

​

​

You can now define and train a NN with

Quick Note about Google Colab

Colab allows access to GPU architectures

• but not by default

For more efficient computing, adjust your code to run on the GPU and select the GPU runtime.

Runtime > Change Runtime type > Hardware Accelerator > GPU > SAVE

Full Tutorial

Sources & Documentation

Tutorials

• Recitation PyTorch Tutorial
• PyTorch Official Tutorial Page
• “Learn the Basics” is great for a more official tutorial on PyTorch with heavy description

Documentation

• PyTorch Dataset & Dataloader Documentation
• PyTorch NN package Documentation
• Module
• various neural network Layers
• Loss functions
• PyTorch Optim package Documentation