Intro to PyTorch

Slides adapted from NYUSH Machine Learning course slides

What is PyTorch?

1. An open source machine learning framework that accelerates the path from research prototyping to production deployment. (from official website)
2. PyTorch is an open source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing.

(from wikipedia)

​

Common deep learning frameworks

Pytorch highlights

• Dynamic Computation Graph
• Debugging is easy
• Linear code flow
• Code style is similar to python
• Has an active forum to discuss questions

​

https://cs.stanford.edu/people/karpathy/

It allows you to create and execute computational graphs

https://pytorch.org/blog/overview-of-pytorch-autograd-engine/

Documentation… our best friend?

https://pytorch.org/docs/torch

​

What you can do with Pytorch

Image classification

Chatbot

https://github.com/czs0x55aa/pytorch-chatbot

Text Generator

https://github.com/Morizeyao/GPT2-Chinese

https://suulnnka.github.io/BullshitGenerator/index.html

Some Pytorch basic syntaxes

Pytorch syntax is similar to Numpy

Import library

>>> import torch

Initialize a 3x4 empty tensor

>>> x = torch.empty(3, 4)

>>> print(x)

tensor([

[ 0.0000e+00, 1.5846e+29, -2.2563e-10, 2.8586e-42],

[ 1.1210e-44, -0.0000e+00, 0.0000e+00, 0.0000e+00],

[ 0.0000e+00, 1.4013e-45, 0.0000e+00, 0.0000e+00]])

Initialize a 3x4 all zero tensor

>>> x = torch.zeros(3,4,dtype=torch.long)

>>> print(x)

tensor([[0, 0, 0, 0],

[0, 0, 0, 0],

[0, 0, 0, 0]])

Create tensor using existing data

>>> x = torch.tensor([1,2,3,4])

>>> print(x)

tensor([1, 2, 3, 4])

Pytorch syntax is similar to Numpy

Check shape

>>> x.size()

torch.Size([3, 4])

in-place operations (modify caller)

>>> x.add_(y)

tensor([

[0.8908, 1.6021, 0.5945, 1.3780],

[0.1862, 1.0999, 0.8705, 1.0570],

[0.6097, 1.4280, 0.8986, 1.3685]])

>>> x.t_()

tensor([[0.0156, 0.0366, 0.3641],

[0.7827, 0.8206, 0.9690],

[0.5584, 0.0220, 0.4738],

[0.4707, 0.7703, 0.4945]])

Non in-place operations (return something new)

>>> x.add(y)

tensor([

[0.8908, 1.6021, 0.5945, 1.3780],

[0.1862, 1.0999, 0.8705, 1.0570],

[0.6097, 1.4280, 0.8986, 1.3685]])

>>> x.t()

tensor([[0.0156, 0.0366, 0.3641],

[0.7827, 0.8206, 0.9690],

[0.5584, 0.0220, 0.4738],

[0.4707, 0.7703, 0.4945]])

Convert Pytorch tensor to Numpy ndarray

Torch tensor and Numpy array share memory!

>>> x = torch.tensor([1,2,3,4])

>>> print(x)

tensor([1, 2, 3, 4])

>>> y = x.numpy()

>>> print(y)

array([1, 2, 3, 4])

Resizing a tensor

Numpy: reshape( ); Pytorch: view( )

>>> x = torch.randn(4, 4)

>>> y = x.view(16)

>>> z = x.view(2, 8)

>>> print(x.size(), y.size(), z.size())

torch.Size([4, 4]) torch.Size([16]) torch.Size([2, 8])

Using GPU: what Numpy cannot do

tensor.to(gpu_device);

>>> if torch.cuda.is_available():

device = torch.device("cuda")

y = torch.ones_like(x, device=device)

x = x.to(device)

z = x + y

print(z)

print(z.to("cpu", torch.double))

​

For other Tensor operations, you can use the official doc.

• transposing;
• indexing;
• slicing;
• mathematical operations;
• linear algebra;
• random numbers

https://pytorch.org/docs/torch

​

Example:

Implementing Simple NN using Pytorch tensor + autograd

Calculating gradient was painful

Let’s use pytorch autograd!

torch.Tensor — PyTorch 1.10.0 documentation

Overview of PyTorch Autograd Engine

Pytorch is designed on computational graph

Overview of PyTorch Autograd Engine

Example 1:

Implementing Simple NN using Pytorch tensor + autograd + nn

Specifying neural network calculations was painful.

Let’s use pytorch.nn module!

Example 2:

Implementing Simple NN using Pytorch tensor + autograd + nn + optimizer

Some other optimization algorithms

• Adadelta
• Adagrad
• Adam
• RMSprop
• SGD
• ……

**If you are interested in math, you can check out: https://ruder.io/optimizing-gradient-descent/

Plotting: matplotlib, or, tensorboard

Pytorch also support plotting with TensorBoard

https://www.tensorflow.org/tensorboard