Neural Networks �and �Deep Learning

Ahmad Kalhor

Associate Professor .

School of Electrical and Computer Engineering –

University of Tehran

Fall 2022

Contents

Ahmad Kalhor - University of Tehran

2

1. Introduction
• Natural Neural Networks
• Artificial NNs and Applications
• Architectures, Activation Functions and Learning in ANNs
• Mcculloch & Pitz Neuron
2. Fully Connected Neural Networks
• Linear Perceptron, AdaLine and MadaLine
• Multi-Layer Perceptron (MLP)
• Auto-encoders and Restricted Boltzmann Machine
• Deep belief networks
3. Convolutional Neural Networks (CNNs)
• Convolutional Neural Networks (CNNs)
• Developments and Techniques in CNNs
• Some popular Architectures (AlexNet, VGG, ResNet, Inception, MobileNet, DenseNet, and EfficientNet)

4. Region based CNNs

• CNNs for Object Detection (RCNN, Mask RCNN, YOLO,...)
• CNNs for Object Segmentation (UNet, Mask RCNN, ...)

​

Ahmad Kalhor - University of Tehran

3

5. Recurrent Neural Networks

• Recurrent Neural Network(RNN)
• Long Short Term Memory (LSTM)
• Gated Recurrent Units(GRU)
• Some Extensions

6. Transformers

• Attention Mechanism
• Transformers
• Bidirectional Encoder Representations from Transformers (BERT)
• Vision Based Transformers

7. Variational Auto encoders and Generative Adversarial Networks

• Variational Auto-encoders
• Vector Quantization VAEs.
• Generative Adversarial Networks
• Some Architectures (DCGAN, CGAN, ACGAN,SRGAN , InfoGAN...)

Ahmad Kalhor - University of Tehran

4

Ahmad Kalhor- University of Tehran

5

* A few (optional) mini-projects are designed for extra work (bonus points)

Teaching Assistants

• Reza Dehghani, Kimia Alavi – Chief TA
• Daniyal Saidi
• Amir ahmad Davanlu
• Milad Reisi
• Amin Mohammad Mohammadi
• Shahla Daneshi
• Mojtaba Amiri
• Mohammad Nili
• Maedeh Toosi

Ahmad Kalhor- University of Tehran

6

• Mohammad Sepehri
• Amin Sheikhzade
• Siavash Shams
• Abbas Badiei
• Saeed Mohammadi
• Siavash Razmi
• Sajjad Alikhani

Chapter 1

Introduction

Ahmad Kalhor - University of Tehran

7

1. Introduction

• AI is “a learning technology”
• Inspiring from the natural intelligence (in human and animals)
• Environment perception (clustering/classification/regression)
• Providing Memory about it
• Interaction with it (Doing complexity tasks)
• Computation intelligence (math-logic)
• Stablished on branches of mathematic:
• Matrices Algebra in representation and in computation
• Analysis in learning
• Statistical and Geometric in interpretation and design

​

• Widely applicable in our learning activities
• Media, Security and Entertainment
• Cyber physical , hybrid and autonomous systems
• Social Robots and human-robot interaction
• Industrial systems
• Health systems
• Cognitive systems
• Finance, and Energy Systems

​

​

​

​

Ahmad Kalhor- University of Tehran

8

What is Artificial intelligence?

Artificial NN (ANN) and Deep Learning (DL) in AI

Ahmad Kalhor - University of Tehran

9

A simple definition for Neural Networks

• Neural Networks : A set of simple units (neurons) which are wired together in layers, in order to make important (desired) outputs against stimulating inputs.

​

Ahmad Kalhor - University of Tehran

10

Natural Neural Networks

• NNs have been created naturally in body of animals and plants.

Important questions about animals intelligence

• How do they recognize (cluster and classify) foods, enemies and environment?
• How do they make memories about different events?
• How do they learn to define different mechanisms and acts in their environment?

……..

• How do they become intelligent?

​

Ahmad Kalhor - University of Tehran

11

(Trying to answer to aforementioned questions)

Biologists and Scientists (about 200-300 years ago) discovered

the brain, nervous system and spinal cord.

​

(currently) Neuroscientists research about the structure and function of the brain as the most important part of intelligent.

​

​

Ahmad Kalhor - University of Tehran

12

A Biological Neuron (nerve cell)

Ahmad Kalhor- University of Tehran

13

1. Dendrites

to receive the weighted electrochemical signals from adjacent neurons

(2) Cell body (soma)

to make a summation on received signals

(3) Nucleus

to generate an impulsive signal by comparing the absorbed signal with a threshold

(4) Axon

to send the generated signal to other adjacent neurons

(5) Synaptic gaps

to assign a weight to each send signal to adjacent neurons

Communication between neurons by electrochemical signals:

Ahmad Kalhor- University of Tehran

14

A simple mathematical Model for Biological Neuron

Ahmad Kalhor- University of Tehran

15

Dendrites

Synaptic gaps

Soma Nucleus

Types of neurons in human nervous system

1. Sensory neurons

 Get information about what's going on inside and outside of the body and bring that information into the central nervous system (CNS) so it can be processed.

​

2. Interneurons

which are found only in the CNS, connect one neuron to another. Most of interneurons are in the brain.

There are about 100 billiard neurons in the brain.

There are about 10^15 connections among neurons(10000 connections for each neuron on average)

​

3. Motor neurons

get information from other neurons and convey commands to your muscles, organs and glands.

Ahmad Kalhor- University of Tehran

16

A simple Input-Output Model for human nervous system

Ahmad Kalhor- University of Tehran

17

Learning capabilities of NNs in human body

1. Classification

Localization, Detection and Classification of different objects, faces, voices, smells, and approximation and prediction different physical variables: distances, temperatures, smoothness, brightness, and so on…..

(2) Memory

Capability to create memories about different events with long and short dependencies.

Capability to associate sequenced different patterns together.

(3) Complex and difficult tasks/actions

Car driving and parking, Swimming, Playing music,…..

(4) Computational intelligence

Logic, mathematics, Inference

​

Ahmad Kalhor- University of Tehran

18

Learning in natural neural network �(some important facts)

1. The learning process in brain is mainly performed by tuning the synaptic gaps. Information gathered or resulted from environment is coded in synaptic gaps.
2. The communication speed (electrochemical signal transition) for each neuron is low but Since the communications among neurons are performed in parallel, the processing speed of the brain is high on average.
3. The learning process in the brain is not disturbed if some parts of the brain are damaged and hence the robustness of the natural NNs is high (Fault tolerant).
4. Due to high level abstraction and inference in the brain, its generalization in learning is high.

Ahmad Kalhor- University of Tehran

19

Inspiring From natural NNs:

From begging of the 20- century, Scientists and engineers have been interested to design artificial neural networks:

​

“To make solutions for demanded (challenging) learning problems”

Ahmad Kalhor- University of Tehran

20

Artificial Neural Networks

Applications of ANNs

• Classification NNs
• Classification of objects, images, faces, texts, speeches, signatures, videos and ….
• Disease Diagnosis
• Fault Detection (Industrial)
• Fraud/ Anomaly Detection
• Regression NNs
• Function Approximation(Static/ODE/PDE)
• Identification
• Segmentation
• Simulation
• Prediction
• Signal Recovering/ Repairing

​

Ahmad Kalhor- University of Tehran

21

• Memory NNs
• Word/Voice/Video Prediction
• Natural Language Translation
• image captioning/Descriptions
• Sentiment analysis
• {Demand, Transaction and Price} Prediction
• Path Planning, Tele Representation in Social robots
• Mechanism-Based NNs
• Pattern Generation/sorting/clustering/Noise cancellation
• Super-resolution techniques
• Recommender systems
• Image Enhancement
• Data Compressing
• Imitation-based systems
• To describe the mechanism of the Brain

Ahmad Kalhor- University of Tehran

22

Feed-forward and recurrent neural networks

1- Feedforward Neural Networks

In feedforward networks, messages(data flow) are passed forward only

Feedforward neural network is an artificial neural network wherein connections between the nodes do not form a cycle.

Each output is a static of function of inputs.

​

​

Ahmad Kalhor- University of Tehran

23

2- Recurrent Neural Networks

In recurrent networks, at least in one layer , messages(data flow) are returned to the same or former layers.

In recurrent neural networks (RNNs), a notion of time is introduced. The input at time step t depends on an output from time step t − 1.

These networks are suitable to represent dynamic behavior of functions and systems as nonlinear difference or differential equations .

Ahmad Kalhor- University of Tehran

24

Activation Functions

• Each node(neuron) has an activation function by which it responses to the stimulating inputs.

Ahmad Kalhor- University of Tehran

25

Supervised and unsupervised learning

1. Supervised Learning

Supervised learning as the name indicates a presence of supervisor as teacher. Basically supervised learning is a learning in which we teach or train the ANNs using data which is well labeled that means some data is already tagged with correct answer.

Applications: Classification/Regression: Function Approximation and Prediction/Recognition

​

Learning methods

• Error back propagation(Steepest Descends) MSE, MAE, Cross Entropy,…. : Batch based- stochastic point based-stochastic mini batch based

Gradient (Local search): Descent optimization algorithms:

SGD/SGD+Momentum/Nesterov accelerated gradient/Adagrad/RMSprop/Adam/AdaMax/Nadam/AMSGrad

• Evolutionary based (Global search) : Genetic Algorithm, Particle Swarm Optimization, Ant colony
• Intelligence-based (Global search): Simplex- Simulated Annealing

​

​

Ahmad 2- University of Tehran

26

2. Unsupervised Learning

• Unsupervised learning is the training of ANNs using information that is neither classified nor labeled and allowing the algorithm to act on that information without guidance.
• Instead of explicit targets for the network, there are some statistical or geometric properties for the suitable output of the network.
• Some examples of unsupervised learning algorithms include K-Means Clustering, Principal Component Analysis and Hierarchical Clustering.
• Causal Relationship in Regression Problems.
• Applications: Pattern Generation/ Pattern clustering/ Pattern sorting/ Optimization problems/Control tasks

​

Ahmad Kalhor- University of Tehran

27

3. Semi Supervised and Self Supervised Learning�

• Semi-supervised learning is an approach to machine learning that combines a small amount of labeled data with a large amount of unlabeled data during training. Semi-supervised learning falls between unsupervised learning (with no labeled training data) and supervised learning (with only labeled training data).

​

• Self-supervised learning is a representation learning method where a supervised task is created out of the unlabelled data. Self-supervised learning is used to reduce the data labelling cost and leverage the unlabelled data pool. Some of the popular self-supervised tasks are based on contrastive learning.

​

​

​

​

Ahmad Kalhor- University of Tehran

28

What is the difference between unsupervised and self-supervised?

The only difference is that, unlike unsupervised learning, self-supervised learning does not perform the grouping and clustering of data, as is the case with unsupervised learning. This learning type allows machines to examine part of a data example to figure out the remaining part

Mcculloch & Pitz Neuron�Warren MuCulloch (neuroscientist) and Walter Pitts (logician) 1943

Ahmad Kalhor- University of Tehran

29

Some Logic Functions by M&P neurons��AND���OR���AND Not

Ahmad Kalhor- University of Tehran

30

Two Applications

Ahmad Kalhor- University of Tehran

31

• XOR

Two Applications

Ahmad Kalhor- University of Tehran

32

• Hot and Cold

End of the Introduction

​

​

Thank you

Ahmad Kalhor- University of Tehran

33