Lecture 1: �Artificial�Neural Networks

Presenter: Alan K. Nguyen, BUAIS

Sources:

“Deep Learning”, Ian Goodfellow, 2016

“Deep Learning: Foundations and Concepts”, David Bishop, 2023

Carnegie Mellon University, Advanced Deep Learning, Fall 2020, with permission.

Neural Networks Breakthroughs

• Over the past 15 years of “Artificial Intelligence” Development, Neural Nets have become one of the main approaches to AI.
• They have been successfully applied to various pattern recognition, prediction, and analysis problems.
• It can deal with many problems where ML and Statistical Learning methods were shown to be unusable/infeasible
• Many successes in multiple fields: voice/speech recognition, text recognition and NLP, image processing & computer vision…

So, what are they exactly?

Neural

Network

Voice Signal

Text Transcription

Neural

Network

Raw Image

Text Caption

Neural

Network

Game State

Next Best Move

It begins with the magical ability of humans: to Think.

Humans can:

• Learn, solve problems, recognize patterns, create, analyze,… (think)
• This “intelligence” is worthy of emulation
• Can we make a computational model out of this?
• Marvin Minsky came up with this problem about 2700 years after other (important) Greek dudes (and other French, Germans… in the 1700s)
• Quote: “If the brain was simple enough to be understood - we would be too simple to understand it!”

​

Model of Human Cognition: Association

• Associationism: Humans learn through association (of things, events…).
• Timeline (brief): 400BC-1900AD: Plato (the Greek Dude), David Hume, Ivan Pavlov.
• Associationism: Collection of ideas stating a basic philosophy:
• “Pairs of thoughts become associated based on the organism’s past experience”
• Learning is a mental process that forms associations between temporally related phenomena

Dawn of Connectionism: how do we store Associations?

• Are these “associations” in the room with us right now?
• David Hartley’s Observations on man (1749):
• We receive input through vibrations and those are transferred to the brain
• Memories could also be small vibrations (called vibratiuncles) in the same regions
• Our brain represents compound or connected ideas by connecting our memories with our current senses
• Current science (1700s) did not know about neurons!
• CONNECTIONISM is the next step!

Connectionism

• 1850s: People start to realize that the brain is a mass of interconnected neurons: 1) many neurons connect into each neuron, 2) each neuron connects out to many neurons, 3) therefore, the brain is a network of neurons.
• Enter the stage: Alexander Bain’s “Mind and Body”: “The information is in the connection (1873)”

​

Bain’s Two Big Ideas

• 1) Neural Groupings:
• Neurons excite and stimulate each other
• Different combinations of inputs can result in different outputs.:
• Different intensities of activation of A lead to the differences in when X and Y are activated.
• 2) Memories Creations:
• “when two impressions concur, or closely succeed one another, the nerve-currents find some bridge or place of continuity, better or worse, according to the abundance of nerve- matter available for the transition.” (1873) (This is what type of Learning???) (1949)

​

​

​

​

​

​

​

Result: Connectionist Machines.

• Neurons connect to other neurons. The processing/capacity of the brain is a function of these connections. Connectionist machines emulate this structure (Ferrier, D. (1876). The Functions of the Brain.)
• Connectionist Machine: Network of processing elements, where all world knowledge is stored in the connections between the elements
• Neural networks are connectionist machines
• The machine has many non-linear processing units:
• The program is the connections between these units, and connections may also define memory

Figure provided by Carnegie Mellon University, Advanced Deep Learning, Fall 2020.

From Biology to Mathematics: Neurons

• Let’s go and create some models based on these connectionist machines!
• Let’s start with component - a neuron:
• Signals come in through the dendrites into the Soma
• A signal goes out via the axon to other neurons:
• Only one axon per neuron
• Mathematical Representation of a biological neuron: (McCulloch, W.S. & Pitts, W.H. (1943). A Logical Calculus of the Ideas Immanent in Nervous Activity, Bulletin of Mathematical Biophysics, 5:115-137, 1943)
• The McCulloch and Pitts model: A Synaptic Model.
• Simple “networks” of neurons can perform Boolean operation (paper, figure 1.1, 1.4)

​

​

​

Figure provided by Carnegie Mellon University, Advanced Deep Learning, Fall 2020.

What can Pitts Model Do?

• Simple “networks” of neurons can perform Boolean operation (paper, figure 1.1, 1.4)
• Since any Boolean function can be emulated, any Boolean function can be composed.
• Can exhibits illusions of “perception” (paper, figure 2.1)
• Networks with loops can “remember“. This actually parallels with the biological development at that time!!:�Lawrence Kubie (1930): Closed loops in the central nervous system explain memory!

​

Figure provided by Carnegie Mellon University, Advanced Deep Learning, Fall 2020.

Problem: �No Learning Mechanism so far!

• Cool model bro, now what?
• Donald Hebb to the rescue!
• “When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased.” (1949, Organization of Behaviors)
• Neurons that fire together wire together: neurons that are active at the same time strengthen their connections and form memories

Hebbian Learning in a nutshell

Figure provided by Carnegie Mellon University, Advanced Deep Learning, Fall 2020.

Problems with Hebbian Learning and Pitts Model

• Stronger connections will enforce themselves
• No notion of “competition”
• No reduction in weights
• Learning is unbounded
• It is therefore, Fundamentally Unstable
• What now? ENTERS: Rosenblatt!

​

Figure provided by Cornell University, “Professor’s perceptron paved the way for AI – 60 years too soon”, 2019

Simplified Perceptrons

• Simplified perceptron model:
• Association units combine sensory input with fixed weights
• Response units combine associative units with learnable weights
• Universal model: Originally assumed could represent any Boolean circuit and perform any logic!

Figure provided by Carnegie Mellon University, Advanced Deep Learning, Fall 2020.

Learning Algorithm for Perceptron

Figure provided by Carnegie Mellon University, Advanced Deep Learning, Fall 2020.

Learning Boolean Gates!

• AND Gate (X ∧ Y):
• The first perceptron has two inputs, XXX and YYY, each with a weight of 1.
• The threshold is 2, meaning that the perceptron will only activate (output 1) if the weighted sum of inputs is equal to or greater than 2.
• For the AND function, this setup works because the output will only be 1 if both X and Y are 1. If either input is 0, the sum will be less than 2, and the output will be 0.
• NOT Gate (¬X):
• The second perceptron has a single input, X, with a weight of -1.
• The threshold is 0, so if X is 1, the weighted sum will be -1, which is less than 0, resulting in an output of 0 (i.e., ¬X).
• If X is 0, the weighted sum will be 0, which meets the threshold, and the output is 1. This implements the NOT function, flipping the value of X.

Figure provided by Carnegie Mellon University, Advanced Deep Learning, Fall 2020.

What about our Good Friend, XOR Gate?

=> No solution for XOR!

Single Neuron (Rosenblatt) is not enough!

• Individual elements are weak computational elements! (Marvin Minsky, Seymour Papert, 1969, Perceptrons: An Introduction to Computational Geometry)
• Connectionist elements are required!
• Now where have we heard of it before?

Figure provided by Carnegie Mellon University, Advanced Deep Learning, Fall 2020.

Boolean Neural Networks!

• Rosenblatt’s perceptron: A variant of the McCulloch and Pitt neuron with a provably convergent learning rule.
• Individual units are limited in their capacity
• Multi-layer perceptrons can model arbitrarily complex Boolean functions!

Figure provided by Carnegie Mellon University, Advanced Deep Learning, Fall 2020.

Boolean Perceptrons as Linear Classifier

Figure provided by Carnegie Mellon University, Advanced Deep Learning, Fall 2020.

Composing complicated “decision”�boundaries

Figure provided by Carnegie Mellon University, Advanced Deep Learning, Fall 2020.

Figure provided by Carnegie Mellon University, Advanced Deep Learning, Fall 2020.

Figure provided by Carnegie Mellon University, Advanced Deep Learning, Fall 2020.

Now, overlap them all!

Figure provided by Carnegie Mellon University, Advanced Deep Learning, Fall 2020.

Even more complex decision boundaries!

Figure provided by Carnegie Mellon University, Advanced Deep Learning, Fall 2020.

So, what are they exactly?�=> They are Functions Estimators

Function f

Voice Signal

Text Transcription

Function g

Raw Image

Text Caption

Function h

Game State

Next Best Move

f, g and h can be approximated by a neural network!

Interesting AI tasks are functions that can be modelled by the network.

Readings

• Deep Learning, Ian Goodfellow, 2016, Chapter 6: Feed-forward Neural Networks (free)
• https://www.deeplearningbook.org/
• Building a basic Neural Network with PyTorch https://pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html
• If you’re a nerd (like me): Review the ancient sources cited!