Download Introduction to Neural Networks

Introduction to Neural
Networks
Neural Networks in the Brain
• Human brain “computes” in an entirely different
way from conventional digital computers.
• The brain is highly complex, nonlinear, and
parallel.
• Organization of neurons to perform tasks much
faster than computers. (Typical time taken in
visual recognition tasks is 100–200 ms.)
• Key features of the biological brain: experience
shapes the wiring through plasticity, and hence
learning becomes the central issue in neural
networks.
Neural Networks as an Adaptive
Machine
• A neural network is a massively parallel
distributed processor made up of simple
processing units, which has a natural propensity
for storing experimental knowledge and making
it available for use.
• Neural networks resemble the brain:
– Knowledge is acquired from the environment
through a learning process.
– Iner neuron connection strengths, known as
synaptic weights, are used to store the acquired
knowledge.
• Procedure used for learning: learning algorithm.
Weights, or even the topology can be adjusted.
Benefits of Neural Networks
• Nonlinearity: nonlinear components, distributed
nonlinearity
• Input-output mapping: supervised learning,
nonparametric statistical inference (model-free
estimation, no prior assumptions)
• Adaptivity: either retain or adapt. Can deal with nonstationary environments. Must overcome stabilityplasticity dilemma.
• Evidential response: decision plus confidence of the
decision can be provided.
• Contextual information: Every neuron in the network
potentially influences every other neuron, so
contextual information is dealt with naturally
Benefits of Neural Networks
• Fault tolerance: performance degrades gracefully.
• VLSI implementability: network of simple
components.
• Uniformity of analysis and design: common
components (neurons), sharability of theories
and learning algorithms, and seamless
integration based on modularity.
• Neurobiological analogy: Neural nets motivated
by neurobiology, and neurobiology also turning
to neural networks for insights and tools.
Human Brain
• Stimulus → Receptors ⇔ Neural Net ⇔
Effectors → Response: Arbib (1987)
• Neurons are slow: 10−3 s per operation,
compared to 10−9 s of modern CPUs.
• Huge number of neurons and on
nections: 1010 neurons, 6 × 1013
connections in human brain.
• Highly energy efficient: 10−16J in the
brain vs. 10−6 J in modern computers
The Biological Neuron
•
•
•
Each neuron is a cell that uses biochemical reactions to receive, process
and transmit information.
Each terminal button is connected to other neurons across a small gap
called a synapse.
A neuron's dendritic tree is connected to a thousand neighbouring
neurons. When one of those neurons fire, a positive or negative charge is
received by one of the dendrites. The strengths of all the received
charges are added together through the processes of spatial and
temporal summation
Models of Neurons
Bias
x1
w
b
1
Input
values
x2
w2

xm

wm
weights

Induced
Field
v
Summing
function
Activation
function
 ()
Output
y
Activation Functions
a if v  c
b if v  c
 (v )  
a if v  c

 (v )  b if v  d
a  (( v  c )( b  a ) /( d  c )) oth

 (v )  z 
 (v ) 
1
1  exp(  xv  y )
 1  v   2 
1
exp   
 

2 
 2   
Stochastic Model
• Instead of deterministic activation, stochastic
activation can be done.
• State of neuron 𝑥 ∈ {+1,-1}
• Probability of firing 𝑃 𝑣
+1 with probability 𝑃(𝑣)
𝑥=
−1 with probability 1 − 𝑃(𝑣)
1
1+exp(−𝑣/𝑇)
• Typical choice of 𝑃 𝑣 =
where 𝑇 is a
pseudo temperature.
• When 𝑇 → 0, the neuron becomes deterministic.
• Think of computer simulation
Signal Flow Graph
•
•
•
•
•
Nodes and links
Links: synaptic links and activation links.
Incoming edges: summation
Outgoing edges: replication
Architectural graph simplifies the above and
abstracts out internal neuronal function.
Signal Flow Graph Example
• Turn the above into a signal-flow graph
Feedback
• Feedback gives dynamics (temporal aspect), and it
is found in almost every part of the nervous system
in every animal.
𝑥𝑗′ 𝑛 = 𝑥𝑗 𝑛 + 𝐵𝑦𝑘 (𝑛)
𝑦𝑘 𝑛 = 𝐴𝑥𝑗′ 𝑛
𝐴
𝑦𝑘 𝑛 =
𝑥𝑗 𝑛
1 − 𝐴𝐵
Feedback
𝑤
𝑦𝑘 𝑛 =
𝑥𝑗 𝑛 = 𝑤
−1
1 − 𝑤𝑧
𝑖𝑛𝑓
𝑖𝑛𝑓
𝑤 𝑙 𝑧 −𝑙 𝑥𝑗 𝑛
𝑙=0
𝑤 𝑙+1 𝑥𝑗 𝑛 − 𝑙
𝑦𝑘 𝑛 =
𝑙=0
𝑧 −1 : 1 − step delay operator
Feedback
• 𝑤 < 1:
– converge (infinite memory,
fading)
• 𝑤 =1
– linearly diverge
• 𝑤 >1
– exponentially diverge
Definition of Neural Networks
• An information processing system that has been
developed as a generalization of mathematical
models of human cognition or neurobiology,
based on the assumptions that
– Information processing occurs at many simple
elements called neurons.
– Signals are passed between neurons over
connection links.
– Each connection link has an associated weight,
which typically multiplies the signal transmitted.
– Each neuron applies an activation function (usually
non-linear) to its net input (sum of weighted input
signals) to determine its output signal.
Network Architectures
• The connectivity of a neural network is
intimately linked with the learning algorithm.
– Single-layer feedforward networks: one input layer,
one layer of computing units (output layer), acyclic
connections.
– Multilayer feedforward networks: one input layer,
one (or more)hidden layers, and one output layer.
Recurrent networks: feedback loop exists.
– Recurrent networks: feedback loop exists.
• Layers can be fully connected or partially
connected.
Similarity Measures
• Similar inputs from similar classes should produce similar
representations, leading to classification into the same
category.
– Reciprocal of Euclidean distance
1
𝑑 𝑥𝑖 ,𝑥𝑗
𝑥𝑖 = 𝑥𝑖1 , 𝑥𝑖2 , 𝑥𝑖3 , … , 𝑥𝑖𝑚
𝑑 𝑥𝑖 , 𝑥𝑗 = 𝑥𝑖 − 𝑥𝑗 =
𝑚
𝑘=1
𝑥𝑖𝑘 − 𝑥𝑗𝑘
– Dot product(inner product)
𝑥𝑖 , 𝑥𝑗 = 𝑥𝑖𝑡 𝑥𝑗 = 𝑚
𝑘=1 𝑥𝑖𝑘 𝑥𝑗𝑘 = 𝑥𝑖
•
𝑡
2 1/2
𝑥𝑗 cos𝜃𝑖𝑗
The two are related, when 𝑥𝑖 = 𝑥𝑗 =1
𝑑 2 𝑥𝑖 , 𝑥𝑗 =
𝑚
𝑘=1
2
𝑥𝑖𝑘 − 𝑥𝑗𝑘 = 𝑥𝑖 − 𝑥𝑗
𝑡
𝑥𝑖 − 𝑥𝑗 = 2 − 2𝑥𝑖𝑡 𝑥𝑗
Similarity Measure
• When two vectors 𝑥𝑖 and 𝑥𝑗 are drawn
from two distributions:
– Mean vector: 𝜇 = 𝐸 𝑥𝑖
– Mahalanobis distance:
2
𝑑𝑖𝑗
=
𝑥𝑖 − 𝑥𝑗
𝑡 −1
Σ
𝑥𝑖 − 𝑥𝑗
– Covariance matrix
Σ = 𝐸 𝑥𝑖 − 𝜇 𝑥𝑖 − 𝜇
𝑡