Download Neural Networks

Neural Networks
William Lai
Chris Rowlett
What are Neural Networks?
• A type of program that is completely different from
functional programming.
• Consists of units that carry out simple computations
linked together to perform a function
• Modeled after the decision making process of the
biological network of neurons in the brain
The Biology of Neural Networks
• Neural Networks are
models of neuron
clusters in the brain
• Each Neuron has a:
•
•
•
•
Dendrites
Axon
Terminal buds
Synapse
• Action potential is passed
down the axon, which
causes the release of
neurotransmitters
Types of Neural Networks:
General
• Supervised
• During training, error is determined by subtracting
output from actual value
• Unsupervised
• Nothing is known of results
• Used to classify complicated data
• Nonlearning
• Optimization
Types of Neural Networks:
Specific
• Perceptrons
• A subset of feed-forward networks, containing only one input layer,
one output layer, and each input unit links to only output units
• Feed-forward networks
• a.k.a. Directed Acyclic Graphs
• Each unit only links to units in subsequent layers
• Allows for hidden layers
• Recurrent networks
• Not very well understood
• Units can link to units in the same layer or even previous layers
• Example: The Brain
Neural Net Capabilities
• Neural Nets can do anything a normal digital
computer can do (such as perform basic or
complex computations)
• Functional Approximations/Mapping
• Classification
• Good at ignoring ‘noise’
Neural Net Limitations
• Problems similar to Y=1/X between (0,1) on
the open interval
• (Pseudo)-random number predictors
• Factoring integers or determining prime
numbers
• Decryption
History of Neural Networks
• McColloch and Pitts (1943)
• Co-wrote first paper on possible model for a
neuron
• Widrow Hoff (1959)
• Developed MADALINE and ADALINE
• MADALINE was the first neural network to try to
solve a real world problem
• Eliminates echo in phone lines
• vonNeumann architecture took over for about
20 years (60’s-80’s)
Early Applications
• Checkers (Samuel, 1952)
• At first, played very poorly as a novice
• With practice games, eventually beat its author
• ADALINE (Widrow and Hoff, 1959)
• Recognizes binary patterns in streaming data
• MADALINE (same)
• Multiple ADAptive LINear Elements
• Uses an adaptive filter that eliminates echoes on
phone lines
Modern Practical Applications
• Pattern recognition, including
• Handwriting Deciphering
• Voice Understanding
• “Predictability of High-Dissipation Auroral Activity”
• Image analysis
• Finding tanks hiding in trees (cheating)
• Material Classification
• "A real-time system for the characterization of sheep
feeding phases from acoustic signals of jaw sounds"
How Do Neural Networks Relate
to Artificial Intelligence?
• Neural networks are usually geared towards
some application, so they represent the
practical action aspect of AI
• Since neural networks are modeled after
human brains, they are an imitation of human
action. However, than can be taught to act
rationally instead.
• Neural networks can modify their own
weights and learn.
The Future of Neural Networks
•
•
•
•
•
•
•
•
Pulsed neural networks
The AI behind a good Go playing agent
Increased speed through the making of chips
robots that can see, feel, and predict the world
around them
improved stock prediction
common usage of self-driving cars
Applications involving the Human Genome
Project self-diagnosis of medical problems using
neural networks
Past Difficulties
• Single-layer approach limited applications
• Converting Widrow-Hoff Technique for use with
multiple layers
• Use of poorly chosen and derived learning function
• High expectations and early failures led to loss of
funding
Recurring Difficulties
• Cheating
• Exactly what a neural net is doing to get its solutions is
unknown and therefore, it can cheat to find the solution as
opposed to find a reliable algorithm
• Memorization
• Overfitting without generalization
Describing Neural Net Units
• All units have input values, aj
• All input values are weighted, as in each aj is
multiplied by the link’s weight, Wj,i
• All weighted inputs are summed, generating
ini
• The unit’s activation function is called on ini,
generating the activation value ai
• The activation value is output to every
destination of the current unit’s links.
Perceptrons
OR
XOR
• Single layer
neural networks
• Require linearly
separable
functions
• Guarantees the
one solution
Back-Propagation
W j, i  W j, i    aj  Err  g'in i 
• Back-propagation uses a special function to
divide the error of the outputs to all the
weights of the network
• The result is a slow-learning method for
solving many real world problems
Organic vs. Artificial
• Computer cycle times are in the order of
nanoseconds while neurons take milliseconds
• Computers compute the results of each
neuron sequentially, while all neurons in the
brain fire simultaneously every cycle
• Result: massive parallelism makes brains a
billion times faster than computers, even
though computer bits can cycle a million
times faster than neurons
Questions?