Download DSS Chapter 1

Business Intelligence and
Decision Support Systems
(9th Ed., Prentice Hall)
Chapter 6:
Artificial Neural Networks
for Data Mining
Learning Objectives





6-2
Understand the concept and definitions of
artificial neural networks (ANN)
Know the similarities and differences between
biological and artificial neural networks
Learn the different types of neural network
architectures
Learn the advantages and limitations of ANN
Understand how backpropagation learning
works in feedforward neural networks
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Learning Objectives


Understand the step-by-step process of how to
use neural networks
Appreciate the wide variety of applications of
neural networks; solving problem types of





6-3
Classification
Regression
Clustering
Association
Optimization
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Neural Network Concepts




Neural networks (NN): a brain metaphor for
information processing
Neural computing
Artificial neural network (ANN)
Many uses for ANN for


Many application areas

6-4
pattern recognition, forecasting, prediction, and
classification
finance, marketing, manufacturing, operations,
information systems, and so on
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Biological Neural Networks
Synapse
Dendrites
Synapse
Axon
Axon
Soma

6-5
Dendrites
Soma
Two interconnected brain cells (neurons)
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Processing Information in ANN
Inputs
Weights
Outputs
x1
Y1
w1
x2
w2
.
.
.
Neuron (or PE)
S 
f (S )
n

i 1
X iW
Summation
i
Transfer
Function
wn
Y
.
.
.
Y2
Yn
xn

6-6
A single neuron (processing element – PE)
with inputs and outputs
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Biology Analogy
6-7
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Elements of ANN


Processing element (PE)
Network architecture



Network information processing




6-8
Hidden layers
Parallel processing
Inputs
Outputs
Connection weights
Summation function
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Elements of ANN
(PE)
x1
(PE)
x2
Weighted Transfer
Sum
Function
(f)
(S)
x3
Y1
(PE)
(PE)
(PE)
(PE)
(PE)
Input
Layer
6-9
Hidden
Layer
Output
Layer
Neural Network with
One Hidden Layer
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Elements of ANN
(a) Single neuron
(b) Multiple neurons
x1
x1
w11
(PE)
Y1
(PE)
Y2
w1
(PE)
w21
Y
w1
x2
w12
Y  X 1W1  X 2W2
x2
w22
PE: Processing Element (or neuron)
Summation Function for a
Single Neuron (a) and
Several Neurons (b)
6-10
Y1  X1W11  X 2W21
Y2  X1W12  X2W22
Y3  X 2W 23
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
w23
(PE)
Y3
Elements of ANN

Transformation (Transfer) Function



Linear function
Sigmoid (logical activation) function [0 1]
Tangent Hyperbolic function [-1 1]
X1 = 3
W
1
X2 = 1
X3 = 2
6-11
Y = 3(0.2) + 1(0.4) + 2(0.1) = 1.2
Transfer function:
YT = 1/(1 + e-1.2) = 0.77
=0
.2
W2 = 0.4
W
Summation function:
=0
3
Processing
element (PE)
Y = 1.2
YT = 0.77
.1
 Threshold value?
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Neural Network Architectures

Several ANN architectures exist







6-12
Feedforward
Recurrent
Associative memory
Probabilistic
Self-organizing feature maps
Hopfield networks
… many more …
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Neural Network Architectures

Architecture of a neural network is driven by
the task it is intended to address


Most popular architecture: Feedforward,
multi-layered perceptron with
backpropagation learning algorithm

6-13
Classification, regression, clustering, general
optimization, association, ….
Used for both classification and regression type
problems
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Learning in ANN


A process by which a neural network learns
the underlying relationship between input and
outputs, or just among the inputs
Supervised learning



Unsupervised learning



6-14
For prediction type problems
E.g., backpropagation
For clustering type problems
Self-organizing
E.g., adaptive resonance theory
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
A Taxonomy of ANN Learning
Algorithms
Learning Algorithms
Discrete/binary input
Surepvised
· Simple Hopefield
· Outerproduct AM
· Hamming Net
Continuous Input
Unsupervised
· ART-1
· Carpenter /
Grossberg
Surepvised
·
·
·
·
·
Unsupervised
Delta rule
Gradient Descent
Competitive learning
Neocognitron
Perceptor
· ART-3
· SOFM (or SOM)
· Other clustering
algorithms
Architectures
Supervised
Recurrent
· Hopefield
6-15
Unsupervised
Feedforward
·
·
·
·
Nonlinear vs. linear
Backpropagation
ML perceptron
Boltzmann
Extimator
· SOFM (or SOM)
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Extractor
· ART-1
· ART-2
A Supervised Learning Process
ANN
Model
Compute
output
Adjust
weights
No
Is desired
output
achieved?
Three-step process:
1. Allocate random weights
2. Compute temporary
outputs
3. Compare outputs with
desired targets
4. Adjust the weights and
repeat the process
Yes
Stop
learning
6-16
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
How a Network Learns

Example: single neuron that learns the
inclusive OR operation (Page 257)
Learning parameters:
 Learning rate
 Momentum
* See your book for step-by-step progression of the learning process
6-17
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
How a Network Learns
X1 = 0
X2 = 1
6-18
.
W2 = 0.3
Processing
element (PE)
.
Yj = 0
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Learning Model





Architecture of ANN, Single neuron, 2
inputs
Allocate random weights, 0.1 , 0.3
Compute the Output Yj
Error (Delta) = Zj – Yj
Update the weights

6-19
Wi(Final)=Wi(Initial) + alpha X delta X Xi
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Backpropagation Learning
a(Zi – Yi)
error
x1
w1
x2
w2
.
.
.
Neuron (or PE)
S 
n

i 1
f (S )
X iW i
Summation
Y  f (S )
Yi
Transfer
Function
wn
xn

6-20
Backpropagation of Error for a Single Neuron
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Backpropagation Learning

The learning algorithm procedure:
1.
2.
3.
4.
5.
6.
6-21
Initialize weights with random values and set
other network parameters
Read in the inputs and the desired outputs
Compute the actual output (by working forward
through the layers)
Compute the error (difference between the actual
and desired output)
Change the weights by working backward through
the hidden layers
Repeat steps 2-5 until weights stabilize
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Development Process of an ANN
6-22
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Sensitivity Analysis on ANN Models



A common criticism for ANN: The lack of
expandability
The black-box syndrome!
Answer: sensitivity analysis



6-23
Conducted on a trained ANN
The inputs are perturbed while the relative
change on the output is measured/recorded
Results illustrates the relative importance of
input variables
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Sensitivity Analysis on ANN Models
Systematically
Perturbed
Inputs
Trained ANN
“the black-box”
Observed
Change in
Outputs
D1
6-24
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Other Popular ANN Paradigms
Self Organizing Maps (SOM)
 First introduced
by the Finnish
Professor Teuvo
Kohonen
 Applies to
clustering type
problems
Input 1
Input 2
Input 3
6-25
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Other Popular ANN Paradigms
Self Organizing Maps (SOM)

SOM Algorithm –
1.
2.
3.
4.
5.
6.
6-26
Initialize each node's weights
Present a randomly selected input vector to
the lattice
Determine most resembling (winning) node
Determine the neighboring nodes
Adjusted the winning and neighboring nodes
(make them more like the input vector)
Repeat steps 2-5 for until a stopping criteria
is reached
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Other Popular ANN Paradigms
Self Organizing Maps (SOM)

Applications of SOM








6-27
Customer segmentation
Bibliographic classification
Image-browsing systems
Medical diagnosis
Interpretation of seismic activity
Speech recognition
Data compression
Environmental modeling, many more …
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Applications Types of ANN

Classification


Regression


Adaptive Resonance Theory (ART) and SOM
Association

6-28
Feedforward networks (MLP), radial basis function
Clustering


Feedforward networks (MLP), radial basis
function, and probabilistic NN
Hopfield networks
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Advantages of ANN





6-29
Able to deal with (identify/model) highly
nonlinear relationships
Not prone to restricting normality and/or
independence assumptions
Can handle variety of problem types
Usually provides better results (prediction
and/or clustering) compared to its statistical
counterparts
Handles both numerical and categorical
variables (transformation needed!)
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
Disadvantages of ANN


They are deemed to be black-box solutions,
lacking expandability
It is hard to find optimal values for large
number of network parameters



6-30
Optimal design is still an art: requires expertise
and extensive experimentation
It is hard to handle large number of variables
(especially the rich nominal attributes)
Training may take a long time for large
datasets; which may require case sampling
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
ANN Software

Standalone ANN software tool





Part of a data mining software suit



6-31
NeuroSolutions
BrainMaker
NeuralWare
NeuroShell, … for more (see pcai.com) …
PASW (formerly SPSS Clementine)
SAS Enterprise Miner
Statistica Data Miner, … many more …
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
End of the Chapter

6-32
Questions / comments…
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall
All rights reserved. No part of this publication may be reproduced, stored in a
retrieval system, or transmitted, in any form or by any means, electronic,
mechanical, photocopying, recording, or otherwise, without the prior written
permission of the publisher. Printed in the United States of America.
Copyright © 2011 Pearson Education, Inc.
Publishing as Prentice Hall
6-33
Copyright © 2011 Pearson Education, Inc. Publishing as Prentice Hall