Download PPT

ΗΜΥ 653
Ενσωματωμένα Συστήματα και
Συστήματα Πραγματικού Χρόνου
Εαρινό Εξάμηνο 2017
ΔΙΑΛΕΞΕΙΣ 5 - 6: Intelligent Embedded Systems
- S/W and Real-Time O/S
ΧΑΡΗΣ ΘΕΟΧΑΡΙΔΗΣ
Επίκουρος Καθηγητής ΗΜΜΥ
(ttheocharides@ucy.ac.cy)
Slides adopted from Prof. Peter Marwedel
Additional Ack: Chris Gregg,UVA
Ηλεκτρονικος Εγκέφαλος

Κεντρικό Νευρικό Σύστημα

Υλικό: Επεξεργαστές και
Κυκλώματα
Λογισμικό: Αλγόριθμοι
Έλεγχου, Ανάλυσης και
Επεξεργασίας Δεδομένων...

Τεχνητή Νοημοσύνη

Μνήμη

ΗΜΥ653 Δ05-6 Embedded Systems Software.2
© Θεοχαρίδης, ΗΜΥ, 2017
«να καταλαβαίνει και να αντιλαμβάνεται το
περιβάλλον»

Πώς αντιλαμβανόμαστε εμείς το
περιβάλλον μας?

Ποια η διαφορά το «αντιλαμβάνομαι»
και του «καταλαβαίνω»?

Ακούω την ομιλία αυτή
ΑΝΤΙΛΑΜΒΑΝΟΜΑΙ ότι κάποιος
μιλά!
 Αυτή η ομιλία όμως δεν μου
αρέσει!


ΚΑΤΑΛΑΒΑΙΝΩ τί λέει και ΔΕΝ
με αφορά!
ΗΜΥ653 Δ05-6 Embedded Systems Software.3
© Θεοχαρίδης, ΗΜΥ, 2017
Δεδομένα και Αίσθηση!
Σήμερα μπορούμε να μεταφέρουμε σε
ένα υπολογιστή απεριόριστα δεδομένα
•
Ήχος, εικόνα, μετρήσεις
αισθητήρων, κλπ.
Ενσύρματα αλλά και Ασύρματα, με
ΜΕΓΑΛΗ ΤΑΧΥΤΗΤΑ!
ΠΩΣ ΑΝΤΙΛΑΜΒΑΝΕΤΑΙ
ΚΑΙ ΠΩΣ ΚΑΤΑΝΟΕΙ ΤΑ
ΔΕΔΟΜΕΝΑ ΕΝΑΣ
ΥΠΟΛΟΓΙΣΤΗΣ ΟΜΩΣ;
ΗΜΥ653 Δ05-6 Embedded Systems Software.4
© Θεοχαρίδης, ΗΜΥ, 2017
Intelligence! (Ευφυία!)
ΗΜΥ653 Δ05-6 Embedded Systems Software.5
© Θεοχαρίδης, ΗΜΥ, 2017
Ο Εγκέφαλος (ηλεκτρονικός) σήμερα!
ΗΜΥ653 Δ05-6 Embedded Systems Software.6
© Θεοχαρίδης, ΗΜΥ, 2017
Στην πράξη…

Συλλογή, κωδικοποίηση και αποθήκευση δεδομένων

Αποθήκευση και αναγνώριση προτύπων στα
πλαίσια «εκπαίδευσης»

– Τι σημαίνουν δηλαδή τα δεδομένα; Ποια δεδομένα
είναι χρήσιμα; Πώς μπορώ να «εκπαιδεύσω» ένα
ηλεκτρονικό σύστημα;
ΗΜΥ653 Δ05-6 Embedded Systems Software.7
© Θεοχαρίδης, ΗΜΥ, 2017
ΠΡΑΓΜΑΤΙΚΟΣ ΧΡΟΝΟΣ ΑΝΤΑΠΟΚΡΙΣΗΣ
ΗΜΥ653 Δ05-6 Embedded Systems Software.8
© Θεοχαρίδης, ΗΜΥ, 2017
ΑΠΟΦΑΣΕΙΣ!

Τι κάνουμε με τα δεδομένα?

Τι κάνει ένας άνθρωπος όταν δεί στον δρόμο ένα φορτηγό να τρέχει
προς τα πάνω του?
ΑΝΤΙΔΡΑΣΗ!
ΗΜΥ653 Δ05-6 Embedded Systems Software.9
© Θεοχαρίδης, ΗΜΥ, 2017
We want to design some truly intelligent robot
ΗΜΥ653 Δ05-6 Embedded Systems Software.10
© Θεοχαρίδης, ΗΜΥ, 2017
What is AI?
Discipline that systematizes and automates
intellectual tasks to create machines that:
Act like humans
Act rationally
Think like humans
Think rationally
More formal and mathematical
ΗΜΥ653 Δ05-6 Embedded Systems Software.11
© Θεοχαρίδης, ΗΜΥ, 2017
Some Important Achievements of AI

Logic reasoning (data bases)


Search and game playing


Applied in reasoning interactive robot helper
Applied in robot motion planning
Knowledge-based systems

Applied in robots that use knowledge like internet robots

Bayesian networks (diagnosis)

Machine learning and data mining

Planning and military logistics

Autonomous robots
ΗΜΥ653 Δ05-6 Embedded Systems Software.12
© Θεοχαρίδης, ΗΜΥ, 2017
MAIN BRANCHES OF AI APPLICABLE TO ROBOTICS
Artificial
Intelligence
Neural
Nets
ΗΜΥ653 Δ05-6 Embedded Systems Software.13
Fuzzy
Logic
Genetic
Algorithms
© Θεοχαρίδης, ΗΜΥ, 2017
What we’ll be doing

Uncertain knowledge and reasoning


Probability, Bayes rule
Machine learning

Decision trees, computationally learning theory, reinforcement
learning
ini   W j ,i a j  Wi  a j
j
ΗΜΥ653 Δ05-6 Embedded Systems Software.14
© Θεοχαρίδης, ΗΜΥ, 2017
A Generalized Model of Learning
Correct outputs
Training
System
Output
Input
Learning
Element
Knowledge
Base
Performance
Element
Feedback
Element
Training system is used to create learning pairs
(input, output) to train our robot
Starts with some knowledge. The performance element participates in performance. The feedback element
provides a comparison of actual output vs correct output which becomes input to the learning element. It
analyzes the differences and updates the knowledge base.
ΗΜΥ653 Δ05-6 Embedded Systems Software.15
© Θεοχαρίδης, ΗΜΥ, 2017
Correct outputs
Training
System
Output
Input
Learning
Element
Knowledge
Base
Performance
Element
Feedback
Element
Starts with some knowledge.
The performance element participates in performance.
The feedback element provides a comparison of actual output vs
correct output which becomes input to the learning element. It
analyzes the differences and updates the knowledge base.
ΗΜΥ653 Δ05-6 Embedded Systems Software.16
© Θεοχαρίδης, ΗΜΥ, 2017
Introduction to Pattern Recognition
The applications of Pattern Recognition can be found
everywhere. Examples include disease categorization,
prediction of survival rates for patients of specific
disease, fingerprint verification, face recognition, iris
discrimination, chromosome shape discrimination,
optical character recognition, texture discrimination,
speech recognition, and etc. The design of a pattern
recognition system should consider the application
domain. A universally best pattern recognition system
has never existed.
ΗΜΥ653 Δ05-6 Embedded Systems Software.17
© Θεοχαρίδης, ΗΜΥ, 2017
A Pattern Recognition Paradigm
ΗΜΥ653 Δ05-6 Embedded Systems Software.18
© Θεοχαρίδης, ΗΜΥ, 2017
Texture Discrimination
ΗΜΥ653 Δ05-6 Embedded Systems Software.19
© Θεοχαρίδης, ΗΜΥ, 2017
Shape Discrimination
ΗΜΥ653 Δ05-6 Embedded Systems Software.20
© Θεοχαρίδης, ΗΜΥ, 2017
Are They From the Same Person?
ΗΜΥ653 Δ05-6 Embedded Systems Software.21
© Θεοχαρίδης, ΗΜΥ, 2017
What is pattern recognition?
“The assignment of a physical object or event to one of several prespecified
categeries” -- Duda & Hart

A pattern is an object, process or event that can be
given a name.

A pattern class (or category) is a set of patterns sharing
common attributes and usually originating from the same
source.

During recognition (or classification) given objects are
assigned to prescribed classes.

A classifier is a machine which performs classification.
ΗΜΥ653 Δ05-6 Embedded Systems Software.22
© Θεοχαρίδης, ΗΜΥ, 2017
Examples of applications
• Handwritten: sorting letters by postal
code, input device for PDA‘s.
• Optical Character
Recognition (OCR)
• Biometrics
• Diagnostic systems
• Military applications
• Printed texts: reading machines for blind
people, digitalization of text documents.
• Face recognition, verification, retrieval.
• Finger prints recognition.
• Speech recognition.
• Medical diagnosis: X-Ray, EKG analysis.
• Machine diagnostics, waster detection.
• Automated Target Recognition (ATR).
• Image segmentation and analysis
(recognition from aerial or satelite
photographs).
ΗΜΥ653 Δ05-6 Embedded Systems Software.23
© Θεοχαρίδης, ΗΜΥ, 2017
Examples of applications
ΗΜΥ653 Δ05-6 Embedded Systems Software.24
© Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.25
© Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.26
© Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.27
© Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.28
© Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.29
© Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.30
© Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.31
© Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.32
© Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.33
© Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.34
© Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.35
© Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.36
© Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.37
© Θεοχαρίδης, ΗΜΥ, 2017
Overfitting and underfitting
Problem: how rich class of classifications q(x;θ) to use.
underfitting
good fit
overfitting
Problem of generalization: a small emprical risk Remp does not
imply small true expected risk R.
ΗΜΥ653 Δ05-6 Embedded Systems Software.38
© Θεοχαρίδης, ΗΜΥ, 2017
Basic concepts
Patter
n
 x1 
x 
 2  x

 
 xn 
y
Hidden state
Feature vector
xX
- A vector of observations
(measurements).
X .
- x is a point in feature space
y Y
- Cannot be directly measured.
- Patterns with equal hidden state belong to the same
class.
Task
- To design a classifer (decision rule)
q : X Y
which decides about a hidden state based on an onbservation.
ΗΜΥ653 Δ05-6 Embedded Systems Software.39
© Θεοχαρίδης, ΗΜΥ, 2017
Example
height
Task: horse jockey recognition.
weight
 x1 
x   x
 2
The set of hidden stateY
is  {H , J }
The feature space X
is   2
Training
examples
Linear classifier:
{( x1 , y1 ), , (xl , yl )}
yH
x2
H if (w  x)  b  0
q(x)  
 J if (w  x)  b  0
yJ
ΗΜΥ653 Δ05-6 Embedded Systems Software.40
( w  x)  b  0
x
1
© Θεοχαρίδης,
ΗΜΥ, 2017
Components of PR system
Pattern
Sensors and
preprocessin
g
Teacher
Feature
extraction
Classifier
Class
assignment
Learning algorithm
• Sensors and preprocessing.
• A feature extraction aims to create discriminative features good for
classification.
• A classifier.
• A teacher provides information about hidden state -- supervised learning.
• A learning algorithm sets PR from training examples.
ΗΜΥ653 Δ05-6 Embedded Systems Software.41
© Θεοχαρίδης, ΗΜΥ, 2017
Feature extraction
Task: to extract features which are good for classification.
Good features:• Objects from the same class have similar feature
values.
• Objects from different classes have different values.
“Good” features
ΗΜΥ653 Δ05-6 Embedded Systems Software.42
“Bad” features
© Θεοχαρίδης, ΗΜΥ, 2017
Feature extraction methods
Feature extraction
 m1 
m 
 2
  
 
mk 
φ1
φ2
φn
 x1 
x 
 2

 
 xn 
Feature selection
 m1 
m 
 2
 m3 
 
  
mk 
 x1 
x 
 2

 
 xn 
φ(θ)
Problem can be expressed as optimization of parameters of featrure extractor
.
Supervised methods: objective function is a criterion of separability
(discriminability) of labeled examples, e.g., linear discriminat analysis (LDA).
Unsupervised methods: lower dimesional representation which preserves
important characteristics of input data is sought for, e.g., principal component
ΗΜΥ653
Δ05-6 Embedded
Systems Software.43
© Θεοχαρίδης, ΗΜΥ, 2017
analysis
(PCA).
Classifier
A classifier partitions feature space X into class-labeled regions
such that
X  X 1  X 2    X |Y | and X 1  X 2    X |Y |  {0}
X1
X1
X3
X2
X1
X2
X3
The classification consists of determining to which region a feature vector x
belongs to.
Borders between decision boundaries are called decision regions.
ΗΜΥ653 Δ05-6 Embedded Systems Software.44
© Θεοχαρίδης, ΗΜΥ, 2017
Representation of classifier
A classifier is typically represented as a set of discriminant functions
f i (x) : X  , i  1,, | Y |
The classifier assigns a feature vector x to the i-the
class if
f i (x)  f j (x) j  i
f1 (x)
x
Feature
vector
ΗΜΥ653 Δ05-6 Embedded Systems Software.45
f 2 (x)

max
y
Class identifier
f|Y | (x)
Discriminant
function
© Θεοχαρίδης, ΗΜΥ, 2017
Review
Fig.1 Basic components of a pattern recognition system
ΗΜΥ653 Δ05-6 Embedded Systems Software.46
© Θεοχαρίδης, ΗΜΥ, 2017
Steps

Data acquisition and sensing

Pre-processing
 Removal of noise in data.
 Isolation of patterns of interest from the
background.

Feature extraction
 Finding a new representation in terms of features.
(Better for further processing)
ΗΜΥ653 Δ05-6 Embedded Systems Software.47
© Θεοχαρίδης, ΗΜΥ, 2017
Steps

Model learning and estimation
 Learning a mapping between features and pattern
groups.

Classification
 Using learned models to assign a pattern to a
predefined
category

Post-processing
 Evaluation of confidence in decisions.
 Exploitation of context to improve performances.
ΗΜΥ653 Δ05-6 Embedded Systems Software.48
© Θεοχαρίδης, ΗΜΥ, 2017
Table 1 : Examples of pattern recognition applications
ΗΜΥ653 Δ05-6 Embedded Systems Software.49
© Θεοχαρίδης, ΗΜΥ, 2017
Bayes Statistical Classifiers

Consideration
Randomness of patterns


Decision criterion
Pattern x is labeled as class wi if
W
W
 L p(x / w )P(w ) <  L p(x / w )P(w )
ki
k=1
k
k
qj
q
q
q=1
Lij : Misclassification loss function
p(x/wi) : P.d.f. of a particular pattern x comes from class wi
P(wi) : Probability of occurrence of class wi
ΗΜΥ653 Δ05-6 Embedded Systems Software.50
© Θεοχαρίδης, ΗΜΥ, 2017
Bayes Statistical Classifiers

Decision criterion :
Given Lij is symmetrical function

Posterior probability decision rule
p(x / wi)P(wi) > p(x / wj)P(wj)
dj(x)= p(x / wj)P(wj)= P(wj / x)
dj(x) : decision functions
Pattern x classifies to class j if dj(x) yields the largest value
ΗΜΥ653 Δ05-6 Embedded Systems Software.51
© Θεοχαρίδης, ΗΜΥ, 2017
Classification in Statistical PR
• A class is a set of objects having some important
properties in common
• A feature extractor is a program that inputs the
data (image) and extracts features that can be
used in classification.
• A classifier is a program that inputs the feature
vector and assigns it to one of a set of designated
classes or to the “reject” class.
With what kinds of classes do you work?
ΗΜΥ653 Δ05-6 Embedded Systems Software.52
© Θεοχαρίδης, ΗΜΥ, 2017
Feature Vector Representation
 X=[x1,
x2, … , xn],
each xj a real
number
 xj
may be an object
measurement
 xj
may be count of
object parts
 Example:
object rep.
[#holes, #strokes,
moments, …]
ΗΜΥ653 Δ05-6 Embedded Systems Software.53
© Θεοχαρίδης, ΗΜΥ, 2017
Possible features for char rec.
ΗΜΥ653 Δ05-6 Embedded Systems Software.54
© Θεοχαρίδης, ΗΜΥ, 2017
Some Terminology
 Classes:
set of m known categories of objects
(a) might have a known description for each
(b) might have a set of samples for each
 Reject
Class:
a generic class for objects not in any of
the designated known classes
 Classifier:
Assigns object to a class based on features
ΗΜΥ653 Δ05-6 Embedded Systems Software.55
© Θεοχαρίδης, ΗΜΥ, 2017
Discriminant functions

Functions f(x, K)
perform some
computation on
feature vector x

Knowledge K from
training or
programming is
used

Final stage
determines class
ΗΜΥ653 Δ05-6 Embedded Systems Software.56
© Θεοχαρίδης, ΗΜΥ, 2017
Classification using nearest class mean
 Compute
the
Euclidean distance
between feature
vector X and the
mean of each class.
 Choose
closest
class, if close
enough (reject
otherwise)
ΗΜΥ653 Δ05-6 Embedded Systems Software.57
© Θεοχαρίδης, ΗΜΥ, 2017
Nearest mean might yield poor results with
complex structure
ΗΜΥ653 Δ05-6 Embedded Systems Software.58

Class 2 has two
modes; where is its
mean?

But if modes are
detected, two
subclass mean
vectors can be used
© Θεοχαρίδης, ΗΜΥ, 2017
Classifiers
• Decision Tree Classifiers
• Artificial Neural Net Classifiers
• Bayesian Classifiers and Bayesian Networks
(Graphical Models)
• Support Vector Machines
ΗΜΥ653 Δ05-6 Embedded Systems Software.59
© Θεοχαρίδης, ΗΜΥ, 2017
Human Brain
60
ΗΜΥ653 Δ05-6 Embedded Systems Software.60
© Θεοχαρίδης, ΗΜΥ, 2017
Biological Neural Networks

Several neurons are connected to one another to
form a neural network or a layer of a neural
network.
61
ΗΜΥ653 Δ05-6 Embedded Systems Software.61
© Θεοχαρίδης, ΗΜΥ, 2017
Introduction – Neural Nets

What are Neural Networks?



Neural networks are a paradigm of programming computers.
They are exceptionally good at performing pattern recognition
and other tasks that are very difficult to program using
conventional techniques.
Programs that employ neural nets are also capable of learning
on their own and adapting to changing conditions.
ΗΜΥ653 Δ05-6 Embedded Systems Software.62
© Θεοχαρίδης, ΗΜΥ, 2017
Background



An Artificial Neural Network (ANN) is an information processing paradigm
that is inspired by the biological nervous systems, such as the human
brain’s information processing mechanism.
The key element of this paradigm is the novel structure of the information
processing system. It is composed of a large number of highly
interconnected processing elements (neurons) working in unison to solve
specific problems. NNs, like people, learn by example.
An NN is configured for a specific application, such as pattern recognition or
data classification, through a learning process. Learning in biological
systems involves adjustments to the synaptic connections that exist
between the neurons. This is true of NNs as well.
ΗΜΥ653 Δ05-6 Embedded Systems Software.63
© Θεοχαρίδης, ΗΜΥ, 2017
Artificial Neural Networks

Activation function:
 Threshold logic function (Perceptron)
 Sigmoidal function (MLFFNN)
 Soft-max function (MLFFNN for classification)
 Recti-linear function (CNN)
 Gaussian function (RBFNN for regression)
 Spiking function (Pulsed NN)

Structure of network:
 Feedforward neural networks
 Feedback neural networks
 Feedforward and feedback neural networks

Learning method:
 Supervised learning
 Unsupervised learning
 Competitive learning

Pattern analysis task:
ΗΜΥ653 Δ05-6 Embedded Systems Software.64
64
© Θεοχαρίδης, ΗΜΥ, 2017
How the Human Brain learns

In the human brain, a typical neuron collects signals from others
through a host of fine structures called dendrites.

The neuron sends out spikes of electrical activity through a long, thin
stand known as an axon, which splits into thousands of branches.
At the end of each branch, a structure called a synapse converts the
activity from the axon into electrical effects that inhibit or excite
activity in the connected neurons.
ΗΜΥ653 Δ05-6 Embedded Systems Software.65
© Θεοχαρίδης, ΗΜΥ, 2017

Neural Networks
Biological

Biological approach to AI

Developed in 1943

Comprised of one or more layers of neurons

Several types, we’ll focus on feed-forward
networks
Artificial
Neurons
http://faculty.washington.edu/chudler/color/pic1an.gif
http://research.yale.edu/ysm/images/78.2/articles-neural-neuron.jpg
ΗΜΥ653 Δ05-6 Embedded Systems Software.66
© Θεοχαρίδης, ΗΜΥ, 2017
A Neuron

Receives n-inputs

Multiplies each input by
its weight

Applies activation
function to the sum of
results

Outputs result
http://www-cse.uta.edu/~cook/ai1/lectures/figures/neuron.jpg
ΗΜΥ653 Δ05-6 Embedded Systems Software.67
© Θεοχαρίδης, ΗΜΥ, 2017
A Neuron Model

When a neuron receives excitatory input that is sufficiently
large compared with its inhibitory input, it sends a spike of
electrical activity down its axon. Learning occurs by changing
the effectiveness of the synapses so that the influence of one
neuron on another changes.
ΗΜΥ653 Δ05-6 Embedded Systems Software.68
© Θεοχαρίδης, ΗΜΥ, 2017
Activation Functions

Controls when unit is “active” or
“inactive”

Threshold function outputs 1
when input is positive and 0
otherwise

i.e.
Sigmoid function = 1 / (1 + e-x)

Hyperbolic Tangent…etc.
ΗΜΥ653 Δ05-6 Embedded Systems Software.69
© Θεοχαρίδης, ΗΜΥ, 2017
Neural Network Layers
 Each
layer receives
its inputs from the
previous layer and
forwards its outputs
to the next layer
http://smig.usgs.gov/SMIG/features_0902/tualatin_ann.fig3.gif
ΗΜΥ653 Δ05-6 Embedded Systems Software.70
© Θεοχαρίδης, ΗΜΥ, 2017
Multilayer Feedforward Neural Network

Architecture of an MLFFNN
 Input layer: Linear neurons
 Hidden layers (1 or 2): Sigmoidal neurons
 Output layer:
- Linear neurons (for function approximation task)
- Sigmoidal neurons (for pattern classification task)
Input Layer
x1
1
Hidden Layer
Output Layer
1
1
s1o
s2o
x2
2
2
2
.
.
.
.
.
.
.
.
.
.
.
.
xi
i
j
k
sko
K
o
sK
.
.
.
xd
.
.
.
d
ΗΜΥ653 Δ05-6 Embedded Systems Software.71
.
.
.
J
.
.
.
71
© Θεοχαρίδης, ΗΜΥ, 2017
Pattern Recognition

An important application of neural networks is pattern recognition.
Pattern recognition can be implemented by using a feed-forward
neural network that has been trained accordingly.

During training, the network is trained to associate outputs with input
patterns.

When the network is used, it identifies the input pattern and tries to
output the associated output pattern.

The power of neural networks comes to life when a pattern that has
no output associated with it, is given as an input.

In this case, the network gives the output that corresponds to a
taught input pattern that is least different from the given pattern.
ΗΜΥ653 Δ05-6 Embedded Systems Software.72
© Θεοχαρίδης, ΗΜΥ, 2017
Pattern Recognition (cont.)

Suppose a network is trained to recognize the patterns T
and H. The associated patterns are all black and all white
respectively as shown above.
ΗΜΥ653 Δ05-6 Embedded Systems Software.73
© Θεοχαρίδης, ΗΜΥ, 2017
Pattern Recognition (cont.)
Since the input pattern looks more like a ‘T’, when the
network classifies it, it sees the input closely resembling
‘T’ and outputs the pattern that represents a ‘T’.
ΗΜΥ653 Δ05-6 Embedded Systems Software.74
© Θεοχαρίδης, ΗΜΥ, 2017
Pattern Recognition (cont.)
The input pattern here closely resembles ‘H’ with a slight
difference. The network in this case classifies it as an ‘H’
and outputs the pattern representing an ‘H’.
ΗΜΥ653 Δ05-6 Embedded Systems Software.75
© Θεοχαρίδης, ΗΜΥ, 2017
Pattern Recognition (cont.)

Here the top row is 2 errors away from a ‘T’ and 3 errors away from
an H. So the top output is a black.

The middle row is 1 error away from both T and H, so the output is
random.

The bottom row is 1 error away from T and 2 away from H. Therefore
the output is black.

Since the input resembles a ‘T’ more than an ‘H’ the output of the
network is in favor of a ‘T’.
ΗΜΥ653 Δ05-6 Embedded Systems Software.76
© Θεοχαρίδης, ΗΜΥ, 2017
Learning by Back-Propagation: Illustration
ARTIFICIAL NEURAL NETWORKS Colin Fahey's Guide (Book CD)
ΗΜΥ653 Δ05-6 Embedded Systems Software.77
© Θεοχαρίδης, ΗΜΥ, 2017
Computational Complexity

Could lead to a very large number of calculations
Input Units
Influence
Map Layer 1
Hidden Units
Output Units
Influence
Map Layer 2
ΗΜΥ653 Δ05-6 Embedded Systems Software.78
© Θεοχαρίδης, ΗΜΥ, 2017
Standard computers

Referred to as Von Neumann machines

Follows explicit instructions

Sample program
if (time < noon)
print “Good morning”
else
print “Good afternoon”
No learning
No autonomous
intelligence
ΗΜΥ653 Δ05-6 Embedded Systems Software.79
© Θεοχαρίδης, ΗΜΥ, 2017
Neural networks
 Modeled
 Does
 Is
off the human brain
not follow explicit instructions
trained instead of programmed
 Key
papers

McCulloch, W. and Pitts, W. (1943). A logical calculus
of the ideas immanent in nervous activity. Bulletin of
Mathematical Biophysics, 7:115 - 133.

Rosenblatt, Frank. (1958) The Perceptron: A
Probabilistic Model for Information Storage and
Organization in the Brain. Psychological Review,
65:386-408.
ΗΜΥ653 Δ05-6 Embedded Systems Software.80
© Θεοχαρίδης, ΗΜΥ, 2017
Sources for lecture

comp.ai.neural-networks FAQ

Gurney, Kevin. An Introduction to Neural Networks,
1996.
81
ΗΜΥ653 Δ05-6 Embedded Systems Software.81
© Θεοχαρίδης, ΗΜΥ, 2017
Neuron drawing
82
ΗΜΥ653 Δ05-6 Embedded Systems Software.82
© Θεοχαρίδης, ΗΜΥ, 2017
Neuron behavior

Signals travel between neurons through electrical
pulses

Within neurons, communication is through chemical
neurotransmitters

If the inputs to a neuron are greater than its threshold,
the neuron fires, sending an electrical pulse to other
neurons
This is a simplification.
ΗΜΥ653 Δ05-6 Embedded Systems Software.83
© Θεοχαρίδης, ΗΜΥ, 2017
Perceptron (artificial neuron)
inputs
weights
threshold
xa
output
+
>10?
xb
ΗΜΥ653 Δ05-6 Embedded Systems Software.84
© Θεοχαρίδης, ΗΜΥ, 2017
Training

Inputs and outputs are 0 (no) or 1 (yes)

Initially, weights are random

Provide training input

Compare output of neural network to desired output


If same, reinforce patterns
If different, adjust weights
ΗΜΥ653 Δ05-6 Embedded Systems Software.85
© Θεοχαρίδης, ΗΜΥ, 2017
Example
If both inputs are 1, output should be 1.
inputs
weights
threshold
x2
output
+
>10?
x3
ΗΜΥ653 Δ05-6 Embedded Systems Software.86
© Θεοχαρίδης, ΗΜΥ, 2017
Example (1,1)
If both inputs are 1, output should be 1.
inputs
1
weights
threshold
x2
output
+
1
>10?
x3
ΗΜΥ653 Δ05-6 Embedded Systems Software.87
© Θεοχαρίδης, ΗΜΥ, 2017
Example (1,1)
If both inputs are 1, output should be 1.
inputs
1
weights
x2
threshold
2
output
+
1
x3
ΗΜΥ653 Δ05-6 Embedded Systems Software.88
>10?
3
© Θεοχαρίδης, ΗΜΥ, 2017
Example (1,1)
If both inputs are 1, output should be 1.
inputs
1
weights
x2
threshold
2
+
1
x3
ΗΜΥ653 Δ05-6 Embedded Systems Software.89
5
output
>10?
3
© Θεοχαρίδης, ΗΜΥ, 2017
Example (1,1)
If both inputs are 1, output should be 1.
inputs
1
weights
x2
threshold
2
+
1
x3
ΗΜΥ653 Δ05-6 Embedded Systems Software.90
5
output
>10?
0
3
© Θεοχαρίδης, ΗΜΥ, 2017
Example (1,1)
If both inputs are 1, output should be 1.
inputs
1
weights
x2
threshold
2
+
1
x3
ΗΜΥ653 Δ05-6 Embedded Systems Software.91
5
output
>10?
0
3
© Θεοχαρίδης, ΗΜΥ, 2017
Example (1,1)
If both inputs are 1, output should be 1.
inputs
1
weights
x2
threshold
2
+
1
x3
ΗΜΥ653 Δ05-6 Embedded Systems Software.92
5
output
>10?
0
3
© Θεοχαρίδης, ΗΜΥ, 2017
Example (1,1)
If both inputs are 1, output should be 1.
inputs
1
weights
threshold
x2
output
+
1
>10?
x3
Repeat for all inputs until weights stop changing.
ΗΜΥ653 Δ05-6 Embedded Systems Software.93
© Θεοχαρίδης, ΗΜΥ, 2017
Function Learning Formulation

Goal function f

Training set: (x(i), f(x(i))), i = 1,…,n

Inductive inference: find a function h that
fits the points well
f(x)
x
ΗΜΥ653 Δ05-6 Embedded Systems Software.94
© Θεοχαρίδης, ΗΜΥ, 2017
Different types of Neural Networks

Feed-forward networks

Feed-forward NNs allow signals to travel one way only; from
input to output. There is no feedback (loops) i.e. the output of
any layer does not affect that same layer.

Feed-forward NNs tend to be straight forward networks that
associate inputs with outputs. They are extensively used in
pattern recognition.

This type of organization is also referred to as bottom-up or topdown.
ΗΜΥ653 Δ05-6 Embedded Systems Software.95
© Θεοχαρίδης, ΗΜΥ, 2017
Continued

Feedback networks




Feedback networks can have signals traveling in both directions
by introducing loops in the network.
Feedback networks are dynamic; their 'state' is changing
continuously until they reach an equilibrium point.
They remain at the equilibrium point until the input changes and
a new equilibrium needs to be found.
Feedback architectures are also referred to as interactive or
recurrent, although the latter term is often used to denote
feedback connections in single-layer organizations.
ΗΜΥ653 Δ05-6 Embedded Systems Software.96
© Θεοχαρίδης, ΗΜΥ, 2017
Diagram of an NN
Fig: A simple Neural Network
ΗΜΥ653 Δ05-6 Embedded Systems Software.97
© Θεοχαρίδης, ΗΜΥ, 2017
Network Layers

Input Layer - The activity of the input units represents the
raw information that is fed into the network.

Hidden Layer - The activity of each hidden unit is
determined by the activities of the input units and the
weights on the connections between the input and the
hidden units.

Output Layer - The behavior of the output units depends
on the activity of the hidden units and the weights
between the hidden and output units.
ΗΜΥ653 Δ05-6 Embedded Systems Software.98
© Θεοχαρίδης, ΗΜΥ, 2017
Continued

This simple type of network is interesting because the
hidden units are free to construct their own
representations of the input.

The weights between the input and hidden units
determine when each hidden unit is active, and so by
modifying these weights, a hidden unit can choose what
it represents.
ΗΜΥ653 Δ05-6 Embedded Systems Software.99
© Θεοχαρίδης, ΗΜΥ, 2017
Network Structure

The number of layers and of neurons depend on the
specific task. In practice this issue is solved by trial and
error.

Two types of adaptive algorithms can be used:


start from a large network and successively remove some
neurons and links until network performance degrades.
begin with a small network and introduce new neurons until
performance is satisfactory.
ΗΜΥ653 Δ05-6 Embedded Systems Software.100
© Θεοχαρίδης, ΗΜΥ, 2017
Network Parameters

How are the weights initialized?

How many hidden layers and how many neurons?

How many examples in the training set?
ΗΜΥ653 Δ05-6 Embedded Systems Software.101
© Θεοχαρίδης, ΗΜΥ, 2017
Weights

In general, initial weights are randomly chosen, with
typical values between -1.0 and 1.0 or -0.5 and 0.5.

There are two types of NNs. The first type is known as


Fixed Networks – where the weights are fixed
Adaptive Networks – where the weights are changed to reduce
prediction error.
ΗΜΥ653 Δ05-6 Embedded Systems Software.102
© Θεοχαρίδης, ΗΜΥ, 2017

Neurone vs. Node
ΗΜΥ653 Δ05-6 Embedded Systems Software.103
© Θεοχαρίδης, ΗΜΥ, 2017

Structure of a node:
Squashing function limits node output:
ΗΜΥ653 Δ05-6 Embedded Systems Software.104
© Θεοχαρίδης, ΗΜΥ, 2017
Feed-forward nets
Information flow is unidirectional
Data is presented to Input layer
Passed on to Hidden Layer
Passed on to Output layer
Information is distributed
Information processing is parallel
Internal representation (interpretation) of data
ΗΜΥ653 Δ05-6 Embedded Systems Software.105
© Θεοχαρίδης, ΗΜΥ, 2017
 Feeding
data through the net:
(1  0.25) + (0.5  (-1.5)) = 0.25 + (-0.75) = - 0.5
Squashing:
ΗΜΥ653 Δ05-6 Embedded Systems Software.106
1
 0.3775
0.5
1 e
© Θεοχαρίδης, ΗΜΥ, 2017

Data is presented to the network in the form of
activations in the input layer

Examples




Data usually requires preprocessing


Pixel intensity (for pictures)
Molecule concentrations (for artificial nose)
Share prices (for stock market prediction)
Analogous to senses in biology
How to represent more abstract data, e.g. a name?

Choose a pattern, e.g.
- 0-0-1 for “Chris”
- 0-1-0 for “Becky”
ΗΜΥ653 Δ05-6 Embedded Systems Software.107
© Θεοχαρίδης, ΗΜΥ, 2017

Weight settings determine the behaviour of a network
 How can we find the right weights?
ΗΜΥ653 Δ05-6 Embedded Systems Software.108
© Θεοχαρίδης, ΗΜΥ, 2017
Training the Network - Learning

Backpropagation
Requires training set (input / output pairs)
 Starts with small random weights
 Error is used to adjust weights (supervised learning)
 Gradient descent on error landscape

ΗΜΥ653 Δ05-6 Embedded Systems Software.109
© Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.110
© Θεοχαρίδης, ΗΜΥ, 2017

Advantages



Downsides




It works!
Relatively fast
Requires a training set
Can be slow
Probably not biologically realistic
Alternatives to Backpropagation

Hebbian learning
- Not successful in feed-forward nets

Reinforcement learning
- Only limited success

Artificial evolution
- More general, but can be even slower than backprop
ΗΜΥ653 Δ05-6 Embedded Systems Software.111
© Θεοχαρίδης, ΗΜΥ, 2017
Example: Voice Recognition

Task: Learn to discriminate between two different voices
saying “Hello”

Data

Sources
- Steve Simpson
- David Raubenheimer

Format
- Frequency distribution (60 bins)
- Analogy: cochlea
ΗΜΥ653 Δ05-6 Embedded Systems Software.112
© Θεοχαρίδης, ΗΜΥ, 2017

Network architecture

Feed forward network
- 60 input (one for each frequency bin)
- 6 hidden
- 2 output (0-1 for “Steve”, 1-0 for “David”)
ΗΜΥ653 Δ05-6 Embedded Systems Software.113
© Θεοχαρίδης, ΗΜΥ, 2017

Presenting the data
Steve
David
ΗΜΥ653 Δ05-6 Embedded Systems Software.114
© Θεοχαρίδης, ΗΜΥ, 2017

Presenting the data (untrained network)
Steve
0.43
0.26
David
0.73
0.55
ΗΜΥ653 Δ05-6 Embedded Systems Software.115
© Θεοχαρίδης, ΗΜΥ, 2017

Calculate error
Steve
0.43 – 0
= 0.43
0.26 –1
= 0.74
0.73 – 1
= 0.27
0.55 – 0
= 0.55
David
ΗΜΥ653 Δ05-6 Embedded Systems Software.116
© Θεοχαρίδης, ΗΜΥ, 2017

Backprop error and adjust weights
Steve
0.43 – 0
= 0.43
0.26 – 1
= 0.74
1.17
David
0.73 – 1
= 0.27
0.55 – 0
= 0.55
0.82
ΗΜΥ653 Δ05-6 Embedded Systems Software.117
© Θεοχαρίδης, ΗΜΥ, 2017

Repeat process (sweep) for all training pairs





Present data
Calculate error
Backpropagate error
Adjust weights
Repeat process multiple times
ΗΜΥ653 Δ05-6 Embedded Systems Software.118
© Θεοχαρίδης, ΗΜΥ, 2017

Presenting the data (trained network)
Steve
0.01
0.99
David
0.99
0.01
ΗΜΥ653 Δ05-6 Embedded Systems Software.119
© Θεοχαρίδης, ΗΜΥ, 2017

Results – Voice Recognition

Performance of trained network
- Discrimination accuracy between known “Hello”s
– 100%
- Discrimination accuracy between new “Hello”’s
– 100%
ΗΜΥ653 Δ05-6 Embedded Systems Software.120
© Θεοχαρίδης, ΗΜΥ, 2017
Size of Training Data

Rule of thumb:


the number of training examples should be at least five to
ten times the number of weights of the network.
Other rule:
|W|
N
(1 - a)
ΗΜΥ653 Δ05-6 Embedded Systems Software.121
|W|= number of weights
a = expected accuracy on
test set
© Θεοχαρίδης, ΗΜΥ, 2017
Training Basics

The most basic method of training a neural network is
trial and error.

If the network isn't behaving the way it should, change
the weighting of a random link by a random amount. If
the accuracy of the network declines, undo the change
and make a different one.

It takes time, but the trial and error method does produce
results.
ΗΜΥ653 Δ05-6 Embedded Systems Software.122
© Θεοχαρίδης, ΗΜΥ, 2017
Training: Backprop algorithm

The Backprop algorithm searches for weight values that minimize the
total error of the network over the set of training examples (training
set).

Backprop consists of the repeated application of the following two
passes:
 Forward pass: in this step the network is activated on one
example and the error of (each neuron of) the output layer is
computed.
 Backward pass: in this step the network error is used for
updating the weights. Starting at the output layer, the error is
propagated backwards through the network, layer by layer. This
is done by recursively computing the local gradient of each
neuron.
ΗΜΥ653 Δ05-6 Embedded Systems Software.123
© Θεοχαρίδης, ΗΜΥ, 2017
Back Propagation

Learning Methodology

Back-propagation training algorithm
Network activation
Forward Step
Error
propagation
Backward Step

Backprop adjusts the weights of the NN in order to
minimize the network total mean squared error.
ΗΜΥ653 Δ05-6 Embedded Systems Software.124
© Θεοχαρίδης, ΗΜΥ, 2017
The Learning Process (cont.)

Every neural network possesses knowledge which is
contained in the values of the connection weights.

Modifying the knowledge stored in the network as a
function of experience implies a learning rule for
changing the values of the weights.
ΗΜΥ653 Δ05-6 Embedded Systems Software.125
© Θεοχαρίδης, ΗΜΥ, 2017
The Learning Process (cont.)

Recall: Adaptive networks are NNs that allow the change
of weights in its connections.

The learning methods can be classified in two
categories:


Supervised Learning
Unsupervised Learning
ΗΜΥ653 Δ05-6 Embedded Systems Software.126
© Θεοχαρίδης, ΗΜΥ, 2017
Supervised Learning

Supervised learning which incorporates an external
teacher, so that each output unit is told what its desired
response to input signals ought to be.

An important issue concerning supervised learning is the
problem of error convergence, ie the minimization of
error between the desired and computed unit values.

The aim is to determine a set of weights which minimizes
the error. One well-known method, which is common to
many learning paradigms is the least mean square
(LMS) convergence.
ΗΜΥ653 Δ05-6 Embedded Systems Software.127
© Θεοχαρίδης, ΗΜΥ, 2017
Supervised Learning

In this sort of learning, the human teacher’s experience is
used to tell the NN which outputs are correct and which
are not.

This does not mean that a human teacher needs to be
present at all times, only the correct classifications
gathered from the human teacher on a domain needs to
be present.

The network then learns from its error, that is, it changes
its weight to reduce its prediction error.
ΗΜΥ653 Δ05-6 Embedded Systems Software.128
© Θεοχαρίδης, ΗΜΥ, 2017
Unsupervised Learning

Unsupervised learning uses no external teacher and is
based upon only local information. It is also referred to as
self-organization, in the sense that it self-organizes data
presented to the network and detects their emergent
collective properties.

The network is then used to construct clusters of similar
patterns.

This is particularly useful is domains were a instances
are checked to match previous scenarios. For example,
detecting credit card fraud.
ΗΜΥ653 Δ05-6 Embedded Systems Software.129
© Θεοχαρίδης, ΗΜΥ, 2017
Neural Network in Use
Since neural networks are best at identifying patterns or trends
in data, they are well suited for prediction or forecasting
needs including:
 sales forecasting
 industrial process control
 customer research
 data validation
 risk management
ANN are also used in the following specific paradigms:
recognition of speakers in communications; diagnosis of
hepatitis; undersea mine detection; texture analysis; threedimensional object recognition; hand-written word recognition;
and facial recognition.
ΗΜΥ653 Δ05-6 Embedded Systems Software.130
© Θεοχαρίδης, ΗΜΥ, 2017
Multilayer Feedforward Neural Network

Architecture of an MLFFNN
 Input layer: Linear neurons
 Hidden layers (1 or 2): Sigmoidal neurons
 Output layer:
- Linear neurons (for function approximation task)
- Sigmoidal neurons (for pattern classification task)
Input Layer
x1
1
Hidden Layer
Output Layer
1
1
s1o
s2o
x2
2
2
2
.
.
.
.
.
.
.
.
.
.
.
.
xi
i
j
k
sko
K
o
sK
.
.
.
xd
.
.
.
d
ΗΜΥ653 Δ05-6 Embedded Systems Software.131
.
.
.
J
.
.
.
131
© Θεοχαρίδης, ΗΜΥ, 2017
ANNs – The basics

ANNs incorporate the two fundamental components of
biological neural nets:
1. Neurones (nodes)
2. Synapses (weights)
ΗΜΥ653 Δ05-6 Embedded Systems Software.132
© Θεοχαρίδης, ΗΜΥ, 2017
Perceptron
(The goal function f is a boolean one)
+
+
x1
+
+
-
-
-
+
xi wi
x2
S
g
y
w 1 x1 + w 2 x2 = 0
x1
-
xn
y = g(Si=1,…,n wi xi)
ΗΜΥ653 Δ05-6 Embedded Systems Software.133
© Θεοχαρίδης, ΗΜΥ, 2017
Perceptron
(The goal function f is a boolean one)
+
+
x0
xi wi
?
-
S
g
y
+
+
-
+
-
xn
y = g(Si=1,…,n wi xi)
ΗΜΥ653 Δ05-6 Embedded Systems Software.134
© Θεοχαρίδης, ΗΜΥ, 2017
Unit (Neuron)
x1
xi wi
S
g
y
xn
y = g(Si=1,…,n wi xi)
g(u) = 1/[1 + exp(-au)]
ΗΜΥ653 Δ05-6 Embedded Systems Software.135
© Θεοχαρίδης, ΗΜΥ, 2017
Neural Network
Network of interconnected neurons
x1
xi w
x1
i
xi w
i
xn
S
g
y
S
g
y
xn
Acyclic (feed-forward) vs. recurrent networks
ΗΜΥ653 Δ05-6 Embedded Systems Software.136
© Θεοχαρίδης, ΗΜΥ, 2017
Two-Layer Feed-Forward Neural Network
w1j
Inputs
ΗΜΥ653 Δ05-6 Embedded Systems Software.137
w2k
Hidden
layer
Output
layer
© Θεοχαρίδης, ΗΜΥ, 2017
Comments and Issues on ANN



How to choose the size and structure of networks?
•
If network is too large, risk of over-fitting (data caching)
•
If network is too small, representation may not be rich
enough
Role of representation: e.g., learn the concept of an
odd number
Incremental learning
ΗΜΥ653 Δ05-6 Embedded Systems Software.138
© Θεοχαρίδης, ΗΜΥ, 2017
Artificial Neural Networks



Neural networks are composed of:

Layers of nodes (artificial neurons)

Weights connecting the layers of nodes
Different types of neural networks:

Radial basis function networks

Multi-layer feed forward networks

Recurrent networks (feedback)
Learning algorithms:

Modify the weights connecting the nodes
ΗΜΥ653 Δ05-6 Embedded Systems Software.139
© Θεοχαρίδης, ΗΜΥ, 2017
Convolutional Neural Networks
ΗΜΥ653 Δ05-6 Embedded Systems Software.140
© Θεοχαρίδης, ΗΜΥ, 2017
Convolution Neural Networks

Convolutional neural network (CNN) is a special
type of multilayer feedforward neural network
(MLFFNN) that is well suited for pattern
classification.

Development
motivated.

A CNN is an MLFFNN designed specifically to
recognize 2-dimensional shapes with a high
degree of invariance to translation, scaling,
skewing and other forms of distortion.
of
CNN
is
neuro-biologically
S. Haykin, Neural Networks and Learning Machines,
Prentice-Hall of India, 2011
ΗΜΥ653 Δ05-6 Embedded Systems Software.141
© Θεοχαρίδης, ΗΜΥ, 2017
Structure of a CNN
Feature extraction:
Each neuron takes inputs from a local receptive field in the previous layer, thereby
forcing it to extract local features.

Feature mapping:

Each convolutional layer of a CNN is composed of multiple feature maps.

Each feature map is in the form of a plane within which the individual neurons are
constrained to share the same set of synaptic weights. Effects of this constraint
are the following:

Shift invariance, forced into the operation of a feature map through the use of
convolution, followed by a sigmoid function.

Reduction in number of free parameters, accomplished through the use of
weight sharing.
Subsampling:

Each convolutional layer is followed by a subsampling layer that performs local
averaging and subsampling, whereby the resolution of feature map is reduced.
Weights of connections to nodes in the convolutional layers are learned through
training.
ΗΜΥ653 Δ05-6 Embedded Systems Software.142
© Θεοχαρίδης, ΗΜΥ, 2017
LeNet5: CNN for Handwritten Character Recognition
Input
32x32
6
Feature Maps
28x28
Convolution
6
Feature Maps
14x14
16
16
Feature Maps Feature Maps
10x10
5x5
Convolution
Output
26
Subsampling
Subsampling
Input: 32x32 pixel image of a character centered and normalized in size
Overlapping receptive field for a neuron in a convolutional layer with sigmoidal neurons:
–
For a node in the 1st hidden layer: 5x5 units in the input layer
–
For a node in the 3rd hidden layer: 5x5 units in 3, 4 or 6 feature maps in the 2nd
hidden layer
Non-overlapping receptive field for a neuron in a feature map of a subsampling layer (2nd and
4th hidden layers) with linear neurons: 2x2 units in the corresponding feature map in the
previous layer
Weight sharing: All the nodes in a feature map in a convolutional layer have the same synaptic
weights (~278000 connections, but only ~1700 weight parameters)
Output layer: 26 nodes with one node for each character. Each node in the output layer is
connected to the nodes in all the feature maps in the 4th hidden layer.
Y. LeCun, L. Bottou, Y. Bengio and P. Haffner, “Gradient-based learning applied to document
recognition,” Proceedings of IEEE, vol.86, no.11, pp.2278-2324, November 1998.
ΗΜΥ653 Δ05-6 Embedded Systems Software.143
© Θεοχαρίδης, ΗΜΥ, 2017
ROC Curves
(Receiver
operating
characteristics)
Plots trade-off
between false
positives and
false negatives
for different
values of a
threshold
Figure from “Statistical color models with application to skin
detection,” M.J. Jones and J. Rehg, Proc. Computer Vision
and Pattern Recognition, 1999 copyright 1999, IEEE
ΗΜΥ653 Δ05-6 Embedded Systems Software.144
© Θεοχαρίδης, ΗΜΥ, 2017
In what problems ANNs are good?

Noisy data from sensors

overhead camera poor for orientation

Some sensors don’t work well

Can’t add new sensors (rules of the game)

Other parts of the system constrain available information
(eg. radio communications takes 25ms, frame processing
takes 15ms, etc)
ΗΜΥ653 Δ05-6 Embedded Systems Software.145
© Θεοχαρίδης, ΗΜΥ, 2017
Face recognition
146
Steve Lawrence, C. Lee Giles, A.C. Tsoi and A.D. Back. Face Recognition: A Convolutional Neural Network Approach. IEEE Transactions on
Neural Networks, Special Issue on Neural Networks and Pattern Recognition, Volume 8, Number 1, pp. 98-113, 1997. © Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.146
Famous Applications

NASA’s intelligent flight control system (F15As and C19s
– NOT space shuttle)

Handwritten number recognition (postcodes on snail
mail)

Robotic arm control

Number plate recognition on moving cars

Detection of cancerous cells in smear tests
ΗΜΥ653 Δ05-6 Embedded Systems Software.147
© Θεοχαρίδης, ΗΜΥ, 2017
Example:
Application of NN to
Motion Planning
(Climbing Robot)
ΗΜΥ653 Δ05-6 Embedded Systems Software.148
© Θεοχαρίδης, ΗΜΥ, 2017
Transition
one-step planning
initial 4-hold
stance
...
3-hold
stance
...
...
...
4-hold
stance
ΗΜΥ653 Δ05-6 Embedded Systems Software.149
one-step
planning contact / zero force
breaking
© Θεοχαρίδης, ΗΜΥ, 2017
Idea: Learn Feasibility



Create a large database
of labeled transitions
Train a NN classifier
Q : transition  {feasible, not feasible)
Learning is possible because:
Shape of a feasible space is mostly determined
by the equilibrium condition that depends on
relatively few parameters
ΗΜΥ653 Δ05-6 Embedded Systems Software.150
© Θεοχαρίδης, ΗΜΥ, 2017
Creation of Database


Sample transitions at random
(by picking 4 holds at random
within robot’s limb span)
Label each transition – feasible
or infeasible – by sampling with
high time limit
 over 95% infeasible transitions

Re-sample around feasible
transitions
 35-65% feasible transitions

~1 day of computation to create a
database of 100,000 labeled transitions
ΗΜΥ653 Δ05-6 Embedded Systems Software.151
© Θεοχαρίδης, ΗΜΥ, 2017
Training of a NN Classifier



3-layer NN, with 9 input units, 100 hidden
units, and 1 output unit
Training on 50,000 examples
(~3 days of computation)
Validation on the remaining 50,000 examples
 ~78% accuracy (ε = 0.22)
 0.003ms average running time
ΗΜΥ653 Δ05-6 Embedded Systems Software.152
© Θεοχαρίδης, ΗΜΥ, 2017
Other (Linear) Classifiers
a
x
denotes +1
w x + b>0
f
yest
f(x,w,b) = sign(w x + b)
denotes -1
How would you
classify this data?
w x + b<0
ΗΜΥ653 Δ05-6 Embedded Systems Software.153
© Θεοχαρίδης, ΗΜΥ, 2017
Linear Classifiers
a
x
denotes +1
f
yest
f(x,w,b) = sign(w x + b)
denotes -1
How would you
classify this data?
ΗΜΥ653 Δ05-6 Embedded Systems Software.154
© Θεοχαρίδης, ΗΜΥ, 2017
a
Linear Classifiers
x
denotes +1
f
yest
f(x,w,b) = sign(w x + b)
denotes -1
How would you
classify this data?
ΗΜΥ653 Δ05-6 Embedded Systems Software.155
© Θεοχαρίδης, ΗΜΥ, 2017
Linear Classifiers
a
x
denotes +1
f
yest
f(x,w,b) = sign(w x + b)
denotes -1
Any of these
would be fine..
..but which is
best?
ΗΜΥ653 Δ05-6 Embedded Systems Software.156
© Θεοχαρίδης, ΗΜΥ, 2017
a
Linear Classifiers
x
denotes +1
f
yest
f(x,w,b) = sign(w x + b)
denotes -1
How would you
classify this data?
Misclassified
to +1 class
ΗΜΥ653 Δ05-6 Embedded Systems Software.157
© Θεοχαρίδης, ΗΜΥ, 2017
a
Classifier Margin
x
denotes +1
denotes -1
ΗΜΥ653 Δ05-6 Embedded Systems Software.158
f
yest
f(x,w,b) = sign(w x + b)
Define the margin
of a linear
classifier as the
width that the
boundary could be
increased by
before hitting a
datapoint.
© Θεοχαρίδης, ΗΜΥ, 2017
a
Maximum Margin
x
denotes +1
denotes -1
f
1. Maximizing the margin is good
2. Implies that
only support
vectors
f(x,w,b)
= sign(w
x + b)are
important; other training examples
are ignorable. The maximum
margin
linear
3. Empirically it works
very very
well.
classifier is the
linear classifier
with the, um,
maximum margin.
Support Vectors
are those
datapoints that
the margin
pushes up
against
Linear SVM
ΗΜΥ653 Δ05-6 Embedded Systems Software.159
yest
This is the
simplest kind of
SVM (Called an
LSVM)
© Θεοχαρίδης, ΗΜΥ, 2017
Support Vector Machines
 SVMs
pick best separating hyperplane according
to some criterion
 e.g.
maximum margin
 Training
process is an optimisation
 Training
set is effectively reduced to a relatively
small number of support vectors
ΗΜΥ653 Δ05-6 Embedded Systems Software.160
© Θεοχαρίδης, ΗΜΥ, 2017
SVM applications

SVMs were originally proposed by Boser, Guyon and Vapnik in 1992 and
gained increasing popularity in late 1990s.

SVMs are currently among the best performers for a number of classification
tasks ranging from text to genomic data.

SVM techniques have been extended to a number of tasks such as
regression [Vapnik et al. ’97], principal component analysis [Schölkopf et al.
’99], etc.

Most popular optimization algorithms for SVMs are SMO [Platt ’99] and
SVMlight [Joachims’ 99], both use decomposition to hill-climb over a subset of
αi’s at a time.

Tuning SVMs remains a black art: selecting a specific kernel and
parameters is usually done in a try-and-see manner.
ΗΜΥ653 Δ05-6 Embedded Systems Software.161
© Θεοχαρίδης, ΗΜΥ, 2017
Feature Spaces
 We
may separate data by mapping to a higherdimensional feature space
 The
feature space may even have an infinite
number of dimensions!
 We
need not explicitly construct the new feature
space
ΗΜΥ653 Δ05-6 Embedded Systems Software.162
© Θεοχαρίδης, ΗΜΥ, 2017
Kernels
 We
may use Kernel functions to implicitly map to
a new feature space
 Kernel
fn:
K x1 , x2  R
 Kernel
must be equivalent to an inner product in
some feature space
ΗΜΥ653 Δ05-6 Embedded Systems Software.163
© Θεοχαρίδης, ΗΜΥ, 2017
Example Kernels
xz
Linear:
P x  z
Polynomial:
Gaussian:
ΗΜΥ653 Δ05-6 Embedded Systems Software.164


exp  x  z /  2
2

© Θεοχαρίδης, ΗΜΥ, 2017
Perceptron Revisited: Linear Separators
 Binary
classification can be viewed as the task of
separating classes in feature space:
wTx + b = 0
wTx + b > 0
wTx + b < 0
f(x) = sign(wTx + b)
ΗΜΥ653 Δ05-6 Embedded Systems Software.165
© Θεοχαρίδης, ΗΜΥ, 2017
Which of the linear separators is
optimal?
ΗΜΥ653 Δ05-6 Embedded Systems Software.166
© Θεοχαρίδης, ΗΜΥ, 2017
Best Linear Separator?
ΗΜΥ653 Δ05-6 Embedded Systems Software.167
© Θεοχαρίδης, ΗΜΥ, 2017
Best Linear Separator?
ΗΜΥ653 Δ05-6 Embedded Systems Software.168
© Θεοχαρίδης, ΗΜΥ, 2017
Best Linear Separator?
ΗΜΥ653 Δ05-6 Embedded Systems Software.169
© Θεοχαρίδης, ΗΜΥ, 2017
Best Linear Separator?
ΗΜΥ653 Δ05-6 Embedded Systems Software.170
© Θεοχαρίδης, ΗΜΥ, 2017
Find Closest Points in Convex Hulls
d
c
ΗΜΥ653 Δ05-6 Embedded Systems Software.171
© Θεοχαρίδης, ΗΜΥ, 2017
Plane Bisect Closest Points
wT x + b =0
w=d-c
d
c
ΗΜΥ653 Δ05-6 Embedded Systems Software.172
© Θεοχαρίδης, ΗΜΥ, 2017
Classification Margin
wT x  b
separator ris
w

Distance from example data to the

Data closest to the hyperplane are support vectors.

Margin ρ of the separator is the width of separation between
ρ
classes.
r
ΗΜΥ653 Δ05-6 Embedded Systems Software.173
© Θεοχαρίδης, ΗΜΥ, 2017
Maximum Margin Classification

Maximizing the margin is good according to intuition and
theory.

Implies that only support vectors are important; other
training examples are ignorable.
ΗΜΥ653 Δ05-6 Embedded Systems Software.174
© Θεοχαρίδης, ΗΜΥ, 2017
Margins and Complexity
Skinny margin
is more flexible
thus more complex.
ΗΜΥ653 Δ05-6 Embedded Systems Software.175
© Θεοχαρίδης, ΗΜΥ, 2017
Margins and Complexity
Fat margin
is less complex.
ΗΜΥ653 Δ05-6 Embedded Systems Software.176
© Θεοχαρίδης, ΗΜΥ, 2017
Nonlinear SVM - Overview



SVM locates a separating hyperplane in the
feature space and classify points in that
space
It does not need to represent the space
explicitly, simply by defining a kernel
function
The kernel function plays the role of the dot
product in the feature space.
ΗΜΥ653 Δ05-6 Embedded Systems Software.177
© Θεοχαρίδης, ΗΜΥ, 2017
Properties of SVM

Flexibility in choosing a similarity function

Sparseness of solution when dealing with large data
sets
- only support vectors are used to specify the separating
hyperplane

Ability to handle large feature spaces
- complexity does not depend on the dimensionality of the
feature space

Overfitting can be controlled by soft margin approach

Nice math property: a simple convex optimization problem
which is guaranteed to converge to a single global solution

Feature Selection
ΗΜΥ653 Δ05-6 Embedded Systems Software.178
© Θεοχαρίδης, ΗΜΥ, 2017
SVM Applications

SVM has been used successfully in many real-world
problems
- text (and hypertext) categorization
- image classification
- bioinformatics (Protein classification,
Cancer classification)
- hand-written character recognition
ΗΜΥ653 Δ05-6 Embedded Systems Software.179
© Θεοχαρίδης, ΗΜΥ, 2017
Additional Resources

An excellent tutorial on VC-dimension and Support
Vector Machines:
C.J.C. Burges. A tutorial on support vector machines for pattern
recognition. Data Mining and Knowledge Discovery, 2(2):955974, 1998.

The VC/SRM/SVM Bible:
Statistical Learning Theory by Vladimir Vapnik, WileyInterscience; 1998
http://www.kernel-machines.org/
ΗΜΥ653 Δ05-6 Embedded Systems Software.180
© Θεοχαρίδης, ΗΜΥ, 2017
Application Knowledge
Embedded System Hardware
2:
Specification
3:
ES-hardware
4: system
software (RTOS,
middleware, …)
ΗΜΥ653 Δ05-6 Embedded Systems Software.181
Design
repository
6: Application
mapping
Design
8:
Test
7: Optimization
5: Validation &
Evaluation (energy,
cost, performance, …)
© Θεοχαρίδης, ΗΜΥ, 2017
Overview
Application Space
OPERATING SYSTEM
O/S Tasks
Architectural Space
PROCESSOR
ΗΜΥ653 Δ05-6 Embedded Systems Software.182
CUSTOM H/W
I/O
MEMORY
© Θεοχαρίδης, ΗΜΥ, 2017
Ιs programming for an embedded system different?
 Speed
 Size
 Correctness
 Portability
 Real-Time
 Other
or not
Hardware Considerations
ΗΜΥ653 Δ05-6 Embedded Systems Software.183
© Θεοχαρίδης, ΗΜΥ, 2017
How does one actually program an embedded system?

Has this changed over the years? Has there been significant
progress?

How does Ford program their Fuel Injection System computer?

How does GE program their Microwave Ovens (and was this different
30 years ago?)

How does Nikon program their digital SLR cameras?

How does Apple/RIM/Microsoft program their
iPhone/Blackberry/SmartPhone?

How does a hobbyist program a Pic microprocessor?

How does NASA program (and reprogram) the Mars Rover?

How did James Gosling at Sun want his set-top boxes programmed?

(How was the original 128K Macintosh programmed?)
ΗΜΥ653 Δ05-6 Embedded Systems Software.184
© Θεοχαρίδης, ΗΜΥ, 2017
Programming Languages
Machine Code
Fixed Rom, Ram,
Firmware
Assembly
Code
Higher Level
Languages
Compiled
C, nesC, C++,
Ada, Forth, etc.
Interpreted (?)
Perl, Python,
Javascript
Java
ΗΜΥ653 Δ05-6 Embedded Systems Software.185
Markup
HTML, XML
All Eventually End up
as Machine Code
© Θεοχαρίδης, ΗΜΥ, 2017
So you’ve chosen C. Let’s Go!
(Not so fast.)
What does a programmer need to know about
programming for an embedded system?
•
S/he’d better know about the hardware.
•
Purpose
•
How data flows (to include getting the program onto the
system, I/O), how to interface with sensors, actuators, etc.
•
Whether there is an operating system, and how it runs
programs
•
Limitations: memory, speed, upgradability (firmware?)
•
How are hardware errors handled? (think Mars Rover)
•
Plan on debugging hardware issues…
ΗΜΥ653 Δ05-6 Embedded Systems Software.186
© Θεοχαρίδης, ΗΜΥ, 2017
So you’ve chosen C. Let’s Go!
(Not so fast.)
What does a programmer need to know about programming
for an embedded system?
•
S/he’d better know about the software tools related to
programming for the specific hardware.
•
Is there a compiler/linker/assembler? (hopefully!)
•
How is the memory addressed, what bit-level knowledge of the
hardware is necessary, how does one access pins, etc.?
•
How will debugging be managed?
•
How will testing be done?
•
What licensing needs to be organized (this can be tricky!)
•
Does the software need to be portable? (using C is probably
going to help you—see the second point above).
ΗΜΥ653 Δ05-6 Embedded Systems Software.187
© Θεοχαρίδης, ΗΜΥ, 2017
The Embedded Software Development Process
Barr, M. & Massa, A. Oram, A. (ed.) Programming Embedded Systems
in C and C++, 2nd Edition. O'Reilly & Associates, Inc., 2006 , p.55
ΗΜΥ653 Δ05-6 Embedded Systems Software.188
© Θεοχαρίδης, ΗΜΥ, 2017
The Tools

Compiler: Translates human readable code into assembly
language or opcodes for a particular processor (or
possibly into machine-independent opcodes a la Java).
Produces an object file.

Assembler: Translates assembly language into opcodes
(it is really a compiler, too). Also produces an object file.

Linker: Organizes the object files, necessary libraries,
and other data and produces a relocatable file.

Locator: Takes the relocatable file and information about
the memory of the system and produces an executable.
(By the way: gcc takes care of all of these functions at once)
ΗΜΥ653 Δ05-6 Embedded Systems Software.189
© Θεοχαρίδης, ΗΜΥ, 2017
The Tools: Embedded System Specifics
All of the tools run on the host computer, not the embedded
computer.

Compiler: Has to know about the specific hardware (except in very
trivial cases). Should be able to optimize for size.

Assembler: Produces “startup code”; not inserted automatically as
in general purpose computers (i.e., the programmer needs to
compile it independently).

Linker: Needs the correct libraries (open source c libraries, such
as newlib, are available).

Locator: Needs programmer input for information about memory.
Bottom Line: There can be a lot of extra work for the programmer,
although certain systems (e.g. Pic programming) tools can automate
most of it.
ΗΜΥ653 Δ05-6 Embedded Systems Software.190
© Θεοχαρίδης, ΗΜΥ, 2017
Moving the program onto the
embedded system

Remember, the program is written (and possibly run
in an emulator) on a host computer, but it still needs
to get onto the embedded system.

Methods:



Build/burn the program into the hardware (firmware or other
flash memory)
Bootloader: a bootloader resides on the embedded system
and facilitates loading programs onto the system.
Debug Monitor: The debug monitor is a more robust
program on an embedded system that helps with
debugging and other chores, and can include a bootloader
as well.
ΗΜΥ653 Δ05-6 Embedded Systems Software.191
© Θεοχαρίδης, ΗΜΥ, 2017
Debugging

Debugging embedded systems can be facilitated with a
Debug Monitor, or through a remote debugger on the host
computer. A serial link is normally set up, and the debugger
acts more or less like a general purpose debugger.

Emulators can be used to test the system without utilizing the
actual hardware (but this has many caveats, and nothing
beats testing on the real system).

Software Simulators allow the programmer to debug
completely on the host system, which can be quicker and can
allow faster code turnaround.

When it comes down to it, an oscilloscope and a multimeter
can be your best friend for debugging.
ΗΜΥ653 Δ05-6 Embedded Systems Software.192
© Θεοχαρίδης, ΗΜΥ, 2017
Reuse of standard software components

Knowledge from previous designs to be
made available in the form of intellectual
property (IP, for SW & HW).



Operating systems
Middleware
….
ΗΜΥ653 Δ05-6 Embedded Systems Software.193
© Θεοχαρίδης, ΗΜΥ, 2017
Embedded operating systems
- Requirement: Configurability -

Configurability
No single RTOS will fit all needs, no overhead for
unused functions tolerated  configurability needed.





simplest form: remove unused functions (by linker ?).
Conditional compilation (using #if and #ifdef commands).
Dynamic data might be replaced by static data.
Advanced compile-time evaluation useful.
Object-orientation could lead to a derivation subclasses.
ΗΜΥ653 Δ05-6 Embedded Systems Software.194
© Θεοχαρίδης, ΗΜΥ, 2017
Verification of derived OS?

Verification a potential problem of systems
with a large number of derived OSs:


Each derived OS must be tested thoroughly;
potential problem for eCos (open source RTOS from Red Hat),
including 100 to 200 configuration points [Takada, 01].
ΗΜΥ653 Δ05-6 Embedded Systems Software.195
© Θεοχαρίδης, ΗΜΥ, 2017
Requirement: Disc and network handled by tasks



Disc & network handled by tasks instead of integrated drivers.
Relatively slow discs & networks can be handled by tasks.
Many ES without disc, a keyboard, a screen or a mouse.
Effectively no device that needs to be supported by all versions
of the OS, except maybe the system timer.
Embedded OS
Standard OS
kernel
ΗΜΥ653 Δ05-6 Embedded Systems Software.196
© Θεοχαρίδης, ΗΜΥ, 2017
- Requirement: Protection is optional
Protection mechanisms not always necessary:
ES typically designed for a single purpose,
untested programs rarely loaded, SW considered reliable.
(However, protection mechanisms may be needed for safety
and security reasons).
Privileged I/O instructions not necessary and
tasks can do their own I/O.
Example: Let switch be the address of some switch
Simply use
load register,switch
instead of OS call.
ΗΜΥ653 Δ05-6 Embedded Systems Software.197
© Θεοχαρίδης, ΗΜΥ, 2017
- Requirement: Interrupts not restricted to OS 
Interrupts can be employed by any process
For standard OS: serious source of unreliability.
Since






embedded programs can be considered to be tested,
since protection is not necessary and
since efficient control over a variety of devices is required,
it is possible to let interrupts directly start or stop tasks (by storing the
tasks start address in the interrupt table).
More efficient than going through OS services.
Reduced composability: if a task is connected to an interrupt, it may be
difficult to add another task which also needs to be started by an event.
ΗΜΥ653 Δ05-6 Embedded Systems Software.198
© Θεοχαρίδης, ΗΜΥ, 2017
- Requirement: Real-time capability
Many embedded systems are real-time (RT) systems and,
hence, the OS used in these systems must be real-time
operating systems (RTOSes).
ΗΜΥ653 Δ05-6 Embedded Systems Software.199
© Θεοχαρίδης, ΗΜΥ, 2017
- Real-time OS (1) 
Def.: (A) real-time operating system is an operating system
that supports the construction of real-time systems

The following are the three key requirements
1.
The timing behavior of the OS must be predictable.
 services of the OS: Upper bound on the execution time!
RTOSs must be deterministic:

unlike standard Java,

short times during which interrupts are disabled,

contiguous files to avoid unpredictable head movements.
[Takada, 2001]
ΗΜΥ653 Δ05-6 Embedded Systems Software.200
© Θεοχαρίδης, ΗΜΥ, 2017
- Real-time OS (2) 2.
OS must manage the timing and scheduling

OS possibly has to be aware of task deadlines;
(unless scheduling is done off-line).

OS must provide precise time services with high resolution.
[Takada, 2001]
ΗΜΥ653 Δ05-6 Embedded Systems Software.201
© Θεοχαρίδης, ΗΜΥ, 2017
Time services
Time plays a central role in “real-time” systems.
 Actual time is described by real numbers.
 Two discrete standards are used in real-time equipment:



International atomic time TAI
(french: temps atomic internationale)
Free of any artificial artifacts.
Universal Time Coordinated (UTC)
UTC is defined by astronomical standards
UTC and TAI identical on Jan. 1st, 1958.
 30 seconds had to be added since then.
 Not without problems: New Year may start twice per night.

ΗΜΥ653 Δ05-6 Embedded Systems Software.202
© Θεοχαρίδης, ΗΜΥ, 2017
Internal synchronization



1.
2.
3.




Synchronization with one master clock
Typically used in startup-phases
Distributed synchronization:
Collect information from neighbors
Compute correction value
Set correction value.
Precision of step 1 depends on how information is
collected:
Application level:
~500 µs to 5 ms
Operation system kernel: 10 µs to 100 µs
Communication hardware: < 10 µs
ΗΜΥ653 Δ05-6 Embedded Systems Software.203
© Θεοχαρίδης, ΗΜΥ, 2017
External synchronization

External synchronization guarantees consistency with
actual physical time.

Trend is to use GPS for ext. synchronization

GPS offers TAI and UTC time information.

Resolution is about 100 ns.
GPS mouse
© Dell
ΗΜΥ653 Δ05-6 Embedded Systems Software.204
© Θεοχαρίδης, ΗΜΥ, 2017
Problems with external synchronization

Problematic from the perspective of fault tolerance:
Erroneous values are copied to all stations.
Consequence: Accepting only small changes to local time.

Many time formats too restricted;
e.g.: NTP protocol includes only years up to 2036
For time services and global synchronization of clocks synchronization see
Kopetz, 1997.
ΗΜΥ653 Δ05-6 Embedded Systems Software.205
© Θεοχαρίδης, ΗΜΥ, 2017
- Real-time OS (3) 3.
The OS must be fast
Practically important.
[Takada, 2001]
ΗΜΥ653 Δ05-6 Embedded Systems Software.206
© Θεοχαρίδης, ΗΜΥ, 2017
RTOS-Kernels

Distinction between

real-time kernels and modified kernels of standard OSes.
Distinction between
 general RTOSes and RTOSes for specific domains,
 standard APIs (e.g. POSIX RT-Extension of Unix, ITRON, OSEK) or
proprietary APIs.
ΗΜΥ653 Δ05-6 Embedded Systems Software.207
© Θεοχαρίδης, ΗΜΥ, 2017
Functionality of RTOS-Kernels

Includes





processor management,
memory management,
resource management
and timer management;
task management (resume, wait etc),
inter-task communication and synchronization.
ΗΜΥ653 Δ05-6 Embedded Systems Software.208
© Θεοχαρίδης, ΗΜΥ, 2017
Requirements for RTOS

Predictability of timing




The timing behavior of the OS must be predictable
For all services of the OS, there is an upper bound on the
execution time
Scheduling policy must be deterministic
The period during which interrupts are disabled must be short
(to avoid unpredictable delays in the processing of critical
events)
209
ΗΜΥ653 Δ05-6 Embedded Systems Software.209
© Θεοχαρίδης, ΗΜΥ, 2017
Requirements for RTOS

OS should manage timing and scheduling


OS possibly has to be aware of task deadlines;
(unless scheduling is done off-line).
Frequently, the OS should provide precise time services with
high resolution.
- Important if internal processing of the embedded system is linked
to an absolute time in the physical environment

Speed:

The OS must be fast
210
ΗΜΥ653 Δ05-6 Embedded Systems Software.210
© Θεοχαρίδης, ΗΜΥ, 2017
Functionality of RTOS Kernel

Processor management

Memory management

Timer management

Task management (resume, wait etc)

Inter-task communication and synchronization
resource management
211
ΗΜΥ653 Δ05-6 Embedded Systems Software.211
© Θεοχαρίδης, ΗΜΥ, 2017
Why Use an RTOS?

Can use drivers that are available with an RTOS

Can focus on developing application code, not on
creating or maintaining a scheduling system

Multi-thread support with synchronization

Portability of application code to other CPUs

Resource handling by RTOS

Add new features without affecting higher priority
functions

Support for upper layer protocols such as:


TCP/IP, USB, Flash Systems, Web Servers,
CAN protocols, Embedded GUI, SSL, SNMP
212
ΗΜΥ653 Δ05-6 Embedded Systems Software.212
© Θεοχαρίδης, ΗΜΥ, 2017
Classification of RTOS

RT kernels vs modified kernels of standard OS


Fast proprietary kernels: may be inadequate for complex
systems, because they are designed to be fast rather than to be
predictable in every respect, e.g., QNX, PDOS, VCOS, VTRX32,
VxWORKS
RT extensions to standard OS: RT-kernel runs all RT-tasks and
standard-OS executed as one task on it

General RTOS vs RTOS for specific domains

Standard APIs vs proprietary APIs

e.g. POSIX RT-Extension of Unix, ITRON, OSEK)
213
Source: R. Gupta, UCSD
ΗΜΥ653 Δ05-6 Embedded Systems Software.213
© Θεοχαρίδης, ΗΜΥ, 2017
Classes of RTOSes according to R. Gupta
1. Fast proprietary kernels

Fast proprietary kernels
For complex systems, these kernels are inadequate,
because they are designed to be fast, rather than to be
predictable in every respect

[R. Gupta, UCI/UCSD]
Examples include
QNX, PDOS, VCOS, VTRX32, VxWORKS.
ΗΜΥ653 Δ05-6 Embedded Systems Software.214
© Θεοχαρίδης, ΗΜΥ, 2017
Classes of RTOSes according to R. Gupta
2. Real-time extensions to standard OSs

Real-time extensions to standard OSes:
Attempt to exploit comfortable main stream OSes.
RT-kernel running all RT-tasks.
Standard-OS executed as one task.
+ Crash of standard-OS does not affect RT-tasks;
- RT-tasks cannot use Standard-OS services;
less comfortable than expected
ΗΜΥ653 Δ05-6 Embedded Systems Software.215
© Θεοχαρίδης, ΗΜΥ, 2017
Example: RT-Linux

Init
Bash
Mozilla
scheduler
Linux-Kernel
RT-tasks cannot use
standard OS calls.
Commercially available
from fsmlabs
(www.fsmlabs.com)
RT-Task
RT-Task
driver
interrupts
RT-Linux
RT-Scheduler
I/O
interrupts
interrupts
Hardware
ΗΜΥ653 Δ05-6 Embedded Systems Software.216
© Θεοχαρίδης, ΗΜΥ, 2017
Example: Posix 1.b RT-extensions to Linux

Standard scheduler can be replaced by POSIX scheduler
implementing priorities for RT tasks
RT-Task
RT-Task
Init
Bash
POSIX 1.b scheduler
Linux-Kernel
Mozilla
Special RT-calls and
standard OS calls available.
Easy programming, no
guarantee for meeting
deadline
driver
I/O, interrupts
Hardware
ΗΜΥ653 Δ05-6 Embedded Systems Software.217
© Θεοχαρίδης, ΗΜΥ, 2017
Evaluation (Rajesh Gupta)

According to Gupta, trying to use a version of a standard
OS:

not the correct approach because too many basic and
inappropriate underlying assumptions still exist such as
optimizing for the average case (rather than the worst
case), ... ignoring most if not all semantic information, and
independent CPU scheduling and resource allocation.

Dependences between tasks not frequent for most
applications of std. OSs & therefore frequently ignored.

Situation different for ES since dependences between
tasks are quite common.
ΗΜΥ653 Δ05-6 Embedded Systems Software.218
© Θεοχαρίδης, ΗΜΥ, 2017
Classes of RTOSes according to R. Gupta
3. Research systems trying to avoid limitations
Research systems trying to avoid limitations.
Include MARS, Spring, MARUTI, Arts, Hartos, DARK,
and Melody
 Research issues [Takada, 2001]:







low overhead memory protection,
temporal protection of computing resources
RTOSes for on-chip multiprocessors
support for continuous media
quality of service (QoS) control.
Competition between

Market

traditional vendors (e.g. Wind River Systems) and
Embedded Windows XP and Windows CE
ΗΜΥ653 Δ05-6 Embedded Systems Software.219
© Θεοχαρίδης, ΗΜΥ, 2017
Virtual machines


Emulate several processors on a single real processor
Running
- As Single process (Java virtual machine)
- On bare hardware

- Allows several operating systems to be executed on top
- Very good shielding between applications
Temporal behavior
ΗΜΥ653 Δ05-6 Embedded Systems Software.220
© Θεοχαρίδης, ΗΜΥ, 2017
Resource access protocols
Critical sections: sections of code at which
exclusive access to some resource must be guaranteed.
 Can be guaranteed with semaphores S or “mutexes”.

Task 1
P(S)
V(S)
P(S) checks semaphore to see if resource
is available
and if yes, sets S to “used“.
Uninterruptible operations!
If no, calling task has to wait.
Task 2
Mutually
exclusive
access
to resource
guarded by
S
ΗΜΥ653 Δ05-6 Embedded Systems Software.221
P(S)
V(S): sets S to “unused“ and starts
sleeping task (if any).
V(S)
© Θεοχαρίδης, ΗΜΥ, 2017
Blocking due to mutual exclusion

Priority T1 assumed to be > than priority of T2.

If T2 requests exclusive access first (at t0), T1 has to wait
until T2 releases the resource (time t3), thus inverting the
priority:
In this example:
blocking is bounded by length of critical section of T2.
ΗΜΥ653 Δ05-6 Embedded Systems Software.222
© Θεοχαρίδης, ΗΜΥ, 2017
Blocking with >2 tasks can exceed
the length of any critical section

Priority of T1 > priority of T2 > priority of T3.

T2 preempts T3:

T2 can prevent T3 from releasing the resource.
normal execution
critical section
Priority inversion!
ΗΜΥ653 Δ05-6 Embedded Systems Software.223
© Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.224
© Θεοχαρίδης, ΗΜΥ, 2017
The MARS Pathfinder problem (1)

“But a few days into the mission, not
long after Pathfinder started gathering
meteorological data, the spacecraft
began experiencing total system
resets, each resulting in losses of data.
The press reported these failures in
terms such as "software glitches" and
"the computer was trying to do too
many things at once".” …
http://research.microsoft.com/~mbj/Mars_Pathfinder/Mars_Pathfinder.html
ΗΜΥ653 Δ05-6 Embedded Systems Software.225
© Θεοχαρίδης, ΗΜΥ, 2017
The MARS Pathfinder problem (2)

“VxWorks (the OS used) provides preemptive priority
scheduling of threads. Tasks on the Pathfinder spacecraft
were executed as threads with priorities that were assigned
in the usual manner reflecting the relative urgency of these
tasks.”

“Pathfinder contained an "information bus", which you can
think of as a shared memory area used for passing
information between different components of the spacecraft.”

A bus management task ran frequently with high priority to move
certain kinds of data in and out of the information bus. Access to the
bus was synchronized with mutual exclusion locks (mutexes).”
http://research.microsoft.com/~mbj/Mars_Pathfinder/Mars_Pathfinder.html
ΗΜΥ653 Δ05-6 Embedded Systems Software.226
© Θεοχαρίδης, ΗΜΥ, 2017
The MARS Pathfinder problem (3)


The meteorological data gathering task ran as an infrequent, low
priority thread, … When publishing its data, it would acquire a
mutex, do writes to the bus, and release the mutex. ..
The spacecraft also contained a communications task that ran with
medium priority.”

High priority:
retrieval of data from shared memory
Medium priority: communications task
Low priority:
thread collecting meteorological data
http://research.microsoft.com/~mbj/Mars_Pathfinder/Mars_Pathfinder.html
ΗΜΥ653 Δ05-6 Embedded Systems Software.227
© Θεοχαρίδης, ΗΜΥ, 2017
The MARS Pathfinder problem (4)

“Most of the time this combination worked fine. However, very
infrequently it was possible for an interrupt to occur that caused
the (medium priority) communications task to be scheduled
during the short interval while the (high priority) information bus
thread was blocked waiting for the (low priority) meteorological
data thread. In this case, the long-running communications
task, having higher priority than the meteorological task, would
prevent it from running, consequently preventing the blocked
information bus task from running. After some time had
passed, a watchdog timer would go off, notice that the data bus
task had not been executed for some time, conclude that
something had gone drastically wrong, and initiate a total
system reset. This scenario is a classic case of priority
inversion.”
http://research.microsoft.com/~mbj/Mars_Pathfinder/Mars_Pathfinder.html
ΗΜΥ653 Δ05-6 Embedded Systems Software.228
© Θεοχαρίδης, ΗΜΥ, 2017
Coping with priority inversion:
the priority inheritance protocol




Tasks are scheduled according to their active priorities. Tasks with the
same priorities are scheduled FCFS.
If task T1 executes P(S) & exclusive access granted to T2: T1 will
become blocked.
If priority(T2) < priority(T1): T2 inherits the priority of T1.
 T2 resumes.
Rule: tasks inherit the highest priority of tasks blocked by it.
When T2 executes V(S), its priority is decreased to the highest priority
of the tasks blocked by it.
If no other task blocked by T2: priority(T2):= original value.
Highest priority task so far blocked on S is resumed.
Transitive: if T2 blocks T1 and T1 blocks T0,
then T2 inherits the priority of T0.
ΗΜΥ653 Δ05-6 Embedded Systems Software.229
© Θεοχαρίδης, ΗΜΥ, 2017
Example

How would priority inheritance affect our example with 3
tasks?
T3 inherits the priority of
T1 and T3 resumes.
V(S)
ΗΜΥ653 Δ05-6 Embedded Systems Software.230
© Θεοχαρίδης, ΗΜΥ, 2017
P(a)
V(a)
P(b)
P(a) P(b)
ΗΜΥ653 Δ05-6 Embedded Systems Software.231
V(b)
V(b)
V(a)
[P/V added@unido]
© Θεοχαρίδης, ΗΜΥ, 2017
V(a)
P(a)
P(a) P(b)
P(b)
ΗΜΥ653 Δ05-6 Embedded Systems Software.232
V(b) V(a)
V(b)
[P/V added@unido]
© Θεοχαρίδης, ΗΜΥ, 2017
ΗΜΥ653 Δ05-6 Embedded Systems Software.233
© Θεοχαρίδης, ΗΜΥ, 2017
Priority Inversion Problem

Pathfinder mission on Mars in 1997


Used VxWorks, an RTOS kernel, from WindRiver
Software problems caused the total system resets of the
Pathfinder spacecraft in mission
- Watchdog timer goes off, informing that something has gone
dramatically wrong and initiating the system reset
234
ΗΜΥ653 Δ05-6 Embedded Systems Software.234
© Θεοχαρίδης, ΗΜΥ, 2017
Priority Inversion Problem

VxWorks provides preemptive priority scheduling of
threads

Tasks on the Pathfinder spacecraft were executed as threads
with priorities that were assigned in the usual manner reflecting
the relative urgency of these tasks.
Task 1 tries to get the semaphore
Task 1 preempts Task3
Task 1 gets the semaphore
and execute
Priority Inversion
Task 1
(highest priority)
Task 2
(medium priority)
Task 2 preempts task 3
Task 3
(lowest priority)
Task 3 is resumed
Task 3 gets semaphore
Task 3 is resumed
Time
Task 3 releases the semaphore
235
ΗΜΥ653 Δ05-6 Embedded Systems Software.235
© Θεοχαρίδης, ΗΜΥ, 2017
Priority Inheritance

A chain of processes could all be accessing resources that the highpriority process needs


All these processes inherit the high priority until they are done with the resource
When they are finished, their priority reverts to its original value
Task 1 tries to get the semaphore
(Priority of Task 3 is raised to Task 1’s)
Task 1 preempts Task3
Priority Inversion
Task 1 completes
Task 1
(highest priority)
Task 2
(medium priority)
Task 3
(lowest priority)
Task 3 gets semaphore
Task 3 is resumed
with the highest priority
Time
Task 3 releases the semaphore
236
ΗΜΥ653 Δ05-6 Embedded Systems Software.236
© Θεοχαρίδης, ΗΜΥ, 2017
Priority inversion on Mars

Priority inheritance also solved the Mars Pathfinder
problem: the VxWorks operating system used in the
pathfinder implements a flag for the calls to mutex
primitives. This flag allows priority inheritance to be set to
“on”. When the software was shipped, it was set to “off”.
The problem on Mars was corrected by
using the debugging facilities of
VxWorks to change the flag to “on”,
while the Pathfinder was already on the
Mars [Jones, 1997].
ΗΜΥ653 Δ05-6 Embedded Systems Software.237
© Θεοχαρίδης, ΗΜΥ, 2017
Remarks on priority inheritance protocol

Possible large number of tasks with high priority.

Possible deadlocks.

Ongoing debate about problems with the protocol:
Victor Yodaiken: Against Priority Inheritance, Sept. 2004,
http://www.fsmlabs.com/resources/white_papers/priority-inheritance/

Finds application in ADA: During rendez-vous,
task priority is set to the maximum.

Protocol for fixed set of tasks: priority ceiling protocol.
ΗΜΥ653 Δ05-6 Embedded Systems Software.238
© Θεοχαρίδης, ΗΜΥ, 2017
Summary

General requirements for embedded operating systems
- Configurability
- I/O
- Interrupts


General properties of real-time operating systems
- Predictability
- Time services,
- Synchronization
- Classes of RTOSs,
- Device driver embedding
Priority inversion
- The problem
- Priority inheritance
ΗΜΥ653 Δ05-6 Embedded Systems Software.239
© Θεοχαρίδης, ΗΜΥ, 2017
Complexity trends in OS

OS functionality drives the
complexity
Small controllers
sensors
Home appliances
Mobile phones
PDAs
Game Machines
Router
240
ΗΜΥ653 Δ05-6 Embedded Systems Software.240
© Θεοχαρίδης, ΗΜΥ, 2017
Requirements of EOS

Memory Resident: size is important consideration


Data structure optimized
Kernel optimized and usually in assembly language

Support of signaling & interrupts

Real-time scheduling


tight-coupled scheduler and interrupts
Power Management capabilities


Power aware schedule
Control of non-processor resources
241
ΗΜΥ653 Δ05-6 Embedded Systems Software.241
© Θεοχαρίδης, ΗΜΥ, 2017
Embedded OS Design approach


Traditional OS: Monolithic or Distribute
Embedded: Layered is the key (Constantine D. P, UIUC
2000)
Basic Loader
Power
Management
Interrupt & signalling
Real-Time Scheduler
Memory Management
Networking support
Custom device
support
242
ΗΜΥ653 Δ05-6 Embedded Systems Software.242
© Θεοχαρίδης, ΗΜΥ, 2017
Power Management by OS

Static Approach: Rely on pre-set parameters


Example: switch off the power to devices that are not in use for a
while (pre-calculated number of cycles). Used in laptops now.
Dynamic Approach: Based on dynamic condition of
workloads and per specification of power optimization
guidelines

Example: Restrict multitasking progressively, reduce context
switching, even avoid cache/memory access, etc..
243
ΗΜΥ653 Δ05-6 Embedded Systems Software.243
© Θεοχαρίδης, ΗΜΥ, 2017
H/W-S/W Architecture
A
significant part of the problem is deciding which
parts should be in s/w on programmable
processors, and which in specialized h/w
 Today:
Ad hoc approaches based on earlier experience with
similar products, & on manual design
 H/W-S/W partitioning decided at the beginning, and
then designs proceed separately

ΗΜΥ653 Δ05-6 Embedded Systems Software.244
© Θεοχαρίδης, ΗΜΥ, 2017
Embedded System Design

CAD tools take care of h/w fairly well


Although a productivity gap emerging
But, S/W is a different story…

HLLs such as C help, but can’t cope with complexity and
performance constraints
Holy Grail for Tools People: H/W-like synthesis &
verification from a behavior description of the
whole system at a high level of abstraction using
formal computation models
ΗΜΥ653 Δ05-6 Embedded Systems Software.245
© Θεοχαρίδης, ΗΜΥ, 2017
Embedded System Design from a Design
Technology Perspective
 Intertwined
subtasks
 Specification/modeling
 H/W
ASIC
Processor
Analog I/O
Memory
& S/W partitioning
 Scheduling & resource
allocations
 H/W & S/W implementation
 Verification
DSP
Code
ΗΜΥ653 Δ05-6 Embedded Systems Software.246
& debugging
 Crucial
is the co-design and
joint optimization of hardware
and software
© Θεοχαρίδης, ΗΜΥ, 2017
Embedded System Design Flow

Environ
-ment


ASIC
Analog I/O
Processor
Memory


DSP
Code
ΗΜΥ653 Δ05-6 Embedded Systems Software.247
Modeling
 the system to be designed, and experimenting with
algorithms involved;
Refining (or “partitioning”)
 the function to be implemented into smaller,
interacting pieces;
HW-SW partitioning: Allocating
 elements in the refined model to either (1) HW units,
or (2) SW running on custom hardware or a suitable
programmable processor.
Scheduling
 the times at which the functions are executed. This is
important when several modules in the partition share
a single hardware unit.
Mapping (Implementing)
 a functional description into (1) software that runs on
a processor or (2) a collection of custom, semicustom, or commodity HW.
© Θεοχαρίδης, ΗΜΥ, 2017
Microsoft Mobility Platforms
Pocket PC
Smart
Personal
Objects
• One-way
network
• Information
consumption
Windows CE
• Information
consumption
• View and some data
entry
Smartphone
• Integrated PDA with
• Information
phone
consumption
• Interoperability with
• Primarily data
Office, Exchange
viewing
• Integrated phone and SQL Server
• .NET Compact
with PDA
Framework
• Interoperability
• ASP.NET mobile
with Exchange
controls
• .NET Compact
Framework
• ASP.NET mobile
controls
Tablet PC
Notebook PC
• Keyboard and mouse
input methods
• Keyboard centric at the
desk, pen & keyboard
away from the desk
• Keyboard, mouse plus
pen, ink, and speech
input methods
• Full .NET framework
preinstalled
• Full .NET framework
available
• Pen, ink, handwriting and
speech recognition API’s
• Complex document
authoring, editing and
reading
• Keyboard centric at the
desk
Windows Mobile
ΗΜΥ653 Δ05-6 Embedded Systems Software.248
• Complex document
authoring, editing and
active reading
• Note taking and ink
annotating
Increased Functionality
Windows XP/XPE
© Θεοχαρίδης, ΗΜΥ, 2017
Windows Embedded Family
ΗΜΥ653 Δ05-6 Embedded Systems Software.249
© Θεοχαρίδης, ΗΜΥ, 2017
Embedded Linux in
various devices:
ΗΜΥ653 Δ05-6 Embedded Systems Software.250
NASA personal
assistant
© Θεοχαρίδης, ΗΜΥ, 2017
Thread vs. Process
http://www.lynuxworks.com/products/posix/processes.php3
ΗΜΥ653 Δ05-6 Embedded Systems
Software.251
© Θεοχαρίδης, ΗΜΥ, 2017
Thread vs. Process
http://www.lynuxworks.com/products/posix/processes.php3
ΗΜΥ653 Δ05-6 Embedded Systems
Software.252
© Θεοχαρίδης, ΗΜΥ, 2017
Thread vs. Process
http://www.lynuxworks.com/products/posix/processes.php3
ΗΜΥ653 Δ05-6 Embedded Systems
Software.253
© Θεοχαρίδης, ΗΜΥ, 2017
Developing Process
From system
design
t
e
s
t
Linux OS
select
OS Porting and
improvement
Driver and
Application
software
development
ΗΜΥ653 Δ05-6 Embedded Systems Software.254
Rehat,bluecat,RT
Linux,Monta Vista
Linux,RTAI,…
http://linux.org
http://www.gn
u.org…
Tekram,HP,Intel,
…
© Θεοχαρίδης, ΗΜΥ, 2017
Windows embedded VS
Cont.
Linux Embedded
Both
on
strong
uptake
curve!
Oses targeted in current and next
embedded projects,2002,data from EDC
ΗΜΥ653 Δ05-6 Embedded Systems Software.255
© Θεοχαρίδης, ΗΜΥ, 2017
Application Knowledge
Embedded System Software
2:
Specification
3:
ES-hardware
4: system
software (RTOS,
middleware, …)
ΗΜΥ653 Δ05-6 Embedded Systems Software.256
Design
repository
6: Application
mapping
Design
8:
Test
7: Optimization
5: Validation &
Evaluation (energy,
cost, performance, …)
© Θεοχαρίδης, ΗΜΥ, 2017
Mapping of Applications to Platforms
© Renesas, Thiele
ΗΜΥ653 Δ05-6 Embedded Systems Software.257
© Θεοχαρίδης, ΗΜΥ, 2017
Distinction between mapping problems
ΗΜΥ653 Δ05-6 Embedded Systems Software.258
© Θεοχαρίδης, ΗΜΥ, 2017
Problem Description

Given



A set of applications
Scenarios on how these applications will be used
A set of candidate architectures comprising
- (Possibly heterogeneous) processors
- (Possibly heterogeneous) communication architectures
- Possible scheduling policies

Find




A mapping of applications to processors
Appropriate scheduling techniques (if not fixed)
A target architecture (if DSE is included)
Objectives


Keeping deadlines and/or maximizing performance
Minimizing cost, energy consumption
ΗΜΥ653 Δ05-6 Embedded Systems Software.259
© Θεοχαρίδης, ΗΜΥ, 2017
Related Work






Mapping to EXUs in automotive design
Scheduling theory:
Provides insight for the mapping task  start times
Hardware/software partitioning:
Can be applied if it supports multiple processors
High performance computing (HPC)
Automatic parallelization, but only for
- single applications,
- fixed architectures,
- no support for scheduling,
- memory and communication model usually different
High-level synthesis
Provides useful terms like scheduling, allocation, assignment
Optimization theory
ΗΜΥ653 Δ05-6 Embedded Systems Software.260
© Θεοχαρίδης, ΗΜΥ, 2017
Scope of mapping algorithms
Useful terms from hardware synthesis:
 Resource Allocation
Decision concerning type and number of available resources
 Resource Assignment
Mapping: Task  (Hardware) Resource
 xx to yy binding:
Describes a mapping from behavioral to structural domain, e.g. task to
processor binding, variable to memory binding
 Scheduling
Mapping: Tasks  Task start times
Sometimes, resource assignment is
considered being included in scheduling.
ΗΜΥ653 Δ05-6 Embedded Systems Software.261
© Θεοχαρίδης, ΗΜΥ, 2017
Classes of mapping algorithms
(considered in this course)

Classical scheduling algorithms
Mostly for independent tasks & ignoring communication, mostly for
mono- and homogeneous multiprocessors

Hardware/software partitioning
Dependent tasks, heterogeneous systems,
focus on resource assignment

Dependent tasks as considered in architectural synthesis
Initially designed in different context, but applicable

Design space exploration using genetic algorithms
Heterogeneous systems, incl. communication modeling
ΗΜΥ653 Δ05-6 Embedded Systems Software.262
© Θεοχαρίδης, ΗΜΥ, 2017
Real-time scheduling

Assume that we are given a task graph G=(V,E).

Def.: A schedule s of G is a mapping
VT
of a set of tasks V to start times from domain T.
G=(V,E)
V1
V2
V3
V4
s
T
t
Typically, schedules have to respect a number of constraints, incl. resource
constraints, dependency constraints, deadlines.
Scheduling = finding such a mapping.
ΗΜΥ653 Δ05-6 Embedded Systems Software.263
© Θεοχαρίδης, ΗΜΥ, 2017
Hard and soft deadlines

Def.: A time-constraint (deadline) is called hard if not
meeting that constraint could result in a catastrophe
[Kopetz, 1997].

All other time constraints are called soft.

We will focus on hard deadlines.
ΗΜΥ653 Δ05-6 Embedded Systems Software.264
© Θεοχαρίδης, ΗΜΥ, 2017
Periodic and aperiodic tasks

Def.: Tasks which must be executed once every p units of
time are called periodic tasks. p is called their period. Each
execution of a periodic task is called a job.

All other tasks are called aperiodic.

Def.: Tasks requesting the processor at unpredictable
times are called sporadic, if there is a minimum separation
between the times at which they request the processor.
ΗΜΥ653 Δ05-6 Embedded Systems Software.265
© Θεοχαρίδης, ΗΜΥ, 2017
Preemptive and non-preemptive scheduling


Non-preemptive schedulers:
Tasks are executed until they are done.
Response time for external events may be quite long.
Preemptive schedulers: To be used if
- some tasks have long execution times or
- if the response time for external events to be short.
ΗΜΥ653 Δ05-6 Embedded Systems Software.266
© Θεοχαρίδης, ΗΜΥ, 2017
Centralized and distributed scheduling

Centralized and distributed scheduling:
Multiprocessor scheduling either locally on 1 or on several processors.

Mono- and multi-processor scheduling:
- Simple scheduling algorithms handle single processors,
- more complex algorithms handle multiple processors.
– algorithms for homogeneous multi-processor systems
– algorithms for heterogeneous multi-processor systems (includes HW
accelerators as special case).
ΗΜΥ653 Δ05-6 Embedded Systems Software.267
© Θεοχαρίδης, ΗΜΥ, 2017
Dynamic/online scheduling

Dynamic/online scheduling:
Processor allocation decisions (scheduling) at
run-time; based on the information about the
tasks arrived so far.
ΗΜΥ653 Δ05-6 Embedded Systems Software.268
© Θεοχαρίδης, ΗΜΥ, 2017
Static/offline scheduling

Static/offline scheduling:
Scheduling taking a priori knowledge about arrival times,
execution times, and deadlines into account.
Dispatcher allocates processor when interrupted by timer.
Timer controlled by a table generated at design time.
ΗΜΥ653 Δ05-6 Embedded Systems Software.269
© Θεοχαρίδης, ΗΜΥ, 2017
Time-triggered systems (1)

In an entirely time-triggered system, the temporal control
structure of all tasks is established a priori by off-line
support-tools. This temporal control structure is encoded in a
Task-Descriptor List (TDL) that contains the cyclic
schedule for all activities of the node. This schedule
considers the required precedence and mutual exclusion
relationships among the tasks such that an explicit
coordination of the tasks by the operating system at run time
is not necessary. ..

The dispatcher is activated by the synchronized clock tick. It
looks at the TDL, and then performs the action that has been
planned for this instant [Kopetz].
ΗΜΥ653 Δ05-6 Embedded Systems Software.270
© Θεοχαρίδης, ΗΜΥ, 2017
Time-triggered systems (2)

… pre-run-time scheduling is often the only practical
means of providing predictability in a complex system.
[Xu, Parnas].

It can be easily checked if timing constraints are met.
The disadvantage is that the response to sporadic events
may be poor.
ΗΜΥ653 Δ05-6 Embedded Systems Software.271
© Θεοχαρίδης, ΗΜΥ, 2017
Schedulability

Set of tasks is schedulable under a set of
constraints, if a schedule exists for that set
of tasks & constraints.

Exact tests are NP-hard in many
situations.

Sufficient tests: sufficient conditions for
schedule checked. (Hopefully) small
probability of not guaranteeing a schedule
even though one exists.

Necessary tests: checking necessary
conditions. Used to show no schedule
exists. There may be cases in which no
schedule exists & we cannot prove it.
ΗΜΥ653 Δ05-6 Embedded Systems Software.272
sufficient
schedulable
necessary
© Θεοχαρίδης, ΗΜΥ, 2017
Cost functions

Cost function: Different algorithms aim at minimizing
different functions.

Def.: Maximum lateness =
maxall tasks (completion time – deadline)
Is <0 if all tasks complete before deadline.
T1
T2
t
ΗΜΥ653 Δ05-6 Embedded Systems Software.273
© Θεοχαρίδης, ΗΜΥ, 2017
Aperiodic scheduling
- Scheduling with no precedence constraints 
Let {Ti } be a set of tasks. Let:




ci be the execution time of Ti ,
di be the deadline interval, that is,
the time between Ti becoming available
and the time until which Ti has to finish execution.
ℓi be the laxity or slack, defined as ℓi = di - ci
fi be the finishing time.
i
di
ci
ℓi
t
ΗΜΥ653 Δ05-6 Embedded Systems Software.274
© Θεοχαρίδης, ΗΜΥ, 2017
Uniprocessor with equal arrival times

Preemption is useless.

Earliest Due Date (EDD): Execute task with earliest due
date (deadline) first.
fi
fi
fi
EDD requires all tasks to be sorted by their (absolute) deadlines. Hence, its
complexity is O(n log(n)).
ΗΜΥ653 Δ05-6 Embedded Systems Software.275
© Θεοχαρίδης, ΗΜΥ, 2017
Optimality of EDD

EDD is optimal, since it follows Jackson's rule:
Given a set of n independent tasks, any algorithm that
executes the tasks in order of non-decreasing (absolute)
deadlines is optimal with respect to minimizing the
maximum lateness.

Proof (See Buttazzo, 2002):

Let  be a schedule produced by any algorithm A

If A  EDD   Ta, Tb, da ≤ db, Tb immediately precedes Ta in .

Let ' be the schedule obtained by exchanging Ta and Tb.
ΗΜΥ653 Δ05-6 Embedded Systems Software.276
© Θεοχαρίδης, ΗΜΥ, 2017
Exchanging Ta and Tb cannot increase lateness

Max. lateness for Ta and Tb in  is Lmax(a,b)=fa-da
Max. lateness for Ta and Tb in ' is L'max(a,b)=max(L'a,L'b)
Two possible cases
1.
L'a ≥ L'b:  L'max(a,b) = f'a – da < fa – da = Lmax(a,b)
since Ta starts earlier in schedule '.
2.
L'a ≤ L'b:  L'max(a,b) = f'b – db = fa – db ≤ fa – da = Lmax(a,b) since fa=f'b and
da ≤ db
 L'max(a,b) ≤ Lmax(a,b)

Tb
'
Ta
Ta
Tb
fa=f'b
ΗΜΥ653 Δ05-6 Embedded Systems Software.277
© Θεοχαρίδης, ΗΜΥ, 2017
EDD is optimal

Any schedule  with lateness L can be transformed into an
EDD schedule n with lateness Ln ≤ L, which is the
minimum lateness.

EDD is optimal (q.e.d.)
ΗΜΥ653 Δ05-6 Embedded Systems Software.278
© Θεοχαρίδης, ΗΜΥ, 2017
Earliest Deadline First (EDF)
- Horn’s Theorem -

Different arrival times: Preemption potentially reduces
lateness.

Theorem [Horn74]: Given a set of n independent tasks with
arbitrary arrival times, any algorithm that at any instant
executes the task with the earliest absolute deadline among
all the ready tasks is optimal with respect to minimizing the
maximum lateness.
ΗΜΥ653 Δ05-6 Embedded Systems Software.279
© Θεοχαρίδης, ΗΜΥ, 2017
Earliest Deadline First (EDF)
- Algorithm 
Earliest deadline first (EDF) algorithm:




Each time a new ready task arrives:
It is inserted into a queue of ready tasks, sorted by their absolute
deadlines. Task at head of queue is executed.
If a newly arrived task is inserted at the head of the queue, the
currently executing task is preempted.
Straightforward approach with sorted lists (full comparison
with existing tasks for each arriving task) requires run-time
O(n2); (less with binary search or bucket arrays).
Sorted queue
Executing task
ΗΜΥ653 Δ05-6 Embedded Systems Software.280
© Θεοχαρίδης, ΗΜΥ, 2017
Earliest Deadline First (EDF)
- Example -
Earlier deadline
 preemption
Later deadline
 no preemption
ΗΜΥ653 Δ05-6 Embedded Systems Software.281
© Θεοχαρίδης, ΗΜΥ, 2017
Optimality of EDF

To be shown: EDF minimizes maximum lateness.

Proof (Buttazzo, 2002):








Let  be a schedule produced by generic schedule A
Let EDF: schedule produced by EDF
Preemption allowed: tasks executed in disjoint time intervals
 divided into time slices of 1 time unit each
Time slices denoted by [t, t+1)
Let (t): task executing in [t, t+1)
Let E(t): task which, at time t, has the earliest deadline
Let tE(t): time (t) at which the next slice of task E(t) begins its
execution in the current schedule
ΗΜΥ653 Δ05-6 Embedded Systems Software.282
© Θεοχαρίδης, ΗΜΥ, 2017
Optimality of EDF (2)

If   EDF, then there exists time t: (t)  E(t)

Idea: swapping (t) and E(t) cannot increase max.
lateness. t=4; (t)=4; E(t)=2; tE=6; (tE)=2
T1
t
T2
t
T3
t
T4
0
2
4
6
8
10
12
14
16
(t)=2; (tE)=4
T1
t
t
T2
t
T3
t
T4
0
2
4
6
8
10
12
14
16
t
If (t) starts at t=0 and D=maxi{di } then EDF can be obtained from  by at most
D transpositions.
ΗΜΥ653 Δ05-6 Embedded Systems Software.283
[Buttazzo,
© Θεοχαρίδης,2002]
ΗΜΥ, 2017
Optimality of EDF (3)

Algorithm
interchange:

{ for (t=0 to D-1) {

if ((t)  E(t)) {

(tE) = (t);

(t) = E(t); }}}
Using the same argument as in the proof of
Jackson’s algorithm, it is easy to show that
swapping cannot increase maximum
lateness; hence EDF is optimal.
Does interchange preserve schedulability?
1. task E(t) moved ahead: meeting deadline in new schedule if meeting
deadline in 
2. task (t) delayed: if (t) is feasible, then (tE+1) ≤ dE, where dE is the earliest
deadline. Since dE ≤ di for any i, we have tE+1 ≤ di, which guarantees
schedulability of the delayed task.
[Buttazzo, 2002]
ΗΜΥ653 Δ05-6 Embedded Systems Software.284
q.e.d.
© Θεοχαρίδης, ΗΜΥ, 2017
Least laxity (LL), Least Slack Time First (LST)

Priorities = decreasing function of the laxity (the less laxity,
the higher the priority); dynamically changing priority;
preemptive.
ℓ
ℓ
ℓ
ℓ
ℓ
ℓ
ℓ
ΗΜΥ653 Δ05-6 Embedded Systems Software.285
ℓ
ℓ
ℓ
ℓ
ℓ
© Θεοχαρίδης, ΗΜΥ, 2017
Properties







Not sufficient to call scheduler & re-compute laxity just at task arrival times.
Overhead for calls of the scheduler.
Many context switches.
Detects missed deadlines early.
LL is also an optimal scheduling for mono-processor systems.
Dynamic priorities  cannot be used with a fixed prio OS.
LL scheduling requires the knowledge of the execution time.
ΗΜΥ653 Δ05-6 Embedded Systems Software.286
© Θεοχαρίδης, ΗΜΥ, 2017
Scheduling without preemption (1)
Lemma: If preemption is not allowed, optimal schedules may have to leave the
processor idle at certain times.
Proof: Suppose: optimal schedulers never leave processor idle.
ΗΜΥ653 Δ05-6 Embedded Systems Software.287
© Θεοχαρίδης, ΗΜΥ, 2017
Scheduling without preemption (2)

T1: periodic, c1 = 2, p1 = 4, d1 = 4

T2: occasionally available at times 4*n+1, c2= 1, d2= 1

T1 has to start at t=0

 deadline missed, but schedule is possible (start T2 first)

 scheduler is not optimal  contradiction! q.e.d.
ΗΜΥ653 Δ05-6 Embedded Systems Software.288
© Θεοχαρίδης, ΗΜΥ, 2017
Scheduling without preemption

Preemption not allowed:  optimal schedules may leave
processor idle to finish tasks with early deadlines arriving
late.

Knowledge about the future is needed for optimal
scheduling algorithms
No online algorithm can decide whether or not to keep
idle.

EDF is optimal among all scheduling algorithms not keeping
the processor idle at certain times.

If arrival times are known a priori, the scheduling problem
becomes NP-hard in general. B&B typically used.
ΗΜΥ653 Δ05-6 Embedded Systems Software.289
© Θεοχαρίδης, ΗΜΥ, 2017
Scheduling with precedence constraints

Task graph and possible
schedule:
ΗΜΥ653 Δ05-6 Embedded Systems Software.290
© Θεοχαρίδης, ΗΜΥ, 2017
Simultaneous Arrival Times:
The Latest Deadline First (LDF) Algorithm
LDF [Lawler, 1973]: reads the task graph and among the
tasks with no successors inserts the one with the latest
deadline into a queue. It then repeats this process, putting
tasks whose successor have all been selected into the
queue.
 At run-time, the tasks are executed in the generated total
order.
 LDF is non-preemptive and is optimal for mono-processors.

If no local deadlines exist, LDF performs just a topological sort.
ΗΜΥ653 Δ05-6 Embedded Systems Software.291
© Θεοχαρίδης, ΗΜΥ, 2017
Asynchronous Arrival Times:
Modified EDF Algorithm

This case can be handled with a modified EDF algorithm.
The key idea is to transform the problem from a given set of
dependent tasks into a set of independent tasks with
different timing parameters [Chetto90].

This algorithm is optimal for mono-processor systems.

If preemption is not allowed, the heuristic algorithm
developed by Stankovic and Ramamritham can be used.
ΗΜΥ653 Δ05-6 Embedded Systems Software.292
© Θεοχαρίδης, ΗΜΥ, 2017
© L. Thiele, 2006
ΗΜΥ653 Δ05-6 Embedded Systems Software.293
© Θεοχαρίδης, ΗΜΥ, 2017
Summary

Definition mapping terms





Resource allocation, assignment, binding, scheduling
Hard vs. soft deadlines
Static vs. dynamic TT-OS
Schedulability
Classical scheduling

Aperiodic tasks
- No precedences
– Simultaneous (EDD)
& Asynchronous Arrival Times (EDF, LL)
- Precedences
– Simultaneous Arrival Times ( LDF)
– Asynchronous Arrival Times ( mEDF)
ΗΜΥ653 Δ05-6 Embedded Systems Software.294
© Θεοχαρίδης, ΗΜΥ, 2017