Download MARKOV CHAIN

MARKOV CHAIN
A M AT 1 8 2 2 N D S E M AY 2 0 1 6 - 2 0 1 7
THE RUSSIAN MATHEMATICIAN
Andrei
Andreyevich
Markov
https://en.wikipedia.org/wiki/Andrey_Markov#/media/File:AAMarkov.jpg
TWO NICE BOOKS
HOW IMPORTANT IS HISTORY
Suppose we have a stochastic process: 𝑋𝑡 . For simplicity, let
us consider a discrete-time process (but we can also consider
continuous-time).
• It is possible that the random variable 𝑋𝑡+1 does not depend
on 𝑋𝑡 , 𝑋𝑡−1 , 𝑋𝑡−2 ,… (similar to my example in the last
lecture)
• It is also possible that the random variable 𝑋𝑡+1 does
depend on 𝑋𝑡 , 𝑋𝑡−1 , 𝑋𝑡−2 ,…
HOW IMPORTANT IS HISTORY
What if I only consider only the present, NOT THE PAST, to
affect the future?
That is, the random variable 𝑋𝑡+1 does depend on 𝑋𝑡 but not
on 𝑋𝑡−1 , 𝑋𝑡−2 ,…
This property is called ”memoryless”, “lack of memory”,
“forgetfulness”.
HOW IMPORTANT IS HISTORY
The process following such memoryless property is called a
MARKOV PROCESS.
Actually, the memoryless property is also called Markov
property. A system following this property can be called
“Markovian”.
The memoryless property makes it possible to easily predict
the behavior of a Markov process.
If we consider a chain with memoryless
property then we have a MARKOV CHAIN.
“Markov chains are the simplest mathematical models for
random phenomena evolving in time”.
“The whole of the mathematical study of stochastic
processes can be regarded as a generalization in one way or
another of the theory of Markov chains”.
- Norris
MARKOV CHAINS
In this lecture, we will focus on discrete-time Markov chains,
but to give you a hint: Poisson process and Birth process are
examples of continuous-time Markov chains.
Continuous-time Markov chains in Queueing Theory
Sample Notation in AMAT 167:
(M/M/2):(FCFS/100/∞)
DISCRETE -TIME
MARKOV CHAIN
MARKOVIAN PROPERTY IN SYMBOLS
𝑷 𝑿𝒕+𝟏 = 𝒋 𝑿𝟎 = 𝒌𝟎 , 𝑿𝟏 = 𝒌𝟏 , … , 𝑿𝒕−𝟏 = 𝒌𝒕−𝟏 , 𝑿𝒕 = 𝒊
= 𝑷 𝑿𝒕+𝟏 = 𝒋 𝑿𝒕 = 𝒊
for all 𝒕 and for any 𝒊, 𝒋, 𝒌𝒋,𝒋=𝟎,𝟏,𝟐…,𝒕−𝟏
TRANSITION PROBABILITIES
The conditional probability
𝑃 𝑋𝑡+1 = 𝑗 𝑋𝑡 = 𝑖 for a Markov Chain is called a
one-step transition probability.
For simplicity, we denote 𝑃 𝑋𝑡+1 = 𝑗 𝑋𝑡 = 𝑖 = 𝑝𝑖𝑗 .
TRANSITION PROBABILITIES
In a Markov Chain, if 𝑃 𝑋𝑡+1 = 𝑗 𝑋𝑡 = 𝑖 =
𝑃 𝑋1 = 𝑗 𝑋0 = 𝑖 for all 𝑡 then the one-step
transition probability is said to be stationary.
TRANSITION PROBABILITIES
The conditional probability
𝑃 𝑋𝑡+𝑛 = 𝑗 𝑋𝑡 = 𝑖 for a Markov Chain is called
an n-step transition probability.
For simplicity, we denote 𝑃 𝑋𝑡+𝑛 = 𝑗 𝑋𝑡 = 𝑖 =
𝑝𝑖𝑗 (𝑛) .
TRANSITION PROBABILITIES
If we have a stationary one-step transition
probability, it follows that
𝑃 𝑋𝑡+𝑛 = 𝑗 𝑋𝑡 = 𝑖 = 𝑃 𝑋𝑛 = 𝑗 𝑋0 = 𝑖 for any
𝑛.
Note: we will just use the term “stationary
transition probability”.
TRANSITION PROBABILITIES
Note that
• 𝑝𝑖𝑗 (1) = 𝑝𝑖𝑗
• 𝑝𝑖𝑗
•
(0)
1, 𝑗 = 𝑖
= 𝑃 𝑋𝑡 = 𝑗 𝑋𝑡 = 𝑖 =
0, 𝑗 ≠ 𝑖
𝑀
(𝑛)
𝑝
𝑗=0 𝑖𝑗
= 1, for all 𝑖, 𝑛 where 𝑀 is the total
number of possible outcomes/states
N-STEP TRANSITION PROBABILITY MATRIX
(N-STEP TRANSITION MATRIX)
State
0
𝐏 (𝑛) =
1
⋮
M
0
…
1
𝑛
𝑝00
(𝑛)
𝑝10
⋮
(𝑛)
𝑝𝑀0
𝑛
𝑝01
(𝑛)
𝑝11
⋮
(𝑛)
𝑝𝑀1
…
…
⋱
…
M
(𝑛)
𝑝0𝑀
(𝑛)
𝑝1𝑀
⋮
(𝑛)
𝑝𝑀𝑀
If n=1, we call this matrix “transition matrix”.
OUR FOCUS
In this lecture,
we will focus on Markov Chains with
• Finite number of states
• Stationary transition probabilities
• Initial probabilities 𝑃 𝑋0 = 𝑖 are known for all 𝑖.
EXAMPLE 1 (TAHA)
A Weather Example
The weather in the town of Centerville can change rather quickly from
day to day. However, the chances of being dry (no rain) tomorrow are
somewhat larger if it is dry today than if it rains today. In particular, the
probability of being dry tomorrow is 0.8 if it is dry today, but is only 0.6 if
it rains today.
Assume that these probabilities do not change if information
about the weather before today is also taken into account.
For 𝑡 = 0, 1, 2, … , the random variable 𝑋𝑡 takes on the values,
0 if day t is dry
𝑋𝑡 =
1 if day t has rain
EXAMPLE 1
𝑝00 = 𝑃 𝑋𝑡+1 = 0 𝑋𝑡 = 0 = 0.8
𝑝01 = 𝑃 𝑋𝑡+1 = 1 𝑋𝑡 = 0 = 0.2
𝑝10 = 𝑃 𝑋𝑡+1 = 0 𝑋𝑡 = 1 = 0.6
𝑝11 = 𝑃 𝑋𝑡+1 = 1 𝑋𝑡 = 1 = 0.4
EXAMPLE 1
Transition matrix:
0.8
𝐏=
0.6
0.8
State
transition
diagram:
State
0
0.2
0.4
0.2
0.6
0.4
State
1
EXAMPLE 2 (TAHA)
An Inventory Example
Dave’s Photography Store has the following
inventory problem. The store stocks a particular
model camera that can be ordered weekly.
For 𝑡 = 1, 2, … , the i.i.d. random variable
𝐷𝑡 ~𝑃𝑜𝑖𝑠𝑠𝑜𝑛(1) is
𝐷𝑡 = demand for camera during week 𝑡.
EXAMPLE 2
For 𝑡 = 0,1, 2, … , let the random variable
𝑋𝑡 = number of cameras on hand at the end of week 𝑡
where 𝑋0 is the initial stock.
At the end of each week, the store places an order that is
delivered in time for the next opening of the store. The
store uses the following order policy:
If 𝑋𝑡 = 0, order 3 cameras.
If 𝑋𝑡 > 0, do not order any cameras.
EXAMPLE 2
The inventory level fluctuates between a minimum
of zero cameras and a maximum of three cameras.
Possible states of 𝑋𝑡 are 0, 1, 2, 3.
The random variables 𝑋𝑡 are dependent and may be
evaluated iteratively by the expression
max{3−𝐷𝑡+1 , 0} if 𝑋𝑡 = 0
𝑋𝑡+1 =
.
max{𝑋𝑡 −𝐷𝑡+1 , 0} if 𝑋𝑡 ≥ 1
EXAMPLE 2
What are the elements of the transition matrix
related to the Markov Chain 𝑋𝑡 ?
𝑝00
𝑝10
𝐏= 𝑝
20
𝑝30
𝑝01
𝑝11
𝑝21
𝑝31
𝑝02
𝑝12
𝑝22
𝑝32
𝑝03
𝑝13
𝑝23
𝑝33
EXAMPLE 2
Since 𝐷𝑡 ~𝑃𝑜𝑖𝑠𝑠𝑜𝑛 𝜆 = 1 and using
𝑃 𝐷𝑡+1 = 𝑛 =
𝜆𝑛 𝑒 −𝜆
,
𝑛!
• 𝑃 𝐷𝑡+1 = 0 = 𝑒
−1
we have
≈ 0.368
• 𝑃 𝐷𝑡+1 = 1 = 𝑒 −1 ≈ 0.368
• 𝑃 𝐷𝑡+1 = 2 =
𝑒 −1
2
≈ 0.184
• 𝑃 𝐷𝑡+1 ≥ 3 = 1 − 𝑃 𝐷𝑡+1 ≤ 2 ≈ 0.08
EXAMPLE 2
For the transition from state 0 to state 𝑖 =
0,1,2,3 (1st row of the transition matrix): Since
𝑋𝑡+1 = max{3−𝐷𝑡+1 , 0} if 𝑋𝑡 = 0, then
• 𝑝00 = 𝑃 𝐷𝑡+1 ≥ 3 ≈ 0.08
• 𝑝01 = 𝑃 𝐷𝑡+1 = 2 ≈ 0.184
• 𝑝02 = 𝑃 𝐷𝑡+1 = 1 ≈ 0.368
• 𝑝03 = 𝑃 𝐷𝑡+1 = 0 ≈ 0.368
EXAMPLE 2
For the transition from state 1 to state 𝑖 =
0,1,2,3 (2nd row of the transition matrix): Since
𝑋𝑡+1 = max{𝑋𝑡 −𝐷𝑡+1 , 0} if 𝑋𝑡 ≥ 1, then
• 𝑝10 = 𝑃 𝐷𝑡+1 ≥ 1 ≈ 0.632 (why?)
• 𝑝11 = 𝑃 𝐷𝑡+1 = 0 ≈0.368
• 𝑝12 = 0
• 𝑝13 = 0
EXAMPLE 2
Doing similar computations to the rest of the
rows, we will have
0.08
0.632
𝐏≈
0.264
0.08
0.184
0.368
0.368
0.184
0.368
0
0.368
0.368
0.368
0
0
0.368
EXAMPLE 2
State transition diagram