Download Introduction to Probability Theory

Applied Probability with R
Sourish Das
sourish@cmi.ac.in
Class Time:
Tuesday – 14:00-15:15 (NKN Hall)
Thursday – 14:00-15:15 (NKN Hall)
Office Hour:
Tuesday – 15:45 -16:45 (New Building - Office No. 410)
Thursday – 15:45 -16:45 (New Building - Office No. 410)
Grading Scheme:
R Assignment (20%)
Class Test (20%)
Midterm Exam (30%)
Final Exam (30%)
Chapter 1: Introduction to Probability
Theory
Subjective Probability
1. It is very probable that the Mangalyaan will land on Mars
2. It is rather improbable that the Chennai weather will improve
by tomorrow
In both sentences probability has the meaning of degree of belief
(in a proposition of the person making statement). This kind of
probability may generally be regarded as subjective.
Objective Probability
1. The probability of getting a head from a throw of a fair coin is
0.5
2. The probability for a newborn to be male is slightly higher
than 0.5
3. The probability for a lamp produced by factory to be defective
is no more than 0.05
Here the term is being used in the context of an experiment. It
now means the long-run relative frequency of some outcome of the
experiment
In our course we shall be concerned on ‘objective probability’.
Random Experiment
By ‘Random Experiment’ we mean an operation that can at least
conceivably be repeated an infinite number of times under
essentially similar conditions and whose outcome cannot be
predited exactly. E.g., Tossing a coin.
Relative Frequency
I
Event It is any outcome of the given random experiment.
This will generally be denoted by one of the letters A,B,C etc.
I
Relative Frequency Let our experiment be repeated n times
and suppose A occurs on fn (A) occasions. Then fn (A) will be
will be called
called the frequency of A and the ratio of fn (A)
n
relative frequency.
Relative Frequency
I
N: Number of times the given experiment is performed
I
fN (A): Number of times the event A occurs among the
repeatations (frequency of A)
I
fN (A)
N
proportion of time A occurs in the set of representations
(aka relative frequency of A in the set of representations)
Understanding Probability through an Experiment
Example 1
Suppose A and B enters into a game where A will first throw a die.
Then B will throw a die. If B’s number greater than A then B will
win. Otherwise A will win. They decided that they will play on 5
different days and each day they will play N times. We will varry
the value of N as 1,10,100,1000 and 10000.
R Code for Example 1
for(i in 1:1000){
A=sample(1:6,1)
B=sample(1:6,1)
if(B>A)win[i]=1
if(B<=A)win[i]=0
}
Example 1 (contd.)
N
1
10
100
1000
10000
Day 1
0.0000
0.4000
0.4500
0.4270
0.4274
Day 2
0.0000
0.4000
0.4100
0.4270
0.4193
Day 3
0.0000
0.2000
0.4100
0.3990
0.4133
Day 4
1.0000
0.3000
0.4900
0.3980
0.4203
Day 5
0.0000
0.6000
0.4000
0.4340
0.4083
1. Note that these are relative frequency.
2. Relative frequency stabilises with increasing N.
3. Guess what would be the probability that B will win in a
particular game?
4. What is the exact probability that B will win in a particular
game?
Example 1 (contd.)
1. This feature of the relative frequencies of any event of the
experiment is called statistical regularity.
2. Further fixed level around which the the relative frequency
stabilises with increasing N is called probability.
3. This is what we meant by saying that the probability of A is
the long run relative frequency of A. Probability is the ‘limit’
of relative frequency.
fN (A)
N→∞ N
P(A) = lim
Event, Elementary Event and Sample Space
By an elementary event of an experiment we mean an outcome
that cannot be decomposed into simpler outcomes. This collection
of all elementary events is called sample space (or sample
description space) of the experiment.
Example 2
In the throwing of a die, we considered two cases:
1. The appearance of a ‘six’ as a possible outcome (example of
elementary event)
2. appearance of an even number (example of just an event)
Figure : Black area reprsents A ∩ B or joint occurance of A and B
Figure : Shaded area reprsents A ∪ B or joint occurance of A or B
Figure : Gray area is complement of A, i.e., Ac
Figure : Shaded area represenrs A difference B, A − B = A ∩ B c of A
–
1
2
3
4
5
Set Theory
Elements or points
Set
Empty Set
Unversal Set
A is a subset of B
6
ω is an element of A
7
A is superset of B
8
A and B are equal
Probability Theory
Elementary events
Event
Impossible event
Sample Space
Occurance of A imples
occurance of B
ω is an elemntary event
favourable to A
Occurance of A implies
occurance of B
A and B are equivalent
Symbol Used
ω, e
A, B, C etc.
φ
Ω
A⊂B
ω∈A
A⊃B
A=B
–
9
Set Theory
Union of A and B
10
Intersection of
A and B
Complement of A
Difference of A and B
11
12
Probability Theory
Occurance of either A
or B or both
Occurance of A and B
Symbol Used
A∪B
Non-occurance of A
Occurance of A without
occurance of B
Ac
A−B
A∩B
Certain other symbol to be used
⇒
‘Implies’
⇐
‘Implied by’
⇔
if and only if (iff)
∃
There exists
∀
‘for all’
Laws of Union and Intersection Operation
1. Complementory: (i) A ∪ B = B ∪ A and (ii) A ∩ B = B ∩ A
2. Associative: (i) (A ∪ B) ∪ C = A ∪ (B ∪ C ) and (ii)
(A ∩ B) ∩ C = A ∩ (B ∩ C )
3. Distributive:
(i) (A ∪ B) ∩ C = (A ∩ C ) ∪ (B ∩ C )
(ii) (A ∩ B) ∪ C = (A ∪ C ) ∩ (B ∪ C )
4. Idempotency: (i) A ∪ A = A and (ii) A ∩ A = A
5. DeMorgan’s Law:
(i) (∪i Ai )c = ∩i Aci
(ii) (∩i Ai )c = ∪i Aci
Exhaustive Events
The events (A1 , A2 , ..., An ) or (A1 , A2 , ...) are said to be exhaustive
(or collectively exhaustive) of some one of the events is sure to
occur,
i.e.,
∪i Ai = Ω.
Mutually Exclusive Events
The events (A1 , A2 , ..., An ) or (A1 , A2 , ...) are said to be mutually
exclusive (or mutually disjoint) if no two events can occur can
simultaneously.
Ai ∩ Aj = φ ∀i, j(i 6= j)
Partition of Sample Space Ω
The events (A1 , A2 , ..., An ) or (A1 , A2 , ...) are said to form partition
of sample space if they are mutually exclusive as well as exhustive.
For any event A, A and Ac necessarily form a parttion of sample
space. Because A and Ac are mutually exclusive as well as
exhustive.
Exercise: Home Work
1. Let Ω = {a, b, c, d, e, f , g , h}, we define three events as
A = {a, b, c} , B = {a, e, f , g } and C = {d, e, g , h} then
what are: (i) A ∪ B, (ii) A ∩ B, (iii) Ac , (iv) (A ∪ B) ∩ C (v)
A − B , (vi) B − A , (vii) Are A, B and C exhaustive? (viii)
Are A, B and C mutually exclusive?
i
) ∀i ∈ N = {1, 2, 3, ...} forms
2. Suppose Ai = [0, i+1
exhaustive events.
2.1 Then define Ω.
2.2 Do Ai form partition of sample space?
Example 1 (Contd.)
Throwing a die twice (or throwing two dice), i.e., A will first throw
a die, then B will throw the same die.
Ω = {(i, j)|i = 1, 2, 3, 4, 5, 6 and
j = 1, 2, 3, 4, 5, 6}
i.e.,




Ω=



(1, 1)
(2, 1)
(3, 1)
(4, 1)
(5, 1)
(6, 1)
(1, 2)
(2, 2)
(3, 2)
(4, 2)
(5, 2)
(6, 2)
(1, 3)
(2, 3)
(3, 3)
(4, 3)
(5, 3)
(6, 3)
(1, 4)
(2, 4)
(3, 4)
(4, 4)
(5, 4)
(6, 4)
(1, 5)
(2, 5)
(3, 5)
(4, 5)
(5, 5)
(6, 5)
(1, 6)
(2, 6)
(3, 6)
(4, 6)
(5, 6)
(6, 6)








Example
Ex 3 In tossing a coin Ω = {H, T }
Ex 4 In tossing three coins (or tossing a coin 3 times)
Ω = {HHH, HHT , HTH, HTT , THH, THT , TTH, TTT }
Ex 5 In tossing a die,
Ω = {1, 2, 3, 4, 5, 6}
Example
Ex 6 Throwing a coin till head apears, define Ω
Ω = {H, TH, TTH, TTTH, ...}
Ex 7 In measuring heights (in cm) of students in CMI, define Ω
Ω = {ω | 0 < ω < ∞}
where ω is in centimiters.
How Example 6 and Example 7 different from other?
Classical Definition of Probability
Suppose
1. all elementary events are equally likely,
2. N(A) denote the number of elementary event favourable to
event A and
3. N(Ω) = N is the total number of elementary event in sample
space Ω.
Then probability of the event A is
P(A) =
N(A)
.
N
Example 1 (Contd.)
Throwing a die twice (or throwing two dice), i.e., A will first throw
a die, then B will throw the same die. i.e.,


(1, 1) (1, 2) (1, 3) (1, 4) (1, 5) (1, 6)
 (2, 1) (2, 2) (2, 3) (2, 4) (2, 5) (2, 6) 


 (3, 1) (3, 2) (3, 3) (3, 4) (3, 5) (3, 6) 


Ω=

 (4, 1) (4, 2) (4, 3) (4, 4) (4, 5) (4, 6) 
 (5, 1) (5, 2) (5, 3) (5, 4) (5, 5) (5, 6) 
(6, 1) (6, 2) (6, 3) (6, 4) (6, 5) (6, 6)
The cases where B wins the game are marked as red. We define
the event B as
B: B wins the game,
N(B) = 15 and N(Ω) = 36, therefore
P(B) =
15
= 0.4166667
36
Criticism of Classical Definition
1. Classical definition assumes Ω is countably finite. Like in
example 1, total number of elementary event is 36.
2. But if Ω is an open set (like Example 7) or Ω is countably
infinite set (like in Example 6), then classical definition of
probability cannot handle such cases.
3. Look at the phrase “all elementary events are equally likely ” the word “equally likely” means same probability. We cannot
define probability using the probability itself. It creates a
circular logic, hence defination is fundamentally flawed.
Therefore we need a better definition of Probability.
Axiomatic Definition of Probability
σ-field of events
By σ-field of events we mean a class of events which have the
following properties
(i) Non-empty,
(ii) Closed under complementary,
(iii) Closed under countable union.
Axiomatic Definition of Probability
σ-field of events
By σ-field of events we mean a class of events which have the
following properties
(i) Non-empty,
(ii) Closed under complementary,
(iii) Closed under countable union.
Hence if A be a σ-filed of event then,
(i) ∃ one event A in Ω, such that A ∈ A or in otherwords atleast
φ ∈ A and Ω ∈ A;
(ii) If A ∈ A, then Ac ∈ A;
(iii) If Ai ∈ A ∀i = 1, 2, ... then ∪∞
i=1 Ai ∈ A
Axiomatic Definition of Probability
Ex 8 A = {φ, Ω} is the trivial σ-field
Ex 9 If A is a subset of sample space Ω, then σ-field generated by
A is
σ(A) = {φ, A, Ac , Ω}
Ex 10 If Ω = R then σ-field of R generated by intervals of (−∞, a]
is known as Borel σ-field.
Axiomatic Definition of Probability
Let Ω be the sample space of an random experiment and A be a
σ-field of events of Ω. The pair (Ω, A) is called a probability
space. Then probability function defined on a σ-field A of Ω is a
function P : A → [0, 1] such that,
1. P(A) ≥ 0 for every A ∈ A ;
2. P(Ω) = 1;
3. If Ai ∈ A ∀i = 1, 2, ... are mutually exclusive events then
P(∪∞
i= Ai )
=
∞
X
i=1
P(Ai )
Rational of Axiomatic Definition
1. First we note that for any event A what so ever fN (A) ≥ 0
when n > 0, =⇒ fNN(A) ≥ 0. So non-nagativity of relative
frequency is equivalent to the first axioms.
2. if A is a sure event, i.e., A = Ω, then fN (A) = N so that
fN (A)/N = 1.
3. For any mutually exclusive events A1 , A2 , ..., An we have
fN (∪ni=1 Ai ) =
n
X
fN (Ai ),
i=1
implying
Pn
fN (∪ni=1 Ai )
fN (Ai )
= i=1
;
N
N
so relative frequency obeys all the three axioms of the
axiomatic definition
Rational of Axiomatic Definition
I
As classical definition of probability is defined on the concept
of relative frequency; therefore classical definition is a special
case of axiomatic definition of probability.
Example
Ex 6 Throwing a coin till head apears, define Ω
Ω = {H, TH, TTH, TTTH, ...}
Ex 7 In measuring heights (in cm) of students in CMI, define Ω
Ω = {ω | 0 < ω < ∞}
where ω is in centimiters.
Clearly, classical definition does not work here! Does axiomatic
definition works here?
Fundamenta Theorems of Probability
If P is a probability function on A, then
1 P(φ) = 0
2 P is finitely additive. That is,
P(∪ni=1 Ai ) =
n
X
P(Ai )
i=1
3 Any events Ai ∈ A are exhautive as well as mutually exclusive.
Then,
X
P(Ai ) = 1
i
Fundamental Theorems of Probability
4 If A be any event in A, then A and Ac are both events in A
which are exhaustive as well as mutually exclusive. Hence,
P(A) + P(Ac ) = 1
P(Ac ) = 1 − P(A)
5 P is monotone function, means if A and B are events in A,
such that
A ⊂ B, then P(A) ≤ P(B).
Fundamental Theorems of Probability
6 P is subtractive function, means if A and B are events in A,
such that
A ⊂ B =⇒ P(B − A) = P(B) − P(A).
7 If A and B are any event in A then,
P(A ∪ B) = P(A) + P(B) − P(A ∩ B)
Fundamental Theorems of Probability
8 (Poincares Thm) If Ai , i = 1, 2, 3, ..., r are any event in A then
P(∪ri=1 Ai ) = s1 − s2 + s3 − ... + (−1)r −1 sr ,
P
wherePs1 = i P(Ai ),
s2 = Pri,j=1 i<j P(Ai ∩ Aj ),
s3 = ri,j,k=1 i<j<k P(Ai ∩ Aj ∩ Ak ) and so on.
9 (Boole’s Inequality) If Ai ∈ A then
X
P(∪i Ai ) ≤
P(Ai ).
i
Fundamental Theorems of Probability
Definition: If {An } be a sequence of events (of Ω) then the
sequence is said to be monotone of either An ⊂ An+1 for every
n = 1, 2, ... or An ⊃ An+1 for every n = 1, 2, ....
In the former case the sequence is said to be monotone
nondecreasing or expanding.
In the later case it is said to be monotone nonincreasing or
contracting.
Fundamental Theorems of Probability
Definition: If {An } be a sequence of events (of Ω) then the
sequence is said to be monotone of either An ⊂ An+1 for every
n = 1, 2, ... or An ⊃ An+1 for every n = 1, 2, ....
In the former case the sequence is said to be monotone
nondecreasing or expanding.
In the later case it is said to be monotone nonincreasing or
contracting.
Definition: If {An } is an expanding sequence then by limit of
sequence An we mean
lim An = ∪n An .
On the other hand if {An } is an contracting sequence then by
limit of the sequence An we mean
lim An = ∩n An .
Fundamental Theorems of Probability
10 (Continuity Theorem) Let P be probability measure, and let
An be sequence of events in A, then A ∈ A and
lim P(An ) = P( lim An )
n→∞
n→∞
Conditional Probability
I
The kind of probability that we have considered so far may be
called “unconditional probability ”.
I
While assigning a probability to event A, we have not
considered any information that may affect the chance of
occurance of A.
I
In other words, suppose we may have information that the
event B has already occured, then the probability of A may
have to be evaluated differently.
Conditional Probability
Suppose B is an event for which P(B) > 0,
I
fN (B) is the number of times B occurs among the N
repeatations of the random experiment
I
fN (A ∩ B) is the number of times A occurs among the
repeatation in which B occurs
I
fN (A∩B)
fN (B)
conditional relative frequency of A given B. Note that
conditional relative frequency is defined only when fN (B) > 0.
→ P(A ∩ B) and fNN(B) → P(B).
As N → ∞, fN (A∩B)
N
Consequently
I
fN (A∩B)
fN (B)
=
fN (A∩B)
N
fN (B)
N
→
P(A∩B)
P(B)
It indicates that probability of A given B is the limiting value
of conditional relative frequency.
Definition of Conditional Probability
Let (Ω, A, P) be the probability space and let B ∈ A be such that
P(B) > 0. If A ∈ A then conditional probability of A given B (or
under the condition that B has occured) is denoted by P(A|B) is
defined as
P(A ∩ B)
.
P(A|B) =
P(B)
In case P(B) = 0 the conditional probability is not defined or is
not meaningful.
Example 11
In Table 1 we presented the data on the child who who survived or
lost in the Titanic disaster. Suppse we define the following events,
Class
1st , 2nd
3rd
Total
Survived
29
27
56
Lost
1
52
53
Total
30
79
109
Table : Example 11: In Titanic diaster Child who survived or lost
presented by class
Results on Conditional Probability
1 Let A, B ∈ A and P(B) > 0, then P(A ∩ B) = P(B)P(A|B)
2 In general, Ai ∈ A, i = 1, 2, 3, ..., r such that P(∩ri=1 Ai ) > 0
then
−1
P(∩ri=1 Ai ) = P(A1 )P(A2 |A1 )P(A3 |A ∩ A2 )...P(Ar | ∩ri=1
Ai )
Results on Conditional Probability
3 Let A, B, C ∈ A, such that P(B ∩ C ) > 0 then
P(A ∩ B|C ) = P(B|C )P(A|B ∩ C )
4 Let Bi ∈ A, i = 1, 2, ..., n are exhaustive and mutually
exclusive events such that P(Bi ) > 0 for each i, then A ∈ A.
Then we shall have
P(A) =
n
X
P(Bi )P(A|Bi ).
i=1
P
Note that P(Bi ) > 0 and ni=1 P(Bi ) = 1 may look upon
weighted arithmetic mean of conditional probability (where
P(Bi )’s can be viewed as weight).
Baye’s Theorem
Baye’s Theorem: Let Bi ∈ Abe exhaustive as well as mutually
exclusive events such that P(Bi ) > 0. If A ∈ A be any other event
such that P(A) > 0 then
P(Bi )P(A|Bi )
P(Bi |A) = Pn
j=1 P(Bj )P(A|Bj )
Exercise
1. Donated blood is screened for HIV positive. Suppose the test
has 99% accuracy, and the test that one in tes thousand
people in the same age group are HIV positive. The test has a
5% false-positive rating, as well. Suppose the test screens
someone as positive. What is the probability that the patient
really is HIV positive?
Hint: 99% accuracy of the test refers to P(test positive |
patient is HIV positive). But we want to estimate that
P(patient is HIV positive | test positive).
Independence
Suppose (Ω, A, P) be a probability space. Let A and B be any two
events in A. Events A and B are said to be independent if
P(A ∩ B) = P(A)P(B).
Independence
Suppose P(B) > 0 so that P(A|B) is defined. Then we have
P(A|B) =
P(A)P(B)
P(A ∩ B)
=
= P(A),
P(B)
P(B)
then the conditional probability of {A|B} is same as unconditional
probability of {A}. In other words, the additional information that
B has already occurred makes no difference in the probability of
{A}
Example 12
Suppose a card is drawn at random from a full deck. Two events
are defined as follows:
I
A: the card is sprade
I
B: the cars is queen
Show that A and B are independent.