Download Rules of Probability

Rules of Probability
The additive rule
P[A  B] = P[A] + P[B] – P[A  B]
and
P[A  B] = P[A] + P[B] if P[A  B] = f
The additive rule for more than two events
n  n
P  Ai    P  Ai    P  Ai  Aj 
i
j
 i 1  i 1
 P  Ai  Aj  Ak  
i
j
  1
k
n 1
P  A1  A2 
and if Ai  Aj = f for all i ≠ j.
then
n  n
P  Ai    P  Ai 
 i 1  i 1
 An 
The Rule for complements
for any event E
P  E   1  P  E 
Conditional Probability,
Independence
and
The Multiplicative Rue
Then the conditional probability of A given B is
defined to be:
P  A B  
if P  B  0
P  A  B
P  B
The multiplicative rule of probability

 P  A P  B A if P  A  0
P  A  B  


P
B
P
A
B
if
P
B

0








and
P  A  B  P  A P  B
if A and B are independent.
This is the definition of independent
The multiplicative rule for more than
two events
P  A1  A2 
 An  
P  A1  P  A2 A1  P  A3 A2  A1 
P  An An 1  An 2
 A1 
Proof
P  A1  A2 
 An   P  A1  A2 
 An 1   An 
 P  A1  A2 
 An 1  P  An A1  A2 
 P  A1  A2 
 An 2  P  An 1 A1  A2 
 An 1 
P  An A1  A2 
 An 2 
 An 1 
and continuing we obtain
 P  A1  P  A2 A1  P  A3 A2  A1 
P  An An 1  An  2
 A1 
Example
What is the probability that a poker hand is a
royal flush
i.e.
1. 10
2. 10
3. 10
4. 10
,J
,J
,J
,J
,Q
,Q
,Q
,Q
, K ,A
, K ,A
, K ,A
, K ,A
1.
2.
3.
4.
10
10
10
10
,J
,J
,J
,J
,Q
,Q
,Q
,Q
, K ,A
, K ,A
, K ,A
, K ,A
Solution
Let A1 = the event that the first card is a “royal
flush” card.
Let Ai = the event that the ith card is a “royal
flush” card. i = 2, 3, 4, 5.
P  A1  
, P  A2 A1   514 , P  A3 A2  A1   503 ,
P  A4 A3  A2  A1   492 , P  A5 A4  A3  A2  A1  
20
52
1
48
P  Royal Flush  
20
P  A1  A2  A3  A4  A5    52
 524  503  492  481 
Another solution is by counting
4
4
P  Royal Flush  

 52   52  51 50  49  48 
 
5  4  3  2 1

5
 
4  5  4  3  2 1
20  4  3  2 1


52  51 50  49  48 52  51 50  49  48
The same result
Independence
for more than 2 events
Definition:
The set of k events A1, A2, … , Ak are called
mutually independent if:
P[Ai1 ∩ Ai2 ∩… ∩ Aim] = P[Ai1] P[Ai2] …P[Aim]
For every subset {i1, i2, … , im } of {1, 2, …, k }
i.e. for k = 3 A1, A2, … , Ak are mutually independent if:
P[A1 ∩ A2] = P[A1] P[A2], P[A1 ∩ A3] = P[A1] P[A3],
P[A2 ∩ A3] = P[A2] P[A3],
P[A1 ∩ A2 ∩ A3] = P[A1] P[A2] P[A3]
P[A1] = .4, P[A2] = .5 , P[A3] = .6
A1
P[A1∩A2] = (0.4)(0.5) = 0.20
A2
P[A1 ∩ A3] = (0.4)(0.6) = 0.24
0.08
0.08
0.12
P[A2 ∩ A3] = (0.5)(0.6) = 0.30
0.12
0.12
0.18
0.12
0.18
A3
P[A1 ∩ A2 ∩ A3] =
(0.4)(0.5)(0.6) = 0.12
Definition:
The set of k events A1, A2, … , Ak are called
pairwise independent if:
P[Ai ∩ Aj] = P[Ai] P[Aj] for all i and j.
i.e. for k = 3 A1, A2, … , Ak are pairwise independent if:
P[A1 ∩ A2] = P[A1] P[A2], P[A1 ∩ A3] = P[A1] P[A3],
P[A2 ∩ A3] = P[A2] P[A3],
It is not necessarily true that P[A1 ∩ A2 ∩ A3] = P[A1]
P[A2] P[A3]
P[A1] = .4, P[A2] = .5 , P[A3] = .6
A1
P[A1∩A2] = (0.4)(0.5) = 0.20
A2
P[A1 ∩ A3] = (0.4)(0.6) = 0.24
0.10
0.06
0.14
P[A2 ∩ A3] = (0.5)(0.6) = 0.30
0.14
0.10
0.16
0.10
0.20
A3
P[A1 ∩ A2 ∩ A3] = 0.14
≠ (0.4)(0.5)(0.6) = 0.12
Bayes Rule
• Due to the reverend T.
Bayes
• Picture found on website:
Portraits of Statisticians
• http://www.york.ac.uk/depts/
maths/histstat/people/welco
me.htm#h
P  A P  B A
P  A B  
P  A P  B A  P  A  P  B A 
Proof:
P  A B  

P  A  B
P  B
P  A  B
P  A  B   P  A  B 
P  A P  B A

P  A P  B A  P  A  P  B A 
Example:
We have two urns. Urn 1 contains 14 red balls and
12 black balls. Urn 2 contains 6 red balls and 20
black balls.
An Urn is selected at random and a ball is selected
from that urn.
Urn 1
Urn 2
If the ball turns out to be red what is the probability
that it came from the first urn?
Solution:
Let A = the event that we select urn 1
A = the event that we select urn 2
1
P  A  P  A  
2
Let B = the event that we select a red ball
14
6


P  B A  , P  B A  
26
26
We want P  A B  .
Note: the desired conditional probability is in the
reverse direction of the given conditional probabilities.
This is the case when Bayes rule should be used
Bayes rule states
P  A P  B A
P  A B  
P  A P  B A  P  A  P  B A 
1
14



14
14
2
26 
 1 14


 0.70
6
1
 2   26    2   26  14  6 20
Example: Testing for a disease
Suppose that 0.1% of the population have a certain
genetic disease.
A test is available the detect the disease.
If a person has the disease, the test concludes that he
has the disease 96% of the time. It the person doesn’t
have the disease the test states that he has the disease
2% of the time.
Two properties of a medical test
Sensitivity = P[ test is positive | disease] = 0.96
Specificity = P[ test is negative | disease] = 1 – 0.02 = 0.98
A person takes the test and the test is positive, what is the
probability that he (or she) has the disease?
Solution:
Let A = the event that the person has the disease
A = the event that the person doesn’t have the
disease
P  A  0.001, P  A   1  0.001  0.999
Let B = the event that the test is positive.
P  B A  0.96, P  B A   0.02
We want P  A B  .
Note: Again the desired conditional probability is in
the reverse direction of the given conditional
probabilities.
Bayes rule states
P  A P  B A
P  A B  
P  A P  B A  P  A  P  B A 
0.001 0.96 


 0.001 0.96    0.999  0.02 
0.00096

 0.0458
0.00096  .01998
Thus if the test turns out to be positive the chance of
having the disease is still small (4.58%).
Compare this to (.1%), the chance of having the disease
without the positive test result.
An generlization of Bayes Rule
Let A1, A2 , … , Ak denote a set of events such that
S  A1  A2 
 Ak and Ai  Aj  f
for all i and j. Then
P  Ai  P  B Ai 
P  Ai B  
P  A1  P  B A1    P  Ak  P  B Ak 
If A1, A2 , … , Ak denote a set of events such that
S  A1  A2 
 Ak and Ai  Aj  f
for all i and j. Then A1, A2 , … , Ak is called a
partition of S.
S
A1 A2
Ak
…
Proof
B   B  A1     B  Ak 
and  B  Ai    B  Aj   f
A1 A2
for all i and j.
Ak
B
Then
P  B  P  B  A1  
 P  B  Ak 
 P  A1  P  B A1  
 P  Ak  P  B Ak 
and
P  Ai B  
P  Ai  B 
P  B
P  Ai  P  B Ai 

P  A1  P  B A1    P  Ak  P  B Ak 
Example:
We have three urns. Urn 1 contains 14 red balls and
12 black balls. Urn 2 contains 6 red balls and 20
black balls. Urn 3 contains 3 red balls and 23 black
balls.
An Urn is selected at random and a ball is selected
from that urn.
Urn 1
Urn 3
Urn 2
If the ball turns out to be red what is the probability
that it came from the first urn? second urn? third Urn?
Solution:
Let Ai = the event that we select urn i
S  A1  A2  A3
1
P  A1   P  A2   P  A3  
3
Let B = the event that we select a red ball
14
6
3
P  B A1   , P  B A2   , P  B A3  
26
26
26
We want P  Ai B  for i  1, 2,3.
Note: the desired conditional probability is in the
reverse direction of the given conditional probabilities.
This is the case when Bayes rule should be used
Bayes rule states
P  A1  P  B A1 
P  A1 B  
P  A1  P  B A1   P  A2  P  B A2   P  A3  P  B A3 
P  A2
1
14


14
14
3
26 
 1 14


6
3
1
1
 3  26    3  26    3  26  14  6  3 23
6
1


6
6
3
26 
B   1 14


6
3
1
1
 3  26    3  26    3  26  14  6  3 23
3
1


3
3
3
26 
P  A3 B   1 14


6
3
1
1
 3  26    3  26    3  26  14  6  3 23
Example:
Suppose that an electronic device is
manufactured by a company.
During a period of a week
–
–
–
–
–
15% of this product is manufactured on Monday,
23% on Tuesday,
26% on Wednesday ,
24% on Thursday and
12% on Friday.
Also during a period of a week
– 5% of the product is manufactured on Monday is
defective
– 3 % of the product is manufactured on Tuesday is
defective,
– 1 % of the product is manufactured on Wednesday
is defective ,
– 2 % of the product is manufactured on Thursday is
defective and
– 6 % of the product is manufactured on Friday is
defective.
If the electronic device manufactured by this plant turns
out to be defective, what is the probability that is as
manufactured on Monday, Tuesday, Wednesday,
Thursday or Friday?
Solution:
Let
A1 = the event that the product is manufactured on
Monday
A2 = the event that the product is manufactured on
Tuesday
A3 = the event that the product is manufactured on
Wednesday
A4 = the event that the product is manufactured on
Thursday
A5 = the event that the product is manufactured on
Friday
Let B = the event that the product is defective
Now
P[A1] = 0.15, P[A2] = 0.23, P[A3] = 0.26, P[A4] = 0.24
and P[A5] = 0.12
Also
P[B|A1] = 0.05, P[B|A2] = 0.03, P[B|A3] = 0.01,
P[B|A4] = 0.02 and P[B|A5] = 0.06
We want to find
P[A1|B], P[A2|B], P[A3|B], P[A4|B] and P[A5|B] .
We will apply Bayes Rule
P  Ai  P  B Ai 
P  Ai B  
P  A1  P  B A1    P  A5  P  B A5 
i
P[Ai]
P[B|Ai]
P[Ai]P[B|Ai]
P[Ai|B]
1
0.15
0.05
0.0075
0.2586
2
0.23
0.03
0.0069
0.2379
3
0.26
0.01
0.0026
0.0897
4
0.24
0.02
0.0048
0.1655
5
0.12
0.06
0.0072
0.2483
Total
1.00
0.0290
1.0000
The sure thing principle and
Simpson’s paradox
The sure thing principle
Suppose
P  A C   P  B C  and
P  A C   P  B C  then
P  A  P  B
Example – to illustrate
Let A = the event that horse A wins the race.
B = the event that horse B wins the race.
C = the event that the track is dry
C = the event that the track is muddy
Proof:
P  A C   P  B C  implies
or P  A  C   P  B  C 
P  A C   P  B C  implies
or P  A  C   P  B  C 
P A  C
P C 
P  A  C 
P C 


PB  C
P C 
P  B  C 
P  A  P  A  C   P  A  C  
P  B  C   P  B  C   P  B 
P C 
Simpson’s Paradox
Does
P  D S  C   P  D S  C  and
P  D S  C   P  D S  C  imply
P  D S   P  D S  ?
Example to illustrate
D = death due to lung cancer
S = smoker
C = lives in city, C = lives in country
Solution
P  D S  

PD  S 
P S 

P  D  S  C   P  D  S  C 
P S 
P  D  S  C  P S  C 
P S  C 
P S 
P  D  S  C  P  S  C 

P S 
P  S  C 
 P  D S  C  P C S   P  D S  C  P C S 
similarly
P  D S   P  D S  C  P C S   P  D S  C  P C S 
if P  D S  C   P  D S  C  and
P  D S  C   P  D S  C 
whether
P  D S   P  D S  C  P C S   P  D S  C  P C S 
is greater than
P  D S   P  D S  C  P C S   P  D S  C  P C S 
depends also on the values of
P C S  , P C S   1  P C S  ,
P C S  and P C S   1  P C S 
Suppose P  D S  C   0.90  P  D S  C  =0.60 and
P  D S  C   0.40  P  D S  C   0.10
whether
P C S   .10, P C S   1  P C S   .90,
P C S  =.80 and P C S   1  P C S   0.20
than
P  D S   P  D S  C  P C S   P  D S  C  P C S 
 .90.10  .40.90  .45
and
P  D S   P  D S  C  P C S   P  D S  C  P C S 
 .60.80  .10.20  .50