Download Notes - kaharris.org

Sums of Discrete Random Variables
Problem involving sums
Problem. Let X and Y be two independent discrete random variables
with known probability mass functions pX and pY and whose possible
values are only nonnegative integers.
What is the probability mass function for X + Y ?
Math 425
Intro to Probability
Lecture 27
Consider a possible event {X + Y = n}. Then
pX +Y (n) = P {X + Y = n} = P {Y = n − X }
n
X
=
P {X = k , Y = n − k }
Kenneth Harris
kaharri@umich.edu
Department of Mathematics
University of Michigan
=
March 20, 2009
=
k =0
n
X
k =0
n
X
P {X = k } · P {Y = n − k }
pX (k ) · pY (n − k ).
k =0
Kenneth Harris (Math 425)
Math 425 Intro to Probability Lecture 27
March 20, 2009
1/1
Kenneth Harris (Math 425)
Sums of Discrete Random Variables
Example. A die is rolled twice. Let X and Y be the outcomes, so they
have the common mass function
(
p(i) =
p(2) = p1 (1) · p2 (1) =
pX (k ) · pY (n − k ).
0
if 1 ≤ i ≤ 6
othewise
1 1
1
· =
6 6
36
p(3) = p1 (1) · p2 (2) + p1 (2) · p2 (1) =
pX ∗ pY is the probability mass function for the sum X + Y .
Note. Generalizing the convolution to discrete random variables with
integer or rational possible values is straightforward. However, the
cases we are interested in (Poisson, Geometric, Binomial, Uniform) fit
this definition.
Math 425 Intro to Probability Lecture 27
1
6
The convolution pX +Y = pX ∗ pY is computed as follows
k =0
Kenneth Harris (Math 425)
3/1
Example
Definition
Let X and Y be discrete random variables taking only nonnegative
values.
The convolution of X and Y is the probability mass function
p = pX ∗ pY given by
n
X
March 20, 2009
Sums of Discrete Random Variables
Convolutions: discrete case
p(n) =
Math 425 Intro to Probability Lecture 27
March 20, 2009
4/1
2
36
3
p(4) = p1 (1) · p2 (3) + p1 (2) · p2 (2) + p1 (3) · p2 (1) =
36

1

if 2 ≤ n ≤ 7
 36 (n − 1)
1
p(n) =
(13
−
n)
if 8 ≤ n ≤ 12
36


0
otherwise.
Kenneth Harris (Math 425)
Math 425 Intro to Probability Lecture 27
March 20, 2009
5/1
Sums of Discrete Random Variables
Sums of Discrete Random Variables
Example
Key Properties
Sum of two uniform discrete random variables on {1, 2, 3, 4, 5, 6}.
The convolution has several important properties which are
especially useful for computing the probability mass function of the
sum of several independent random variables.
0.15
0.10
Theorem
Let X , Y and Z be discrete random variables taking only nonnegative
values.
0.05
(a) (Commutativity). pX ∗ pY = pY ∗ pX .
(b) (Associativity). (pX ∗ pY ) ∗ pZ = pX ∗ (pY ∗ pZ ).
2
Kenneth Harris (Math 425)
4
6
8
Math 425 Intro to Probability Lecture 27
10
12
March 20, 2009
6/1
Kenneth Harris (Math 425)
Sums of Discrete Random Variables
{1, 2, 3, 4, 5, 6}. It is a bit tedious to compute ,
Looks more and more like a Bell curve. As n increases the distribution
approaches the normal distribution with µ = n · 3.5 and σ 2 = n · 35
12 .
independent and identically distributed with distribution pX (x).
Consider the random variable which sums these
Xk
0.07
n ≥ 1.
k =1
n=20
0.05
0.04
= pX1 ∗ pX2 ∗ · · · ∗ pXn
= pSn−1 ∗ pXn
0.03
= pSn−1 ∗ pX .
0.02
Math 425 Intro to Probability Lecture 27
March 20, 2009
n=30
0.01
It is possible in some cases to calculate the distribution for Sn by
recursion from the distribution for Sn−1 .
Kenneth Harris (Math 425)
n=10
0.06
whose distribution is
pSn
7/1
Example
Sum of n = 10, 20, 30 uniform discrete random variables on
Let X1 , X2 , . . . , Xn , be discrete random variables which are
n
X
March 20, 2009
Sums of Discrete Random Variables
Sums of random variables
Sn =
Math 425 Intro to Probability Lecture 27
50
8/1
Kenneth Harris (Math 425)
100
Math 425 Intro to Probability Lecture 27
150
March 20, 2009
9/1
Sums of Binomial Random Variables
Sums of Binomial Random Variables
Theorem
Proof
A Bernoulli random variable gives the number of successes in one
Bernoulli trial:
p(0) = 1 − p
p(1) = p.
Inductive Step. Suppose that the sum Sn of n independent and
identically distributed Bernoulli random variables with probability
p ∈ (0, 1) has the distribution
A
binomial random variable gives the number of successes in n
Bernoulli trials:
p(k ) =
n k
p (1 − p)n−k
k
0 ≤ k ≤ n.
pSn (k ) =
Let X1 , X2 , . . . , Xn be independent and identically distributed Bernoulli
random variables with parameter p ∈ (0, 1).
The sum Sn of these random variables is a binomial random
variable with pararmeters n and p.
Math 425 Intro to Probability Lecture 27
March 20, 2009
n+1 k
pSn+1 (k ) =
p (1 − p)n+1−k
k
Kenneth Harris (Math 425)
11 / 1
n
=
k
X
pSn (k − j) · pXn+1 (j)
12 / 1
I owe you , but this show that the sum of n + 1 independent
Bernoulli random variables is a binomial random variable on n + 1
trials.
March 20, 2009
n
n
=
+
.
k
k −1
{1, 2, . . . , n + 1}?
1
When n + 1 IS NOT in the subset, then choose the k members
from {1, 2, . . . , n}, so
n
possibilities.
k
2
Math 425 Intro to Probability Lecture 27
n+1
k
How many ways are there of choosing a set of size k from
pSn (k ) · (1 − p) + pSn (k − 1) · p
n
n k
p (1 − p)n+1−k +
pk (1 − p)n+1−k
k
k −1
n+1 k
p (1 − p)n+1−k
k
where 0 ≤ k ≤ n + 1.
Kenneth Harris (Math 425)
March 20, 2009
Justify (when k ≤ n)
∗ pXn+1 .
j=0
=
Math 425 Intro to Probability Lecture 27
Proof – continued
Compute Sn+1 = Sn + Xn+1 using pS
=
0 ≤ k ≤ n + 1.
Sums of Binomial Random Variables
Proof – continued
=
0 ≤ k ≤ n.
and with the same distribution. Show pSn+1 = pSn ∗ pXn+1 has
distribution
Sums of Binomial Random Variables
P {Sn + Xn+1 = k }
n k
p (1 − p)n−k
k
Let Xn+1 be a Bernoulli random variable independent of those in Sn
Theorem (Binomial Random Variables)
Kenneth Harris (Math 425)
Basis. S1 is a single Bernoulli random variable, so a binomial random
variable on 1 trial.
13 / 1
When n + 1 IS in the subset, then choose the other k − 1
members from {1, 2, . . . , n}, so
n
possibilities.
k −1
Kenneth Harris (Math 425)
Math 425 Intro to Probability Lecture 27
March 20, 2009
14 / 1
Sums of Geometric Random Variables
Sums of Geometric Random Variables
Theorem
Proof
A geometric random variable gives the waiting time for the first
Basis. S1 is a single geometric random variable, so a negative
binomial random variable with n = 1.
success
p(m) = p(1 − p)m−1
m = 1, 2, 3, . . .
Inductive Step. Suppose that the sum Sn of n independent and
identically distributed geometric random variables with probability
p ∈ (0, 1) has the distribution
A negative binomial random variable with parameter n gives the
waiting time for the nth success.
p(m) =
m−1 n
p (1 − p)m−n
n−1
m = n, n + 1, n + 2, . . .
pSn (m) =
Let X1 , X2 , . . . , Xn be independent and identically distributed geometric
random variables with probability p ∈ (0, 1).
The sum Sn of these random variables is a negative binomial
random variable with parameters n and p.
Math 425 Intro to Probability Lecture 27
March 20, 2009
Sn and with the same distribution. Show pSn+1 = pSn ∗ pXn+1 has
distribution
m − 1 n+1
pSn+1 (m) =
p (1 − p)m−n−1
n
Kenneth Harris (Math 425)
16 / 1
Sums of Geometric Random Variables
Math 425 Intro to Probability Lecture 27
March 20, 2009
17 / 1
Proof – continued
Let m ≥ n + 1 (the minimal possible value). Compute pS
n
=
m
X
Justify (when m ≥ n)
∗ pXn+1
m X
m
j −1
=
n
n−1
pSn (j) · pXn+1 (m − j)
j=0
=
m−1
X
j=n
j=n
j −1 n
p (1 − p)j−n · p(1 − p)m−j−1
n−1
How many ways are there of choosing a set of size n from
m−1
X
{1, 2, . . . , m}?
j − 1 n+1
=
p (1 − p)m−n−1
n−1
j=n
m−1 m
=
p (1 − p)n−m
n
I owe you , but this show that the sum of n + 1 independent
geometric random variables is a negative binomial random variable
with parameters n + 1 and p.
Kenneth Harris (Math 425)
m = n + 1, n + 2, n + 3, . . . ,
Sums of Geometric Random Variables
Proof – continued
P {Sn + Xn+1 = m}
m = n, n + 1, n + 2, . . .
Let Xn+1 be a geometric random variable independent of those in
Theorem (Geometric Random Variables)
Kenneth Harris (Math 425)
m−1 n
p (1 − p)m−n
n−1
Math 425 Intro to Probability Lecture 27
March 20, 2009
18 / 1
1
Choose the largest possible value in {n, n + 1, . . . , m}.
2
For each j = n, . . . m (the largest value in the subset of size n),
choose a subset of size n − 1 from {1, . . . , j − 1}. There are
j −1
possibilities.
n−1
Kenneth Harris (Math 425)
Math 425 Intro to Probability Lecture 27
March 20, 2009
19 / 1
Sums of Continuous Random Variables
Sums of Continuous Random Variables
Definition: Convolution
Theorem: convolutions and sums of r.v.’s
How are sums of independent random variables distributed?
Analogous to the definition for discrete random variables, we define
the convolution of continuous random varariables.
Definition
Let X and Y be two continuous random variables with densities fX (x)
and fY (y ).
The convolution f = fX ∗ fY is the function given by
Z ∞
f (z) =
fX (z − y )fY (y ) dy
−∞
Theorem
Let X and Y be independent continuous random variables with density
fX (x) and fY (y ).
The sum X + Y is a continuous random variable with density
fX +Y = fX ∗ fY . That is
Z ∞
fX +Y (a) =
fX (a − y )fY (y ) dy
−∞
Note. The convolution is commutative and associative: for any density
functions f , g and h,
f ∗g = g∗f
(f ∗ g) ∗ h = f ∗ (g ∗ h).
Kenneth Harris (Math 425)
Math 425 Intro to Probability Lecture 27
March 20, 2009
21 / 1
Sums of Continuous Random Variables
Kenneth Harris (Math 425)
Math 425 Intro to Probability Lecture 27
March 20, 2009
22 / 1
Sums of Continuous Random Variables
Proof
Proof – continued
Proof. Let X and Y be independent with joint density fX ,Y (x, y ).
FX +Y (a) =
=
Z
FX +Y (a)
We get the density for X + Y by differentiating FX +Y
−∞
Z a−y
fX +Y (a)
fX (x)fY (y ) dx dy
=
−∞
∞
Z
=
−∞
∞
−∞
a−y
hZ
i
fX (x) dx fY (y ) dy
−∞
Z
FX (a − y )fY (y ) dy .
=
FX (a − y )fY (y ) dy .
−∞
P {X + Y ≤ a} = P {X ≤ a − Y }
Z ∞ Z a−y
fX ,Y (x, y ) dx dy
−∞
Z ∞
∞
=
−∞
d
FX +Y (a)
da Z
∞
d
=
FX (a − y )fY (y ) dy
da −∞
Z ∞
d
=
FX (a − y )fY (y ) dy
−∞ da
Z ∞
=
fX (a − y )fY (y ) dy = fX ∗ fY (a)
=
−∞
Kenneth Harris (Math 425)
Math 425 Intro to Probability Lecture 27
March 20, 2009
23 / 1
Kenneth Harris (Math 425)
Math 425 Intro to Probability Lecture 27
March 20, 2009
24 / 1
Sums of Independent Exponential Random Variables
Sums of Independent Exponential Random Variables
Theorem
Proof
An exponential random variable gives the waiting time for the first
Basis. S1 is a single exponential random variable, so a gamma
random variable with (α = 1, λ).
success in a Poisson process
f (t) = λe−λt
t ≥0
Inductive Step. Suppose that the sum Sn of n independent and
identically distributed exponential random variables with probability
p ∈ (0, 1) has the distribution
A gamma random variable with parameters (α = n, λ) is the
waiting time for the n success in a Poisson process
f (t) = λe−λt
(λt)n−1
(n − 1)!
fSn (t) = λe−λt
t ≥ 0.
in Sn and with the same distribution. Show pSn+1 = pSn ∗ pXn+1 has
distribution
n
Let X1 , X2 , . . . , Xn be independent and identically distributed
exponential random variables with parameter λ > 0.
Then the sum Sn of these random variables is a gamma random
variable with pararmeters (α = n, λ).
Math 425 Intro to Probability Lecture 27
March 20, 2009
fSn+1 (t) = λe−λt
26 / 1
Sums of Independent Exponential Random Variables
t ≥ 0.
Math 425 Intro to Probability Lecture 27
March 20, 2009
27 / 1
Theorem
Compute
Z
The sum of independent gamma random variables is a gamma
∞
random variable.
fSn (y ) · fXn+1 (a − y ) dy
=
−∞
Z
=
a
λe−λy
0
= λe−λa
Z
= λe−λa
Theorem (Gamma Random Variables)
(λy )n−1
· λe−λ(a−y ) dy
(n − 1)!
a
λ
0
Let X1 , X2 , . . . , Xn be independent gamma distributed random variables
with parameters (αi , λ), i = 1, . . . , n.
Then their sum
PnSn is also a gamma distributed random variable but
with parameter ( i=1 αi , λ).
(λy )n−1
dy
(n − 1)!
(λa)n
n!
See Ross Proposition 3.1 for the case of n = 2, but the proof is similar to
This show that the sum of n + 1 independent exponential random
variables is a gamma random variable with parameters (α = n + 1, λ).
Kenneth Harris (Math 425)
Kenneth Harris (Math 425)
(λt)
(n)!
Sums of Independent Exponential Random Variables
Proof – continued
fSn +Xn+1 (a)
t ≥ 0.
Let Xn+1 be an exponential random variable independent of those
Theorem (Exponential Random Variables)
Kenneth Harris (Math 425)
(λt)n−1
(n − 1)!
Math 425 Intro to Probability Lecture 27
March 20, 2009
28 / 1
the case of exponential random variables. When the αi are integers, this
theorem follows from the result about sums of exponential random variables.
Kenneth Harris (Math 425)
Math 425 Intro to Probability Lecture 27
March 20, 2009
29 / 1
Sums of Normal Random Variables
Sums of Normal Random Variables
Theorem
Proof of a special case
Ross proves the most general case. I will prove the case of two
standard normal random variables, X and Y , whose density is
Theorem (Normal Random Variables)
2
1
fX (x) = fY (x) = √ e−x /2
2π
Let X1 , X2 , . . . , Xn be independent normal random variables with
respective parameters µi , σi2 , i = 1, . . . , n.
Then the sum Sn of these
is a normal random
Pn random variables
Pn
2
variable with pararmeters i=1 µi and i=1 σi .
with mean µ = 0 and variance σ 2 = 1.
When X2 is a normally distributed random variable with mean µ and
variance σ , the density is
See Ross Proposition 6.3.2, page 283.
Kenneth Harris (Math 425)
Math 425 Intro to Probability Lecture 27
fX (x) = √
March 20, 2009
31 / 1
Sums of Normal Random Variables
Kenneth Harris (Math 425)
1
2πσ
e−(x−µ)
2
/2σ 2
Math 425 Intro to Probability Lecture 27
March 20, 2009
32 / 1
Sums of Normal Random Variables
Proof of a special case
Proof of a special case
Then the density of X + Y is
fX +Y (a)
=
=
1
=
2
=
=
Z ∞
2
2
1
e−(a−y ) /2 e−y /2 dy
2π −∞
Z
1 −a2 /2 ∞ −(y 2 −ay )
e
e
dy
2π
−∞
Z
1 −a2 /4 ∞ −(y −(a/2))2
e
e
dy
2π
−∞
Z ∞
i
2
1 −a2 /4 √ h 1
e
π √
e−(y −(a/2)) dy
2π
π −∞
2
1
√ √ e−a /4
normal: µ = 0, σ 2 = 2
2π 2
When X and Y 2are normally distributed random variables with
mean µ = 0 and σ = 1, their sum X + Y has density
fX +Y (a) = √
1
√ e−a
2π 2
2
/4
and is a normally distributed random variable with µ = 0 and σ 2 = 2.
(1): Complete the square: −(y 2 − ay ) = −(y 2 − ay + (a/2)2 ) + (a/2)2
(2): [. . .] = 1 : normal with µ = a2 and σ 2 = 21 .
Kenneth Harris (Math 425)
Math 425 Intro to Probability Lecture 27
March 20, 2009
33 / 1
Kenneth Harris (Math 425)
Math 425 Intro to Probability Lecture 27
March 20, 2009
34 / 1