Download Document

EMIS 7300
SYSTEMS ANALYSIS METHODS
FALL 2005
Dr. John Lipp
Copyright © 2002 - 2005 John Lipp
Session 1 Outline
• Part 1: The Statistics You Thought You Knew.
• Part 2: Probability Theory.
• Part 3: Discrete Random Variables.
• Part 4: Continuous Random Variables.
S1P4-2
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Today’s Session Topics
• Part 4: Continuous Random Variables.
–
–
–
–
–
–
–
–
–
–
–
Continuous Random Variables.
Continuous PDF and CDF.
Stochastic Expectation.
Mixed Random Variables.
Uniform Random Variables.
Gaussian Random Variables.
Central Limit Theorem.
Chi-Square Random Variables.
Exponential Random Variables
Rayleigh Random Variables.
Cauchy Random Variables.
S1P4-3
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Continuous Random Variables
• The elementary outcomes are mapped to real numbers, i.e.,
x = X(E), where x is a real number and E is an outcome in a
sample space S.
E
S
x = X(E)
-3
-2
-1
0
1
x
2
3
S1P4-4
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Continuous Random Variables (cont.)
• The number of possible x values is infinite.
• Question: can the mean calculation

   xi f X ( xi )
i 1
be performed when the xi are real numbers?
• Or put differently, can the real numbers be made into a list
such that {x1, x2, x3, …, x} contains all possible real numbers
to which the random variable maps?
S1P4-5
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Continuous Random Variables (cont.)
• Proof by contradiction.
– Consider just the real numbers between 0 and 1.
– Write them as decimals, e.g., 0.598248766…
– Assume that the (infinite) list of numbers shown below
contains all the real numbers between 0 and 1.
{0.887871659879013691591325…,
0.698173286190238298539754…,
0.213498077356359723983947…,
0.162390283987279837459734…,
0.903743761618759869834982…,
…}
– Find a real number between 0 and 1 not in the list!
S1P4-6
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Continuous RV - Cumulative Distribution Function
• For a discrete random variable, P(X = xi) = fX[xi].
• For a (pure) continuous random variable P(X = x) = 0!
• Therefore the definition of PDF and CDF needs to be
redefined for continuous random variables.
• Start by defining the cumulative distribution function (CDF)
as P(X  x) = FX(x).
S
E
X(E)  x
-3
-2
-1
0
x
1
2
3
S1P4-7
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Continuous RV - Cumulative Distribution Function (cont.)
• The CDF for a continuous RV has some basic properties:
–
–
–
–
–
FX(-) = P(X  -) = 0.
FX(+) = P(X  +) = 1.
FX(x)  FX(y) when x  y (monotone increasing).
FX(x) may contain jump discontinuities (special case).
FX(x) is right-continuous.
1
x
-3
-2
-1
0
1
2
3
S1P4-8
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Continuous RV - Probability Distribution Function
• Use the Fundamental Theorem of Calculus,
a
FX (a )  FX (b)   f X ( x)dx
b
where
d FX ( x)
f X ( x) 
dx
• The function fX(x) is the probability density function (PDF) for
a continuous random variable. Note
FX ( x) 
x
 f X ( y)dy

• The PDF really has no meaning outside of an integral.
Specifically, P(X = x)  fX(x). [Exception: the special case of
jump discontinuities in the CDF.].
S1P4-9
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Continuous RV - Probability Distribution Function (cont.)
• The PDF has some basic properties:
– fX(x)  0 (because FX(x) is monotone increasing).
– fX(x) may contain Dirac delta functions (from the jump
discontinuities in FX(x)).
FX(x)

fX(x)

 a ( x)dx  a 

– The area under fX(x) is unity,

 f X ( x)dx  FX ()  FX ()  1  0  1

• Expectation is now an integral which involves the PDF:

Eg ( X )   g ( x) f X ( x)dx

S1P4-10
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
The Mean, Mode, Median, and Variance
• The mean of a continuous random variable is

 x  EX    x f X ( x)dx

• The mode is the value of x at the maximum (peak) of fX(x).
• The median is the value of x at which it is equally probable
for X to have a value above or below the median, that is,
median


median
 f X ( x)dx   f X ( x)dx  FX (median)  0.5
• The mode, median, or mean are not necessarily the same
values!
• The variance of a continuous random variable is


  EX      x   x  f X ( x)dx   x 2 f X ( x)dx   x2
2
x
2
2
x
2


S1P4-11
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Mixed Distributions
• The CDF can be separated as the sum of two functions: a
continuous function and a sum of step functions.
1
FXc(x)
x
-3
-2
1
0
-1
2
3
1
FXd(x)
x
-3
-2
-1
0
1
2
3
S1P4-12
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Mixed Distributions (cont.)
• The continuous portion represents a “pure” continuous
random variable and the step portion represents a “pure”
discrete random variable.
• The latter’s continuous PDF is a sum of Dirac delta functions,
M
f X d ( x)   P( X  xi ) ( x  xi )
i 1
1
fXd(x) = a(x - x1)
x
-3
-2
-1
0
1
2
3
S1P4-13
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Mixed Distributions (cont.)
• Recall the definition of discrete PDF, fX[xi] = P(X = xi). Then,
Md
f X d ( x)   f X d [ xi ] ( x  xi )
i 1
• Using the sifting property of the Dirac delta function yields
the expected result for the discrete portion of a continuous
CDF,
FX d ( x) 
x
x Md
f X [ yi ] ( y  xi )dy
 f X ( y )dy   
i 1

Md x

d

d
Md
f X [ xi ]u ( x  xi )
 f X [ yi ] ( y  xi )dy  
i 1
i 1 
d
d
where u(x) is the unit step function.
S1P4-14
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Time to Gather some Data
• Aim at the bull’s eye using the dart gun’s sights.
• Measure the X and Y displacements from the bull’s eye.
Y
X
• Collect 10 data sample pairs for your dart gun.
• Plot the data on the provided probability paper.
S1P4-15
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Distribution: Uniform
• A most simple distribution. The name comes from its PDF.
1
1
b-a
x
a
-3
-2
0
-1
1
b
2
3
• Vital statistics (mean, variance, median, NO mode):
x
ab
dx 
 median
2
aba
b
  E{ X }  
2
2
x
a

b
(
b

a
)


 2  E{ X 2 }   2  
dx  
 
12
 2 
aba
b
2
S1P4-16
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Distribution: Gaussian
• The Gaussian distribution is also known as the normal
distribution and is often written in shorthand as N(,  2).
• PDF:
1
f X ( x) 
2
e
2
 x   2 2 2
 = 1.5
=1
0.35
0.3
0.25
95.5%
0.2
0.15
99.7%
68.3%
0.1
0.05
0
-4
-3
-2
-1
0
1
2
3
4
5
6
S1P4-17
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Distribution: Gaussian (cont.)
• CDF:
x
1
x


FX ( x)  
  1  erfc


2

2




Insert Plot of CDF here.
• Mean is  and variance is  2.
• The sum of any number of Gaussian random variables (and
constants), independent or otherwise, and most any linear
operation on a Gaussian random variable, is a Gaussian
random variable.
S1P4-18
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Distribution: Gaussian (cont.)
Example:
• Let X be N(x, x2). Define Y = mX + b. Then Y is Gaussian
with mean and variance
y = E{Y} = E{mX + b} = mx + b
y2 = E{Y 2}- y2 = E{(mX + b)2} – (mx + b)2
= E{m 2 X 2 + 2mbX + b 2} – (mx + b)2
= m 2(x2 + x2) + 2mbx + b 2 – (mx + b)2
= m2x2
• Thus, the PDF of Y is fY ( y ) 
1
2 m 
2
e
 y m x b 2 2  m 2
S1P4-19
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Distribution: Gaussian (cont.)
0.4
0.35
0.3
X
f (x)
0.25
0.2
0.15
0.1
0.05
0
-5
-4
-3
-2
-1
0
x
1
2
3
4
5
• A normal random variable N(0,1) is known as the standard
normal random variable. It is often denoted as Z.
X  x
• If X ~ N(x, x2) then a standard RV is Z 
.
x
S1P4-21
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Gaussian (Normal) Probability Distribution (cont.)
• The sum of any number of normally distributed data values
(and constants), from independent experiments or otherwise,
and most any linear operation on normally distributed data, is
also normally distributed.
• The sample mean of normally distributed data {x1, x2, …, xn}
is normally distributed:
– Sum the data, then
– Divide by a scalar, n, a linear operation.
• Neither the sample variance nor the sample standard deviation
of normally distributed data {x1, x2, …, xn} are normally
distributed:
– Both involve the sum of squared normal data.
S1P4-22
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Central Limit Theorem
• Let Y be the sum (or average) of a “very large” number of
mutually independent random variables Xi with identical PDFs
and finite means and variances.
• “Remove” the mean and variance from Y, that is, standardize:
N
1 N
X i  N x
Xi  x

Y  y 
Z
 i1
 N i1
y
N x
x N
• Z tends towards a standard normal random variable.
• In general, the sum of even a small number of independent
identically distributed (i.i.d.) random variables tends towards
Needs Better
a Gaussian distribution.
Explanation!
(Work out more details)
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
S1P4-23
Central Limit Theorem (cont.)
•
Consider the random variable UN defined to be the sum of
i.i.d. uniform random variables ranged between -1 and +1,
N
•
U N   U (1,1) Show Sum
i 1
As Pictures
Use the following theorem:
If X and Y are independent random variables, the PDF of
Z = X + Y is the convolution of the PDFs of X and Y,
f Z ( z )  f X ( x)  fY ( y ) 
•
z
 f X (u ) fY ( z  u )du

Repeatedly applying this theory to generate UN ...
S1P4-24
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Central Limit Theorem (cont.)
0.4
0.4
U
U
2
0.6
1
0.6
0.2
0
-4
0.2
-2
0.6
0
z
2
0
-4
4
0
z
2
4
-2
0
z
2
4
0.6
U
U
4
0.4
3
0.4
0.2
0
-4
EMIS 7300
-2
0.2
-2
0
z
2
4
0
-4
Copyright  2002 - 2005 Dr. John Lipp
S1P4-25
Statistical Modeling (cont.)
Increasing Analysis Difficulty
• It is possible for both the response’s mean and variance to be
functions of the independent variables, the distribution to be
other than the normal, in various combinations:
Mean
Variance
PDF
Constant
Constant
Normal
(x1,x2,…,xm)
Constant
Normal
Constant
 2(x1,x2,…,xm)
Normal
(x1,x2,…,xm)
 2(x1,x2,…,xm)
Normal
Constant
Constant
Other
(x1,x2,…,xm)
Constant
Other
Constant
 2(x1,x2,…,xm)
Other
(x1,x2,…,xm)
 2(x1,x2,…,xm)
Other
Data
Transforms
approximate
simpler
forms
S1P4-26
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Distribution: Chi-Square
• Let X be a zero mean (x = 0) Gaussian random variable with
variance x2.
• Then Y = X 2 has a chi-square distribution (with one degree of
y
freedom).
 2
1
2 x
f
(
y
)

e
, y0
Y
2
• PDF:
2 x y
Insert Plot of PDF here.
S1P4-27
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Distribution: Chi-Square (cont.)
• The mean is x2 and the variance is 2x4.
Show how it is easy to show the mean.
• The median is  0.4550 x2. (Found numerically.)
• Mode = 0.
S1P4-28
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Distribution: Chi-Square (cont.)
1.4
=1
=2
1.2
1
Y
f (y)
0.8
0.6
0.4
0.2
0
0
1
2
3
4
5
y
6
7
8
9
10
S1P4-29
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Distribution: Exponential
• Let X1 and X2 be zero mean (x = 0) independent Gaussian
random variables with equal variances x2.
• Then Y = X12 + X22 has an exponential distribution with
parameter  = 2 x2.
y
1 
fY ( y )  e
, y0
• PDF:

Insert Plot of PDF here.
S1P4-30
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Distribution: Exponential (cont.)
• CDF:
FY ( y )  1  e

y

, y0
Insert Plot of CDF here.
• The mean is , the variance is 2 2, the median is - ln(2),
and the mode = 0.
• Let U be a random variable uniformly distributed between
0 and 1. Then Y = -  ln(U) is exponentially distributed.
• Note: The textbook has a different “definition.”
S1P4-31
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Distribution: Chi-Square with n-degrees of freedom
• Let X1, X2, …, Xn be zero mean (x = 0) independent Gaussian
random variables with equal variances x2.
• Then Y = X12 + X22 + … + Xn2 has a chi-square distribution
with n-degrees of freedom.
1
( n 2 )1 ( y / 2 x2 )
• PDF:
fY ( y )  n / 2 n
y
e
,y0
2  x (n / 2)
3
n = 10
n = 25
n = 100
2.5
1.5

f 2(2)
2
1
0.5
0
0
0.5
1
1.5

EMIS 7300
2
2.5
3
2
Copyright  2002 - 2005 Dr. John Lipp
S1P4-32
Distribution: Chi-Square with n-degrees of freedom (cont.)
• The gamma function (r) is a “continuous” generalization of
the factorial (needed to count continuous things)
r is an integer 
 (r  1)!
(r )   x e dx  1/ 2

 (r  3 / 2)! r mulitple of 1 2
0

r 1  x
• The mean is nx2 and the variance is 2nx4.
• Show that for n = 2 it is consistent with exponential.
• Show motivation: analysis of standard deviation / variance.
S1P4-33
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Distribution: Rayleigh
• Let X1 and X2 be zero mean (x = 0) independent Gaussian
random variables with equal variances x2.
• Then Y  X 12  X 22 has a Rayleigh distribution ( = x2).
• The PDF :
fY ( y ) 
y

e
y2

2
, y0
0.35
=4
0.3
0.2
Y
f (y)
0.25
0.15
0.1
0.05
0
0
1
2
3
4
5
y
6
7
8
9
10
S1P4-34
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Distribution: Rayleigh (cont.)
FY ( y )  1  e
• CDF:
1
y2

2
, y0
=4
0.8
Y
F (y)
0.6
0.4
0.2
0
0
1
2
3
4
5
y
6
7
8
9
10
• The mean is  2 , the variance (2 - /2), the mode  , the
median 2 ln(2).
• The Rayleigh distribution has important applications in miss
distance calculations, radar and other coherent signal systems.
S1P4-35
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Distribution: Rayleigh (cont.)
• A ballistic shell is required to hit within 1 meter of a target.
• After several trials it is observed that the downrange and
crossrange miss distances are independent and Gaussian with
a standard deviation of 0.8 meters.
• Thus, the radial miss distance, the RSS (root sum squares) of
the downrange and crossrange, is Rayleigh distributed with
 = 0.64.
• The Ph is then the CDF of 1,
1
y
Ph  P(Y  1)  FY (1)  
e
0.64
0
y2

1.28
dy  1  e
1

1.28
 0.54
S1P4-36
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Distribution: Cauchy
• Let X1 and X2 be zero mean (x = 0) independent Gaussian
random variables with equal variances x2.
• Then Y = X1 / X2 is Cauchy distributed with parameter  = 1.
 1
fY ( y ) 
  2  y2
• PDF:
0.35
0.3
0.2
Y
f (y)
0.25
0.15
0.1
0.05
0
-10
-8
-6
-4
-2
0
y
2
4
6
8
10
S1P4-37
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Distribution: Cauchy (cont.)
• The Cauchy distribution has no mean or variance! Why?
• The mode and median are both zero.
S1P4-38
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp
Homework
• Mandatory (answers in the back of the book):
4-1
4-47
4-13
4-79
4-29
4-41
R
4-101
• Mandatory: Subtract the X and Y
sample means from your dart gun
data and compute the radial miss
distance and angle errors. Using
probability paper, attempt to
determine the appropriate PDF of
the radial and angle errors.

Ri  ( xi  x ) 2  ( yi  y ) 2
 yi  y 

 i  tan 
 xi  x 
1
S1P4-39
EMIS 7300
Copyright  2002 - 2005 Dr. John Lipp