Download Software Prediction Models

Software Prediction Models
Forecasting the costs of
software development
Prediction Study Outcomes Vary






Estimation-by-analogy beats regression
Or not
Classification and regression trees (CART)
beats regression
Or not
Artificial neural networks beat regression
Or not
Why Are The Results Conflicting?





Poor data or research procedure
Complex techniques may require expert
users; hence applications may vary
Small sample size
Measurement process that is flawed
Selective use of differing parameters may
result in different rankings
Key Terms

Accuracy indicator
–
–


Leave-one-out cross-validation
Arbitrary function approximator taxonomy
–
–
–

Some measure of a process
A summary statistic based on that measure
Many-data versus sparse-data
Linear versus nonlinear
Supervised versus unsupervised
Reliability versus validity
Indicator 1: MMRE


Mean magnitude of relative error (MMRE) is an
average where the MRE=|actual-prediction|/actual
Claimed advantages of MMRE
–
–
–
–
Compare across data sets*
Independent of units
Compare across differing prediction models*
Scale independence
*An hypothesis challenged by this paper
Indicator 2: MER

Magnitude of the error relative to the
estimate (MER) is defined as
MER = |actual-prediction|/prediction
Indicator 3: AR

The absolute residual (AR) is defined as
AR = |actual-prediction|
Other Measures





Standard deviation (SD)
Relative standard deviation (RSD)
Log standard deviation (LSD)
Balanced relative error (BRE)
Inverted balanced relative error (IBRE)
Standard Deviation of Residuals,
Denoted SD
n
SD 
2
ˆ


y

y
 i i
i 1
(n  1)
Notes :
1. This formula is NOT the general formula for StDev
2. The simplifica tion occurs because the mean of the
residuals ri  yi  yˆ i is zero
Algebraic Simplification
Let ri  yi  yˆ i represent the i th residual
and r the mean of the residuals.
n
StDev(residuals ) 
n

 r 
i 1
n 1
i 1
n 1
n
2
i
2


r

r
 i

2
ˆ
  yi  yi 
i 1
n 1
Relative Standard Deviation (RSD)
n
RSD 

i 1
 yi  yˆi 
2
xi
(n  1)
Log Standard Deviation (LSD)

 
 ei   


i 1 
 2
LSD 
(n  1)
n
2

 

2
where
  the variance of the residual ei and
2
ei  ln yi  ln yˆ i  ?
Balanced Relative Error (BRE)
 ( yˆ  y )
ˆ
,
y

y

0
 y
BRE  
( yˆ  y )

, yˆ  y  0
 yˆ
Inverted Balanced Relative Error
(IBRE)
 ( yˆ  y )
ˆ
,
y

y

0
 y
IBRE  
( yˆ  y )

, yˆ  y  0
 yˆ