1 of 21

General Error Regression Method

mini-Tutorial

Brad Clark, PhD

BradClark55@gmail.com

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

1

Nov 2022

Source: Tim Anderson, SE3011 Engineering Economics and Cost Estimation course, Naval Postgraduate School, Fall 2020

2 of 21

Estimation Error

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

2

Nov 2022

Additive Error

Multiplicative Error

Reference:

H.L. Eskew and K.S. Lawler, “Correct and Incorrect Error Specifications in Statistical Cost Models,” Journal of Cost Analysis, Spring 1994, page 107

3 of 21

General Error Regression

  • General Error Regression Method (GERM)
  • A better method of regression analysis
    • Has been in the literature for several years, but
    • Has become widely feasible in today’s computing environment
  • General error regression separates the question of whether estimation errors should be additive or multiplicative from the question of whether the shape of the CER should be linear or non-linear
  • Give the cost analyst the freedom to model any CER functional form
    • Y = a + bxc
    • And the choice of additive or multiplicative errors

​

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

3

Nov 2022

4 of 21

General Error Regression

  • Breaks the bond of OLS Regression

​

  • But, requires the analyst to be able to “minimize” an error function using numerical optimization technique

​

  • This is possible using Excel’s solver function

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

4

Nov 2022

5 of 21

Typical CER Function Forms

y = Cost (dependent variable

x = parameter (independent variable)

a, b, c are constant coefficients from historical data

Factor CER: y = ax

Linear CER: y = a + bx (Algebra’s intercept & slope)

Non-Linear CERs: y = axc

y = abx

y = a + bxc (versatile functional form)

models a linear function when c = 1

models a power function when a = 0

​

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

5

Nov 2022

 

6 of 21

Additive Error Model

  •  

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

6

Nov 2022

 

Additive Error

7 of 21

Additive Error Model Regression

  •  

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

7

Nov 2022

8 of 21

Multiplicative Error Model

  •  

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

8

Nov 2022

Multiplicative Error

9 of 21

Multiplicative Error Model Regression

  •  

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

9

Nov 2022

10 of 21

MPE Bias

  •  

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

10

Nov 2022

11 of 21

Minimizing Bias

  • Is a biased CER a bad thing?
  • Depends on whether you would rather have a minimum standard error (SE) with bias (using OLS) or an unbiased CER (using an unbiased MPE procedure) with a slightly larger standard error
  • It is a question of precision (minimum SE) over accuracy (larger SE but lower bias)
  • Generally, cost estimators prefer accuracy over precision

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

11

Nov 2022

12 of 21

Zero Bias MPE

  •  

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

12

Nov 2022

13 of 21

ZMPE Procedure

  • Same as MPE but constrained by holding the net percentage bias to zero (0)
    • Constrained optimization problem
    • Minimize sum of squared errors
  • However, adding the constraint reduces the trade space, leading to a higher standard error
  • ZMPE leads to CERs with zero net percentage bias but increased standard error
    • Choosing accuracy over precision
  • Solve using numerical optimization (Excel solver)

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

13

Nov 2022

14 of 21

GERM Statistics

  • In general error regression we no longer have access to traditional statistics that we once had in the OLS method such as:
    • Coefficient of determination (R2)
    • t statistic
    • F statistic
    • ANOVA
    • Etc.
  • We can compare all general error regression models with three statistics:
    • Standard error of the estimate (SE or SEE)
    • Net percentage bias
      • How well does the CER go through the middle of the data – want this to be as close to zero bias as possible
    • Pearson’s correlation squared (r2) between actuals and estimates
      • Want this to be as close to 1.0 as possible

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

14

Nov 2022

15 of 21

Standard Error of the Estimate

  •  

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

15

Nov 2022

16 of 21

Net Percentage Bias

  •  

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

16

Nov 2022

17 of 21

Pearson’s Correlation Coefficient Squared

  •  

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

17

Nov 2022

18 of 21

GERM Procedure

  • Plot the data (scatter plot)
  • Choose a function form that models the logical impact of independent variables on the dependent variable, cost
    • E.g. y = a + bx, y = axc, y = a + bxc
  • Choose either additive- or multiplicative-error model
    • Additive if you want the error of the estimate to be expressed in uniform dollar amounts across the entire cost range
    • Multiplicative if you want the error of the estimate to be expressed as a uniform percentage error of the estimated cost cost range
  • Select starting coefficients values for a, b, c that minimize the error
  • Constrain the net bias to zero
  • Run the solver tool in Excel

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

18

Nov 2022

19 of 21

GERM Demonstration

Multiplicative Percent Error (MPE)

Zero Multiplicative Percent Error (ZMPE)

Comparison between Log OLS Regression and ZMPE

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

19

Nov 2022

Source: Tim Anderson, SE3011 Engineering Economics and Cost Estimation course, Naval Postgraduate School, Fall 2020

20 of 21

Conclusions

  •  

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

20

Nov 2022

21 of 21

Additional References

  • H.L. Eskew and K.S. Lawler, “Correct and Incorrect Error Specifications in Statistical Cost Models,” Journal of Cost Analysis, Spring 1994
  • Stephen A. Book & Philip H. Young (1997) General-Error Regression for Deriving Cost-Estimating Relationships, The Journal of Cost Analysis, 14:2, 1-28
  • Timothy P. Anderson (2009) A Distribution-Free Measure of the Significance of CER Regression Fit Parameters Established Using General Error Regression Methods, Journal of Cost Analysis and Parametrics, 2:1, 7-22
  • Stephen A. Book , Melvin A. Broder & Daniel I. Feldman (2011) Statistical Foundations of Adaptive Cost-Estimating Relationships, Journal of Cost Analysis and Parametrics, 4:1, 63-90
  • Stephen A. Book (2012) Prediction Bounds for General-Error-Regression Cost-Estimating Relationships, Journal of Cost Analysis and Parametrics, 5:1, 25-51

First International Boehm Forum on COCOMO(R) and Systems and Software Cost Modeling

21

Nov 2022