1 of 28

Evolution of Software Cost Estimating:�Challenges & Insights

Brad Clark, PhD

2024 International Boehm COCOMO

Forum on Systems and Software Cost

Modeling

2 of 28

Boehm Center for Systems and Software Engineering

2

November 2024

My first career!

(40 years ago)

3 of 28

The Problem

Boehm Center for Systems and Software Engineering

3

November 2024

Organizations have a need for credible costs and schedules for budgeting purposes

Platforms and development approaches have put stress on organizations to produce quality cost estimates that meet mission needs

As a result, the cost estimation industry needs to evolve to address these estimation challenges through education, processes, and tools

In this presentation we are going to discuss the wide array of challenges and discuss a few insights to meet them

4 of 28

COST ESTIMATING CHALLENGES

Boehm Center for Systems and Software Engineering

4

November 2024

5 of 28

Software’s Perverseness

You can’t see it – feel it – hold it – weight it
Sometimes it only makes up a small part of the system – other times it IS the system
Increasingly Software-enabled Systems are becoming significant strategic resources

Boehm Center for Systems and Software Engineering

5

November 2024

Technology	Transportation and Logistics
Finance	Education
Healthcare	Entertainment and Social Media
Manufacturing	Energy and Utilities
Retail and e-Commerce	Government and Public Sector (Services, Tax, Safety, Regulatory, Administration)

6 of 28

Advances in Computing Platforms

Boehm Center for Systems and Software Engineering

6

November 2024

We have come a long way, and platforms continue to evolve

+ AI

7 of 28

Computing Platform Evolution

Boehm Center for Systems and Software Engineering

7

November 2024

Big Iron computing hardware (e.g., IBM 360 Mainframe)

Punch cards
Central facility

ENIAC

Mini-computers (e.g., DEC VAX)

Lower cost than mainframes
Personal interactions

Micro-computers and Personal computers

Modem communications

DARPA network

Super-computers &

Parallel computing

Public internet networks – web browsers & text-based email

Collaborative software applications

Server farms & increased computing power

Cloud Computing

Software as a Service (SaaS)

Internet of Things (IoT)

Infrastructure as a Service (IaaS)

Who needs the Internet?

Autonomous systems�(sea, land, air, and space)

Collective of:

Sensors
Control Systems
Automation

Web browser as a computing platform

Evolving computing capability brings new cost estimation challenges

Generative Pre-trained Transformer AI

(ChatGPT, CoPilot, Gemini)

8 of 28

Estimating Applications

As estimation capabilities increase so do their applications

Boehm Center for Systems and Software Engineering

8

November 2024

Quote Analysis

Sustain

Negotiation Analysis

Bid / No Bid

Life Cycle Cost Analysis

Should Cost

Estimate to Complete

Develop

Cost Risk Analysis

Cost As Independent Variable

9 of 28

Early Cost Estimation

75% of System costs are determined early in the lifecycle during the time of greatest uncertainty

Boehm Center for Systems and Software Engineering

9

November 2024

10 of 28

Rapid Change Creates a Late Cone of Uncertainty

Boehm Center for Systems and Software Engineering

10

November 2024

Early lifecycle uncertainty (ConOps) meets late lifecycle uncertainty (changing priorities)

Source: Barry Boehm, USC-CSSE COCOMO Forum, 2014

11 of 28

Brownfield Software Conversion

Development and deployment of new software systems in the immediate presence of existing (legacy) software applications/systems
This implies that any new software architecture must take into account and coexist with live software already in place
There are a lot of large COBOL systems still in operation

Boehm Center for Systems and Software Engineering

11

November 2024

Source: Wikipedia

Challenging to perform conversion due to re-engineering/translation and difficult to estimate the cost

Image source: Scholarly Community Encyclopedia, encyclopedia.pub

12 of 28

Developing Secure Software

Boehm Center for Systems and Software Engineering

12

November 2024

Level 1	Protection is ad-hoc.
Level 2	Protect against casual or coincidental access by unauthenticated entities.
Level 3	Protect against intentional unauthenticated access by entities using simple means with low resources, generic skills and low motivation.
Level 4	Protect against intentional unauthenticated access by entities using sophisticated means with moderate resources.
Level 5	Protect against intentional unauthenticated access by entities using sophisticated means with extended resources.

13 of 28

Digital Engineering -1

Digital transformation changes the way we work – impacts cost
New costs: investment and training
Maturity levels, e.g.,

1 Pre-Digital Engineering
2 Computer-Aided Design (CAD)
3 Computer-Aided Engineering (CAE)
4 Digital Prototyping
5 Model-Based Engineering (MBE)
6 Digital Twins
7 Digital Thread
8 Integrated Digital Acquisition Frameworks
9 Collaborative Digital Ecosystems
10 Digital Engineering Enterprise Solutions

Boehm Center for Systems and Software Engineering

13

November 2024

Image source: appspacesol.com

14 of 28

Digital Engineering -2

Continuous simulation, development, and production

Boehm Center for Systems and Software Engineering

14

November 2024

Image source: mantech.com

15 of 28

Continuous Iterative Development

Continuous development & integration needed for faster delivery

Continuous Integration / Continuous Delivery Development (CI/CD)
Software Factory Ecosystem
Software product lines
Software pipelines
Reuse libraries
Development Operations (DevOps)
Development Security Operations(DevSecOps)

What is the beginning and end for an estimate?

Boehm Center for Systems and Software Engineering

15

November 2024

Image source: embedded.com

16 of 28

AI Assisted Development -1

AI benefits could reduce cost

Boehm Center for Systems and Software Engineering

16

November 2024

Activity	Benefits
Requirements	AI tools can help clarify terminologies and concepts on-the-fly, reducing the need for prolonged meetings or external consultations. Developers or analysts can query AI tools for insights or comparative analysis, which might help in refining requirements.
Design	Developers can quickly consult AI tools for recommended design patterns or architectural practices suitable for their problem. Preliminary design ideas can be discussed with AI tools for quick feedback.
Code	Developers can seek assistance on coding challenges, syntax, and algorithmic solutions. AI tools can assist in code reviews, highlighting potential pitfalls or anti-patterns.
Testing and Integration	AI tools can suggest potential edge cases or testing scenarios. For failing tests or integration issues, developers can discuss potential causes and solutions with AI tools.
Maintenance	AI tools can assist in understanding old codebases, suggesting potential refactoring techniques or identifying deprecated methods. For known errors or bugs, AI tools can suggest common solutions or workarounds.

Source: Ray Madachy, Ryan Bell, Ryan Longshore, “Factoring AI Assistance into Software and Systems Engineering Cost Models,” Naval Postgraduate School, International Boehm COOCOM Forum, Nov 2023

17 of 28

AI Assisted Development -2

AI benefits could increase cost

Boehm Center for Systems and Software Engineering

17

November 2024

Activity	Downfalls
Requirements	Relying too much on AI tools for domain-specific knowledge might lead to missed nuances that an expert in the field would be aware of.
Design Code	Over-relying on AI for design decisions without human review can lead to suboptimal choices.
Testing and Integration	If developers use AI tool suggestions verbatim without understanding, it might introduce bugs or inefficient code. Waiting on AI tool responses for every small issues can become a crutch and delay development if developers stop trying to problem-solve on their own.
Maintenance	If AI tool suggestions are taken without thorough review, it might lead to unnecessary tests or efforts spent on non-issues. If teams over-rely on AI tools for maintenance, they might overlook deeper architectural or design issues that require human expertise.

Source: Ray Madachy, Ryan Bell, Ryan Longshore, “Factoring AI Assistance into Software and Systems Engineering Cost Models,” Naval Postgraduate School, International Boehm COOCOM Forum, Nov 2023

18 of 28

GAO Cost Estimating Challenges

Boehm Center for Systems and Software Engineering

18

November 2024

Program	Data	Cost Estimate
Program instability Unreasonable program baselines Unrealistic projected savings New development or sustainment processes First-time integration Cutting edge technology Complex technology Diminished industrial base	Obtaining data Inadequate data collection Historical cost databases not available Irrelevant or out-of-date data Unreliable data No comparison to similar programs Data not normalized	No defined process for generating the estimate Inexperienced analyst Unrealistic or poorly defined assumptions No supporting documentation Overoptimism Inappropriate estimating methodologies No basis of estimate

Source: GAO Cost Estimating and Assessment Guide, 2020

”Having some data is better than not having any data at all” – Is that true?

19 of 28

COST ESTIMATING INSIGHTS

Boehm Center for Systems and Software Engineering

19

November 2024

20 of 28

Evolving Cost Estimation Requires Better Tools��

Boehm Center for Systems and Software Engineering

20

November 2024

74	75	76	77	78	79	80	81	82	83	84	85	86	87	88	89	90	91	92	93	94	95	96	97	98	99	00	…
																													AI Assistance
																												ACEIT
																										•	COCOMO II
																							•	CostXpert
																					•	Sage
													•	REVIC
												•	SEER
												•	SASET
									•	JS-1
							•	COCOMO
							•	SLIM										Automated Cost Estimating Integrated Tools
					•	Freburger-Basili
					•	Price
				•	Schneider
			•	Walston-Felix
			•	Jones
			•	Halstead
•	Fredrick

Insight: Integrated Cost Estimation Analysis Tools are the future

ACEIT: Automated Cost Estimating Integrated Tools

21 of 28

Development Approaches Drive Estimation

Software Development Lifecycles (SDLC) evolve to match the execution platforms
Software Cost Estimation estimates the effort, cost, and duration of the SDLC & associated processes, e.g.,

Waterfall
Iterative Waterfall
Evolutionary
Iterative Evolutionary

Insight

Understanding the SDLC & processes are key to creating a credible estimate

Boehm Center for Systems and Software Engineering

21

November 2024

Image source: commons.wikipedia.org

22 of 28

Analysis of Alternatives (AoA) LCCE

Life Cycle Cost Estimates (LCCE) provide an important AoA cost input

Multiple alternatives should be considered early in the life cycle

70-75% cost decision are made

LCCEs are labor intensive (multi-disciplinary)

Systems Engineering
Hardware
Software (considering Development approach)
Training
Deployment

Insight

AoAs require robust cost estimation processes
Digital Engineering Tools and Cost Estimation Tools need to be integrated

Boehm Center for Systems and Software Engineering

22

November 2024

23 of 28

Transdisciplinary Cost Estimation

Should be studying the whole system, because no gluing together of partial pieces of a complex nonlinear system can give a good idea of the behavior of the whole
Systems of Systems, e.g.,

Web services
Networked devices/vehicles
Homeland Security
Healthcare

Software platforms - what the application operates on, e.g.,

Web-based applications
Databases
Cloud computing
Embedded computers

Insight

You should study the whole system – then you will know what drives cost and the questions to ask for a cost estimate

Boehm Center for Systems and Software Engineering

23

November 2024

Image source: i2insights.org

24 of 28

Continuous Need for Cost Estimation Training

Challenges

Population turnover: People move up, change organizations, retire
New junior estimators are hired to replace them

Insights

Teach people on-the-job on how to do software cost estimation
ICEAA Software Cost Estimation course should be required formal training
Take transdisciplinary courses – try to understand the “why” of different cost estimation concepts
Exposure yourself to leading edge information – attend conferences, symposiums, forum, and workshops

Boehm Center for Systems and Software Engineering

24

November 2024

Image source: electricalworkbook.com

25 of 28

Mayfly Cost Models

Agile, adaptable, flexible, localized and short-lived
Challenge in the future is to develop a framework for developing models quickly and then teach that framework to cost estimators

Understanding statistics around modeling is important
Understanding how Cost Estimating Relationships (CER) are created

There will continue to be a need for these types of models

Mayfly models for dynamic for changing development technologies
Open models like COCOMO demonstrate how models work

Insight

There should be different modeling approaches for different computing platforms

Boehm Center for Systems and Software Engineering

25

November 2024

Image source: researchgate.net

26 of 28

Stop Pounding a Square Peg Into a Round Hole

Cost estimators have a hammer and see everything as a nail
Factory Concept

Car Factory: raw materials go in – car comes out
Software Factory: requirements go in – capabilities come out

Insight

Factories have investments and fixed costs
Cost As Independent Variable: estimate output given level of effort

Boehm Center for Systems and Software Engineering

26

November 2024

Image source: manufacturingtomorrow.com

27 of 28

In Summary

Organizations have a need for credible costs and schedules for budgeting purposes
Platforms and development approaches have put stress on organizations to produce quality cost estimates that meet mission needs
As a result, the cost estimation industry needs to evolve to address these estimation challenges through education, processes, and tools

Automated Cost Estimating Integrated Tools
Understanding the SDLC & processes are key to creating a credible estimate
Digital Engineering (DE) Tools and Cost Estimation Tools need to be integrated for rapid costing of DE thread’s for Analysis of Alternatives
By studying different domains, you will know what drives cost and the questions to ask for a cost estimate
Continue to seek cost estimation training
Mayfly models are adaptable and flexible to estimate costs for different computing platforms
In a software factor scenario, cost predicts capability

Boehm Center for Systems and Software Engineering

27

November 2024

28 of 28

Thank You��Questions or Comments?

Boehm Center for Systems and Software Engineering

28

November 2024