Leveraging AI for Enhanced Cost Estimation

Matt McDonald

Eric Sick

AI as an Assistant to the Estimator

Increased Complexity is challenging estimators in providing decision makers data for decisions

Challenges /Limitations with AI

AI is NOT THE ANSWER, but a tool to educate and a source of data to increase knowledge

Implementing Practical AI

Assess current estimation processes and identify improvement opportunities.

Introduction

We will discuss practical applications of AI and how AI can assist with challenges faced by Estimators with lack of information or experience.

How AI can be of Value

By increasing accuracy, efficiency, and confidence of estimators who lack data

Evolution and Benefits of AI in Cost Estimation

1950 -1980

AI concept introduced by Alan Turing in 1950.

1990s

IBM's Deep Blue defeats world chess champion in 1997.

2000s

DARPA's autonomous vehicle Grand Challenge held in 2004.

2010s

DeepMind's AlphaGo defeats Go champion in 2016.

2020

GPT-3, a powerful language model, released in 2020.

2024

AI systems achieve significant advances in natural language understanding.

START

Current

Phase 01

Phase 02

Phase 03

Phase 04

Exponential growth in capabilities is leading to transformation across all lines of business.

Leading Innovation in AI Augmentation

4

https://seerai.galorath.com/

Delivers Inputs for Any Cost Model

Provides Inputs Based on AI

SEERai Augments SEER & Other Models with Knowledge & Parameters

Supports SEER Predictive Analytics

Trained on SEER Kbases

Reflects Decades of Investment & Innovation

Deskilling When Expertise Unavailable

How AI Models feel more familiar

• Bayesian Inference
• Broad Applicability
• Uses ranges from heuristic to artificial networks
• Statistical process
• Describes how prior knowledge can be used to predict future probabilities
• Bayes Theorem
• Where predictive probability distribution of the hypothesis, given a set of data, is the product of the prior probability distribution and a likelihood function.

5

P(B|A)P(A)

P(A|B)=

P(B)

The methods behind AI models are based on commonly used practices

Neural and Bayesian Networks

6

Untrained Model

Update Weighting

Trained NLP

01

02

03

Describes learning, or the updating of a hypothesis given additional data

Methodology

7

Systems Engineering

Conceptual Model

Operational Requirements

Constrained Parameter Space

Algorithmic, Rules-Based

Optimize for Solution

Operational Requirements

Unconstrained Parameter Space

Probabilistic, Concept-Based

Marginalize for Distribution

Strengths and Weaknesses

8

Predictive Capability

Reasoned outputs,

Continuous Learning

Improves over time

False Confidence

No uncertainty metrics

Black Box

Unsecure

Prohibitive for some use cases

01

03

02

04

Practical Steps to Implement AI

• Start with Clear Objectives and Pilot Projects:
• Define specific goals for AI integration (e.g., improve accuracy, automate data analysis).
• Begin with pilot projects to test AI models and gather insights before scaling up.
• Ensure High-Quality Data and Continuous Monitoring:
• Collect and cleanse comprehensive historical data to train accurate AI models.
• Continuously monitor AI performance and update models to maintain reliability and relevance.
• Invest in Education and Integration:
• Educate and train staff on AI technologies and their applications in cost estimation.
• Seamlessly integrate AI tools into existing workflows to enhance efficiency without causing disruption.

9

Understanding the challenges with AI will help us create best practices

How do start a network security compliant AI pilot project?

1. Define Pilot Objectives and Scope
2. Select an On-Premises AI Platform
3. Establish a Secure Environment
4. Select and Prepare Data for the Pilot
5. Choose Role-Specific Models
6. Develop and Test the Solution
7. Set Up Pilot-Specific Training and Evaluation
8. Plan for Future Integration and Scalability

10

By following these steps, you can initiate a secure and compliant AI pilot project within a restricted network, enabling innovation while adhering to agency security constraints.

How do you address the issue of hallucinations in AI?

1. Enhanced Data Curation and Preprocessing – High Quality Data Wanted
2. Reinforcement Learning w/ Human Feedback (RLHF) – Get SMEs involved
3. Post-Training Verification Mechanisms – Robust UAT
4. Adaptive Model Architectures – Pre-set prompt techniques with restrictive language
5. Confidence Scoring and Output Filters
6. Continuous Monitoring and Feedback Loops

11

SEERai is designed to provide accurate and reliable information based on the data and knowledge bases it has access to.

How do you work with data at different access/security levels?

1. Role-Based Access Control (RBAC) and Attribute-Based Access Control (ABAC)
2. Segmentation of Data by Security Level
3. Data Labeling and Metadata Tagging
4. Fine-Grained Data Filtering and Prompt Engineering
5. Federated Learning or Multi-Instance LLMs
6. Audit Trails and Anomaly Detection
7. Training LLMs on Synthetic or Anonymized Data

12

Tailoring or combining depends on specific security policies, the nature of the data, and the level of risk associated with accessing different security-level data.

What % of data do you use to train the system versus % for testing?

• We use an 80/20 Split: This split provides a robust training dataset while preserving enough data to reliably evaluate the model’s performance.
• Given the diversity of our data, we’re at a low risk of overfitting, making 80/20 a solid approach.
• Cross-Validation: For smaller datasets, k-fold cross-validation (often with k=5 or k=10) is beneficial. This method divides the data into k subsets, training the model k times while leaving out a different subset each time for testing.

13

Following this approach ensures that a model is well-trained and generalizes effectively to new data.

How do I configure SEER-SEM for Hybrid Development 50/50?

1. Select the Appropriate Knowledge Bases
2. Adjust Effort Allocation
3. Define Labor Rates and Roles
4. Tool Usage and Automation Level
5. Development Process and Environment
6. Risk and Uncertainty Analysis
7. Calibration and Validation

14

Results in a balanced estimate that reflects the unique nature of a hybrid development team utilizing both traditional and AI-driven methodologies.

What can an AI tool deliver? WBS? Risk Ranges? Sizing Recommendations?

15

Building structured estimates from minimal data, aiding informed decision-making.

Develop Requirements

Calculate Conversions

Develop Estimate Ranges

AI in Action - Examples

16

01.

Benefits: Drive Efficiency: Reduces manual effort and saves time. Improve Accuracy: Minimizes human errors in script creation.

Writing Scripts

02.

Benefits: Comprehensive Coverage: Ensures all aspects of the project are covered. Consistency: Maintains consistency across WBS elements.

Building WBS

03.

Benefits: Precision: Enhances the precision of hardware-related cost assessments. Speed: Accelerates the estimation process

Spec Translation

AI can be an Assistant to the Estimator

Challenges must be Overcome

AI can be an immediate force multiplier to estimators

Implement Practical AI that works for your organization

Conclusion and Future Directions

It is a Brave New World

Contact us for more information

• Service Catalog - Agile Approaches for:
• PMO Support
• Process Standardization
• Tool implementation to manage these complex efforts.
• We offer the SEER® Suite of Predictive Analytic Models
• We are a Prime Contractor on SeaPort NxG.

18

• CAGE 978U2
• DUNS Number 07-975-4803
• NAICs Codes:
• 511210, 541330ENG, 54151S, and 611420
• Contact Us:

Matt McDonald

President, Galorath Services mmcdonald@galorath.com

(m) 571.230.5555

Script Example

Backup

19

VBA Script Example Question

“Provide Excel VBA code that will run a Monte Carlo risk simulation based on the following assumptions in the “Risk Assumptions” Tab: 1) Normal Distribution; 2) Low Point is in Cell B2; 3) Mean is in Cell C2; High Point is in Cell D2; and the number of iterations is in Cell E2. The simulation will create a new “Risk Runs” worksheet and save the values iteratively beginning in Cell B2.”

​

Link to Example

20

VBA Script Example: AI response

21

Non-VBA Script Example: Question

“Provide a non-VBA method in Excel that will run a Monte Carlo risk simulation based on the following assumptions in the “Risk Assumptions” Tab: 1) Normal Distribution; 2) Low Point is in Cell B2; 3) Mean is in Cell C2; High Point is in Cell D2; and the number of iterations is in Cell E2. The simulation will create a new “Risk Runs” worksheet and save the values iteratively beginning in Cell B2.”

22

​

​

​

​

​

• “Drag this formula down to the number of iterations specified in cell E2 of the “Risk Assumptions” worksheet.”
• An example of an AI “hallucination” instance when AI generates incorrect response

23

Non-VBA Script Example: Response

WBS Example

Backup

24

WBS Example Prompt

“I am looking to use the SEER suite of models for estimating the development of a military attack helicopter. Please act as an experienced SEER user and provide all results in SEER-compatible format. Specifically, I need the following:

1. Work Breakdown Structure (WBS): Provide a detailed WBS for a military attack helicopter. Use Mil Std 881 where possible
2. Mass Range for Each WBS Element: Estimate the mass range (least, likely, and most) for each WBS element, assuming the use of lightweight composites where appropriate and a speed exceeding 120 miles per hour.
3. SEER KBASES: Identify the appropriate SEER KBASES to be used for each WBS element.
4. Software Size Range: Estimate the software size range (least, likely, and most) for the avionics software, which should be in C++ and include up to 70% software reuse. Tie this estimate into the WBS.
5. Major Risks: Identify the major risks in the estimate.
6. Additional Questions: Provide five additional questions I should ask the Program Manager to better understand the project requirements.”

25

WBS Prompt Example: AI response

1. Airframe
• 1.1 Structure
• 1.1.1 Fuselage
• 1.1.2 Wings/Rotors
• 1.1.3 Empennage
• 1.1.4 Landing Gear
• 1.2 Propulsion
• 1.2.1 Engines
• 1.2.2 Fuel System
• 1.3 Electrical Systems
• 1.3.1 Power Distribution
• 1.3.2 Wiring and Connectors
• 1.4 Avionics
• 1.4.1 Flight Control Systems
• 1.4.2 Navigation Systems
• 1.4.3 Communication Systems
2. Mission Equipment
• 2.1 Weapons Systems
• 2.1.1 Guns and Ammunition
• 2.1.2 Missiles
• 2.2 Sensors
• 2.2.1 Radar
• 2.2.2 Electro-Optical/Infrared (EO/IR) Systems
3. Software Development
• 3.1 Avionics Software
• 3.1.1 Flight Control Software (C++)
• 3.1.2 Navigation Software (C++)
• 3.1.3 Communication Software (C++)
• 3.1.4 Weapons Integration Software (C++)
4. Integration and Test
• 4.1 System Integration
• 4.2 Ground Testing
• 4.3 Flight Testing
5. Support and Training
• 5.1 Maintenance Support
• 5.2 Training Systems
• 5.2.1 Simulators
• 5.2.2 Training Manuals

​

26

Mass Ranges

27

Risks

• Technical Complexity:
• Risk: High complexity in integrating advanced avionics and weapon systems.
• Mitigation: Conduct thorough system integration testing and employ expert system engineers.
• Schedule Delays:
• Risk: Potential delays due to unforeseen technical challenges or supply chain issues.
• Mitigation: Implement robust project management practices and maintain flexibility in the schedule.
• Cost Overruns:
• Risk: Budget overruns due to underestimation of development and testing costs.
• Mitigation: Use detailed cost estimation tools (SEER-H and SEER-SEM) and include contingency reserves.
• Software Integration:
• Risk: Challenges in integrating reused software components with new development.
• Mitigation: Conduct compatibility testing and ensure thorough documentation of reused components.
• Regulatory Compliance:
• Risk: Difficulty in meeting all military and aviation regulatory requirements.
• Mitigation: Engage with regulatory bodies early and conduct regular compliance audits.

28

Questions for the PM

• Scope and Requirements:
• What are the specific mission profiles and operational requirements for the attack helicopter?
• Budget and Funding:
• What is the allocated budget for this project, and are there any funding constraints?
• Schedule and Milestones:
• What are the key milestones and deadlines for the project, and how flexible are these dates?
• Technology and Innovation:
• Are there any specific technologies or innovations that must be incorporated into the helicopter design?
• Stakeholder Expectations:
• Who are the key stakeholders, and what are their primary concerns or expectations regarding the project?

​

29

Translation Example

Backup

30

Spec Translation Example Prompt

“Identify by Bell helicopter model and extract performance data from publicly available specifications data for the identified Bell helicopter and place it in a table that provides data by several lift off weights for hovering altitude, cruise speed, and endurance.”

​

31

Bell Helicopter Performance Data

32