1 of 16

2024 International Boehm COCOMO Forum

on

Systems and Software Cost Modeling

Digital Engineering Measurement Guidebook

November 12, 2024

Salvatore R. Bruno

LM–Aero Engineering and Technology Measurement Lead

INCOSE Measurement Working Group Chair

Six Sigma Certified Green Belt

PRACTICAL SOFTWARE AND SYSTEMS MEASUREMENT

1

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

2 of 16

A collaboration among industry, government and academia

Digital Engineering Measurement Framework – v2.0

Developed and Published by Members of:

​

​

​

​

Practical Software & Systems Measurement

Systems Engineering �Research Center

Aerospace Industries Association

​

​

​

​

​

​

National Defense Industrial Association

International Council on Systems Engineering

Department of Defense�Research & Engineering

​

​

​

​

​

​

The Aerospace Corporation

INCOSE Measurement Working Group

PRACTICAL SOFTWARE AND SYSTEMS MEASUREMENT

2

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

3 of 16

3 Minutes

5 Minutes

8 Minutes

34 Minutes

10 Minutes

​

Digital Engineering Measurement Guidebook

Presentation Agenda

​

  • Key Contributors
  • Brief Guidebook History
  • DE Measurement Framework Objective
  • Version 2.0 Changes
  • Questions and Answers

PRACTICAL SOFTWARE AND SYSTEMS MEASUREMENT

3

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

4 of 16

  • Salvatore Bruno: Digital Engineering Measurement Framework Chair
  • Core Team and Editors
    • Rita Creel
    • Richard Halliger
    • William (Bill) Luk
    • Misak Zetilyan
    • Ryan Noguchi
    • Brad Clark
  • Additional Contributors
    • Frank Salvatore
    • Christoph Neuboeck
    • Nick Kramer
    • Stephanie Sposato
    • Carol Woody

Digital Engineering Measurement Guidebook

Key Contributors

  • Cheryl Jones [1-R]
  • Marilee Wheaton
  • Lennis Bearden
  • Chris Miller
  • Steve Henry
  • Randy Padillo

​

​

  • Gan Wang
  • Natasha (Sally) Shevchenko
  • Drew Miller
  • Paul Embry
  • Gordon Sawyer

​

  • Tom McDermott [1-U]
  • Geoff Draper [1-U]
  • Joe Bradley [1-U]
  • Paul Janusz (PJ) [1-R]
  • Joe Dean

​

​

​

  • William (Bill) Golaz [R]
  • Celia Tseng
  • Jeffrey Nartatez
  • Tony Powell
  • Joanne Arias

More than 33 Individuals

PRACTICAL SOFTWARE AND SYSTEMS MEASUREMENT

4

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

5 of 16

INCOSE/PSM

Digital Engineering

Measurement Framework WG

9/2020 - Present

Digital Engineering Measurement Guidebook

Brief Guidebook History

Supporting OUSD AT&L/R&E objectives

Software Measures

Continuous Iterative Development

Measurement Framework

(part of the emerging Software Acquisition Pathway

of the Adaptive Acquisition Framework)

4/2021

Digital Engineering and

Digital Engineering Strategy

Digital Engineering Metrics

PSM DE Measurement Framework Guidebook 1.0 – May 2022

PSM DE Measurement Framework Guidebook 1.1 – June 2022

May 2023 – May 2025

PSM DE Measurement Framework Guidebook 2.0 – May 2025

PSM DE Measurement Handbook v1.1

Instead

Model-Based Capability Matrix

(Published SERC report SERC-2020-SR-001)

3/2020

PRACTICAL SOFTWARE AND SYSTEMS MEASUREMENT

5

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

6 of 16

Digital Engineering Measurement Guidebook

Brief Guidebook History – DE MF WG History

PRACTICAL SOFTWARE AND SYSTEMS MEASUREMENT

6

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

7 of 16

Provide enterprises a pre, post, and during assessment, evaluation, and comparison to change their current informational storage structure to an advance digital cohesive and holistic operational environment.

​

  • Guide companies through their digital transformation activities.
  • Promote and encourage businesses, companies, agencies, bureaus, … to transition from the traditional document and artifact-based development environment to a digital model-based framework.
  • Assess the measurable impacts and benefits they aspire to achieve with a digital engineering environment.
  • Evaluate the success, improvements, and outcomes of their converted digital engineering framework enterprise.

Digital Engineering Measurement Guidebook

DE Measurement Framework Objective

PRACTICAL SOFTWARE AND SYSTEMS MEASUREMENT

7

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

8 of 16

Digital Engineering Measurement Guidebook

DE Measurement Framework Objective (Cont’d)

DE leverages MBSE ability to associate relationships among system functions, requirements, suppliers, acquirers, and users by:

  • Capturing the relationships, interdependencies, and processes connecting
    • Systems level models
    • Other disciplinary models
  • Lifecycle process flow
  • Business rules

Model-Based Systems

and Software Engineering

(MBSE)

Digital Engineering (DE) Framework

CID

PRACTICAL SOFTWARE AND SYSTEMS MEASUREMENT

8

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

9 of 16

Digital Engineering Measurement Guidebook

Version 2.0 Changes

  • Adaptability and Rework
  • Product Automation
  • Deployment Lead Time
  • Runtime Performance
  • Version 1.1 includes – Eight (8) essential baseline indicators
    • Architecture Completeness and Volatility
    • Model Traceability
    • Product Size
    • Digital Engineering Anomalies

​

  • Version 2.0 includes
    • Enhancing four (4) existing measures (shown above in red text)
    • Adding nine (9) new measures (shown below in green text are workshop indicators)
  • Model Progress
  • Model Coverage
  • Model Completeness
  • Reused Models/Elements
  • Digital Engineering Implementation Costs
  • Digital Engineering Product Value
  • Near-term ROI
  • Workforce Capability
  • Completeness of Safety/Security Requirements

PRACTICAL SOFTWARE AND SYSTEMS MEASUREMENT

9

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

10 of 16

  • Significate underlying guidebook premise to mature is “Model”
    • Version 1.1 has one indicator referencing Model – Model Traceability
    • Terms and Definitions – Ten model terms
      • Model
      • Digital System Model
      • Discipline Specific Model
      • Model-Based
      • Model-Based Development
    • Model definition is
      • A mathematical or physical representation (i.e., simulation) of system relationships for a process, device, or concept. (IEEE Standards Dictionary, IEEE Std 1641)
      • Representation of a real-world process, device, or concept. (IEEE Standards Dictionary, IEEE Std 2413-2019)

Digital Engineering Measurement Guidebook

Version 2.0 Changes (Cont’d)

  • Model-Based Engineering
  • Model Configuration Item
  • Model Element
  • Model Library
  • System Model

PRACTICAL SOFTWARE AND SYSTEMS MEASUREMENT

10

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

11 of 16

  • Significate underlying guidebook presume to mature is “Model” (Cond’t)
    • Need to incorporate the different Model domains and their associate application
      • Discipline-Specific Models
        • MBSE Models
        • Software Models
        • Mechanical Models
        • Electrical/Electronic Models
        • Thermal Models
      • Cross-Cutting Models
        • Reliability Models
        • Performance Models
        • Cybersecurity Models
      • Integrated Models
        • Digital Twin
        • System of Systems (SoS) Models

​

Digital Engineering Measurement Guidebook

Version 2.0 Changes (Cont’d)

Non-DE MF WG

Verified Information

  • Optical Models
  • Acoustic Models
  • Environmental Impact Models
  • Logistics and Sustainment Models
  • Fluid Dynamics Models
  • Structural Models
  • Materials Models
  • Chemical Models
  • Human Factors Models
  • Safety Models
  • Cost Models
  • Schedule Models
  • Test Models
  • Lifecycle Models
  • Mission Model

Examples

PRACTICAL SOFTWARE AND SYSTEMS MEASUREMENT

11

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

12 of 16

Digital Engineering Measurement Guidebook

Version 2.0 Changes (Cont’d)

  • Version 2.0 Additional Indicator – Model Progress

The Model Progress indicator is used to track and measure the effectiveness and profitability of the one of more digital transformation engineering activities. Each digital transformation engineering activity is tracked and measured against the established project timeline. The results of completed digital transformation engineering activities can be used to plan future digital engineering projects and provide insight how to make upcoming digital engineering projects more efficient by being more affordable, and completed on time or sooner with fewer delays, rework, and corrections.

​

The Model Progress Index is defined as

MPI = (MPSA + MPFA) / (MPSP + MPFP)

Where

      • MPI is the Model Coverage Index.
      • MPSA is the total number of actual Model Progress Starts that have begun to date.
      • MPFC is the total number of actual Model Progress Finishes that have completed to date.
      • MPSP is the total number of plan Model Progress Starts that should have begun to date.
      • MPFP is the total number of plan Model Progress Finishes that should have completed to date.

PRACTICAL SOFTWARE AND SYSTEMS MEASUREMENT

12

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

13 of 16

Digital Engineering Measurement Guidebook

Version 2.0 Changes (Cont’d)

  • Version 2.0 Additional Indicator – Model Coverage

The Model Coverage indicator measures the digital engineering transformation completion of the model element requirements of the Digital Engineering Environment. The indicator provides the progress of the model coverage and provides a performance rating. The model coverage progress is for the current point in time and the performance rating is the expected model coverage index on the final delivery date based upon current model coverage productivity.

​

The Model Coverage Index is defined as

MCI = (MERTCA + MERTCC) / (2 * MERTotal)

Where

      • MCI is the Model Coverage Index.
      • MERTCA is the total number of Model Element Requirement with at least one Test Case Assigned.
      • MERTCC is the total number of Model Element Requirement with all Test Cases Completed.
      • MERTotal is the total number of Model Element Requirements.

PRACTICAL SOFTWARE AND SYSTEMS MEASUREMENT

13

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

14 of 16

Digital Engineering Measurement Guidebook

Version 2.0 Changes (Cont’d)

  • Version 2.0 Additional Indicator – Digital Engineering Product Value

The Digital Engineering (DE) Product Value provides an indication of the completeness of the digital transformation activities against the specific Enterprise or Program digital engineering transformation requirements. The DE Product Value measure provides visibility into the digital engineering transformation activities to facilitate early corrective action(s) for specific areas having projected outcome variances from performance values goals. Each Digital Engineering Product Value measure includes measures that are evaluated and reported in accordance with the expected requirement’s performance, as well as any risk and opportunity thresholds.

​

The Digital Engineering Product Value identify areas of risk from achieving the digital transformation set goals and requirements within the constraints of the system hardware architecture. These risk areas often become digital engineering risk mitigation items. Risk mitigation items ….

allow time to correct deficiencies through redesign, procurements, or other mitigation techniques.

​

The reporting that will be used to explain the status of the Digital Engineering Product Value is presented in the figure below. All Digital Engineering Product Values analysis must be updated at the start and conclusion of a key milestone. Additional reporting may be driven by risk burn down plans to ensure the set goals are satisfied. Upon approaching a Digital Engineering Product Value key milestone conclusion, the approved results of the most current analysis date will be reported.

PRACTICAL SOFTWARE AND SYSTEMS MEASUREMENT

14

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

15 of 16

Digital Engineering Measurement Guidebook

Questions and Answers

Ask Me Anything!

PRACTICAL SOFTWARE AND SYSTEMS MEASUREMENT

15

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

16 of 16