Digital Engineering Measurement Framework – v2.0

A collaboration among industry, government and academia

Unclassified

November 15, 2023

Practical Software and Systems Measurement

1

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

Digital Engineering Measurement Framework – v2.0�Introduction and Background

• The PSM Digital Engineering Measurement Framework publication is an Information Need based methodology and/or process; measures to provide feedback on the information needs important to the project and enterprise on their transformation efforts relative to traditional engineering methods
• PSM DE Measurement Framework Version 1.1 completed June 21, 2022
• Digital Transformation (including Model Based Software and Systems Engineering [MBSSE]) was still in the early stages and the processes, tools, methods, and measures were not fully matured to achieve all the benefits of applying digital engineering methods and models across a company’s product and system life cycle
• Today, DE and MBSSE has significantly matured and the PSM DE Measurement Framework needs to be updated to reflect current state of the practice, and to address other important measures and information needs

Practical Software and Systems Measurement

2

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

Measurement Process

• The Framework follows the PSM process to define the Digital Engineering Measurement Framework
• Aligned with ISO/IEC/IEEE 15939 measurement process standard
• Initial Measures Defined
• Architecture Completeness and Volatility
• Model Traceability
• Product Size
• DE Anomalies
• Adaptability and Rework
• Product Automation
• Deployment Lead Time
• Runtime Performance

​

Practical Software and Systems Measurement

3

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

DE Measurement Framework ICM Table (Excerpt)

DE Measurement Framework Summary – Review Version 1.1

Measurement specs written for inclusion in v1.1 release

Practical Software and Systems Measurement

4

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

Example Measurement Information Model – DE Anomalies

Digital engineering measures and indicators are specified in a structured template aligned with the PSM Measurement Information Model

DE Measurement Framework Summary – Review Version 1.1

Practical Software and Systems Measurement

5

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

Example Measurement Specification (Excerpts)

Description

Measures

(Base, Derived)

Definitions

Indicator(s) and Interpretation

Analysis

Guidance

Additional Information and Guidance

DE Measurement Framework Summary – Review Version 1.1

Practical Software and Systems Measurement

6

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

Digital Engineering Measurement Framework – Example Indicators

Architecture Completeness and Volatility

Is the architecture complete to proceed with design?

Model Traceability

What is the traceability and coverage of model elements?

Product Size (Model Elements)

What is the size and scope for the DE project or product?

DE Anomalies

Are we finding and removing anomalies earlier using DE?

How can we reduce the leading causes of anomalies?

Excerpts from the DE Measurement Framework specifications. Some measurement specs have multiple sample indicators. Refer to framework Section 8 – Measurement Specifications for details.

Click titles for details

DE Measurement Framework Summary – Review Version 1.1

Practical Software and Systems Measurement

7

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

Digital Engineering Measurement Framework – Example Indicators

Excerpts from the DE Measurement Framework specifications. Some measurement specs have multiple sample indicators. Refer to framework Section 8 – Measurement Specifications for details.

Product Automation

What percentage of artifacts are automatically model-generated?

Deployment Lead Time

How long does it take to deploy an identified capability?

Runtime Performance

What is the likelihood performance will meet operational needs?

Adaptability and Rework

How much rework is for planned and unplanned changes?

DE Measurement Framework Summary – Review Version 1.1

Practical Software and Systems Measurement

8

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

Plans for Next Steps – v2.0

• Revise and/or add digital engineering information needs and measures
• Revisions can be for:
• Advanced topics (e.g. ROI, size, other topics on slide 17-18)
• Updates to digital engineering approaches across DoD
• Updates to measures to address implementation lessons learned
• Updates to the size measure that was created for v1.1 of the DE Measurement Framework

​

​

Practical Software and Systems Measurement

9

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

v2.0 Meeting Activities

Practical Software and Systems Measurement

10

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

Digital Engineering Measurement Framework Schedule

Practical Software and Systems Measurement

11

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

Working Session – Items of Interest

Review ICM – Update Content

Draft ICM DE 14 Nov 23 Excel Document

Practical Software and Systems Measurement

12

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

Back-Up Slides

Digital Engineering Measurement Framework – v2.0

Practical Software and Systems Measurement

13

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

• Workshops were held at the NDIA SE Conference (Nov 2022) and the INCOSE IW (Jan 2023) to discuss information needs and measures to address. Identified areas included:
• Measures for enterprise DE
• Return on Investment measures: Does DE speed up deployment in upgrades or reduce cost in the system over what we are doing today?
• Measure additional productivity indicators related to velocity and agility (balancing speed, product produced, and quality
• Simple measure of adoption
• Quality, reusability, and knowledge transfer
• Specify leading indicators
• Performance, maturity, and complexity of model elements
• Technical performance measures
• Estimation

​

• What are the information needs (goals, objectives) that you need to address for digital engineering?

Working Session – Items of Interest

Questions for Consideration

Practical Software and Systems Measurement

14

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

Working Session – Items of Interest

Questions for Consideration

Practical Software and Systems Measurement

15

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

Digital Engineering Size

• An important input is the size of the digital engineering artifact – we defined model element as the size measure:
• Atomic (elementary) items that represent individual components, actions, states, messages, properties, relationships, and other items that describe composition, characteristics, or behavior of a system (ISO/IEC/IEEE 24641:2000)
• The project or enterprise must determine the model elements that will be counted
• Model elements should be defined that allow 1) the relative size of the DE effort to be measured and compared to other efforts or plans, and 2) the quality of the DE design decisions (correctness and completeness) to be measured

​

Practical Software and Systems Measurement

16

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

Cost Estimation

• Information needs related to how to cost digital engineering activities and efforts
• What is the up-front investment cost of establishing a digital engineering environment
• What is the return-on-investment for transitioning to digital engineering
• Do the digital engineering artifacts (models, etc.) reduce the costs of future programs?
• Is there an up-front cost to developing models for later reuse?
• What is the overhead cost of maintaining an environment suitable for digital engineering activities?
• How does digital engineering affect software development costs?
• How does digital engineering affect the models such as COCOMO?

​

​

Practical Software and Systems Measurement

17

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified

• Measures for enterprise DE (how does it speed up deployment in upgrades in the system over what we are doing today, can we identify and fix failure modes quicker, speed, TPMs – meeting quicker, non-DE how accurate are TPM projections vs DE project, estimated vs. actual effort)
• Return on Investment measures
• Measure additional productivity indicators related to velocity and agility
• Measure additional indicators that isolate new value to the enterprise through DE, in areas such as quality and knowledge transfer
• Measure enterprise and personnel process adoption
• Analyze breadth of usability and user experience with digital tools, and measure issues with usability
• Supportability and Maintainability measures (impact assessment agility)
• Security related measures
• Identify typical digital artifacts
• Specify leading indicators

Working Session – Items of Interest

Questions for Consideration

Taken from v1.1, Section 6, Next Steps

Practical Software and Systems Measurement

18

DE Measurement Framework v2.0

Unclassified: Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Unclassified