Information Categories

Measurable Concepts

Project Information Needs

Enterprise Information Needs

Prospective Measures

Resources and Cost

Return on Investment

> What is the DE ROI for the project? Does DE improve/reduce effort for a project?�> What up front investments in DE are required to execute the DE activities in the project plan? (Investments may include, e.g., technology, workforce, training processes, policies, organizational interfaces, perhaps even changes to contractual documents/agreements in some cases.)�> How much up front investment in digital engineering is required during the design phase?�> What value (return) do we expect to receive, and are we receiving, for our investment(s) in DE? �> Does DE enable faster, better objective decisions? �> Does DE enable earlier, better portfolio and mission decisions?�> Do decisions at the enterprise level flow to the projects in a timely manner (do the projects have the right level of information)?

> Can the enterprise make more robust and faster decisions based on more comprehensive knowledge and analysis using digital engineering?�> What are our DE investment costs and how do they compare with costs for non-DE/model-based approaches?�> What investments in DE must the enterprise make to support enterprise DE objectives? To support externally imposed DE objectives (mandates, contractual requirements)? To provide enterprise support to projects adopting DE? (Investments may include, e.g., technology, workforce, training processes, policies, organizational interfaces, perhaps even changes to contractual documents/agreements in some cases.)�> Does DE reduce systems engineering (SysE) effort-related costs for a project? > What are our DE investment costs and how do they compare with costs for non-DE/model-based approaches?�> What new value is attached to the enterprise that can be attributed to DE in areas such as quality and knowledge transfer?

> Near-term ROI (tactical efficiencies)�> Long-term ROI (reduced likelihood of program delays, failure, or cancellation; earlier anomaly detection preventing costly downstream impacts; reduced technical debt)�> Data transfers, data usage, graph theory�> Richness of transfers, centrality of data

Information Categories

Measurable Concepts

Project Information Needs

Enterprise Information Needs

Prospective Measures

Resources and Cost

Robust Estimates of DE Costs - DE Implementation Costs

(non-recurring and recurring)��

> What are the non-recurring and recurring DE implementation costs for the project?

�> When do we need to create a new estimate?

�> Are our actual costs consistent with estimates for specified resources and activities, e.g., hardware, software, IT services, engineering labor, and support functions?

�> Are we spending the expected amount of effort (labor costs) on DE activities (e.g., effort performing modeling vs analysis)?

> What are the comparative costs of implementing a centralized DE capability (with project-specific variations) vs. federated DE capabilities, vs. decentralized/ independent DE environments for different projects?

�> Who pays and who benefits, when and how? (These measures would also be used in ROI calculations used to evaluate benefits of the approaches against the costs.)

> DE Implementation Costs (preparing for DE): Estimated vs. actual costs for different approaches (centralized, distributed, obtained as a service, combination, …).

Information Categories

Measurable Concepts

Project Information Needs

Enterprise Information Needs

Prospective Measures

Resources and Cost

Training Resources and Cost�

> What training is available? ��> What is the training cost (effort for trainees, training materials/equipment/labor)?��> Do we have an adequate budget for training?

> What are the training costs for the enterprise?��> Do we have the training resources needed to increase capability and capacity in the areas needed?��> Are people properly trained with the tools? Are the handoffs effective?

> Current, Required, Planned/Actual workforce development resources needed to meet anticipated DE needs��> Cost of Needed DE Training vs. Budget Allocated��> Planned vs. Actual Expenditures on DE Training

Information Categories

Measurable Concepts

Project Information Needs

Enterprise Information Needs

Prospective Measures

Resources and Cost

Workforce Capability and Capacity

Do we have the workforce (quantity and expertise) needed to perform DE activities on schedule, within budget, and at the needed quality?

> What is our current, anticipated, and planned workforce capability and capacity needed to achieve Enterprise-level DE goals and objectives? ��> Are we accounting for the necessary people and tools resources?�

> Workforce Capability (Number of Personnel and Costs (Current, Required, and Planned/Actual over time))��> Tool Resources

Measure Introduction

Description

[Provide an introduction to the measure specification.]

Relevant Terminology

[Provide any diagram or terminology that is relevant to this measure. Reference figures 1-4 in white paper as pertinent.]

Information Need and Measure Description

Information Need

[InfoNeed]

Base Measure 1

[BaseMeasure] [(scale: Nominal, Ordinal, Interval, Ratio)][Unit(s) of Measure]

Base Measure 2

[BaseMeasure] [(scale: Nominal, Ordinal, Interval, Ratio)] [Unit(s) of Measure]

Base Measure …

Derived Measure 1

[Derived Measure and how it is calculated from the Base Measure(s)]

Indicator Specification

Indicator Description and Sample

[Description of the indicator graph or chart.]��[Include a sketch of the indicator. Ensure it has units of measurement and axis labels. Program names can be masked.]��

Analysis Model

[Describe how this indicator is analyzed.]

Decision Criteria

[Describe the decision criteria and the actions that will be taken when breached. Provide a specific example.]

Indicator Interpretation

[Describe the interpretation of the indicator provided in the description box above.]

Additional Information

Additional Analysis Guidance

[Provide any variations of this measure, or any additional guidance or advanced analysis that may be considered. List any variations of this measure at the team, product, or enterprise levels.]

Implementation Considerations

[List any process or implementation requirements that are necessary for successful implementation.]

Measure Introduction

Description

Digital Engineering (DE) leverages advanced digital tools to model and simulate system development and sustainment. There are a variety of costs with DE implementation:

1.     Software Licensing/Subscriptions such as CAD tools, simulation tools, data analytic platforms, and collaboration tools.

2.     Hardware/Infrastructure such as workstations, servers, networking equipment, and cloud services.

3.     Customization/Integration for tailoring software and system integration to existing enterprise systems.

4.     Data management and Security for data storage systems, implementation of cybersecurity solutions, and compliance to relevant data protection regulations and industry standards.

5.     Maintenance and Support for software (updates, patches, tech support) and hardware (upgrades), and IT support to ensure to handle issues that arise. Either the vendor makes the updates (must be available 24/7) or the customer is trained (more flexible) by the vendor to make the updates.

6.     Process Reengineering for development and sustainment processes to align with new DE practices as well as transitioning employees to new roles and workflows.

Notes:

∙      Not all of these costs may apply and are included here as a list to choose from.

∙      This measure may be challenging to collect data because implementation costs may be paid by the company, organization, business unit and not a single project. The data you collect depends on whether it is a project information need or an enterprise information need.

∙      A comprehensive budget should consider both the initial outlay (non-recurring costs) and ongoing expenses (recurring costs).

∙      These costs are similar to normal costs associated with owning and operating applications on an owned computing platform. The costs are included in the specification to act as a reminder.

Relevant Terminology

Information Need and Measure Description

Information Need

∙       What are the DE implementation costs  (both non-recurring and recurring) for the project?

∙       Are our actual costs consistent with estimates for specified resources and activities, e.g., Software, Hardware, Customization, Data management/cybersecurity, maintenance and support, process re-engineering, and engineering labor?

∙       How much effort (labor costs) is spent on DE activities (e.g., effort performing modeling vs analysis)?

∙       When do we need to create a new estimate, i.e., what is the threshold to create a new estimate?

Base Measure 1

Level of DE Implementation (refer to the DE ROI measurement specification)

Base Measure 2

Software Model Licensing/Subscription Costs (Recurring & Non-Recurring)

Scale: Interval

Unit of Measure: Dollars

DE Implementation Level

Consider collecting the following contextual information for planning and detailed analysis:

∙       License name/version

∙       License vendor: The vendor that sold the license

∙       License manufacturer: The maker of the software (may be the same as the vendor

∙       License model: Perpetual (one-time purchase, non-recurring generally, may be recurring for upgrades), Subscription (periodically recurring payment, e.g., monthly, annually), Freemium (basic features are free, payment for advanced features addons could be one-time, regular consulting would be recurrent), and Open Source (free source code, optional paid support service or additional features, regular consulting would be recurrent)

∙       License types: Single-User, Multi-User (with limits), Site (unlimited use within organization), Floating (shared license with limits), Named User (restricted to a person’s use), Device (restricted to a device)

∙       Payment options: One-time (single payment), Monthly, Annual, Pay-as-you-go (based on usage), Tiered (different levels of access)

∙       Purchase and expiration dates

∙       License cost

Information Need and Measure Description

Base Measure 3

Hardware/Infrastructure Costs (Recurring / Non-Recurring)

Scale: Interval

Unit of Measure: Dollars

Consider collecting the following contextual information for planning and detailed analysis:

∙       Servers: Expenses related to purchasing (non-recurring) or leasing servers (recurring) for hosting digital engineering applications and data.

∙       Workstations: Costs associated with desktops or laptops used by engineers for design, simulation, and analysis tasks (non-recurring) and replenishment / tech upgrades (recurring).

∙       Networking equipment: Expenses for routers, switches, cables, and other networking hardware required for connecting workstations and servers (initially non-recurring) and replenishment / tech upgrades (recurring)..

∙       Cloud Services: Costs associated with using cloud computing services for hosting digital engineering applications, data storage, and processing: self-hosted clouds (recurring) or commercial cloud solutinos (recurring).

Information Need and Measure Description

Base Measure 4

Customization/Integration Costs (Recurring / Non-Recurring)

Scale: Interval

Unit of Measure: Dollars

∙ Upfront architecture setup and data migration (non-recurring)  

UI/UX design and development of customization and integration (non-recurring) 

∙       Workflow customization and process alignment (non-recurring) 

∙       Data model mapping and transformation (non-recurring) 

∙       Custom code development and scripting (non-recurring) 

∙       Testing and validation of tools including cybersecurity, tool chains/integration, communication between models (non-recurring) 

∙       Enterprise system integration (ERP, PLM, ALM, etc.): including integration architecture design, data mapping and synchronization, API development and management, and integration testing. (non-recurring)

∙       Legacy system migration and consolidation: costs for migrating data/processes from older systems to the new DE platform. (non-recurring)

∙       Cloud integration and hybrid deployment: expenses for integrating on-premises systems with cloud services and enabling hybrid deployment models. (non-recurring)

∙       Identity and access management integration: costs for integrating with existing Identification Access Management systems for consistent user authentication and authorization. (non-recurring)

Information Need and Measure Description

Base Measure 5

Data management and Security Costs (Recurring / Non-Recurring)

Scale: Interval

Unit of Measure: Dollars

∙       Data storage (Storage infrastructure, storage software, cloud storage services) (non-recurring for self-hosted, recurring for services)

∙       Security systems (firewalls, encryption: intrusion detection/prevention systems, antivirus software, etc.) (recurring for tech upgrades)

• Security Personnel Costs upgrade and patches to security systems (recurring)

∙       Identity and Access Management (IAM): Expenses for IAM solutions to manage user identities, access permissions, and authentication mechanisms for accessing digital engineering systems and data.(non-recurring)

∙       Regulatory compliance: Expenses related to ensuring compliance with relevant data protection regulations. (recurring)

∙       Data privacy: Costs associated with implementing measures to protect the privacy of sensitive digital engineering data and personal information stored or processed within digital engineering systems.(non-recurring)

∙       Audit and certification: Expenses for conducting audits, assessments, or certifications to validate compliance with regulatory requirements and industry standards related to data management and security. (recurring)

∙       Data backup (Backup solutions, backup storage, disaster recovery planning) (self-backups non-recurring or as a service recurring)

Information Need and Measure Description

Base Measure 6

Maintenance and Support Costs (Recurring)

Scale: Interval

Unit of Measure: Dollars

∙       Insurance: Protection against risks including equipment failure, data breaches, or natural disasters. (recurring)

∙       Software maintenance: Routine maintenance tasks (software model updates, patches, security fixes) to ensure the continued functionality and performance of the customized or integrated solution. (recurring)

​

​

∙       Testing and Quality Assurance (Test plan development, testing resources, test execution, defect management, and quality assurance.) (recurring)

∙       Technical support: Expenses for providing technical support services to users of digital engineering software and systems, including helpdesk support and troubleshooting assistance. (recurring)

∙       IT support: Costs for IT staff or third-party providers to handle issues and maintain the overall IT infrastructure supporting digital engineering operations. (recurring)

∙       Documentation production and storage (User manuals, technical documentation, training materials, and knowledge base) (recurring)

∙       Impact of either the vendor making the updates (must be available 24/7) or the customer is trained (more flexible) by the vendor to make the updates (recuring)

Information Need and Measure Description

Base Measure 7

Process Reengineering Costs: (Recurring / Non-Recurring )

Scale: Interval

Unit of Measure: Dollars

∙       Digital Engineering Process: Costs associated with using the DE tools in the development process. (self-hosted non-recurring, service is recurring)

∙       Change control process: Costs associated with establishing and managing a change control process to document, review, and approve changes to the customized or integrated software.(recurring)

∙       Change requests with respect to the DE Infrastructure: Expenses for evaluating and implementing change requests, including assessing the impact on the project scope, timeline, and budget. (recurring)

∙       Communication and training: Costs of communicating changes to stakeholders and providing training or guidance on how to adapt to the changes effectively. (move to training costs)

∙       Change Management: Expenses related to maintaining version control of the customized software or integrated system to track changes and revisions over time. (recurring)

∙       Process optimization: Costs for reengineering development and sustainment processes to align with new digital engineering practices and maximize efficiency and effectiveness. (recurring)

Information Need and Measure Description

Derived Measure 1

Total implementation costs: sum of each of the different costs broken out by recurring and non-recurring

∙       What does it cost to build it? (non-recurring)

∙       What does it cost to sustain it? (recurring)

∙       What the costs for each Level of DE Implementation

Indicator Specification

Indicator Description and Sample

 Comments

Focus on one cost group, e.g, Customization/Integration graph of costs�This is a look-back indicator. Is there an indicator that forecast costs?

Graph of recurring and non-recurring

Analysis

Multiple costs are displayed to show where the highest implementation cost occurs at a specific time period. Cost growth and cost decline can be observed.

Model

Decision Criteria

If a budget threshold is added to the indicator, a growing trend to the threshold may be used as a leading indicator that the budget will be exceeded. To find the cause of the growing investment, find the implementation costs that are expanding.

Indicator Interpretation

This indicator requires the consistent collection of cost data for each time period for each of the implementation costs.

Measure Introduction

Description

The purpose of digital engineering training is to equip professionals with the knowledge, skills, and competencies necessary to effectively leverage digital tools, methodologies, and technologies in the engineering domain. This training ensures that individuals and teams can optimize processes, enhance productivity, and drive innovation within their organizations.

​

DE Training costs encompass various types of expenses required to ensure that personnel are proficient in using digital engineering tools, methodologies, and practices. There are several categories of cost:

∙       Direct training expenses

∙       Training infrastructure

There are other costs that could be included in training costs. This varies by organization. These costs include indirect costs (e.g. employee time and training materials), continuous learning costs (e.g., conferences and seminars), memberships in professional organizations (e.g., INCOSE), mentoring programs, and on-the-job training,

Relevant Terminology

[Provide any diagram or terminology that is relevant to this measure. Reference figures 1-4 in white paper as pertinent.] TBD

Information Need and Measure Description

Information Need

Have people been trained?

What are the training costs for the enterprise?

Are people properly trained with the methods and tools? Are they used effectively?

What is the training cost for DE?

Labor Hours

Employee Training Time

Scale: Interval

Unit of Measure: Hours

Fill this in….

∙       Levels of Implementation

∙       New person

∙       Previously trained person

Information Need and Measure Description

Base Measure 1

Direct Training Expenses (Recurring & Non-Recurring)

Scale: Interval

Unit of Measure: Dollar

Consider collecting the following contextual information for planning and detailed analysis:

∙      Online Courses: Fees for enrolling in online courses offered by platforms like Coursera, Udemy, or specialized engineering training providers. This can be periodically recurring payment, e.g., monthly, annually, or one-time payment.

∙       In-Person Workshops: Costs for attending in-person workshops, which may include travel and accommodation expenses.

• Training communciation

∙       Certification Programs: Fees for certification exams and preparatory courses (e.g., certifications in CAD software, project management, data analytics).

Organization Training Sessions

∙       Vendor-Provided Training: Costs associated with training sessions provided by software vendors or consultants, often customized to the organization's specific needs.

∙       Internal Training Programs: Expenses related to developing and conducting internal training programs, including materials and instructor fees.

Assessment Tools

∙       Testing and Evaluation Software: Costs for software tools that assess the proficiency and progress of trainees.

∙       Certification Exam Fees: Fees for taking industry-standard certification exams. This may include costs associated with failed certifications or retakes.

∙       Certification Renewal Fees: Fees to maintain certifications such as renewal fees and continuing online credits. This can be periodically recurring payment, e.g., annually.

∙      Training Program Adjustments: Costs for modifying and improving organization training programs based on feedback and assessment results.

Information Need and Measure Description

Base Measure 2

Training Infrastructure (Recurring & Non-Recurring)

Scale: Interval

Unit of Measure: Dollar

Consider collecting the following contextual information for planning and detailed analysis:

Training Facilities

∙       Classroom Rentals: Costs for renting classrooms or training centers.

∙       Equipment: Expenses for purchasing or renting equipment needed for hands-on training, such as computers, projectors, and simulation devices.

Online Training Platforms

∙       Subscription Fees: Costs for subscribing to online training platforms that offer courses relevant to digital engineering.

∙       Learning Management Systems (LMS): Expenses for implementing and maintaining an organization LMS to manage and deliver training programs.

Information Categories

Measurable Concepts

Project Information Needs

Enterprise Information Needs

Prospective Measures

Resources and Cost

Workforce Capability and Capacity

Do we have the workforce (quantity and expertise) needed to perform DE activities on schedule, within budget, and at the needed quality?

> What is our current, anticipated, and planned workforce capability and capacity needed to achieve Enterprise-level DE goals and objectives? ��> Are we accounting for the necessary people and tools resources?�

> Workforce Capability (Number of Personnel and Costs (Current, Required, and Planned/Actual over time))��> Tool Resources

Prerequisite DAU ETM 1070

Computer Based Training

AFIT LS SysML Overview (201) Video Based Course (June Execution)

AFIT Applied MBSE Using SysML (301) – Live Course (July Execution)

Aerospace University MBSE Learning Modules (August/September Execution)

Digital Engineering Strategy

SYsML Package Diagrams

SYsML Package Diagrams

AU Digital Engineering Overview

Modeling to Support Digital Engineering Lifecycle Activities

Block Definition Diagram

Block Definition Diagram

AU Model Based Systems Engineering

Modeling and Sim to Support Mission Systems & DE

SysML Use Case Design

SysML Use Case Design

MBSE Tools Familiarization Video

IDing, Communicating, Preserving Data & Information

SysML Requirement Diagram

SysML Requirement Diagram

AU How to Read Sims Models

Components of the Digital Concept Lifecycle

SysML Activity Diagram & Activity Allocation

SysML Activity Diagram Part 1

AU Model Based Reviews

Continuum of Models to Support Digital Engineering Lifecycle

SysML State Machine Modeling

SysML Activity Diagram Part 2

Lessons Learned

​

SysML Internal Block Diagram

SysML State Machine Modeling

Applied MBSE for Spacecraft

Capstone Event

​

SysML Parametric Diagram

Modeling Structure/Logical Architecture

​

​

SYsML Sequence Diagram

Custom Stereotypes

​

​

​

SysML Internal Block Diagram

​

​

​

Modeling Structure/Physical Architecture

​

​

​

SysML Sequence Diagram

​

​

​

System Analysis & Requirement Satisfaction

​

​

​

MBSE Simulator Exercise

​

BLOCK 1: 3 Hours

BLOCK 2: 4 Hours

BLOCK 3: 12 hours

BLOCK 4: 6-8 Hours

Measure Introduction

Description

[Provide an introduction to the measure specification.]

Relevant Terminology

[Provide any diagram or terminology that is relevant to this measure. Reference figures 1-4 in white paper as pertinent.]

Information Need and Measure Description

Information Need

[InfoNeed]

Base Measure 1

[BaseMeasure] [(scale: Nominal, Ordinal, Interval, Ratio)][Unit(s) of Measure]

Base Measure 2

[BaseMeasure] [(scale: Nominal, Ordinal, Interval, Ratio)] [Unit(s) of Measure]

Base Measure …

Derived Measure 1

[Derived Measure and how it is calculated from the Base Measure(s)]

Measure Introduction

Description

A workforce’s capability framework defines the common processes relating to workforce development that support an organization's purpose. Most capability frameworks are written to describe the behaviors, skills and knowledge expected of employees in both their day-to-day work and specific job roles.

​

A capability framework is made up of five key components that are crucial for its effectiveness.

• Capabilities: These can be core business capabilities or role-specific skills.
• Levels of competency: Levels of expertise or proficiency for each capability is another component. These range from beginner to expert and assist in performance reviews.
• Role-specific requirements: A capability framework also includes detailed job descriptions and descriptions of the capabilities required for different roles. It outlines clear expectations for each role, which can be supported by capability development.
• Development plans: Guidelines, resources, and steps for employees to develop and enhance their capabilities are important as well. The best pathway is dependent on the type of role — for example, creative roles may benefit from collaborative learning experiences, while roles with more technical skill requirements wouldn’t.
• Assessment tools: Assessment tools are the methods and tools used to evaluate employees’ current capabilities, including performance reviews, capability self-assessments, or 360° feedback.

​

Workforce capacity planning is a process that ensures a company has sufficient personnel to deliver against its current and future workload. It involves aligning a company's human resources with its operational requirements to meet customer demands, achieve business objectives, and maintain a competitive edge. Capacity is frequently measured in available employee hours in the context of the organization’s total workload. Employee capacity means the potential ability to perform work over a certain period, which can be measured in terms of time, skills, energy, or quality.

​

Relevant Terminology

[Provide any diagram or terminology that is relevant to this measure.] TBD

Information Need and Measure Description

Information Need

​

​

​

​

Measure Description

​

​

​

​