1 of 21

Advancing AI Engineering using Lessons Learned from Real Mission Scenarios

JUNE 2023

Carrie Gardner

Tyler Brooks

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

1

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

2 of 21

Document Markings

This material is based upon work funded and supported by the Department of Defense under Contract No. FA8702-15-D-0002 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.

NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu.

Carnegie Mellon^® is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.

DM23-0567

2

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

3 of 21

Welcome

Carrie Gardner

SEI AI Division

Technical Manager – AI Mission Center

cgardner@sei.cmu.edu

Tyler Brooks

SEI AI Division

Software Developer

tdbrooks@sei.cmu.edu

3

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

So before we move into the substantive portion of this talk, let us briefly introduce ourselves and share some information about the applied AI engineering work we do.

As [ Annadi ] mentioned, my name is Carrie Gardner, and I lead the AI Mission Center portfolio as the Technical Manager. Together with my group, I partner with our Mission Sponsors to rapidly investigate and experiment with how AI can bring game-changing advantage to U.S. national security interests. My expertise blends engineering and management science to guide my perspective in creating transformative and trustworthy AI solutions. Previous to my transition into AI, I was an engineer and team lead for our sister cybersecurity division, CERT, and developed technical solutions to mitigate cyber threats.

And today I am joined by my friend and colleague, Tyler –

Tyler – < bio >

Thanks Tyler

4 of 21

The Software Engineering Institute (SEI) is a trusted advisor to the U.S. Government on AI

As a Federally Funded Research and Development Center (FFRDC), we are committed to conducting technical analyses and providing an objective perspective on urgent issues and to develop creative, cost-effective solutions.

4

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

5 of 21

Within the SEI AI Division, we translate the state of the possible for AI technology into the state of practice for AI engineering

AI Engineering Best Practices

Workforce Development

Counter / Adversarial AI

Next-Gen Hardware

Human-Machine Teaming

Prototype Development

Test & Evaluation

5

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

6 of 21

Guiding Themes of Well-Engineered AI Systems

	Scalable Accommodate the size, speed, and complexity of mission needs	Scalable management of data and models Enterprise scalability of AI development and deployment Scalable algorithms and infrastructure
	Robust and Secure Operate reliably when faced with uncertainty or threat	Robustness of AI components and systems Designing for security challenges in modern AI systems Testing, evaluating, and analyzing AI systems
	Human-Centered Designed with the goal of working with, and for, people	Understand context of use, sense changes over time Scope and facilitate human-machine teaming Methods, mechanisms, and mindsets for critical oversight

6

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

7 of 21

AI for Mission focuses on how to…

Design: Synthesize Warfighter needs, technology capabilities, and industry best practices to create reference design material

Prototype: Implement code, infrastructure, and configuration to create functional systems for demonstration

Field: Conduct test and evaluate activities and define quality acceptance criteria and requirements

Scale: Conceptualize how might a mission partner productionize AI system development, controlling for quality and efficiency (like an AI Factory)

7

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

8 of 21

Learning from Recent Missions

Three unique missions deploying AI (land, air, sea)
Three sets of unique and overlapping challenges
Dozens of lessons learned

8

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

9 of 21

Mission 1 – Land — Context

Mission Objective

Prototype & field AI-enabled threat detection capability for constrained edge warfare

Engineering Activities

Collect and label data with new targets
Fine-tune model with collected data
Test and evaluate candidate models

Challenges

Edge hardware
Brittle software
Mixed-Skilled stakeholders

Operational Process

9

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

Speaker: Carrie

To give ground-based warfighters enhanced threat intelligence for more effective threat avoidance, the objective of this mission was to prototype and field an adaptable AI-enabled surveillance capability .

As the figure on the right shows, the operational context was to have a unit deploy with a pre-trained computer vision model that had generalized threat intelligence. As an example, the pre-trained model could identify standard military ground vehicles or weapons.

Our aim, was to conduct fine-tuning of the computer vision model in the rugged edge environment so that it could learn new targets and amidst a potentially a new operational environment.

To develop this capability, we had to

Develop a data curation process and pipeline – allowing the unit to collect field data, label it, and then store it in a database designed for CV model processing
Develop a model training process and pipeline – re-training, or more specifically fine-tuning, an existing TF model with the new data we captured and labeled
Develop a model test and evaluation process – instrumenting an experiment tracking solution using tooling like MLFlow to observe and record metrics so that we could select the optimal new model

Complicating the mission, we had to navigate:

A DDIL, or otherwise known as a disrupted, disconnected, intermittent, and low-badnwidth, environment that limited us to rely upon local edge compute
Inherited, brittle software – specifically creating sticky situations with designing the right glue code between modules
Supporting a customer with mixed-skilled stakeholders – some where ML phds and very little information science training

10 of 21

Mission 1 – Land — Lessons Learned

Technical Debt vs Speed to Launch – design and code for extensibility

Match Train Context to Deploy Context – development data and models should match production context

Automate & Enforce Quality Management Practices – automate unit and integration tests, conduct a peer review coding process

10

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

Speaker: Carrie

Upon Reflection,

I had 3 primary takeaways:

Always monitor and reflect on the technical debt vs speed to launch trade-off with implementing prototypes. This connects to the brittle software challenge we faced. There were many times in implementing new functionality with the prototype we had to back-track and re-design an interface because the existing interface was not designed to be extended, requiring more additional engineering time than originally anticipated
Match Train Context to Deploy Context – I originally deployed to the field thinking we would focus on re-training a CV model embedded in a ground vehicle. Upon arrival, the need switched to re-training a CV model embedded on a drone. This context change required us to change many configurations and required-us to re-build and test many parts of the pipeline resulting in a late operational start.
Automate and Enforce Quality Management Practices – this is not a surprise, though this is also something often overlooked as critical with “prototype” development – but unit/integration tests should be automated – think CI/CD hooks or a stand alone software test module; and a code peer review process should be in place and enforced.

Bottom line, even with prototype development, maintain good engineering habits is critical to avoid unnecessary pain down the line.

Next, let’s turn to Tyler’s Air experience.

11 of 21

Mission 2 – Air — Context

Mission Objective

Design and Prototype MLOps pipeline for AI/ Autonomy T&E

Engineering Activities

Design and build CI/CD pipeline
Deploy and evaluate MLOps Services
Design and implement ML prototypes

Challenges

Secure computing environments
Low AI Literacy
Data ownership / sharing

Example MLOps Pipeline

11

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

Managing, evaluating, and deploying AI systems requires new processes and procedures that many organizations currently don’t support. Machine learning models and autonomous systems also require the establishment of new evaluation metrics and procedures. Our second mission focuses on designing and implementing these procedures within an MLOps pipeline.

This mission involved designing and building a CI/CD pipeline for software or systems that will be generated internally or received from a third party. Defining what metrics and processes are necessary to test and evaluate AI components is a new area for most organizations, so we worked to establish several initial evaluation methods. This processes involved deploying and testing open-source MLOps tools that can be leveraged to manage and evaluate machine learning models and their supporting datasets. Testing these tools requires plugging them directly into systems that are using machine learning or autonomy. In order to do this, we developed several relevant prototype systems machine learning systems, such as an object detection system that uses computer vision to detect airplanes in images or in video streams. Leveraging these custom prototypes enabled us to identify metrics and processes that support proper testing and evaluation of AI systems.

We encountered several blockers during this mission. A blocker that is inherent to working in this space is having to work within the confines of secure computing environments. There are only a handful of cloud environments that can be leveraged in the government space to handle the size and speed needed to support machine learning. Those environments also tend to be locked down based on various government mandates and security protocols. A common challenge that also appears here is a lack of literacy in how AI is built or how it can be utilized. This lack of knowledge significantly slows down the pace of designing and implementing any of these systems or pipelines. One last blocker I’ll mention involves architecting this program in a way that promotes data and model reuse and sharing. It requires more effort and planning upfront, but enabling teams to share models, datasets, and services will significantly cut back time wasted on overlapping or siloed work being done.

12 of 21

Mission 2 – Air — Lessons Learned

AI & ML Literacy – educate everyone managing, building, and using ML pipelines and systems on the fundamentals of AI Engineering

Spot & Bridge Silos Between Stakeholders – enable multi-disciplinary collaboration and communication between team members, as well as stakeholders

Design for sharing/reuse – implement systems that foster reuse of data, models, or pipelines across multiple teams and/or organizations

12

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

This mission really highlighted how AI literacy can impact every part of an AI-centric project. It is incredibly important for people at all levels, from management to software developer to end user, to understand the realistic capabilities of AI as well as the scale, feasibility, and risks of deploying and managing AI systems. This is also crucial for understand how to integrate novel MLOps processes and pipelines into existing infrastructure, and how to adopt current evaluation procedures to the novel risk areas that come with using AI. AI Literacy is especially important at the managerial level. Many decision makers are attempting to apply AI anywhere they can without understanding if using AI is practical or even possible.

Another key takeaway is, when distributing larger projects across multiple people and teams, it is easy for people at the team or even at the individual level to become siloed away from the other work being done on the project. When building pipelines to manage, test, evaluate, deploy, and monitor AI systems, it is crucial for everyone involved in the project to understand, at least at a base level, what the other project members are doing. This, combined with clear and consistent lines of communication, prevents several big problems from occurring, like a person becoming a single point of failure for a specific part of the project, and makes it easier for the disparate teams to integrate their work together when the time comes.

Lastly, and I’ve touched on this already, systems and pipelines should be designed to support the sharing and reuse of models, data, and other shareable artifacts. This often manifests as a centralized data store or model registry that all teams in an organization can access. Curating datasets and training models are time consuming tasks that often require significant manual effort. Sharing data and models across an organization can significantly decrease time dedicated to those tasks across all teams.

13 of 21

Mission 3 – Sea — Context

Mission Objective

Integrate and field Unmanned Surface Vessels (USVs) for maritime domain awareness

Engineering Activities

Integrate disparate USV systems
Visualize live object detection
Collect raw data for future use

Challenges

Edge hardware
Network latency, bandwidth
Interoperability, capability management

Functional USV Network Map

Design view

13

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

Speaker: Tyler

The last mission we’ll touch on here involved spending several days on a foreign naval base participating in an exercise whose purpose was to test autonomous and machine learning systems out on the water. A dozen or so private sector companies were brough in to integrate and deploy uncrewed surface vessels (USVs). Several of these vessels were also performing live object detection using machine learning. During this event we acted as observers and advisors, identifying what the current state of the art is for maritime domain awareness and machine learning on the edge. We also identified what challenges have been or still need to be overcome to make deploying machine learning at the edge more feasible.

Achieving a cohesive autonomous ecosystem required significant effort. To start, each vessel came from a different vendor and used its own proprietary software and data formats. To solve this, individual docker containers were created for each vessel, which would translate that vessel’s data stream into a universal data format.

As I mentioned, some of these vessels were performing live object detection using on-board sensors and computers. In order to achieve this, YOLO models were pretrained on open-source and synthetic datasets and then loaded onto Nvidia Jetson computers that plugged directly into the sensor infrastructure on the actual vessel.

And as you might imagine, this scenario is prime for capturing a wide array of sensor data. To support this, systems were built to display and store detected object data, as well as normalize, curate, and store raw data for future processing.

The major challenges during this mission are very common for edge deployments. The vessels can only support so much additional weight, heat, and power consumption, so the computers deployed for object detection were only powerful enough to support small computer vision models. Operating in a constrained environment also generally involves restricted network access. When dealing with raw image and sensor data, network bandwidth and latency very quickly become difficult to manage. The challenge unique to this mission was dealing with how to properly integrate the data streams and capabilities of each vendor’s vessels. Without extensive work, it would be impossible for these systems to form a cohesive and productive ecosystem.

14 of 21

Mission 3 – Sea — Lessons Learned

Define the right requirements – flesh out use cases with input from all stakeholders before finalizing system requirements

Match test and evaluation criteria to quality and functional needs – design requirements/systems to meet appropriate standards for operationalization

Design and implement for robust data management needs – engineer pipelines to efficiently capture, store, utilize, and share data sets/streams

14

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

There were many lessons learned from observing this mission, but one of the biggest ones centers around the importance of properly defining system requirements. This mission relied heavily on acquiring third-party autonomous systems and custom software stacks. There was a clear difference in what capabilities the mission partner expected to be delivered versus what the vendors actually delivered. We learned that this stemmed directly from unclear or vague system requirements. Being able to properly define requirements for AI systems is a huge challenge, which brings us back to how a lack of AI literacy can negatively impact mission outcomes. In the case that decision makers aren’t literate in the capabilities and feasibility of AI systems, they should exercise the option to consult end users such as actual mission operators or systems engineers.

Good requirements for any type of system should include that these systems meet some a performance and safety baseline. We learned that requests for AI systems are often vague about or entirely lacking these evaluation requirements. Testing and evaluation of machine learning models and autonomous systems seems to be an after thought. This mindset needs to change if organizations want to move systems out of the research lab and into actual mission scenarios. Being able to evaluate the performance and safety of a system is key to assessing its technology readiness level and its ability to be operationalized.

Lastly, implementing this ecosystem of interconnected sensors and object detection models provided an incredible opportunity to gather and utilize large amounts of data. This highlights the need for data management pipelines to be implemented alongside these ecosystems. This is especially true when operating on edge devices. The storage space on these devices is typically relatively limited, and raw data is usually discarded in order to prevent overfilling disk space. Data is one of, if not the most, importance resources when it comes to machine learning. Organizations should always be looking for ways to easily capture, process, store, and share it.

15 of 21

Using Lessons Learned to shape the �Discipline of AI Engineering

Engaging with Mission enables the advancement of AI Engineering
Lessons learned help improve internal processes and drives internal strategy
Mission experience improves the ability to support other customer work and helps direct research and engineering efforts

Identified areas for Improvement

Matching training context to deployment context
Performing system tests that match deployment conditions
Enabling proper data management
Writing good acquisition requirements
Implementing appropriate Training and Evaluation processes
AI & ML Literacy

15

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

Speaker: Tyler

At the Software Engineering Institute, part of our mission is to establish, advance, and promote the discipline of AI Engineering. Engaging in missions like the ones we just described is crucial to all three of those processes. Lessons learned, such as matching training context to deployment context and improving AI & ML literacy, directly contribute the body of knowledge that establishes and advances the foundation of AI Engineering. And engaging with customers at the edge allows us to directly promote good AI engineering practices to those who could be using them.

These engagements also give us a fresh look at what ”state of the art” looks like in different contexts, and helps us understand current and future challenges. Spending most of your job in a research laboratory can skew your perception of what types of solutions are useful to our mission partners, and what solutions are actually feasible given the available edge resources. Our edge missions are genuinely helping us grow as an organization to better support our customers, and they are helping us direct our research and engineering efforts toward areas that will provide genuine, practical benefit in this space.

16 of 21

Integrating Lessons Learned into our AI Engineering Curriculum

Mission Engagement enhances Workforce Development

Mission 1:

Designing for the use case, identifying potential risk (i.e. data drift), planning for constrained environments

Mission 2:

Multi-disciplinary teams, proper project planning, defining and building appropriate pipelines

Mission 3:

Data management/governance, properly defining requirements, necessity of T&E processes

Above: Flyer for Introduction to AI Engineering course

16

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

In the AI Division we are directly using mission engagements to advance the discipline of AI Engineering. Part of our work focuses on improving AI literacy through the creation of a workforce development curriculum. We are able to pull out our lessons learned and turn them into learning objectives. For instance, our first mission has shown us the importance of knowing how to plan your system design and development phases to more closely match the environment you will be deploying your system to. We have extracted out other key areas for education, such as learning how to properly plan a project and its pipelines, learning how to adequately define system requirements, and understanding how risk can manifest within these systems (and also how it can be mitigated).

We are actually integrating these key takeaways directly into our Introduction to AI Engineering educational course that we will be offering later this year. We’ve combined internal and external expertise with lessons learned from our engagements to build a course that aims to increase AI literacy across a very diverse audience. We hope to help everyone from managers, to designers, to engineers, to ethicists understand the basics of what it takes to build and maintain trustworthy AI.

17 of 21

The Value of Mission Deployments

Mission deployments are a key part of designing and implementing strategy and research in this space

Without a clear view into the actual “state of the art”, R&D strategy is directed by assumption or internal interest

Lessons learned from these deployments directly identify potential areas of work, research, or collaboration

Facetime with stakeholders enables the establishment and/or strengthening of a community of partners

17

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

Speaker: Tyler and handoff to Carrie

I’d like to quickly reiterate some of their less technical benefits.

As we’ve mentioned previously, mission deployments are essential to keeping internal development and research strategy aligned with the needs of the space. It’s too easy for organizations to base their R&D strategy off of local bias, assumptions, or personal interest. These mission engagements provide the opportunity to look over the fence to perform some strategic course correction and stay ahead of the state of the art. This enables our organization to best support our customers and the AI community as a whole. The AI community is rapidly growing throughout the public and private sectors. These missions are also providing us insights on areas where we can collaborate with other organizations in this space, which is crucial for building out this community of partners. The facetime you get with stakeholders on these deployments is invaluable in strengthening our relationships and building out that community.

To wrap up…

*Ultimately, to bridge the valley of death between prototype and full-scale production, researchers and engineers must closely align with mission partners to deeply understand the operational context. *

(pass to carrie)

18 of 21

Thanks + Contact

Carrie Gardner

SEI AI Division

Technical Manager – AI For Mission

cgardner@sei.cmu.edu

Tyler Brooks

SEI AI Division

Software Developer

tdbrooks@sei.cmu.edu

18

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

19 of 21

Mission 1 – Land — Original

Mission Objective:

Assist with the CV prototype development and fielding at the edge

Model fine-tuning
T&E for candidate model selection

Practices, Processes, Tools

?

Identified areas for Improvement

Matching training context to deployment context
Performing system tests that match deployment conditions

19

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

20 of 21

Mission 2 – Air

Mission Objective:

Assist in the planning and deployment of MLOps and DevSecOps pipelines

Practices, Processes, Tools

?

Identified areas for Improvement

Building AI and ML literacy for decision makers
Removing silos/walls between scientists, developers, and engineers

20

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

21 of 21

Mission 3 – Sea

Mission Objective:

Deploy and interconnecting commercial USVs in the maritime domain while leveraging live CV to assist in decision-making

Practices, Processes, Tools

?

Identified areas for Improvement

Enabling proper data management
Writing good acquisition requirements
Implementing appropriate Training and Evaluation processes

21

[DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.