1 of 36

Overview of the GAO Agile Assessment Guide

Jennifer Leotta, CCEA

November 2024

Slide 1

2 of 36

Outline

GAO Background
Best Practice Guides
Agile Assessment Guide Overview
Key Concepts Related to Program Controls
Case Study: Recent Audit Findings for Cost and Schedule

Slide 2

3 of 36

GAO Background

Slide 3

4 of 36

Overview: About GAO

GAO is an independent, nonpartisan agency serving the Congress to help improve the performance and ensure the accountability of the federal government

Core values are Accountability, Integrity, and Reliability

To ensure independence, the Comptroller General (CG) is appointed to a 15-year term by the President. Other than the CG, there are no political appointees at GAO

Oversight, Insight, Foresight

Slide 4

5 of 36

About GAO: Our Work

GAO work is primarily done at the request of congressional committees or subcommittees or is mandated by public laws or committee reports. We also undertake research under the authority of the Comptroller General.
Some examples of our work include:

Auditing agency operations to determine whether federal funds are being spent efficiently and effectively
Investigating allegations of illegal and improper activities
Reporting on how well government programs and policies are meeting their objectives
Performing policy analyses and outlining options for congressional consideration
Issuing legal decisions and opinions, such as bid protest rulings and reports on agency rules

In 2018, Congress directed the formation of the Science, Technology Assessment, and Analytics (STAA) team, recognizing that the accelerating pace of innovation has created a need for more and deeper analysis of science and technology

More information about GAO can be found at www.gao.gov

Slide 5

6 of 36

Best Practice Guides

Slide 6

7 of 36

Overview: Best Practice Guides

Legislators, government officials, and the public want to know whether government programs are achieving their goals and what these programs are expected to cost and when they are expected to be finished.

Best practice guides provide clear criteria to establish the quality of program artifacts and whether they provide managers and oversight organizations enough information to make informed decisions.�

Developing reliable program cost and schedule estimates is critical to

Effectively using public funds.
Meeting OMB’s capital programming process.
Avoiding cost overruns, missed deadlines, and performance shortfalls.

The Guides help provide a framework for managing the government’s acquisition efforts and ensuring the successful development and integration of cutting-edge technologies and their integration into large and complex systems.

Why develop best practice guides?

Slide 7

8 of 36

Overview: Best Practice Guides

Cost Estimating and Assessment Guide

Issued: March 2020, GAO-20-195G

Schedule Assessment Guide

Issued: December 2015, GAO-16-89G

Technology Readiness Assessment Guide

Issued: January 2020, GAO-20-48G

Agile Assessment Guide

Issued: November 2023, GAO-24-105506

Slide 8

9 of 36

The Need for An Agile Assessment Guide

Organizations are challenged to implement Agile as a new methodology. Among others, challenges include:

Lack of clarity about what Agile is. Participants assume everyone has the same understanding, but there is still confusion about what Agile means.
Lack of cross-departmental buy-In. Without buy-in across departmental lines, the team will be constantly asked for information that not aligned with Agile processes and methods.
Fitting the right approaches to unique situations. There is no single way to embrace Agile adoption; it is a set of principles that lead to faster outcomes, but only if those principles are built for your organizational culture.

GAO Agile Assessment Guide is intended to address these and other challenges.

Slide 9

10 of 36

Agile Assessment Guide Overview

Slide 10

11 of 36

GAO Agile Assessment Guide: What is Agile?

Agile software development is an approach to developing and delivering software that allows stakeholders to validate requirements, processes, and system functionality in increments, and deliver functionality to users in shorter cycles

Software development in the government has traditionally followed a “waterfall” approach. This approach typically involves sequential phases, often with delivery of working software years after development begins

Slide 11

12 of 36

Series of Best Practices

Adoption

Execution

Oversight

13 of 36

GAO Agile Assessment Guide: Chapters

Chapter 1: Background
Chapter 2: Agile Adoption Challenges in the Federal Government and Actions Taken in Response
Chapter 3: Agile Adoption Best Practices

Team activities, Program processes, and Agency environment

Chapter 4: Overview of Agile Execution and Controls
Chapter 5: Requirements Development and Management in Agile
Chapter 6: Agile and the Federal Acquisition Process
Chapter 7: Agile Program Monitoring and Control

Work Breakdown Structure (WBS), Cost estimating, Scheduling, and Earned Value Management (EVM)

Chapter 8: Agile Metrics

Slide 13

14 of 36

GAO Agile Assessment Guide: Appendixes

Appendix I: Objective, Scope, and Methodology
Appendix II: Key Terms
Appendix III: Related Terms
Appendix IV: Auditor’s Key Questions and Effects
Appendix V: Common Agile Frameworks
Appendix VI: Debunking Agile Myths
Appendix VII: Background for Case Studies and Agile in Action
Appendix VIII: Specialists Who Helped Develop this Guide
Appendix IX: GAO Contacts and Staff Acknowledgements

Slide 14

15 of 36

GAO Agile Assessment Guide: Key Terms

Road map: A high-level plan that outlines a set of releases and the associated features. The road map is intended to be continuously revised as the plan evolves.

Release: A planning segment of requirements that deploys needed capabilities. The release is a time boxed event that consists of a set number of iterations that are determined by the program. The release plan is where different sets of usable functionality or products are scheduled to be delivered to the customer.

Backlog: The backlog is a list of features, user stories, and tasks to be addressed by the team, program or portfolio and is ordered from the highest priority to the lowest priority. A backlog can occur at varying levels; for example, a product backlog is a high-level backlog that contains all the requirements for the entire program.

For more terms and definitions see the Agile Guide’s Appendix II: Key Terms

It is important to define the terms with the program early because terms can vary

Slide 15

16 of 36

Key Concepts Related to Program Controls

Slide 16

17 of 36

Debunking Agile Myths (Appendix 6)

Myth: Agile does not require planning

As with any approach, planning is a vital aspect that will greatly diminish the effectiveness of a successful implementation if not done appropriately.
Agile spreads planning activities (e.g. what specific functionality will be delivered when) more evenly throughout the program life cycle.
High-level planning is completed at the beginning of an Agile program and is continuously elaborated on throughout the program as new information becomes available.

Slide 17

18 of 36

Agile Hierarchy (Chapter 4)

Start with a high-level program goal and priority requirements and end development when the program goal has been met.

19 of 36

Recasting the WBS for Agile Development

Cost elements align with WBS and reflect release schedule

WBS supports cost estimating

WBS elaboration continues during development

20 of 36

Matching Development to the Roadmap

Releases are not an Agile schedule artifact to track progress, but they are an aspect of management and an essential adjunct to planning.

21 of 36

Relationship: Roadmap and WBS

In Chapter 7, we highlight how Agile practices and processes can be used to meet the cost estimating and earned value management best practices in the Cost Guide (GAO-20-195G) and the schedule best practices in the Schedule Guide (GAO-16-89G).

We start the chapter with a discussion of using a Work Breakdown Structure (WBS) to unite the program’s monitoring and control tools.

22 of 36

Agile and Program Control Best Practices

THE WORK BREAKDOWN STRUCTURE (WBS) IS A USEFUL TOOL TO CONNECT PROGRAM CONTROLS TO THE AGILE CADENCE

BEST PRACTICES TO DEVELOP A RELIABLE COST ESTIMATE AND SCHEDULE (AS DETAILED IN THE GAO COST AND SCHEDULE GUIDES) STILL APPLY FOR AN AGILE PROGRAM

23 of 36

Cost Estimating Considerations

COST

Relative estimating is typically used by developers, these methods can vary from team to team
Relative estimating does not provide a consistent measure to develop a cost estimate

Consistent Sizing

Cost estimators should participate in release planning sessions to understand the relationship between the backlog and developers’ relative estimating techniques so that they can further reefing the total program’s cost estimate

Integrate

A reliable cost estimate provides many benefits to an Agile program
For example, the cost estimate can be used to support the government budgeting process and help inform management decisions

Benefits

Slide 23

24 of 36

Scheduling Considerations

Planning

While Agile emphasizes detail planning for near term work, programs should define their overall goal in a vision and plan releases to satisfy that vision

Constraints

Using constraints reduces the utility of the schedule as a coordination tool among Agile teams, management, and others

Assign Resources

The schedule should include labor and non-labor resources needed to accomplish work.
The level of detail should be commensurate with the level of detail included in the schedule

Schedule Risk Analysis

A schedule risk analysis should be conducted throughout an Agile development program’s iterative process to identify the risks, paths, and activities most likely to delay the program and to serve as a basis to determine schedule risk contingencies.

Schedule Baseline

In creating a baseline using the rolling wave planning process, updates should contain enough detail to enable a collaborative agreement between product owners and developers without making schedule updates overly frequent or cumbersome.

Slide 24

25 of 36

Case Study: Recent Audit Findings for Cost & Schedule

Personnel Vetting: DOD Needs a Reliable Schedule and Cost Estimate for the National Background Investigation Services Program

(GAO-23-105670)

Slide 25

26 of 36

Background: National Background Investigation Services Program

The Department of Defense, through its Defense Counterintelligence and Security Agency (DCSA), conducts personnel vetting for the majority of the federal workforce

Since 2016, DOD has delivered some capabilities through a new information technology system- the National Background Investigation Services (NBIS) program. DOD has spent over half a billion dollars on NBIS.

DOD has deployed some NBIS capabilities; however, NBIS was originally slated to be fully operational in 2019. At the time of this report, the program projects that legacy systems will be decommissioned by the end of 2024.

In 2021, GAO recommended that DCSA develop a reliable schedule. In this report, we found that they failed to develop a reliable schedule and that their 2022 cost estimate is not reliable.

Slide 26

27 of 36

Findings: Cost

The cost estimate was based on prior budgets rather than programmatic requirements and historic program data.

The estimate did not document all ground rules and assumptions and the WBS was based on budget categories rather than NBIS deliverables.

While some cost drivers were examined for sensitivity, the estimate did not include a risk and uncertainty analysis, did not employ cross-checks, and was not compared to an independent cost estimate.

There was no technical baseline and it was unclear from the documentation whether the estimate was approved/reviewed by management

Characteristic	Score/Summary
Accurate	Minimally Met
Comprehensive	Minimally Met
Credible	Not Met
Well-Documented	Minimally Met

Slide 27

28 of 36

Findings: Schedule

We examined nine schedules as part of the assessment.

There was no WBS to trace the work in the schedule to key documents, not all schedules properly used calendars, and there were no resources loaded in the schedule.
Only three of the nine schedules had a baseline date. Of those three, only two had a status date and only one had a status date that occurred after the baseline date.
The schedules had incomplete logic and schedule data was not consistent with briefings. Further, there was no schedule risk analysis.
All nine schedules had sequencing and logic problems. None of the schedules had a valid critical path or reasonable total float

Characteristic	Score/Summary
Comprehensive	Minimally Met
Controlled	Minimally Met
Credible	Minimally Met
Well-Constructed	Minimally Met

Slide 28

29 of 36

Conclusion

The findings in this report resulted in a Matter for Congress:

Congress should consider requiring the Secretary of Defense to direct the NBIS Program Management Office to develop a reliable program schedule and cost estimate for NBIS as defined in GAO’s Schedule Assessment Guide, Cost Estimating Assessment Guide, and Agile Assessment Guide.

Slide 29

30 of 36

Thank you

Guides Available Online and Downloadable in PDF:

GAO Cost Estimating and Assessment Guide:

https://www.gao.gov/products/gao-20-195g

GAO Schedule Assessment Guide:

https://www.gao.gov/products/gao-16-89g

GAO Technical Readiness Assessment Guide:

https://www.gao.gov/products/gao-20-48g

GAO Agile Assessment Guide:

https://www.gao.gov/products/gao-24-105506

Slide 30

31 of 36

�GAO on the Web

Web site: http://www.gao.gov/

Facebook: https://www.facebook.com/usgao

Twitter: https://twitter.com/usgao

YouTube: https://youtube.com/user/usgao

Podcasts: http://www.gao.gov/podcast/watchdog.html

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.

Slide 31