1 of 30

GAO Cost Guide and 10 Years of Cost Estimate Assessments

Benjamin Wilder

Slide 1

2 of 30

Outline

Background (GAO)
Best Practice Guides
10 Years of Assessments
Case Study: Credible Characteristic

Slide 2

3 of 30

Background

Slide 3

4 of 30

Overview: About GAO

GAO is an independent, nonpartisan agency serving the Congress to help improve the performance and ensure the accountability of the federal government.

Core values are Accountability, Integrity, and Reliability.

To ensure independence, the Comptroller General (CG) is appointed to a 15-year term by the President. Other than the CG, there are no political appointees at GAO.

Oversight, Insight, Foresight

Slide 4

Who is GAO?

We are an independent, nonpartisan agency serving as part of the legislative branch serving the Congress to help improve performance and ensure accountability of the federal government.

At the core of our mission is our values. Accountability describes the nature of our work as oversight organization for federal programs, policies, and operations. Integrity refers to the high standards that GAO sets for ourselves as being a strictly objective, fact-based, nonpartisan organization in all of our activities. And reliability speaks to our goal for how our work is viewed and that all of our products and services are timely, accurate, useful, clear, and candid.

To speak to our independence, we have one political appointee in our agency at the top level, the Comptroller General who is appointed to a 15-year term by the President from a recommendation list submitted by a bipartisan congressional commission, who then must be senate approved.

5 of 30

Overview: About GAO (Our Work)

GAO work is primarily done at the request of congressional committees or subcommittees or is mandated by public laws or committee reports. We also undertake research under the authority of the Comptroller General.
Some examples of our work include:

Auditing agency operations to determine whether federal funds are being spent efficiently and effectively
Investigating allegations of illegal and improper activities
Reporting on how well government programs and policies are meeting their objectives
Performing policy analyses and outlining options for congressional consideration
Issuing legal decisions and opinions, such as bid protest rulings and reports on agency rules

In 2018, Congress directed the formation of the Science, Technology Assessment, and Analytics (STAA) team, recognizing that the accelerating pace of innovation has created a need for more and deeper analysis of science and technology.

More information about GAO can be found at www.gao.gov

Slide 5

So what do we do?

As mentioned, our work is primarily done in service to the Congress. Much of our directives come at the request of congressional committees or subcommittees or is mandated by laws or committee reports. Separate from the requests and mandates from congress the Comptroller General also has the authority to direct additional research.

On the slide you’ll see some examples of what our work includes. My efforts primarily fall under the auditing side, particularly in those bolded points where we analyze and report on how well government programs are meeting their objectives. This work often lead to a review of agency policy and procedures where our findings may drive options for congressional consideration or recommendations for agency implementation.

I work in our Science, Technology Assessment, and Analytics team in our Engineering Sciences division. While STAA was established in 2018 when congress recognized a need in GAO for more and deeper analysis of science and technology our group existed well before that under our Applied Research and Methods Group as the Cost Engineering Sciences Division. The name was rebranded when we were part of the formation of this new team to better reflect our full portfolio of work within project controls beyond just cost.

6 of 30

Best Practice Guides

Slide 6

7 of 30

Overview: Best Practice Guides

Legislators, government officials, and the public want to know whether government programs are achieving their goals and what these programs are expected to cost and when they are expected to be finished.

Best practice guides provide clear criteria to establish the quality of program artifacts and whether they provide managers and oversight organizations enough information to make informed decisions.�

Developing reliable program cost and schedule estimates is critical to

Effectively using public funds
Meeting OMB’s capital programming process
Avoiding cost overruns, missed deadlines, and performance shortfalls

The Guides help provide a framework for managing the government’s complex efforts and ensuring the successful development and integration of cutting-edge technologies and their integration into large and complex systems.

Why develop best practice guides?

Slide 7

8 of 30

Overview: Best Practice Guides

Cost Estimating and Assessment Guide

Issued: March 2020, GAO-20-195G

Schedule Assessment Guide

Issued: December 2015, GAO-16-89G

Technology Readiness Assessment Guide

Issued: January 2020, GAO-20-48G

Agile Assessment Guide

Issued: November 2023, GAO-24-105506

Slide 8

Here the current versions of our four published guides

Our oldest running guide is the cost guide: GAO cost guide exposure draft (GAO-07-1134) July 2007, Original published guide (GAO-09-2SP) March 2009. It not only provides criteria for cost estimating that we’ll go into in more detail in this presentation but also includes criteria for earned value management and conducting analysis of alternatives
Schedule issued December 2015
Technology Readiness Assessment in 2020
Agile Assessment Guide published last year

In addition, we have some in progress efforts that are not as of now being developed as full up guides that include a report on private sector leading practices within the cloud computing space as well as a technical appendix that we will apply to a future report on broader best practices within program and project management as a whole

9 of 30

Cost Estimating and Assessment Guide

GAO Criteria

GAO Cost Guide: https://www.gao.gov/products/GAO-20-195G

Through the evaluation of 18 best practices

By determining the extent to which the estimates are:

Comprehensive

Well-documented

Accurate

Credible

Assess program life cycle cost estimates

Cost Assessment

Slide 9

10 of 30

Cost Estimating 12 Step Process

Slide 10

The Cost Guide presents the best practices in the context of a 12-step cost estimating process. The cost estimating process provides the foundational guidance for initiating, researching, assessing, analyzing, and presenting a cost estimate. Each of the 12 steps is important for ensuring that cost estimates are developed and delivered in time to support important program decisions. The 12-step process represents a consistent methodology based on industry and government best practices that can be used across the federal government to develop, manage, and evaluate program cost estimates. By following a process of repeatable methods, agencies should be able to produce reliable estimates that can be clearly traced, replicated, and updated to better manage their programs and inform decision-makers of the risks involved.

11 of 30

Comprehensive Characteristic

Slide 11

12 of 30

Well Documented Characteristic

Slide 12

13 of 30

Accurate Characteristic

Slide 13

14 of 30

Credible Characteristic

Slide 14

15 of 30

10 Years of Assessments

Slide 15

16 of 30

10 Years of Assessments

We performed cost estimate assessments for 117 programs across the federal government contributing to the publication of 75 GAO reports.
Program selection is primarily done at the request of Congress.
Often programs are selected due to risk level or existing performance issues.
Majority of reports lead to recommendations for areas of improvement.
When recommendations are addressed by the agency a follow up assessment is performed and a determination is made whether to close the recommendation.

Slide 16

From 2014 through 2023 our group has performed cost estimate assessments for 117 programs across the federal government contributing to the publication of 75 GAO reports. This includes 88 full reliability assessments which look at all four characteristics of a reliable cost estimate and 29 abridged analyses which is more targeted and looks at three or fewer of the characteristics.

A key thing to keep in mind, when we get the data, is the context of the programs that we review. Selection of programs is primarily done through those request letters, mandates, and laws directed by Congress. As a result, the programs that are selected for review often already tend to be of a higher risk level or have known performance issues that warrant a deep dive, which would trend towards programs that hypothetically would score lower or have additional findings.

The majority of the reports that our group has input to leads to recommendations for areas of improvement, as you’ll be able to correlate to the data later 95% of reports that we contribute to result in one ore more recommendations. This data only looks at the scores and findings at the time of the published report, however, when we issue recommendations and those are addressed by the agency a follow up assessment is performed to determine whether to close the recommendation or not and that is reported on our website where each report can be found.

The graphic on the right shows a time phase of the 117 programs our group has reviewed. You’ll see a lot of variation over the years which can be attributed to a few things. 1) GAO reports are long term projects often taking a year or more from initiation to publication so you get situations where things stack up on each other. Another thing is this only represents our reports on our cost criteria and not our full portfolio of work and there is some variation year over year of the amount of each guide area our team supports.

Full Analysis – 88, 62

Abridged Analysis – 29, 14

88 Full Analysis

6 Reliable (7%)

82 Not Reliable (93%)

15 Credible Near Misses (17%)

1 Comprehensive Near Misses (1%)

2 Well Documented Near Misses (2%)

29 Abridged Analysis

17 of 30

Who Have We Assessed

23 Agencies

10 Air Force

6 Army

6 Navy

6 Marine Corps

6 Other DoD Entities

10 National Nuclear Security Administration

5 Office of Environmental Management

6 Other DOE Entities

U.S. Department of Homeland Security

5 Customs and Border Protection

4 Transportation Security Administration

4 U.S. Coast Guard

7 Other DHS Entities

Other Government Agencies

5 Housing and Urban Development

5 Veterans Affairs

3 Commerce

3 Agriculture

3 State

3 Administrative Office of the U.S. Courts

2 NASA�2 Labor

2 Treasury

2 General Services Administration�2 Architect of the Capitol

2 Library of Congress

8 Other Government Entities

Cost Estimate Assessments 2014-2023

Slide 17

18 of 30

How Are Assessments Scored

Each best practice has associated criteria.
We assess evidence against these criteria to determine how well they were met.

Criteria can be found at the end of each Step chapter in the Cost Guide.

We rate each best practice based on the fulfillment of its criteria.
We average the rating of each best practice to determine the score of the overall characteristic.
A cost estimate is considered reliable if the overall assessment ratings for each of the four characteristics are substantially or fully met.
If any of the characteristics are not met, minimally met, or partially met, then the cost estimate does not fully reflect the characteristics of a high-quality estimate and cannot be considered reliable.

Slide 18

19 of 30

Best Practice Scoring

The organization provided no evidence that satisfies any of the criterion.

Not Met = 1

The organization provided evidence that satisfies a small portion of the criterion.

Minimally Met = 2

The organization provided evidence that satisfies about half of the criterion.

Partially Met = 3

The organization provided evidence that satisfies a large portion of the entire criterion.

Substantially Met = 4

The organization provided complete evidence that satisfies the entire criterion.

Fully Met = 5

Best Practice	Individual Assessment	Score
The cost estimate includes all life cycle costs	Substantially Met	4
The cost estimate is based on a technical baseline description that completely defines the program, reflects the current schedule, and is technically reasonable.	Partially Met	3
The cost estimate is based on a WBS that is product-oriented, traceable to the statement of work, and at an appropriate level of detail to ensure that cost elements are neither omitted nor double-counted.	Met	5
The cost estimate documents all cost-influencing ground rules and assumptions.	Minimally Met	2
Average Score (4 + 3 + 5 + 2) / 4 = 14 / 4		3.5
Comprehensive Overall Assessment	Substantially Met

Slide 19

20 of 30

Agency Cost Estimate Ratings Varied Over Past 10 Years

Scores are reflections of the program estimates assessed, not the agency itself.

Scores are presented as of the assessment published in the GAO report.

Recommendation follow-ups occur after the published report that may lead to a higher score and closure of the recommendation.

Program cost estimates most consistently scored lowest in the best practices associated with the Credible characteristic.

Slide 20

Here is a bit of an eye chart but this is the summary data of agency cost estimate ratings over the past 10 years.

I want to reaffirm that these scores are strictly reflections of the pool of programs assessed by GAO and is not fully reflective of the agency itself. The scores are presented as of the assessment published in the original GAO report so it doesn’t capture any improvement made after the report through those recommendation follow-up efforts.

Some points of interest when looking at this data

We can see that within our scope of programs that DHS has consistently scored the highest while the agencies outside of our big 3 sample consistently scored the lowest. This is of course is even more difficult to make any agency wide determinations on the lower count agencies since in many cases an agency has only had one of our reports done on one or two programs.
You’ll also notice the difference in the grand total at the characteristic level falling lower in each area than the full 117 program set. This again is a result that not all of our assessments are full reliability assessments looking at all four characteristics and the set does include those abridged analyses that looks at three or fewer. Within this data we almost always have looked at the comprehensive characteristic. This is a big one for no other reason than the first best practice being that the lifecycle cost estimate includes all costs. When developing our reports and our findings it is generally very important to report on if the estimate fully covers the program scope and lifecycle.
At the characteristic comparison level you can see agencies most consistently struggle with the Credible characteristic. As a reminder that characteristic includes the best practices related to risk and uncertainty analysis, sensitivity analysis, performing crosschecks, and reconciling with an independent cost estimate. We have a case study in a bit deepdiving this characteristic a little bit more.
We chose not present any time phased data here for a few reasons. First the data didn’t have any discernable trends and again given the nature of program selection it would be difficult to try to justify an explanation for any of the trends that may have presented itself.

21 of 30

Reliability Assessment - Data in Detail

A cost estimate is considered reliable if the overall assessment ratings for each of the four characteristics are substantially or fully met.

88 Full Reliability Assessments

6 Reliable (7%)
82 Not Reliable (93%)

20% of “Not Reliable” assessments are Near Misses – Only one of the four characteristics was scored lower than Substantially Met

15 Credible Near Misses (17%)
2 Well Documented Near Misses (2%)
1 Comprehensive Near Miss (1%)

Slide 21

Here is a further breakout of the data. Across our 117 program reviews we performed 400 individual characteristic assessments. Of those:

137 score Substantially or Fully Met (34%)
263 scored Partially Met or Below (66%)

As mentioned previously over this 10 year span we performed 88 full reliability assessments looking at all four characteristics. It’s not the rosiest picture but only 6 of those 88 or 7% met the bar of fully reliable. Remember that to achieve that each of the four characteristics must hit at least the substantially me mark.

One silver lining within the data is what I calling ‘near misses’. That is only one of the four characteristics fell short of that substantially met mark resulting in the ‘not reliable’ determination. There were an additional 18 programs that are those ‘near misses’ and as you can see the vast majority of those fell short because of the credible characteristic.

22 of 30

Reliability Assessment - Data in Detail (SW and IT)

Slide 22

23 of 30

Case Study:�Credible Characteristic

Slide 23

24 of 30

Revisiting the Credible Characteristic

Slide 24

25 of 30

Credible Characteristic Case Study: GAO-23-106067

As of June 2023, cost estimates for several Shipyard Improvement Optimization Program (SIOP) efforts had increased significantly from their original baseline estimate. One such project, the Portsmouth Naval Shipyard drydock grew 4x in 2.5 years. GAO assessed the extent to which the Portsmouth Naval Shipyard dry dock cost estimate met GAO’s best practices.

Slide 25

26 of 30

Credible Characteristic Case Study: GAO-23-106067

GAO Found:

Cost risk and uncertainty analysis and sensitivity analysis was performed on the January 2019 project estimate, representing a maturity level of 15 percent, but was not updated to reflect the final design estimate that informed the project budget request.
The project estimate did not contain any documented cross-checks
The Navy did not conduct an Independent Cost Estimate (ICE)

Substantially Met

Fully Met

Minimally Met

GAO recommended the Navy update their cost risk and uncertainty and sensitivity analyses of key SIOP projects throughout the design process as well as ensure that high-value cost elements are cross-checked using different estimating methodologies

Slide 26

SIOP is an example of one of those near miss programs that only scored less than substantially met on the credible characteristic. The estimate construction, documentation, and methodologies were all largely positive. However, they fell short in conducting that next level analysis that adds credibility to the estimate and thus was considered not fully reliable.

In particular, we found that the Navy had performed risk and uncertainty analysis at an early design milestone representing 15% maturity but had not updated it for their near full design package that informed the initial budget submissions. In that initial risk analysis they had identified several risks but failed to quantify them. Of the risks they identified several were cited as leading drivers of the budget increase requests across the following two years. Additionally, the program failed to perform and document crosschecks or perform an Independent Cost Estimate. We made two recommendations related to the cost estimate on updating the risk analyses throughout the design and estimating stages as well as documenting cross-checks.

We did not make a recommendation for an ICE because at that point conversations between the Navy and OSD CAPE on developing ICE’s for these efforts were occurring and we generally don’t seek to make recommendations for things that are in the works to fix

27 of 30

Thank you

Guides Available Online and Downloadable in PDF:

GAO Cost Estimating and Assessment Guide:

https://www.gao.gov/products/gao-20-195g

GAO Schedule Assessment Guide:

https://www.gao.gov/products/gao-16-89g

GAO Technical Readiness Assessment Guide:

https://www.gao.gov/products/gao-20-48g

GAO Agile Assessment Guide:

https://www.gao.gov/products/gao-24-105506

Slide 27

28 of 30

�GAO on the Web

Web site: http://www.gao.gov/

Facebook: https://www.facebook.com/usgao

Twitter: https://twitter.com/usgao

YouTube: https://youtube.com/user/usgao

Podcasts: http://www.gao.gov/podcast/watchdog.html

Copyright

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.

Slide 28

29 of 30

Back Up

Slide 29

30 of 30

Cost Estimating Best Practices

Comprehensive

Includes all life cycle costs
Based on a technical baseline description
Documents cost-influencing ground rules and assumptions
Based on a product oriented WBS

Well-Documented

Shows the source data used
Describes step by step how the estimate was developed
Discusses the technical baseline description
Provides evidence that the cost estimate was reviewed and accepted

Accurate

Adjusted properly for inflation
Based on a historical record
Uses the best methodology from the data collected
Contains few, if any, minor mistakes
Documents and explains variances
Regularly updated

Credible

Conduct sensitivity analysis
Conduct risk and uncertainty analysis
Perform cross checks
Compare to an independent estimate

Slide 30