DOD COST ESTIMATING GUIDE

Version 1.0

OFFICE OF THE SECRETARY OF DEFENSE

Cost Assessment and Program Evaluation

December 2020

FOREWORD

In an environment of growing threats, competing priorities, and fiscal pressures, the Department of

Defense (DoD) must spend the DoD budget on the right things, in the right amounts, at the right time.

DoD cost analysts play a critical role in this by producing cost estimates that support the planning,

programming, budgeting, acquisition, and requirements generation processes. The cost estimating

community of ~1500 government analysts supports an annual budget of more than $700 billion, with

160 major weapons systems and information systems, countless smaller acquisition programs, and

ongoing generation of requirements for future capabilities. Cost estimating is a unique skill set that

combines the best of science and art into a single role. The work relies on sound mathematical and

analytical skills, while also requiring critical thinking, communication, and nuance. Cost estimators have

a depth and breadth of knowledge that is unrivaled in many other career fields.

Every cost estimate is unique, but the overarching process for producing a credible, high-quality

estimate is not. With the help of cost estimating stakeholders from across the national security

community, this guide takes the reader through the steps of the cost estimating process and introduces

topics and concepts that are important for every DoD cost estimator to understand. Special thanks to all

of the organizations that helped CAPE to prepare this guide: DASA-CE, DON estimating community,

AFCAA, MDA, NRO, NPS, AFIT, DAU, GAO, and NASA. The input provided by these stakeholders is

invaluable to the finished product.

The guide provides an overview of important cost estimating topics, and then points the reader to other

resources for detailed theory and explanation, mathematical mechanics, and training opportunities.

Version 1 of this DoD Cost Estimating Guide reflects the current policies and practices as of March 15,

2020. CAPE will endeavor to update the guide as necessary to remain current as these policies and

practices inevitably will evolve in the future.

“No one can predict the future” is an often-used cliché, and yet this is what the DoD asks its cost

estimating community to do every day, albeit in a highly structured and disciplined way. Whether a new

cost estimator or seasoned analyst, this guide will assist with projects and analyses so that the cost

estimating community will continue to provide leaders and decision makers with relevant assessments

and sound recommendations.

TABLE OF CONTENTS

1.0 PURPOSE, POLICY, PROPERTIES, AND DEFINITIONS ............................................................... 7

Purpose of the Department of Defense (DoD) Cost Estimating Guide ............................................... 7

Cost Estimating and Analysis Policy .................................................................................................... 7

1.2.1 Cost Estimating and Analysis Statutes ......................................................................................... 7

1.2.2 Cost Estimating and Analysis DoDDs ........................................................................................... 9

1.2.3 Cost Estimating and Analysis DoDIs ........................................................................................... 10

Cost Estimate Program Category, Studies, and Types ...................................................................... 11

1.3.1 Program Category/Events Requiring a Cost Estimate ............................................................... 11

1.3.2 Studies ........................................................................................................................................ 12

1.3.3 Cost Estimate Type .................................................................................................................... 13

Properties of a Good Cost Estimate .................................................................................................. 14

Definitions ......................................................................................................................................... 15

1.5.1 Cost Analysis vs. Cost Estimating ............................................................................................... 15

1.5.2 Work Breakdown Structure and Estimate Structure ................................................................. 15

1.5.3 Inflation vs. Escalation ............................................................................................................... 16

1.5.4 Cost vs. Price .............................................................................................................................. 17

1.5.5 Direct vs. Indirect ....................................................................................................................... 17

1.5.6 Cost Model vs. Cost Estimate .................................................................................................... 17

1.5.7 Cost Contributors vs. Cost Drivers ............................................................................................. 17

1.5.8 Risk/Opportunity, and Uncertainty ........................................................................................... 18

Cost Estimating and Analysis Policy References ............................................................................... 18

Cost Estimating and Analysis Policy Training .................................................................................... 18

2.0 THE COST ESTIMATING PROCESS ........................................................................................... 20

DoD Cost Estimating Process ............................................................................................................ 20

2.1.1 Policy .......................................................................................................................................... 21

2.1.2 Program Definition ..................................................................................................................... 21

2.1.3 Cost Estimate Basis .................................................................................................................... 21

2.1.4 Data ............................................................................................................................................ 21

2.1.5 Methods ..................................................................................................................................... 21

2.1.6 Model ......................................................................................................................................... 21

2.1.7 Initial Results and Iterate as Necessary ..................................................................................... 22

2.1.8 Final Results and Documentation .............................................................................................. 22

2.1.9 Next Analysis .............................................................................................................................. 22

Component Guidance Documents .................................................................................................... 22

Cost Estimating Process References ................................................................................................. 23

Cost Estimating Process Training ...................................................................................................... 23

3.0 PROGRAM DEFINITION .......................................................................................................... 25

Establish a Program Definition .......................................................................................................... 25

3.1.1 Cost Analysis Requirements Description (CARD) ....................................................................... 27

3.1.2 Understanding the Program and Contract WBS ........................................................................ 27

3.1.3 Program WBS, Contract WBS, O&S CES and the Estimate Structure ........................................ 28

Start Building a Cost Model ............................................................................................................... 29

Program Definition References ......................................................................................................... 29

Program Definition Training .............................................................................................................. 30

4.0 COST ESTIMATE BASIS ............................................................................................................ 31

Cost Estimate Plan............................................................................................................................. 31

4.1.1 Establishing the Purpose and Scope .......................................................................................... 33

4.1.2 Define the Estimate Structure ................................................................................................... 33

4.1.3 Creating a Cost Estimate Schedule ............................................................................................ 33

Framing Assumptions, Ground Rules, and Cost Estimate Assumptions ........................................... 34

4.2.1 Framing Assumptions ................................................................................................................ 34

4.2.2 Ground Rules ............................................................................................................................. 35

4.2.3 Cost Estimate Assumptions ....................................................................................................... 35

Documentation of the Cost Estimate Basis ....................................................................................... 36

Cost Estimate Basis References ........................................................................................................ 36

Cost Estimate Basis Training ............................................................................................................. 37

5.0 IDENTIFY, COLLECT, VALIDATE, NORMALIZE, AND ANALYZE DATA ..................................... 38

Characterizing Data ........................................................................................................................... 38

Data Types ......................................................................................................................................... 39

5.2.1 Cost Data .................................................................................................................................... 39

5.2.2 Programmatic Data .................................................................................................................... 40

5.2.3 Performance and Technical Data ............................................................................................... 40

5.2.4 Schedule Data ............................................................................................................................ 40

Data Sensitive to Duration or Quantity ............................................................................................. 41

Identify Data ...................................................................................................................................... 41

5.4.1 Data Repositories ....................................................................................................................... 42

5.4.2 Deliverables and Reports ........................................................................................................... 44

Collect, Validate, Normalize, and Analyze Data ................................................................................ 48

5.5.1 Data Collection Plan ................................................................................................................... 48

5.5.2 Collecting Data ........................................................................................................................... 49

5.5.3 Validate Data .............................................................................................................................. 51

5.5.4 Normalize Data .......................................................................................................................... 51

5.5.5 Analyze Data .............................................................................................................................. 52

Data References ................................................................................................................................ 53

Data Training ..................................................................................................................................... 53

6.0 SELECT COST/SCHEDULE ESTIMATING METHODS ................................................................ 55

Basic Estimating Methods ................................................................................................................. 55

6.1.1 Analogy Estimating Method ...................................................................................................... 55

6.1.2 Build-up Estimating Method ...................................................................................................... 56

6.1.3 Extrapolation from Actuals Method .......................................................................................... 57

6.1.4 Parametric Estimating Method .................................................................................................. 57

6.1.5 Comparing Basic Estimating Methods ....................................................................................... 58

Other Estimating Methods ................................................................................................................ 60

Additional Considerations ................................................................................................................. 61

6.3.1 Correlation ................................................................................................................................. 61

6.3.2 Cost Improvement Curve ........................................................................................................... 62

6.3.3 Linear Without Intercept ........................................................................................................... 64

6.3.4 Outliers ....................................................................................................................................... 64

Introduction to Estimating Method Uncertainty .............................................................................. 64

Estimating Methods References ....................................................................................................... 65

Estimating Methods Training ............................................................................................................ 66

7.0 BUILD COST ESTIMATE MODEL .............................................................................................. 68

Anatomy of a Cost Estimate Model .................................................................................................. 68

7.1.1 Characteristics to Simplify the Cost Estimate Model ................................................................. 69

7.1.2 Phasing ....................................................................................................................................... 70

7.1.3 Sunk Cost.................................................................................................................................... 72

7.1.4 Cost Modeling Tools .................................................................................................................. 73

7.1.5 Multiple Cost Models for One Program ..................................................................................... 73

7.1.6 Common Cost Metrics ............................................................................................................... 74

Develop and Interpret the Baseline Cost Estimate ........................................................................... 75

7.2.1 Develop the Baseline Cost Estimate .......................................................................................... 75

7.2.2 Interpreting the Baseline Cost Estimate Results ........................................................................ 75

Review the Initial Results .................................................................................................................. 78

7.3.1 Crosschecks ................................................................................................................................ 78

7.3.2 Sensitivity Analysis ..................................................................................................................... 79

7.3.3 What-If Analysis ......................................................................................................................... 80

Addressing Risk/Opportunity, and Uncertainty ................................................................................ 80

7.4.1 Risk/Opportunity ....................................................................................................................... 81

7.4.2 Uncertainty ................................................................................................................................ 82

Iterate as Necessary .......................................................................................................................... 83

Build Cost Model References ............................................................................................................ 83

Build Cost Estimate Model Training .................................................................................................. 84

8.0 FINAL RESULTS AND DOCUMENTATION ............................................................................... 85

Documentation Contents .................................................................................................................. 85

Generate Final Documentation Report ............................................................................................. 86

Present and Defend Results .............................................................................................................. 88

8.3.1 Sand Chart .................................................................................................................................. 88

8.3.2 Pareto Chart ............................................................................................................................... 90

8.3.3 Tornado Charts .......................................................................................................................... 90

8.3.4 Cost Element Chart .................................................................................................................... 92

8.3.5 Program Funding and Quantities Chart ..................................................................................... 93

8.3.6 S-Curve ....................................................................................................................................... 94

GAO Cost Assessment Checklist ........................................................................................................ 95

Lessons Learned ................................................................................................................................ 95

Documentation and Results References ........................................................................................... 96

Documentation and Results Training ................................................................................................ 97

9.0 NEXT ANALYSIS ....................................................................................................................... 98

APPENDIX ..................................................................................................................................... 99

APPENDIX A ACRONYMS ............................................................................................................ 100

APPENDIX B SAMPLE COST ESTIMATING FLOWCHARTS .......................................................... 107

B.1 Government Accountability Office ................................................................................................. 107

B.2 CAPE ................................................................................................................................................ 107

B.3 Department of the Army ................................................................................................................. 108

B.4 Department of the Navy ................................................................................................................. 108

B.5 Department of the Air Force ........................................................................................................... 109

B.6 Joint Space Cost Council (JSCC) ....................................................................................................... 111

B.7 NASA ............................................................................................................................................... 111

APPENDIX C SAMPLE QUESTIONS TO GET STARTED ................................................................. 112

C.1 Sample Kickoff Meeting Questions ................................................................................................. 112

C.2 Sample Program Definition Questions ............................................................................................ 112

APPENDIX D DEPARTMENT OF THE AIR FORCE COST ESTIMATE DOCUMENTATION CHECKLIST

FOR ACAT I, II, AND III COST ESTIMATES ................................................................................... 114

D.1 Introduction .................................................................................................................................... 114

D.2 Body ................................................................................................................................................ 114

D.3 Additional checklist considerations identify whether: ................................................................... 115

APPENDIX E SAMPLE SME INTERVIEW FORM ........................................................................... 116

APPENDIX F SAMPLE ASSESSMENTS OF ESTIMATING METHOD APPLICATION ....................... 117

APPENDIX G GAO BEST PRACTICE LIST ...................................................................................... 119

FIGURES

Figure 1: DoD Cost Estimating Process ........................................................................................................................ 20

Figure 2: Estimating Method Applicability .................................................................................................................. 60

Figure 3: Notional Correlation Matrix Example ........................................................................................................... 62

Figure 4: Notional Major Capability Acquisition Budget Profile vs. a Notional MTA Program Schedule .................... 72

Figure 5: Total Ownership Cost Composition .............................................................................................................. 74

Figure 6: Point Estimate Location Within a Range of Possible Outcomes ................................................................... 76

Figure 7: Sand Chart (Layered) (notional) .................................................................................................................... 89

Figure 8: Sand Chart (Stacked Bar) (notional) ............................................................................................................. 89

Figure 9: Pareto Chart (notional) ................................................................................................................................. 90

Figure 10: Tornado for Cost Drivers Chart (notional) .................................................................................................. 91

Figure 11: Tornado for Cost Contributors Chart (notional) ......................................................................................... 92

Figure 12: O&S Cost Element Chart (notional) ............................................................................................................ 92

Figure 13: Program Funding and Quantities (Spruill) Chart (notional) ........................................................................ 94

Figure 14: S-Curve Example (notional) ........................................................................................................................ 95

Figure 15: GAO Cost Estimating Process .................................................................................................................... 107

Figure 16: CAPE Recommended Analytic Approach for O&S Cost Estimate ............................................................. 107

Figure 17: Department of the Army Cost Estimating Process ................................................................................... 108

Figure 18: DON Cost Estimating Process Flow ........................................................................................................... 108

Figure 19: NAVAIR Life-Cycle Cost Estimating Process Flow (Sep 2019) ................................................................... 109

Figure 20: AF Basic Cost Estimating Process .............................................................................................................. 109

Figure 21: AF Cost Estimating Overview .................................................................................................................... 110

Figure 22: Joint Space Cost Council (JSCC) Cost Estimating Process.......................................................................... 111

Figure 23: NASA Cost Estimating Process .................................................................................................................. 111

Figure 24: Example SME Documentation (Provided by the Missile Defense Agency, 2019) ..................................... 116

Figure 25: AFCAA: Selection of Methods ................................................................................................................... 117

Figure 26: Missile Defense Agency: Selection of Methods ........................................................................................ 117

Figure 27: NASA: Use of Cost Estimating Methodologies by Phase .......................................................................... 118

TABLES

Table 1: Key Inflation/Escalation Terms ...................................................................................................................... 16

Table 2: Information to Include in a Cost Estimate Plan ............................................................................................. 32

Table 3: Data Types and Generic Sources (not exhaustive) ......................................................................................... 41

Table 4: CADE Data ...................................................................................................................................................... 43

Table 5: DoD-level Data Repositories .......................................................................................................................... 44

Table 6: Potential Data Available in Required Acquisition Documents ....................................................................... 45

Table 7: Potential Data Available in Identified Government Data Sources ................................................................. 47

Table 8: Industry Data Sources to Consider ................................................................................................................. 48

Table 9: Summary of Advantages and Disadvantages of Basic Estimating Methods .................................................. 59

Table 10: Common Cost Estimate Documentation Organization ................................................................................ 87

1.0 PURPOSE, POLICY, PROPERTIES, AND DEFINITIONS

Purpose of the Department of Defense (DoD) Cost Estimating Guide

This guide provides consolidated information on the DoD cost estimating process and points the reader

to additional references and training for specific estimating topics. It does not replace DoD Component

guides and training materials. It does make direct references to existing cost estimating or guidance

documents that describe processes, methods, and procedures specific to that environment. This guide:

• applies to all types of cost analyses performed within the DoD,

• bridges the gap between the DoD Directives/Instructions (DoDDs/DoDIs) and the

Component/ Agency-level guidance/resources,

• focuses on major defense acquisition programs (MDAPs), but also applies to acquisition

category (ACAT) II and smaller programs, business system programs, services acquisition

programs, and other estimates including Middle Tier of Acquisition (MTA

) programs and

Nunn-McCurdy requirements, and

• provides a starting point for new analysts across DoD and a resource for seasoned analysts.

Cost Estimating and Analysis Policy

The United States Congress conferred primary DoD acquisition program cost estimation and cost

analysis responsibility to the Office of the Secretary of Defense (OSD) Cost Assessment and Program

Evaluation (CAPE). This responsibility includes the authority to establish DoD policy through DoDIs.

Therefore, the Director of CAPE (DCAPE) has prescribed policies and procedures for the conduct of cost

estimation and cost analysis, to include Independent Cost Estimates (ICEs), Analysis of Alternatives

(AoA), multiyear procurements

(MYP), data collection, etc. The following sections discuss the laws and

policies that govern cost estimating requirements.

1.2.1 Cost Estimating and Analysis Statutes

The United States Congress passes cost estimating and analysis statutes and incorporates them into

various titles and sections of the United States Code (USC). There are also four fiscal laws that govern

how the government spends money and indirectly impact cost estimating. The primary statutes and the

associated directives that establish policy relevant to cost estimating are discussed below.

Four primary fiscal laws relevant to cost estimating are:

• 10 USC Code Sec 114, “Annual authorization of appropriations”: Identifies appropriations

for military spending. Analysts must understand the military appropriations in order to

partition a cost estimate into the proper budget categories.

• Antideficiency Act: Creates various laws for expenditures, obligations, and voluntary

service, which are necessary for analysts to understand. These laws include:

o 31 USC Sec 1341(a)(1)(A) – prohibits authorizing expenditures in excess of the amount

appropriated,

o 31 USC Sec 1341(a)(1)(B) – prohibits spending of funds prior to funds being

appropriated,

o 31 USC Sec 1342 – prohibits voluntary service to the government, and

o 31 USC Sec 1517(a) – prohibits expenditures in excess of apportionment amounts.

MTA programs are a result of the 2016 National Defense Authorization Act (NDAA) in Section 804.

See 10 USC Section (Sec) 2306b “Multiyear contracts: acquisition of property”.

• 31 USC Sec 1301, “Application”: Requires that appropriated funds be applied only to the

objects for which the appropriations were made. This Appropriations statute, commonly

known as the “Misappropriation Act”, contains language about limitations placed on the

use of appropriated funds, which might become an issue during the cost estimating

process.

• 31 USC Sec 1502, “Balances available”: Requires appropriated funds be used only for

goods and services for which a need arises during the period of that appropriation’s

availability for obligation. Known as the “Bona Fide Need” rule, this law contributes to an

analyst’s understanding of obligation requirements.

Other laws directly applicable to cost estimating and analysis include:

• 10 USC Sec 2306b, “Multiyear contracts: acquisition of property”: Establishes the criteria

for entering into multiyear contracts. Includes requirements for a preliminary (prior to

authorization) and final (prior to contract award) CAPE savings forecast. DoD submits the

final savings forecast to Congress, and the contract may not be awarded until 30 days after

that submission.

• 10 USC Sec 2334, “Independent cost estimation and cost analysis”: Includes the Weapon

Systems Acquisition Reform Act of 2009 (Public Law 111-23) which established the DCAPE

statutory authority for independent cost estimation and cost analysis including providing

realistic acquisition cost estimates, conducting/approving MDAP cost estimates, reviewing

Component cost estimates (CCE), analyses, and records, discussing cost estimate risks, and

establishing data collection guidelines. Additionally, 10 USC Sec 2334 provides the

authority for DCAPE to issue cost estimating policy, procedures, and guidance. The

implementing directive for 10 USC Sec 2334 is DoDD 5105.84, “Director of CAPE”.

• 10 USC Sec 2337a, “Assessment, management, and control of operating and support costs

for major weapon systems”: Establishes, in conjunction with 10 USC Sec 2334(g), the

DCAPE authority to collect cost data. 10 USC 2334(g) is specific to acquisition data while 10

USC Sec 2337a(c) provides DCAPE statutory authority to retain Operating and Support

(O&S) data along with the responsibility to establish a database to collect O&S estimates,

documentation, and costs. The DoD published DoDD 5105.84, “Director of Cost

Assessment and Program Evaluation (DCAPE)” on May 11, 2012, before 10 USC Sec 2337a

became law. The next revision to DoDD 5105.84 will capture the content of 10 USC 2337a.

• 10 USC Sec 2366a, “Major defense acquisition programs: determination required before

Milestone A approval”: Defines the responsibilities, determination, and submissions

required for an MDAP to receive Milestone A approval. As part of the determination prior

to granting Milestone A approval, the DCAPE must concur, for the submitted program cost

estimate, that the level of resources required to develop, procure, and sustain the program

is sufficient for successful program execution. Additionally, within 15 calendar days of

granting Milestone A approval, the program Milestone Decision Authority (MDA) is required

to submit the program cost and schedule estimates, as well as the ICE, to the congressional

defense committees. This statute also defines a requirement for an AoA.

• 10 USC Sec 2366b, “Major defense acquisition programs: certification required before

Milestone B approval”: Defines the certifications, determinations, submissions, and

applicable waivers for an MDAP to receive Milestone B approval. As part of the

determination prior to granting Milestone B approval, the DCAPE must concur, for the

submitted program cost estimate, that reasonable cost and schedule estimates have been

developed to execute the program product development and production plan.

Additionally, within 15 calendar days of granting Milestone B approval, the program MDA is

required to submit the program cost and schedule estimates, as well as the ICE, to the

congressional defense committees. This statute also requires the completion of an AoA.

• 10 USC Sec 2366c, “Major defense acquisition programs: submissions to Congress on

Milestone C”: Defines the Congressional submissions required after Milestone C approval.

Within calendar 15 days of granting Milestone C approval, the program MDA is required to

submit a brief summary of the dollar values estimated for the program acquisition unit cost

(PAUC), average procurement unit cost (APUC), the total life-cycle cost, the planned dates

for initial operational test and evaluation (IOT&E) and initial operational capability (IOC),

and the ICE to the congressional defense committees.

• 10 USC Sec 2430, “Major defense acquisition program defined”: Defines an MDAP and

designates the MDA for such programs as the relevant Service Acquisition Executive, unless

otherwise designated by the Secretary of Defense. This definition and designation has a

significant impact on the level of cost estimating detail and documentation required at

milestone decision reviews. This law excludes rapid prototyping/rapid fielding programs

defined as MTA programs in the 2016 NDAA and some defense business systems (DBS)

from the definition of MDAP.

• 10 USC Sec 2433, “Unit Cost Reports”: Establishes the terms procurement program,

significant cost growth threshold, and critical cost growth threshold and their relationship

to the PAUC and APUC for an MDAP or any designated major subprogram. These

relationships form the basis for a Nunn-McCurdy breach that analysts should understand.

• 10 USC Sec 2441, “Sustainment reviews”: Establishes a statutory requirement for ongoing

reviews during system sustainment, which includes an ICE and other cost related analyses

of major weapon systems.

1.2.2 Cost Estimating and Analysis DoDDs

A DoDD is a broad policy document containing what is required by statute, the President, or the

Secretary of Defense to initiate, govern, or regulate actions or conduct by the DoD Components within

their specific areas of responsibilities. DoDDs establish or describe policy, programs, and organizations;

define missions; provide authority; and assign responsibilities. DoDDs directly applicable to cost

estimating and analysis include:

• DoDD 5000.01, “The Defense Acquisition System” (2018): Establishes the management

process by which the DoD provides effective, affordable, and timely systems to the users. It

addresses topics such as acquisition program accountability for cost, schedule, and

performance reporting, reducing cost, cost and affordability, total ownership costs, cost

realism, and the most cost-effective solution over the system life cycle. Cost estimating and

cost analysis play extremely important roles in acquiring new capabilities for the warfighter.

• DoDD 5105.84, “Director of Cost Assessment and Program Evaluation (DCAPE)” (2012):

Assigns the responsibilities, functions, relationships, and authorities of the DCAPE. DCAPE

responsibilities include acquisition support, resource planning, analysis and advice, annual

reports to Congress, and other duties as assigned by the Secretary or Deputy Secretary of

Defense. Acquisition support contains DCAPE responsibilities for cost analysis, AoAs, and

analytic competency.

• DoDD 5134.01, “Under Secretary of Defense for Acquisition, Technology, and Logistics

(USD(AT&L))” (2008): Assigns the responsibilities, functions, relationships, and authorities

of the USD(AT&L). The 2017 NDAA separated the USD(AT&L) into the Undersecretary of

Defense for Research and Engineering (USD(R&E)) and the Undersecretary of Defense for

Acquisition and Sustainment (USD(A&S)). While policies are still under revision for the

duties of these two organizations, their collective impact upon cost estimating including

acquisition and sustainment process improvements and research, technology, and

engineering improvements can lead to changes in a cost estimate.

• DoDD 5144.02, “DoD Chief Information Officer (CIO)” (2017): Assigns the responsibilities,

functions, relationships, and authorities of the DoD CIO. This directive establishes top-level

guidance that contributes to information system cost estimating requirements.

The brief summary of these statutes and directives highlight the many requirements placed upon DCAPE

in directing and establishing the DoD cost estimating policies and procedures which are further

conveyed via DoDIs.

1.2.3 Cost Estimating and Analysis DoDIs

DoDIs implement the policy or prescribe the manner for carrying out the policy, operating a program or

activity, and assigning responsibilities. DoDIs directly applicable to cost estimating and analysis include:

• DoDI 5000.02, “Operation of the Adaptive Acquisition Framework” (2020): Prescribes

procedures for managing acquisition programs and assigns program management

responsibilities. Describes the purpose and characteristics of six acquisition pathways.

Each of the pathways has associated cost estimating requirements. These requirements

are further described in the DoDI 5000.73.

• DoDI 5000.02T, “Operation of the Defense Acquisition System” (2020): Provides the

detailed procedures that guide the operation of the Defense Acquisition System and is the

implementation instruction for DoDD 5000.01. It addresses cost estimating requirements

at a very high level within the context of the acquisition process. It also states that DCAPE

establishes procedural guidance for the collection of cost data on acquisition programs.

This policy is a transition document and its Table 1 outlines new policy documents that are

in development.

• DoDI 5000.73, “Cost Analysis Guidance and Procedures” (2020): Establishes policy, assigns

responsibilities, and provides procedures for the conduct of cost estimation and analysis in

the DoD. This is the implementing instruction for DoDD 5105.84. It is the primary

instruction on cost estimating and cost analysis across the DoD and its Components. This

instruction instantiates cost estimating requirements for many types of cost analysis.

• DoDI 5000.74, “Defense Acquisition of Services” (2020): Establishes policy, assigns

responsibilities, and provides direction for the acquisition of contracted services. This is the

implementation instruction for DoDD 5134.01. It assigns responsibility to DCAPE for

policies and procedures associated with cost estimating and cost analysis for the acquisition

of contracted services.

• DoDI 5000.75, “Business Systems Requirements and Acquisition” (2020): Establishes

policy for the use of the business capability acquisition cycle (BCAC) for business systems

requirements and acquisition. This is the implementation instruction under DoDD 5134.01,

DoDD 5000.01, and DoDD 5144.02. It assigns responsibility to DCAPE for policies and

procedures associated with data collection, cost estimating, and cost analysis for the

acquisition of business systems. (The DoDI 5000.75 supersedes DoDI 5000.02T for all

business system acquisition programs that are not designated as an MDAP.)

• DoDI 5000.80, "Operation of the Middle Tier of Acquisition (MTA)" (2019): Establishes

policy, assigns responsibilities, and prescribes procedures for rapid prototyping and rapid

fielding as defined in Section 804 of Public Law 114-92. This is an implementation

instruction under DoDD 5134.01. It assigns responsibility to DCAPE for advising the

USD(A&S) on schedule, resource allocation, affordability, systems analysis, cost estimation,

and the performance implications of proposed MTA programs. Additionally, DCAPE is to

establish policies and prescribe procedures for the collection of cost data and cost

estimates for MTA programs.

• DoDI 5000.81, "Urgent Capability Acquisition" (2019): Establishes policy, assigns

responsibilities, and provides procedures for acquisition programs that fulfill urgent

operational needs and quick reaction capabilities. This instruction does not include any

specific responsibilities for DCAPE. However, an acquisition program must meet specific

cost and schedule criteria in order to utilize the Urgent Capability Acquisition pathway.

• DoDI 7041.03, “Economic Analysis for Decision Making” (2017): Establishes policy, assigns

responsibilities, and provides procedures for conducting cost-effective economic analyses

(EA). These analyses evaluate the costs and benefits of any government decision to initiate,

renew, or expand program or project alternatives under the Office of Management and

Budget (OMB) Circular No A-94, “Guidelines and Discount Rates for Benefit-Cost Analysis of

Federal Programs.” DoDI 7041.03 is an implementing instruction under DoDD 5105.84. It is

applicable to decisions regarding the use of real property, acquisition of information

systems, and the acquisition of weapon systems and weapons systems support. With

respect to the acquisition of weapons system and weapons systems support, analytic

studies and Business Case Analysis (BCA) may also be considered EAs if they deal with cost

and effectiveness considerations.

Analysts can find the latest versions of DoDDs/DoDIs under DoD Issuance/Directives and DoD Issuance/

Instructions at: https://www.esd.whs.mil/DD/DoD-Issuances/. These DoDIs are not the end of the policy

and guidance chain. DoD Manuals (DoDM), specifically the DoDM 5000.04 Cost and Software Data

Reporting (CSDR) Manual, and the many guides and manuals referenced throughout this document

directly relate to the statutes, directives, and instructions already mentioned. All of these documents

work together to address how the DoD accomplishes cost estimating.

Cost Estimate Program Category, Studies, and Types

The purpose and scope of a cost estimate are a function of program category, events, and type. These

program categories, events, and types help define the amount of detail, the timeline, the approval

process, and other requirements for the specified cost estimate.

1.3.1 Program Category/Events Requiring a Cost Estimate

While 10 USC Sec 2430(d)(1) gives MDA authority to the Component Acquisition Executives (CAEs) for

most MDAPs, DoDI 5000.02T identifies the USD(AT&L) as the Defense Acquisition Executive (DAE) and

MDA for the remaining MDAPs. MDAPs at the DAE level are usually very high dollar value or of special

interest to the Secretary of Defense. DoDI 5000.02T, Enclosure 1, Table 2 also identifies the CAE as the

MDA for ACAT II and III programs and provides definitions for each ACAT level. In some cases, the CAEs

delegate approval authority for lower level ACAT programs to Program Executive Officers (PEOs).

Therefore, the analyst should consult Component level guidance for any recent changes to the MDA

since the MDA is responsible for approving the cost estimates required for the following:

• ACAT I – IV programs: ACAT I – III programs are described in Enclosure 1, Table 2 of DoDI

5000.02T, and ACAT IV programs are Component specific (usually limited to the

Department of the Navy (DON)). The MDA for ACAT I programs will review an ICE and/or

Component Cost Position (CCP) and approve the most appropriate estimate for the

program at milestone reviews. The MDA for ACAT II – IV will review a CCE and/or program

office estimate (POE) for the specified program at milestone reviews. An ACAT program will

have multiple reviews over its life cycle.

• Business System Categories (BCAT) I – III: Table 1 of DoDI 5000.75 describes these non-

ACAT DBS categories where the associated cost estimates are reviewed/approved by the

MDA at authority to proceed (ATP) decision points, which are milestone-like events. A

BCAT will have multiple ATP decision points over its life cycle.

• Service Acquisition Categories (S-CATs) I – IV: These service acquisitions are described in

Table 1 of DoDI 5000.74 where the particular S-CAT level is determined by an independent

government cost estimate (IGCE). Following the initial review, there are no milestones or

decision points within a service acquisition, but there may be other reviews if contract

performance becomes a concern.

• MTA Program: The funding levels for these non-ACAT programs, which may surpass MDAP

thresholds, determines the type of cost estimate(s) required. The expected five-year or less

timeline to finish requires at least one MDA cost estimate review process and possibly more

depending on program cost and schedule performance. MTA Rapid Prototyping programs

require a cost estimate specific to the cost of the rapid prototyping. MTA Rapid Fielding

programs require a full life-cycle cost estimate.

• Nunn-McCurdy Breach: Congress made the Nunn-McCurdy Act permanent in 1983 via 10

USC Sec 2433 by defining significant and critical breaches

for MDAPs to curtail growth in

weapon systems programs. In addition to several certifications from across the acquisition

entities, a Nunn-McCurdy critical breach requires the CAPE to develop an ICE on the revised

program on a reduced timeline and present it to the MDA

1.3.2 Studies

There are acquisition studies containing cost estimates that require decision authority (possibly the

MDA) approval. The program office reviews and approves the cost estimates in these study documents.

Depending on the ACAT, BCAT, or S-CAT level, approval by the Component/DoD may also be required.

These include:

• AoA: A technical and cost assessment to objectively evaluate different potential courses of

action. In DoDD 5105.84, DCAPE requires that an AoA consider trade-offs among life-cycle

costs. While this is an ACAT I requirement, the Components have implemented similar

requirements on lower ACAT programs.

• EA: A systematic approach to identifying, analyzing, and comparing costs and benefits of

alternative courses of action. In DoDI 7041.03, DCAPE establishes the requirement for cost

and benefit analysis to support acquisition decisions. These decisions involve selecting the

best alternative from multiple criteria, including life-cycle costs in net present value

(NPV)

terms. Analytic studies and BCAs including cost and effectiveness considerations for the

acquisition of weapons systems and weapons systems support are types of EA.

• BCA: Used to determine if a new approach should be undertaken. DoDI 5000.02T includes

various requirements for BCAs associated with earned value management (EVM), Milestone

B approval, a product support (PS) BCA as part of the life-cycle sustainment plan, and cloud

computing services. The DoD has issued BCA guidebooks (e.g., PS BCA) and templates (e.g.,

When MDAPs experience cost growth of 15% percent from their current baseline or 30% percent from their

original baseline, they are in a “significant” Nunn-McCurdy Unit Cost Breach. Similarly, a 25% current or 50%

original baseline growth results in a “critical” Nunn-McCurdy Unit Cost Breach. These breaches are based on

growth to the PAUC or the APUC.

NPV analysis account for the time-value of money based on the assertion that dollars received in the future are

worth less than dollars in available today. The OMB promulgates Circular-94 “Guidelines and Discount Rates For

Benefit-Cost Analysis Of Federal Programs” annually.

Information Technology (IT) BCA)). Components have also issued guidance for BCAs. In all

cases, the requirement to include cost estimates in the BCA exists. The BCA addresses the

question: Should I invest or not? A PS BCA guidebook can be found at:

https://www.dau.edu/tools/t/Product-Support-Business-Case-Analysis-(BCA)-Guidebook.

• Source Selection/Proposal Evaluation: The source selection criteria issued by the Director

of Defense Pricing and Contracting (DPC) requires that the program manager

develop an

IGCE prior to the release of the final request for proposal (RFP) in order to help evaluate

proposal cost reasonableness and realism.

While there are other analytic studies concerning cost and effectiveness considerations that require cost

estimates, these are the major types. The Components have issued specific guidance for the types of

analysis they require. For example, Air Force Instruction (AFI) 65-501, “Economic Analysis” states that

implementing the EA approach is applicable to a variety of comparative analyses including EA, lease vs.

buy decisions, BCA, PS BCA, cost benefit analysis, and AoAs and then proceeds to provide guidance on

the implementation of these comparative analyses. The DON, alternatively, has separate EA and BCA

templates. The analyst must be familiar with the respective Component requirements for cost

estimates in these types of studies.

1.3.3 Cost Estimate Type

Regardless of the type of analysis it supports, every estimate should be realistic, defendable,

comprehensive, and well documented. The cost estimate type is a function of the program category,

events, its purpose, and the organization responsible for its development. The following are broad cost

estimate types:

• ICE: A life-cycle cost estimate

is statutorily required for all MDAPs during acquisition and

sustainment decision reviews and other significant out-of-cycle reviews such as Critical

Nunn-McCurdy breaches. For an MDAP in the acquisition process, the CAPE produces an

ICE or reviews and approves the ICE if produced by a Component. For non-MDAP

programs, the Component Cost Agency performs the ICE.

• DoD CCP: The CCP is the outcome of the reconciliation between the CCE and the POE,

except for the DON. It serves as the program official cost position from that Component.

For the DON, the POE serves as its official cost position, in the absence of a CCP.

• DoD CCE: A life-cycle cost estimate developed by one of the Components typically

developed by the Component Cost Agency.

• POE: A cost estimate developed by the program office and used as a tool for life-cycle cost

management throughout the life of the program. A program updates its POE as required to

capture actual incurred costs to date and refined estimating methods. The program

manager uses the POE to inform the acquisition and O&S management processes. The POE

is a consideration during the creation of the CCP.

• Cost Capability Analysis (CCA): An estimate typically developed by the program office to

support the program manager in the delivery of cost-effective solutions through deliberate

trade-off analysis between operational capability and affordability.

• IGCE: Pertains mostly to services acquisitions, specifically contracts, as mentioned in DoDI

5000.74. It provides a government developed cost estimate of an individual contract. The

This guide does not use the acronym PM. Program manager is spelled out to avoid confusion with the term

project manager.

A life-cycle cost estimate is the estimated cost of developing, producing, deploying, maintaining, operating and

disposing of a system over its entire lifespan.

analyst conducts an IGCE to check the reasonableness of a contractor’s cost proposal and to

make sure that the offered prices are within the budget range for a particular program. The

IGCE may assist in cost realism analysis

• Should Cost Estimate

(SCE): A management tool associated with the OSD Better Buying

Power initiative to control and reduce cost throughout the lifecycle, often referred to as a

Should Cost Initiative. The objective is to proactively target cost reduction through process

and productivity improvements. Over time, the SCE has evolved in intent and purpose and

therefore the reader is encouraged to seek out the relevant Component definitions and

policies for this type of cost estimate.

• Sufficiency Review: A review to ensure a program or cost estimate has sufficient

information for a formal milestone review. These reviews are typically component specific.

For example, the Air Force Life Cycle Management Center conducts program sufficiency

reviews “culminating in a final outbrief of the results of those assessments to obtain

approval of a program baseline

” and there is a sufficiency review checklist for cost

estimates scoring documentation, reasonableness and relevance, completeness and

consistency, and risk.

Properties of a Good Cost Estimate

Regardless of the type of cost estimate produced, the analyst can expect leaders and other analysts to

assess it against how well it:

• predicts, analyzes, and evaluates system cost and schedule resources,

• facilitates decision making, and

• assists program managers with program control planning and execution.

Due to the wide variety of cost estimate purposes and types, it is impossible to build a one-size-fits-all

cost estimate evaluation metric. However, the following are fundamental characteristics of any good

cost estimate:

• It is realistic, comprehensive, believable, and all-inclusive.

• It can be audited via traceability in the work breakdown structure (WBS), source data, and

cost model.

• It contains clear and concise definitions.

• It can be replicated by other estimators via well-defined documentation.

• It identifies and substantiates the costs of program resources (e.g., time, materiel,

manpower).

• It discloses any excluded costs along with the rationale.

The 2018 Independent Government Cost Estimate (IGCE) Handbook for Services Acquisition defines cost

reasonableness and cost realism.

“Joint Memorandum on Savings Related to “Should Cost”” signed by USD(AT&L) and USD Comptroller/Chief

Financial Officer (C/CFO) April 22, 2011

2016 Air Force Life Cycle Management Center (AFLCMC) Internal Process Guide to Conduct Program Sufficiency

Reviews (PSR)

Inspired by the Department of the Air Force Cost Analysis Agency (AFCAA), Cost Analysis Handbook, 2008,

Chapter 1, “Properties of a Good Estimate”, pg.1-20 and Government Accountability Office (GAO), Cost Estimating

and Assessment Guide, 2009, Chapter 1, “The Characteristics of Credible Cost Estimates and a Reliable Process for

Creating Them”, pg. 5

• It results in a specific mathematical answer, but that answer is framed within the context of

risks/opportunities and uncertainty.

• It includes comparisons to previous cost estimates and the available (or expected) budget.

• It addresses key stakeholder requirements including tables and charts that support

decision-making.

• It is structured to be easily modified to provide answers for unplanned program changes.

• It has been independently reviewed.

• It is completed on time.

These properties are not a complete list, but analysts should consider them individually and in total

when developing a cost estimate of any type.

Definitions

This section provides key definitions that are particularly important to the ensuing content in this guide

and discussions with other analysts. A comprehensive list of acronyms used throughout this document

is found in Appendix A Acronyms. The Defense Acquisition University (DAU) maintains a comprehensive

glossary of Defense acquisition acronyms and terms (

https://www.dau.edu/glossary/Pages/Glossary.aspx).

1.5.1 Cost Analysis vs. Cost Estimating

CAPE policies are consistent in distinguishing between cost analysis and cost estimating. Cost analysis

encompasses the entire range of activities in the cost estimating process. Cost analysis includes

activities such as sensitivity and what if analysis that are performed on the results of a cost estimate.

(See Sections 7.3.2 for sensitivity and 7.3.3 for what-if analysis.) Cost estimating itself is a blend of art

and science to develop a realistic cost forecast of proposed products or services. In this guide, cost

analysis refers to any effort performed in the support of generating a cost estimate and its

documentation. For example, assessing the benefit of a MYP (rather than annual procurement) is a cost

analysis activity with various results, some of which the analyst incorporates into the cost estimate.

1.5.2 Work Breakdown Structure and Estimate Structure

The 2018 military standard Work Breakdown Structures for Defense Materiel Items (MIL-STD-881D)

describes WBS” as a consistent and visible framework for product-oriented materiel items and contracts

within a defense program. Analysts use MIL-STD-881 WBSs as the basis for acquisition cost estimates.

The 2014 CAPE O&S Cost-Estimating Guide defines an O&S CES that categorizes and defines cost

elements covering the full range of O&S costs that could occur in any defense system. This guide uses

the following terms:

• Program WBS: Refers to a WBS that describes the program and is based on the current

version of MIL-STD-881 inclusive of all government costs.

• Contract WBS: Refers to the agreed-to contract reporting level and includes any

discretionary extensions to lower-levels for reporting. It should be closely aligned with the

program WBS.

• O&S CES: Refers to the CES as defined in the 2014 CAPE O&S Cost-Estimating Guide.

• Estimate Structure: Refers to a program WBS and/or O&S CES that has been expanded

and/or rearranged to support the required cost estimate.

See Section 3.1.2 for a more extensive discussion on the program and contract WBS.

1.5.3 Inflation vs. Escalation

Inflation is the rise in an economy-wide average (general) price level over time; there is only one rate of

inflation that applies to all goods and services in the US economy. Escalation is the change in price (to

include inflation) of particular goods and services in specific sectors of the economy. Escalation has two

components: inflation and real price change (RPC). RPC is the portion of escalation unexplained by

inflation such as market-specific supply and demand.

To account for inflation and escalation, cost can be expressed in a number of different ways, each

suitable for a specific purpose. Table 1 displays terms that the cost community uses to characterize or

modify cost to the proper context.

The 2017 CAPE Inflation and Escalation Best Practices For Cost Analysis: Analyst Handbook contains

more information on calculations associated with the terms in Table 1.

Table 1: Key Inflation/Escalation Terms

Term

Definition

Inflation Index

A series of multipliers that measure the percentage

change in the general price level over time, relative to a

particular year. Costs normalized using an inflation index

are Constant Year (CY) dollars.

Escalation Index

A series of multipliers that measure the percentage

change in price for particular goods and services over

time, relative to a particular year. Costs normalized

using an escalation index are Constant Price (CP) dollars.

Fiscal Year (FY) Dollars

Costs expressed in terms of a particular government FY.

Dollars

Cost normalized for inflation only (not normalized for

RPC) to a specific FY.

CP Dollars

Cost normalized for escalation, including both inflation

and RPC.

Base Year (BY) Dollars

Equivalent to CY dollars for specific point-of-reference

year, often selected for a program’s formal reporting

documents to maintain a constant basis of comparison.

Outlay Profile

In percentage terms, the rate at which a budget is spent

over time (years).

Then Year (TY) Dollars

Costs that include an outlay profile

to cover escalation

as obligations are expended over a multiyear period.

Primarily used for budgeting purposes (e.g., Total

Obligation Authority (TOA)).

See the 2017 CAPE DoD Inflation and Escalation Best Practices for Cost Analysis for authoritative details on

inflation, escalation, and other terms that characterize cost,

CY can also be the acronym for “current year” or “calendar year”. CY refers to “constant year” in this guide.

Some appropriations are required to be obligated within one year fully expended by the second year (e.g., O&S).

Others are spent over a period of up to seven years (e.g., shipbuilding). The outlay profile specifies the percent

spent in each year.

1.5.4 Cost vs. Price

Cost is the expense incurred for a product or service. Price represents the amount of money the

government intends to pay for that product or service. The difference between cost and price is fee

(commonly referred to as profit). Calculating fee is a function of contract type, and there are many

variations. A comparison of major contract types is found at:

https://www.acq.osd.mil/dpap/ccap/cc/jcchb/Files/Topical/Contract_Type_Comparison_Table/resources/contract

_type_table.docx

1.5.5 Direct vs. Indirect

Direct costs are costs attributable to a single product and generally categorized as labor, material, and

other direct cost (ODC). ODC includes items or services, such as tooling or consulting, that are neither

material nor direct labor but are attributable to a single product.

Indirect costs are service or expense costs that benefit multiple products such as utilities and facilities

and are therefore difficult to allocate to a single effort. Companies typically prorate these costs across

multiple contracts. An analyst may allocate indirect costs to different efforts based on relative direct

cost.

1.5.6 Cost Model vs. Cost Estimate

The cost model is what the analyst builds and utilizes to characterize the behavior of the program and

produce a credible cost estimate. The cost estimate is a product of the cost model and the cost

projection of the subject program, given a set of cost model inputs. Section 2.1.6 describes the basic

elements of a cost model.

1.5.7 Cost Contributors vs. Cost Drivers

The question “What is driving the program cost?” elicits different answers depending on who is

answering the question. For some, the answer is the element(s) of the estimate structure that

contribute the most to the total cost of interest. For others it is the programmatic, technical,

performance, or schedule element that has the greatest impact on the total cost of interest. These

concepts can be summarized as:

• Program cost contributors: The element(s) of the estimate structure (generally at a level

lower than acquisition or O&S) that contribute the greatest cost to the program. Finding

data to support elements of the estimate structure that contribute only a small fraction to

the total cost are not as important as those that contribute significantly more to the total

cost interest. For example, CAPE O&S CES 2.1.1 Energy (Fuel, Petroleum, Oil and

Lubricants, Electricity) may be a high cost contributor to the overall O&S estimate.

• Program cost drivers: The inputs (hours, labor rates, quantities, weight, power, etc.) to

cost estimate methods that have the most influence on the total cost of interest. Using the

same 2.1.1 Energy example, either the price of a gallon of fuel or the fuel consumption rate

of the system is likely to drive the total fuel cost.

The notion of contributors and drivers applies to not only their influence on the point estimate

but

also their influence on cost or schedule risk/opportunity and uncertainty. A review of similar programs

This guide does not use the acronym PE. Point estimate is spelled out to avoid confusion with the budgeting

term program element.

and the benefit of subject matter expert (SME) guidance helps to identify potential program cost

contributors and drivers and, in turn, may influence the data collection focus.

1.5.8 Risk/Opportunity, and Uncertainty

A risk is a potential future event or condition that may have a negative effect on cost, schedule, and/or

performance. An opportunity is a potential future event or condition that may have a positive effect on

cost, schedule, and/or performance

. Risk/opportunities have three characteristics: a triggering event

or condition, the probability that event or condition will occur, and the consequence of the event or

condition should it occur.

Analysts often use the terms risk and uncertainty interchangeably. In fact, they are distinct from one

another. Uncertainty is the indefiniteness of the outcome of a situation

. Uncertainty captures the

entire range of possible positive and negative outcomes associated with a given value or calculated

result. In a cost estimating model, an analyst generally addresses uncertainty first. The analyst then

addresses risks/opportunities if and only if the uncertainty assessment has not already captured them.

Cost Estimating and Analysis Policy References

• CAPE, Operating and Support Cost-Estimating Guide, 2014, para. 5, “Overview of Life-Cycle

Costs” pg. 2-1

• Naval Center for Cost Analysis (NCCA)/ AFCAA, Software Development Cost Estimating

Handbook, 2008, Chapter 2.1 “The Defense Acquisition System”, pg. 2-1

• Department of the Army, Cost Analysis Manual, 2020, Chap 2 “Cost Analysis References”,

pg. 8

• AFCAA, AFI 65-508, 2018, Chapter 1 “Overview, Roles, And Responsibilities” pg. 4

• Missile Defense Agency

, Cost Handbook, 2012, Chap 1 “Missile Defense Agency Cost

Estimating Process Overview” pg.7

• GAO, Cost Estimating and Assessment Guide, 2009, Chapter 2 “Why Government Programs

Need Cost Estimates and the Challenges in Developing Theme” pg. 15

• National Aeronautics and Space Administration (NASA), Cost Estimating Handbook, 2015,

para. 1.2 “The NASA Acquisition and Management Processes”, pg. 1

Cost Estimating and Analysis Policy Training

The DAU Cost Estimating certification program for members of the Defense Acquisition Workforce offers

training relevant to cost estimating policy. Additional information on each course may be found in the

DAU iCatalog (

https://icatalog.dau.edu/).

• Business, Cost Estimating, Financial Management (BCF) 130 Fundamentals of Cost Analysis,

Lesson 2

• BCF 216 Applied Operating and Support Cost Analysis, Lesson 2

• BCF 250 Applied Software Cost Estimating, Lesson 4

• BCF 331 Advanced Concepts in Cost Analysis, Lesson 1

DoD, Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, 2017, para. 1.1,

“Purpose”, pg. 3

NCCA, Joint Agency Cost Schedule Risk and Uncertainty Handbook (JA CSRUH), 2014, para. 1.2.2 “The Difference

Between Risk, Opportunity, and Uncertainty”, pg. 2

Missile Defense Agency is spelled out to avoid confusion with Milestone Decision Authority (MDA).

• Continuous Learning, Business (CLB) 009 Planning, Programming, Budgeting, and Execution

and Budget Exhibits (focuses on explaining the Planning, Programming, Budgeting and

Execution (PPBE) process, including the relationship of each phase to the systems acquisition

process)

• CLB 011 Budget Policy (focuses on appropriations and the funding policies associated with

each appropriation)

• CLB 014 Acquisition Reporting Concepts and Policy Requirements (introduces terms,

policies, and requirements)

• CLB 039 Cost Estimation Terminology (defines key cost estimating terms that are often

confused in cost estimating)

The International Cost Estimating and Analysis Association (ICEAA) publishes the Cost Estimating Body of

Knowledge (CEBoK). The follow modules are relevant to cost estimating policy:

• CEBoK v1.2, 2013, Module 1 “Cost Estimating Basics”

• CEBoK v1.2, 2013, Module 2 “Cost Estimating Techniques”

• CEBoK v1.2, 2013, Module 4 “Inflation”

• CEBoK v1.2, 2013, Module 14 “Contract Pricing”

The following course numbers starting with FMF refer to the course number assigned by the Financial

Management (FM) Certification process. Information on these courses (including eligibility

requirements) can be found in the FM myLearn system:

https://fmonline.ousdc.osd.mil/FMmyLearn/Default.aspx.

• FMF 1546 Business Case Analysis

• FMF 1558 DoD FM 101 - Fiscal Law

• FMF 4069 Budget Concepts, Policies, and Principles

• FMF 6599 DoD Basic Fundamentals and Operations of Budget

• FMF 1559 DoD FM 101 - Acquisition & Contracting

• FMF 1560 DoD FM 101 - Cost Analysis

• FMF 4050 Business Case Analysis - Mini-Course

• FMF 1551 QMT 490 - Current Topics in Cost Estimating

2.0 THE COST ESTIMATING PROCESS

This chapter provides an overview of the cost estimating process, and subsequent chapters provide

more detail on each step in the process. The analyst should always tailor the process to his/her specific

estimate or project.

DoD Cost Estimating Process

Analysts can have very different opinions on how best to arrive at a realistic cost estimate because the

number of viable paths to get there and the hurdles to surmount can appear endless. Over the course

of several years, the GAO worked diligently with dozens of national and international experts, both

government and industry, to develop a consensus on a clearly defined cost estimating process and to

document the best practices supporting that process. The result was the 2009 GAO Cost Estimating and

Assessment Guide. The GAO guide includes a process of 12 steps, which, if followed correctly, should

result in reliable cost estimates. It is common for DoD Components to reference this flow chart directly

or to provide a modified version adapted to their environment.

In deference to the many organizations that have developed flow charts to suit their unique

requirements (several of them can be found in Appendix B), Figure 1 defines a generalized cost

estimating process for DoD. This DoD version captures all of the steps in the GAO process and most of

the elements from Component guides, handbooks, and manuals. (See Appendix B.1 for the GAO

process.) The graphic in Figure 1 provides the framework for the discussions in this guide and gives the

reader a comprehensive overview of a DoD-centric process.

Figure 1: DoD Cost Estimating Process

Key features of Figure 1 include:

• Policy and the program definition tend to be products produced by authorities other than

the analyst, although it is important to have analysts participate in these efforts.

• The process recognizes the effort related to Data as fundamental to the success of any cost

estimate and often the most time/effort intensive activity. Figure 1 emphasizes that data is

at the center of the other steps in the process.

• The steps in the process are necessarily overlapping and iterative. It is common to be

performing parts of two or more steps simultaneously, and at any point, returning to

previous steps. A precise and repeatable serial flow for every cost estimating circumstance

simply does not exist.

The remainder of this section introduces the key iterative steps of the DoD cost estimating process.

2.1.1 Policy

The statutes, policies and guidance summarized in Chapter 1.0 identify the requirements for various

types of cost estimates, cost data collection, and other cost estimating related processes.

2.1.2 Program Definition

The program definition is a detailed description of a DoD program for use in preparing a cost estimate.

The primary elements, including the Cost Analysis Requirements Description (CARD), baseline system,

and program WBS, are examined in detail in Chapter 3.0.

2.1.3 Cost Estimate Basis

The analyst is responsible for clearly documenting the purpose and scope (including level of detail) of

the estimate. In particular, this step includes the framing assumptions, ground rules, and assumptions

(e.g., CY to express costs, life-cycle phases to be estimated, level of detail, need for what-if analysis, and

anything else that influences how the estimate is performed), as well as the schedule for the completion

of the cost estimate. (See Chapter 4.0 for more detail.)

2.1.4 Data

Data is the heart of the estimate. The identification, collection, validation, normalization, and analysis of

quality data influence all of the remaining steps in the cost estimating process. (See Chapter 5.0 for

more detail.)

2.1.5 Methods

An analysis of the collected data leads to the selection of the best cost/schedule estimating method(s)

for a specific element of the estimate structure. (See Section 1.5.2 for a definition of “estimate

structure”). The estimating methods address a variety of applicable influences such as the effects of

weight, volume, and power; quantities produced (learning curves and rate effects); quantities per year;

phasing; and many others. The time and availability of data required to implement the method is a

consideration when selecting methods. (See Chapter 6.0 for more detail.)

2.1.6 Model

An analyst produces a cost estimate from a mathematical model that includes all relevant cost elements.

Each lowest level element of the estimate structure has an estimating method. (See Chapter 6.0 for a

discussion of estimating methods). In some cases, the estimating method is a direct function of another

cost in the estimate structure. The analyst should design the cost estimate model to assess the impact

of a change in quantity, phasing, schedule, labor rates, operating/operational/operations tempo

(OPTEMPO), or anything else that could influence one or more element of the estimate structure. (See

Chapter 7.0 for more detail.)

2.1.7 Initial Results and Iterate as Necessary

Once the analyst builds the cost model (including the impacts of risk/opportunity and uncertainty), then

he/she should verify the model serves the intended purpose and validate the model results by

performing the following:

• Cross check: Tests the model’s results for accuracy at various levels in the estimate by

comparing them to the cost and/or schedule of completed projects, or by comparing

against the results of a relevant, alternative cost model that applied different data and/or

methods.

• Sensitivity analysis: Tests the model’s ability to estimate the impact on total cost by

changing a specific cost driver.

• What-if analysis: Tests the model’s ability to estimate the impact of changing a variety of

cost drivers that define a specific alternative.

There are many reasons that make it necessary to iterate through the cost estimating process, including

unexpected results from the cross checks, sensitivity analysis, or what-if analysis. (See Section 7.5 for

more detail.)

2.1.8 Final Results and Documentation

The content and format of results with their associated documentation and presentations are a function

of the estimate purpose and type. Documentation should start at the outset of the cost estimating

process, as shown in Figure 1, to capture all the necessary elements from each step, and be continually

refined throughout the process. (See Chapter 8.0 for more detail.)

2.1.9 Next Analysis

The final step in the cost estimating process is to move on to the next analysis. This could be a

completely new program, additional investigation on the current program, or any other cost estimating

related task. Often, future analysis uses the results of the current analysis.

Component Guidance Documents

Practices and procedures vary between cost analysis organizations according to mission requirements,

workload, staffing, and special circumstances. Components have issued documents that implement

DoDIs and represent a consensus of best practices useful to cost analysis practitioners for their

organizations and cost estimate stakeholders. This is recognition that cost analysis cannot be reduced to

a single linear set of rules to follow. In addition to the DoDD and DoDI documents described in earlier

sections, Component-specific guidance exists in:

• Department of the Army Cost Analysis Manual: Provides basic frameworks for

methodologies and procedures to implement policies for better cost analyses. It is a useful

aid in understanding and participating in the Department of the Army cost and EA process.

htts

ps://www.asafm.army.mil/Portals.72/Documents/Offices/CE/20200330%20CAM.pdf

• AFI 65-508 Cost Analysis Guidance and Procedures: Establishes timelines, documentation

requirements, and review procedures for all Air Force cost estimates, and provides specific

instructions on performing cost analyses.

https://static.e-publishing.af.mil/production/1/saf_fm/publication/afi65-508/afi65-508.pdf

• DON Cost Estimating Guide: Provides a compendium of best practices for life-cycle cost

estimates of weapon system and information systems acquisition programs within the

DON. It strives to improve and standardize processes and procedures while recognizing the

fluidity inherent in the field of defense cost analysis. This, and a variety of additional

relevant references, can be found at:

https://www.ncca.navy.mil/references.cfm

• Missile Defense Agency Cost Estimating and Analysis Handbook: Serves as a desk

reference for the Missile Defense Agency analysts and anyone who interfaces with the

organization analysts or uses its cost estimates. A secondary purpose is to identify and

define a set of standard data requirements for Missile Defense Agency cost estimates. The

handbook can be found at:

https://www.ncca.navy.mil/tools/csruh/References/091_2012.pdf.

Cost Estimating Process References

• DoDI 5000.73, Cost Analysis Guidance and Procedures, 2020, Section 3, “Cost Estimation

Requirements and Procedures”, pg. 6

• DoD Independent Government Cost Estimate Handbook for Service Acquisition, 2018, “Cost

Estimation”, pg. 9

• CAPE, Operating and Support Cost-Estimating Guide, 2014, para. 5, “O&S Cost Estimating

Process” pg. 5-1

• NCCA/AFCAA, Software Development Cost Estimating Handbook, 2008, Chapter 3 “Levels of

Detail in Software Estimates”, pg. 3-1

• Department of the Army, Cost Analysis Manual, 2020, Chapter 3 “Cost Estimating Process”,

pg. 8

• NCCA, Cost Estimating Guide, 2010 Chapter 1 “Overview”, pg. 9

• SPAWAR

, Inst 7110.1 Cost Estimating and Analysis, Encl. 1, 2016, Chapter 2 “Overview”

pg. 2

• United States Marine Corps (USMC), Cost Analysis Guidebook, 2017, Chapter 3 “Cost

Estimating Process”, pg. 37

• AFCAA, Cost Analysis Handbook, 2008, Chapter 3 “Cost Estimating Process and Methods”,

pg. F-4

• Missile Defense Agency, Cost Handbook, 2012, Chap 1 “Missile Defense Agency Cost

Estimating Process Overview” pg.7

• GAO, Cost Estimating and Assessment Guide, 2009, Chapter 7 “The Characteristics of

Credible Cost Estimates and a Reliable Process for Creating Them” pg. 5

• NASA, Cost Estimating Handbook, 2015, Chapter 2 “The Cost Estimating Process”, pg. 4

Cost Estimating Process Training

The DAU Cost Estimating certification program for members of the Defense Acquisition Workforce offers

training relevant to the cost estimating process. Additional information on each course may be found in

the DAU iCatalog (

https://icatalog.dau.edu/).

• BCF 130 Fundamentals of Cost Analysis, Lesson 1

• BCF 132 Applied Cost Analysis, Lessons 1

• BCF 216 Applied Operating and Support Cost Analysis, Lesson 1

• BCF 230 Intermediate Cost Analysis, Lessons 1

• BCF 250 Applied Software Cost Estimating, Lesson 2

Space and Naval Warfare Systems Command (SPAWAR) became the Naval Information Warfare Systems

Command (NAVWAR) June 03, 2019.

• BCF 331 Advanced Concepts in Cost Analysis, Lesson 1

• CLB 007 Cost Analysis (focuses on the basic cost analysis process)

• CLB 025 Total Ownership Cost (provides the framework necessary to estimate total

ownership cost within the acquisition process)

• CLB 032 Force Structure Costing (explains the definition, purpose, and utility of DoD Force

Structure Costing techniques)

• Continuous Learning, Management (CLM) 006 Independent Government Cost Estimate

(IGCE) for Services Acquisition (explains the environment for cost estimating and budgeting,

differentiates the four cost estimating methods and chooses the appropriate method for a

services acquisition program)

• CLM 016 Cost Estimating (focuses on basic cost estimating tools and techniques)

The ICEAA publishes the CEBoK. The follow modules are relevant to cost estimating policy:

• CEBoK v1.2, 2013, Module 2 “Cost Estimating Techniques”

The following course numbers starting with FMF refer to the course number assigned by the FM

Certification process. Information on these courses (including eligibility requirements) can be found in

the FM myLearn system:

https://fmonline.ousdc.osd.mil/FMmyLearn/Default.aspx.

• FMF 1550 QMT 290 - Integrated Cost Analysis

• FMF 1560 DoD FM 101 - Cost Analysis

• FMF 6175 AFIT Cost 669 - Advanced Cost Analysis

• FMF 1546 Business Case Analysis

• FMF 6016 FMA 301 - Business Case Analysis

• FMF 6320 AFM 301 - Cost Estimating for Major Investment Programs

• FMF 1551 QMT 490 - Current Topics in Cost Estimating

The following cost analysis related degrees and certificates are available:

• A 16-course Distance Learning Masters in Cost Estimating and Analysis offered by the Naval

Postgraduate School (NPS) in Monterey, CA

• A four-course Distance Learning Certificate in Cost Estimating and Analysis offered by the

NPS in Monterey, CA

• A two year resident Masters in Cost Estimating and Analysis offered by the Air Force

Institute of Technology (AFIT) in Dayton, OH

• The Defense Acquisition Workforce Improvement Act (DAWIA) Cost Estimating career field

level certifications. Requirements can be found at:

https://icatalog.dau.edu/onlinecatalog/CareerLvl.aspx#

• A Certified Cost Professional (CCP) administered by the Association for the Advancement of

Cost Engineering International (AACEI)

• Certified Estimating Professional (CEP) administered by the AACEI

• An apprentice-level certification for practitioners with at least two years’ experience,

university degree and ICEAA administered Professional Cost Estimator/Analyst Certification

(PCEA®) exam

• A professional certification for practitioners with at least five years’ experience, university

degree and ICEAA administered Certified Cost Estimator/Analyst (CCEA®) exam

3.0 PROGRAM DEFINITION

A key contributor to a sound cost

estimate is an accurate and detailed

program definition. Many formal

program documents address the

goals and content of the envisioned

program (in varying levels of detail

depending on the maturity of the

program). Even so, the analyst

requires a complete and detailed

description of the programmatic,

performance, technical, and schedule

aspects of the program, which should

be suitable for any type of cost

estimate. (See Section 1.3.3 for a

discussion on cost estimate types.)

From the analyst’s perspective, the

program definition contains many

pieces of information that are

essential. However, just knowing the

essentials is insufficient. Understanding the purpose(s) behind the basis for the estimate structure and

its tailoring, estimating method development, time-phasing, normalization, and development and

maintenance costs are just as important.

This chapter and Chapter 4.0 examine additional details behind selecting the necessary essentials for

the type of estimate as well as the purpose for selecting those essentials.

Establish a Program Definition

The program manager and experts throughout the program office are responsible for defining the

program. As such, the program definition is likely not a single document but a synthesis of many

documents and sources. In many settings, this starts with a CARD or a CARD-like document. (See

Section 3.1.1 for a discussion on CARDs.) Ideally, the CARD tables and narrative are a complete, detailed

description of the program. Analysts, however, should not blindly use this information, but take time to

review, understand, and where necessary, question the information to build a full understanding of the

program. The best CARDs unambiguously address all of the analyst’s questions sufficiently so that no

other source of program definition information is required. In situations where the CARD does not exist

or is not sufficient for some reason and the program manager cannot improve it, the analyst can use

other acquisition documents like those listed in Table 6, Table 7 and Table 8 (introduced in Section

5.4.2) to bridge the gap. The analyst can glean necessary program information from those documents

and assemble them into the program definition. This includes general system knowledge and

programmatic information such as:

• an overarching understanding of the program, to guide the development of the estimate

structure and to start thinking about estimating methods,

• program systems engineering/program management (SEPM) personnel by grade and by

fiscal year,

• contractor, subcontractor, and major vendor roles and related information from which to

calculate contract loads by vendor tier, and

• items furnished by the government and other information necessary to identify items that

will not be part of the prime contractor's cost.

The information assembled from source documents includes technical and performance parameters

such as:

• programmatic, performance, technical, and design heritage parameters for use as variables

for cost estimating relationships (CERs), schedule estimating relationships (SERs), scaling, or

analogy selection,

• metrics and cost drivers to enable direct estimation of common elements of the estimate

structure in lieu of estimating them by using a factor of the Prime Mission Product (PMP),

• software parameters necessary for estimating software development cost and software

maintenance cost,

• facility construction and facility conversion data,

• parameters by part for performing commercial-off-the-shelf (COTS)-heavy bottom-up

estimating, or component analysis, and

• end item composition (both uniqueness and commonality), for multiple end-item

configurations.

From the source documents, it is also necessary to assemble schedule and quantity information such as:

• dates for milestone decisions, engineering gates (e.g., Critical Design Review), and other key

program events from which to time-phase and inflate/escalate the cost estimate, and

• phase and contract (annual production lots) quantities and begin/end dates needed to

estimate time-sensitive costs (those elements that vary by duration) and to compute

learning and rate of production methods.

For estimating sustainment, the program’s documentation provides relevant information, including:

• cumulative fielding quantities and expected service life for O&S cost calculations,

• OPTEMPO as a measure of the pace of an operation or operations in terms of equipment

usage (e.g., aircraft flying hours, ship steaming days, or tank driving miles),

• metrics and cost drivers to estimate the cost of maintenance and other O&S costs,

• operators, maintainers, and support personnel by grade and by fiscal year, and

• logistics parameters regarding parts removed for repair/replacement.

The program office is essential to building the program definition, but it is not unusual for the analyst to

spend extensive time and effort reviewing, contributing necessary information, and making

recommendations for improvements. Analysts should work with program/system SMEs and managers

to locate and evaluate program definition information. Analysts should understand and evaluate

framing assumptions that have been central in shaping program expectations. Section 4.2.1 further

discusses framing assumptions. No matter how complete the CARD and other key program documents

may be, the analyst preparing the estimate must attain a solid understanding of the system being

estimated. Key personnel within the program office can assist with the analyst’s understanding. These

include the Program Manager, Deputy Program Manager, Acquisition Manager, Contracting Officer,

Business Financial Manager, Chief Engineer, Chief Tester, and Product Support Manager. Appendix C

provides a list of sample questions suitable for a kick-off meeting and developing an understanding of

the program definition.

3.1.1 Cost Analysis Requirements Description (CARD)

The CARD provides a complete, detailed description of the program baseline prepared by the program

office. If the program has a CARD, or a CARD-like document, it is an important source for most of the

program definition information the analyst requires.

The CARD represents a snapshot of that program. DoDI 5000.73 requires a CARD for all major capability

acquisition programs. The CARD thoroughly describes the programmatic, performance, technical,

operational, sustainment, and schedule characteristics of a program, along with some initial supporting

data sources, and provides program information necessary to develop a cost estimate.

The CARD enables different organizations preparing cost estimates to develop their estimates based on

the same understanding of program requirements. The CARD can serve as a management tool within

the program office and as a common, agreed-upon baseline for all the stakeholders. Therefore, the

program office developing the CARD should include only that information pertinent to the cost estimate.

That is, if the cost estimate focuses on Increment I of a system, then information on Increment II should

not be included in the CARD unless it specifically impacts a cost element in Increment I.

As a program evolves and analysis refines its costs and funding needs, the CARD, as a living document,

evolves with it. The Cost Assessment Data Enterprise (CADE) website (

https://cade.osd.mil/policy/card)

provides guidance and instructions for the preparation and maintenance of the CARD. The CAPE

establishes CARD requirements for ACAT I programs, and the Components establish CARD requirements

for non-ACAT I programs.

For the portions of CARD content that are contextual and descriptive, a CARD narrative is used.

Additionally, recognizing that cost analysis is a quantitative endeavor, the CAPE prescribes that certain

CARD content be in tabular form. In the event that the program does not have a CARD or CARD-like

document (e.g., an MTA program), the CARD tables can nonetheless be a data organization convenience

for the analyst who must assemble the information to compile a program definition. The CAPE-designed

CARD tables are commodity specific and address the following three objectives.

• The tables contain key programmatic and technical data required to estimate costs at a

sufficient level of detail to support program acquisition reviews (e.g. Milestone Reviews) or

PPBE process reviews (e.g., Program Objective Memorandum (POM) submission reviews).

• Over time, the completed tables serve as a record of program evolution.

• The tables support future automation via a database that analysts use for cost estimating,

analysis, and research.

The CARD is an acquisition document written/compiled for the analysts. It should never be the

responsibility of the analyst (at any level) to create the CARD. A common pitfall within program offices

is to ask the program cost estimate team to develop the content of the CARD. Since the CARD contains

the programmatic and technical data of the program, the acquisition and technical professionals in the

program office should develop the content. Program analysts can review the CARD to assess if the

content is detailed enough to support the analysts at the Service and OSD levels.

3.1.2 Understanding the Program and Contract WBS

The primary objective of a program WBS is to achieve a consistent framework for all programmatic

needs, including performance, schedule, risk/opportunity, budget, and contracts. It is also the basis for

an estimate structure across programs and life-cycle phases. The program WBS also facilitates

comparison of estimates performed by different estimators (e.g., ICE vs. CCP).

The contract WBS encompasses only the program WBS elements related to a contract deliverable, but

extended to the agreed-to contract reporting level and any lower level for items considered high-cost,

high-risk, high technical, and/or special interest. While the contract WBS must be closely aligned to the

program WBS, the two are not identical. The program WBS will have elements for Government and

other contractors not contained in the contract WBS. The program WBS serves as a consolidation

mechanism for multiple subordinate contracts and Government elements.

The CADE website (

http://cade.osd.mil/policy/csdr-plan) is a source of extended WBS product-oriented

structures. MIL-STD-881D references this site as a source of extensions to each commodity-specific

appendix. These extensions serve to increase the consistency of the data collection at lower levels of a

contract WBS. In addition to the MIL-STD-881D commodities, this resource has product-oriented

structures for a few additional commodities (e.g., training systems) as well as for sustainment-phase

contracts. The sustainment structure is an extension of the CAPE O&S CES. MIL-STD-881D Appendix L

provides further guidance on how the sustainment cost reporting structure is related to the defense

materiel systems WBS.

DoDI 5000.02T cites Disposal as a program phase. Though MIL-STD-881D does not explicitly address

disposal, a program’s estimate structure should accommodate eventual disposition of the material

items. Considerations include demilitarization, detoxification, long-term waste storage, environmental

restoration, and related elements of transportation and program management.

3.1.3 Program WBS, Contract WBS, O&S CES and the Estimate Structure

A logical, hierarchical structure is necessary to organize the program objectives and the cost estimate by

breaking them both down into manageable elements. Analysts sometimes use the terms WBS and CES

interchangeably. Strictly speaking, they are different but related concepts. This guide introduced the

terms: program WBS, contract WBS, O&S CES, and estimate structure in Section 1.5.2 to help clarify the

use of WBS and CES in this document.

A program office develops a WBS to serve as the framework for specifying objectives. MIL-STD-881

states that this WBS is a hierarchy of product-oriented elements, such as hardware, software, data, and

services that collectively comprise the system. The CAPE requires a program WBS be included in the

CARD as part of the program definition.

Acquisition professionals describe a WBS as either a program WBS or a contract WBS. The program WBS

contains all program acquisition content, but generally not the O&S content. A contract WBS contains

only a portion of the program WBS, and it usually contains a more extensive, lower level breakout of this

program WBS portion. It relates specific program WBS elements to the elements of a contract

statement of work in order to manage the contractor’s work. It may also serve as a contract cost

reporting structure. The program WBS provides the initial structure for the cost estimate.

An estimate structure defines and groups all of the costs of the program in a disciplined hierarchy whose

structure is largely determined by its suitability for cost estimation, i.e., by the availability of data and

the need to perform specific what-if drills. The analyst bases the estimate structure on selected

program WBS elements (e.g., airframe) and may further break it down into functional categories (e.g.,

engineering and manufacturing labor; overhead). Since the program WBS is usually a product-oriented

structure, it may not be sufficiently decomposed to adequately capture all the cost. In these scenarios,

the estimate structure is an extension, or further breakdown, of selected program WBS elements in

order to adequately capture costs and provide a foundation for investigating what-ifs. It is important to

understand that acquisition elements of the estimate structure must roll up into the higher level

program WBS elements. In some cases, the program WBS is sufficient for the cost estimate and O&S

CES elements are not necessary. In this particular case, the program WBS may be identical to the

estimate structure.

Since many cost estimates cover the entire life cycle, the estimate structure is more expansive than the

program or contract WBS. On occasion, multiple estimate structures are required to estimate a

program. For example, in a large program it may be necessary to develop specific estimate structures

separately (e.g., airframe, avionics, propulsion, everything else) and have another estimate structure to

combine them. Additionally, since the cost estimate model will likely be used to explore variations on

the proposed technical solution, the estimate structure is often more granular than either the program

WBS or contract WBS which are based upon the guidance in MIL-STD-881. This could mean more

elements at a particular level and/or more levels of indenture. An O&S WBS does not exist in MIL-STD-

881 because the O&S phase is not product-oriented. Therefore, the 2014 CAPE O&S Cost-Estimating

Guide provides the cost structure for this phase via an O&S CES. The Cost Estimate Basis, Chapter 4.0,

further develops the purpose and utility of an estimate structure.

Start Building a Cost Model

As the program definition begins to take shape, the analyst should start thinking about how to structure

the cost model, the implications for data gathering, and the estimating methods likely to be employed.

Chapter 5.0 describes a data collection process primarily focused on the collection of data from

analogous historical programs similar to the program definition to serve as the basis for estimating the

program costs. Building a simplified cost model at this point can help identify holes in the program

definition and help formulate the data collection plan.

Program Definition References

• DoDI 5000.02T, Operation of the Defense Acquisition System, 2020, Enclosure 10, para. 3

“Cost Analysis Requirements Description (CARD)”, pg. 135

• DoDI 5000.73, Cost Analysis Guidance and Procedures, 2020, Section 3 “Cost Estimation

Requirements and Procedures”, pg. 6

• Performance Assessments and Root Cause Analyses (PARCA)

, MIL-STD-881D, Work

Breakdown Structures for Defense Materiel Items, 2018

• CAPE, Operating and Support Cost-Estimating Guide, 2014, para. 5.2.3, “Define Program

and System Content”, pg. 5-5 and para. 5.2.4 “Select Cost Element Structure” pg. 5-6

• NCCA/AFCAA, Software Development Cost Estimating Handbook, 2008, Chapter 3 “Levels of

Detail in Software Estimates”, pg. 3-1

• Department of the Army, Cost Analysis Manual, 2020, Chap 3 “Cost Estimating Process”, pg.

• NCCA, Cost Estimating Guide, 2010 para. 2.1 “Establish a Program Baseline”, pg. 6

• SPAWAR, Inst 7110.1 Cost Estimating and Analysis, 2016, Enclosure 1, Chapter 4 “Establish

a Program Baseline” pg. 6

• USMC, Cost Analysis Guidebook, 2017, para. 3.1 “Establish A Program Baseline”, pg. 40

• AFCAA, AFI 65-508, 2018, Chapter 5 “Cost Analysis Requirements Description (CARD)” pg. 23

• AFCAA, Tabular Cost Analysis Requirements Description (CARD) Sufficiency Review

Handbook, 2017

• AFCAA, Cost Analysis Handbook, 2008, para. 5-2 “Develop a Technical Baseline”, pg. 5.11

PARCA was superseded by Acquisition, Analytics and Policy (AAP)

• Missile Defense Agency, Cost Handbook, 2012, Chap 1 “Missile Defense Agency Cost

Estimating Process Overview” pg.7

• GAO, Cost Estimating and Assessment Guide, 2009, Chapter 7 “Technical Baseline

Description Definition and Purpose” pg. 57

• NASA, Cost Estimating Handbook, 2015, para. 2.1 “Project Definition Tasks”, pg. 4

Program Definition Training

The DAU Cost Estimating certification program for members of the Defense Acquisition Workforce offers

training relevant to the cost estimate program definition. Additional information on each course may be

found in the DAU iCatalog (

https://icatalog.dau.edu/).

• BCF 130 Fundamentals of Cost Analysis, Lesson 2

• BCF 132 Applied Cost Analysis, Lesson 1

• BCF 216 Applied Operating and Support Cost Analysis, Lesson 2

• BCF 230 Intermediate Cost Analysis, Lesson 1

• BCF 250 Applied Software Cost Estimating, Lesson 4

• BCF 331 Advanced Concepts in Cost Analysis, Lesson 1

• CLM 013 Work-Breakdown Structure (addresses the program and the contract WBS)

The ICEAA publishes the CEBoK. The follow modules are relevant to program definition:

• CEBoK v1.2, 2013, Module 2 “Cost Estimating Basics”

The following course numbers starting with FMF refer to the course number assigned by the FM

Certification process. Information on these courses (including eligibility requirements) can be found in

the FM myLearn system:

https://fmonline.ousdc.osd.mil/FMmyLearn/Default.aspx.

• FMF 4898 ADM 300 - Work Breakdown Structure Review

• FMF 6175 AFIT Cost 669 - Advanced Cost Analysis

• FMF 1550 QMT 290 - Integrated Cost Analysis

• FMF 1551 QMT 490 - Current Topics in Cost Estimating

4.0 COST ESTIMATE BASIS

The cost estimate basis is the

estimate purpose, scope, schedule,

and the framing assumptions, ground

rules, and cost estimating

assumptions. This step in the

estimating process builds on the

program definition and establishes

the basis for the data collection,

estimating method development, and

cost model building. The more

thought and planning performed at

this stage of the cost estimating

process, the more efficient and

successful the remaining steps.

Developing a cost estimate can be a

major effort, and it demands the

attention of experienced, professional

analysts. The cost analysis team must cope with a great deal of uncertainty because the products

and/or services they are estimating may not be precisely defined. Framing assumptions, ground rules,

cost estimate assumptions along with an interpretation of requirements and data bound the estimate.

To successfully navigate, define, and apply them, the analyst team must possess a variety of skills. The

overarching reality is that a quality cost estimate requires significant time, resources, and planning. The

analyst uses the cost estimate basis to substantiate and defend the cost estimate during reviews and

reconciliation sessions.

Cost Estimate Plan

A cost estimate plan organizes the estimators and stakeholders around the purpose, scope, structure,

and schedule of the cost estimate. The analyst should focus this plan on a list of scheduled events that

he/she needs to accomplish to complete the estimate, along with the anticipated timeline to finalize and

deliver the cost estimate. The amount of detail and rigor in these plans varies depending on the number

of people, organizations, and stakeholders involved, as well as the size and complexity of the cost

estimate scope. For example, an ACAT ID milestone POE will likely require a bigger team and more time

than an ACAT III sufficiency review.

The larger the cost estimate team, the more detail the cost estimate plan should include to ensure

everyone is working towards common goals. While developing this plan, all organizations that have a

vested interest in the cost estimate – the stakeholders – need to be identified with their roles and

responsibilities to prevent confusion regarding who is involved and why. For larger programs it is often

a good practice to have the cost estimate plan signed by the program manager, and potentially other

stakeholders, to validate that the plan has been vetted and accepted by the program office and is being

used as the basis for collecting data and developing the cost estimate. CAPE and the Component Cost

Agency may build their own plan, but in many cases they will want to review the program cost estimate

plan for completeness.

Table 2 provides a summary of the information that should be included in a cost estimate plan.

Table 2: Information to Include in a Cost Estimate Plan

Content

Rationale

Policy and procedures References the policies and procedures that drive the cost estimate

and the process used.

Purpose and Scope Provides the reader with an understanding of why the cost estimate

is required and to whom it will be delivered. The scope defines the

boundaries of what is or is not explicitly included in the estimate.

This includes identifying the level of detail required to support every

element in the program, all anticipated what-if excursions, and all

reports.

Define the Estimate

Structure

An estimate structure provides context to the cost estimate and

supports the variety of cost analysis anticipated to deliver the all the

required results. Providing a copy of the estimate structure to Level

2 or Level 3 is helpful. At this level of detail, the estimate structure

should match the program WBS.

Process / Approach Provides a general overview of the process and steps taken to

complete the cost estimate. The analyst should have built a

rudimentary estimate structure and be considering/documenting

estimating methodology options to influence the listing of

desired/required data and data collection efforts. It is also

important for engineers and other SMEs to gain an understanding of

why an analyst is requesting specific data.

Team Members and

Assignments

Include the name, organization, phone number, and email address

for both the team of analysts and the program office. For larger

programs and estimates, it is important to also point out the

responsibility of each team member. If team members require Non-

Disclosure Agreements (NDAs) to accomplish their assignments, this

should be noted.

Travel Many cost estimates require travel to government or industry sites

to collect data and meet with SMEs. This section should detail the

travel dates, locations, and purpose of each trip.

Schedule Defines the timeline for the estimate to be completed, to include

important meeting dates (e.g., kickoff meeting), data collection(s),

draft version dates, review cycles, final delivery dates, and the dates

the documentation will be provided.

Although a cost estimate plan is a living document, the cost estimate team should keep it under

configuration control and change it only with agreement from the stakeholders. Appendix 7 of the 2020

Department of the Army Cost Analysis Manual provides an example of the sorts of documentation

captures in a cost estimate plan. To establish consistency in content and use, guidance on how formal

these plans need to be and their review/approval process should be promulgated by the applicable

authority.

4.1.1 Establishing the Purpose and Scope

The purpose of the cost estimate should be a clear and concise statement that defines the intended use

of the cost estimate. There are various purposes for a cost estimate, including: developing a budget

quality estimate, supporting a POM process, supporting an acquisition milestone decision, performing

an AoA, investigating cost vs. capability trades, conducting an NPV analysis, participating in proposal

evaluation, and conducting a PS BCA, among others.

The scope of the estimate identifies the level of detail required to support not only all elements in the

program, but all anticipated what-if excursions and reports. The scope also defines the boundaries of

what is or is not explicitly included in the estimate being performed. For example, the program manager

may decide that the program includes the cost of a ship but not any additional craft required for security

when that ship is in port. Or, during an AoA, the stakeholders may all agree that cost elements that

remain the same between the different alternatives will not be included in the cost estimate. The

purpose and scope drives the cost estimate schedule and the resources required to complete the

remaining steps of the cost estimating process. The stakeholders who play a role in the development,

review, and the ultimate use of the cost estimate should agree with the estimate purpose and scope.

4.1.2 Define the Estimate Structure

The program manager approves a program WBS as part of the program definition. Once the

stakeholders approve the purpose and scope of the cost estimate, the analyst modifies and/or expands

the program WBS to support the desired cost estimating results. The result is an initial estimate

structure. It is an initial estimate structure because additional detail or different structure may become

apparent as the estimate progresses. The estimate structure may address a complete and detailed life-

cycle cost estimate, e.g., a POE, or be limited to a subset of program scope. For example, a program

completing the Technology Maturation and Risk Reduction (TMRR) phase and working towards a

Milestone B review requires an estimate structure that covers all program cost. In contrast, a program

in the Materiel Solution Analysis (MSA) phase might require an estimate structure that supports an AoA.

In this scenario, only the portions of the estimate structure that highlight the differences among

alternatives are useful. Scopes of work that are assumed to be common among alternatives are often

removed from the AoA study via an agreed upon ground rule and are not included in an MSA cost

estimate.

If an element of the estimate structure is not a sub-element to any program WBS element, it should

remain closely aligned to the current DoD MIL-STD-881 for acquisition elements and the CAPE O&S CES

for O&S elements. Some organizations use Component specific guidance to augment these resources.

For example, the Naval Sea Systems Command (NAVSEA) uses the Expanded WBS Weight Classification

Guidance

, which defines the Expanded Ship WBS (ESWBS). The shipbuilding industry uses ESWBS to

further delineate the scope of work associated with a shipbuilding program. When shipyards and

government program offices use ESWBS as their organizing construct, it improves clarity and facilitates

discussions when the ship analyst adopts it as well. As mentioned in Section 3.1.3, there is no O&S

WBS, but the 2014 CAPE O&S Cost-Estimating Guide provides an O&S CES.

4.1.3 Creating a Cost Estimate Schedule

Once the stakeholders define the purpose and scope of work, the analyst should develop a resource-

loaded schedule

to provide a plan for completing the work. This plan should consider the timeframe in

https://www.sawe.org/files/SAWE%20ESWBS%20RP%2003042011.pdf

Referred to as a Plan of Actions and Milestones (POAM or POA&M) in some contexts.

which the cost estimate is required

, the types of results needed, and the format(s) in which the analyst

needs to provide them. The cost estimate schedule must plan adequate time to complete all steps of

the cost estimating process.

Although it is not necessary to develop a logically linked schedule in a scheduling tool (such as Microsoft

Project or Primavera®), the schedule should provide a sequential set of steps that need to be completed,

many of them iteratively, and identify the resources required. It should include key meetings, dates

when key deliverables are provided (with adequate time for draft reviews), and define the timeline for

completion of the cost estimate. The analyst may “reverse engineer” the schedule dates based on the

desired end state (e.g., the date a program must submit documents to a Milestone C review panel). The

schedule should be vetted with stakeholders for adequacy and availability of resources (both

government and industry) to support it. Once finalized, the schedule is an important component of a

cost estimate plan.

Framing Assumptions, Ground Rules, and Cost Estimate Assumptions

The second part of the cost estimate basis is the framing assumptions, ground rules, and cost estimate

assumptions. The analyst needs to have a clear understanding of each of these and ensure he/she

captures them in the cost estimate documentation. The remainder of this section discusses the

differences among them and importance of each towards the goal of developing a credible and

defendable cost estimate.

4.2.1 Framing Assumptions

A framing assumption is any supposition (explicit or implicit) that could significantly shape cost,

schedule, or performance expectations of the program. The program manager is responsible for

developing the framing assumptions. The concept was introduced by PARCA (now named AAP) in 2012,

when analyzing root causes of Nunn-McCurdy program breaches. PARCA identified false assumptions as

a cause of significant cost growth in some programs, which led to the definition of framing assumptions.

DoDI 5000.02T mentions framing assumptions in the context of acquisition and states that the program

manager is required to present them at Milestone A, Development RFP Release Decision, Milestone B,

and in acquisition strategies. The principles of framing assumptions are applicable to any cost estimate.

In general, there should be a small number (optimally 3-5, but circumstance dependent) of framing

assumptions with the following attributes:

• Critical: Significantly affects program expectations for cost, schedule, or performance.

• No work-arounds: Consequences cannot be easily mitigated.

• Foundational: Not derivative of other assumptions.

• Program specific: Not generically applicable to all programs.

Some sources of framing assumptions include:

• technological and engineering challenges,

• cost, schedule, and requirements trade-offs,

• effectiveness of program-specific managerial or organizational structures (particularly for

joint or combined programs),

• suitability of contractual terms and incentives to deliver specific expected outcomes,

• interdependencies with other programs, and/or

DoDI 5000.73 outlines the timelines for the preparation of a ACAT ID ICE, ACAT IC cost estimate review, MYP

contract cost analysis, cost analysis of Critical Nunn-McCurdy Breach, and others.

• industrial base, market, or political considerations.

Framing assumption examples include:

• legacy performance requirements are adequate for this system,

• threat levels will not significantly change in the next X years,

• requirements will be relaxed as necessary to achieve cost and schedule goals,

• development of X technology will achieve required performance levels, or

• COTS items can be easily integrated and significantly reduce cost.

Framing assumptions are typically a part of program documentation and contained within the program

definition. (See the 2013 PARCA Information Paper on Framing Assumptions at:

https://www.acq.osd.mil/aap/assets/docs/2013-09-13-information-paper-framing-assumptions.pdf,

and DAU, Developing Framing Assumptions (FAs) Job Support Tools (JST) at:

https://www.dau.edu/tools/Lists/DAUTools/Attachments/160/JST_FAs.pdf for more detail.)

4.2.2 Ground Rules

Ground rules represent a common understanding regarding the program that the analyst should not

question or change unless the program office makes formal changes to the program. Ground rules are

different from framing assumptions (Section 4.2.1) in that ground rules characterize the program while

framing assumptions describe an environment within which the program must perform or face

significant problems. The CARD should document the ground rules that are important to the program

office and stakeholders. Ground rules provide a common understanding for activities, constraints,

events, or other concerns that have a major influence on program cost, schedule, and performance.

They may include scheduled events, budget constraints, involve Government Furnished Equipment

(GFE) / Government Furnished Information (GFI), or anything else that may have a major influence but is

open to interpretation. Information commonly addressed in ground rules include:

• boundaries of the program/estimate,

• a production profile for the system,

• the CY for which the cost estimate will be reported,

• how recurring and nonrecurring effort is segregated,

• the expected age or life cycle of an individual platform,

• the year in which a program completes IOC and transitions into sustainment,

• the maintenance approach to maintaining a platform,

• scopes of work that are not included in an estimate (used in AoAs and similar studies),

• how to report sunk cost in a life-cycle cost estimate, and/or

• the discount rate used to conduct NPV / Return on Investment (ROI) calculations (provided

in OMB Circular A-94).

It is important for the analyst to remember that the program manager and his/her technical experts

create the program’s ground rules, not the analyst.

4.2.3 Cost Estimate Assumptions

Separate and distinct from the program definition and framing assumptions developed by the program

manager and ground rules approved by all stakeholders, the analyst develops assumptions to bridge any

gaps resulting from incomplete information. Cost estimate assumptions are never arbitrary, and all

stakeholders should review and understand them. The most important assumptions are often the ones

the analyst makes when there is no ground rule. For example, in the early stages of a program,

decisions regarding the service life of a platform may be unknown. If not provided as a ground rule, an

assumption is required to establish the number of years the platform will be in service, and that is used

as a basis for estimating O&S and disposal cost. Examples of topics often requiring an assumption

include:

• the degree of overlap between the Research and Development (R&D), Production, O&S,

and Disposal phases,

• inflation and escalation rates used to normalize the cost estimate (if not a ground rule),

• where the production units are manufactured or if a production line is shared,

• process/plan disruptions,

• the amount of existing software that will be reused for a new application or purpose,

• the expectation of facility upgrades,

• operating hours per system,

• how a contractor’s accounting cost is allocated across elements of the estimate structure,

• the cost and schedule impacts of Foreign Military Sales (FMS).

The analyst must carefully think through assumptions, as they have a significant impact on the steps that

follow, particularly how to build the cost model and address risk/opportunity and uncertainty.

Documentation of the Cost Estimate Basis

A completed cost estimate includes documentation of its results as well as the process followed to

achieve those results. At this point in the process, the cost estimate basis needs to be clearly defined

and documented. The complete estimate documentation is easier to build when the cost team starts

constructing it upfront and keeps it updated throughout the cost estimating process. As with all the

estimate documentation, the cost estimate basis should be constantly updated, but under a reasonable

level of configuration management.

Cost Estimate Basis References

• DoDI 5000.02T, Operation of the Defense Acquisition System, 2020, Enclosure 10, para. 2

“Cost Estimation”, pg. 132

• DoDI 5000.73, Cost Analysis Guidance and Procedures, 2020, Section 3 “Cost Estimate

Requirements and Procedures”, pg. 6

• CAPE, Operating and Support Cost-Estimating Guide, 2014, para. 5.2.1, “Establish Ground

Rules and Assumptions”, pg. 5-2

• NCCA/AFCAA, Software Development Cost Estimating Handbook, 2008, para. 3.4

“Estimating Process”, pg. 3-6

• Department of the Army Cost Analysis Manual, 2020, Chap 3 “Cost Estimating Process”, pg.

9 and Appendix 7 “Example Documentation”, pg. 81

• NCCA, Joint Agency Cost Estimating Relationship (CER) Development Handbook (JA CER

Handbook), 2018, para. 1.3 “Cost Estimate Purpose and Scope”, pg. 14

• NCCA, Joint Agency Cost Schedule Risk and Uncertainty Handbook, 2014, para. 2.1,

“Strategic Approach”, pg. 6

• NCCA, Cost Estimating Guide, 2010, para. 1.1 “Establish Needs with Stakeholders”, pg. 11

• USMC, Cost Analysis Guidebook, 2017, para. 3.0 “Establish Needs with Stakeholders”, pg. 39

and para. 3.1 “Establish a Program Baseline” pg. 40

• SPAWAR, Inst 7110.1 Cost Estimating and Analysis, 2016, Enclosure 1, para. 3 “Establish

Needs with the Customer” pg. 3

• AFCAA, AFI 65-508, 2018, para. 2.1 “Cost Estimate Types and Expectations” pg. 6

• AFCAA, Cost Analysis Handbook, 2008, para. 5-1 “Understand the Purpose of the Estimate”,

pg. 5.3

• Missile Defense Agency Cost Estimating and Analysis Handbook, 2012, Chapter 2

“Documenting Ground Rules and Assumptions” pg.23

• GAO, Cost Estimating and Assessment Guide, 2009, Chapter 5 “The Cost Estimate’s Purpose,

Scope, and Schedule” pg. 47

• NASA, Cost Estimating Handbook, 2015, Chapter 2 “The Cost Estimating Process”, pg. 22

Cost Estimate Basis Training

The DAU Cost Estimating certification program for members of the Defense Acquisition Workforce offers

training relevant to the cost estimate basis. Additional information on each course may be found in the

DAU iCatalog (

https://icatalog.dau.edu/).

• BCF 130 Fundamentals of Cost Analysis, Lesson 2

• BCF 216 Applied Operating and Support Cost Analysis, Lesson 3

• BCF 230 Intermediate Cost Analysis, Lesson 2

• BCF 250 Applied Software Cost Estimating, Lesson 2

• Continuous Learning, Engineering (CLE) 021 Technology Readiness Assessments (enable

participation in a Technology Readiness Assessment and to determine how to use the TRA

process to enhance program success)

The following course numbers starting with FMF refer to the course number assigned by the FM

Certification process. Information on these courses (including eligibility requirements) can be found in

the FM myLearn system:

https://fmonline.ousdc.osd.mil/FMmyLearn/Default.aspx.

• FMF 1550 QMT 290 - Integrated Cost Analysis

• FMF 1560 DoD FM 101 - Cost Analysis

• FMF 6175 AFIT Cost 669 - Advanced Cost Analysis

• FMF 1551 QMT 490 - Current Topics in Cost Estimating

5.0 IDENTIFY, COLLECT, VALIDATE, NORMALIZE, AND ANALYZE DATA

The core of a quality cost estimate is

defendable, credible, and relevant

data. The best cost estimating

methods are those that rely on

credible and reliable data. For each

cost element within the estimate,

the analyst must identify and use the

best data available. Data needs are

not always clear at the assignment’s

beginning, and data requirements

often evolve during an estimate’s

development. This makes data

collection one of the most difficult,

time-consuming, and costly activities

in cost estimating.

The relevance, currency, and quality

of the data defines its usefulness to

the cost estimate. A small mistake in

the interpretation, analysis, and application of imprecise or irrelevant data can lead to a large error in

the estimate results. Data collection is a top priority for analysts.

The DoD cost estimating process graphic highlights the importance of data by placing it in the center,

influencing and being influenced by every step in the process. The availability and usefulness of data

has a significant influence on the remaining cost estimating steps. The data step in the cost estimating

process includes collection, validation, and normalization processes, which all rely on a strong

foundation built by the program definition and cost estimate basis. The program definition and cost

estimate basis drive the data source identification and collection process. The focus of finding and

collecting data should target the greatest program cost contributors and the cost drivers that have the

most influence on total cost.

This chapter provides guidance on the types of data, where to find that data, how to collect it, and how

to validate it. This chapter also introduces the data normalization process and data analysis techniques

that support the cost estimating process.

Characterizing Data

Data is either quantitative or qualitative. Both quantitative and qualitative data is also either objective

or subjective. Relevant, accurate, and objective quantitative data is the most useful, but subjective,

qualitative data may also provide valuable context for the cost estimate.

• Quantitative data are measures of values or counts and are expressed as numbers.

Weight, power, labor rates, quantities, and rate of production are all examples of

quantitative data.

• Qualitative data approximates and characterizes the item(s) of interest. SMEs illuminate

essential details not immediately apparent in the objective quantitative data. Analysts

collect qualitative data through one-to-one interviews, focus group meetings, and similar

methods. An example of qualitative data are descriptions of how the program of interest

compares to others by describing relative measures of complexity, production efficiencies,

differences in resource capabilities, and identifying programs that are “similar”.

• Objective data is an observable or measurable fact and comes with a pedigree, a well-

documented source. Facts are without bias and rely on relevant, accurate, and actual

historical data. Cost analyses become meaningless if the data behind them are incomplete,

irrelevant, or simply wrong. An analyst should invest time to find objective data sources.

When an analyst learns near the end of the cost estimating process that a source of

objective data was in fact available, but missed, it can impede a good estimating outcome

and the approval process. An example of objective, quantitative data is the weight of an

existing item. A description of implemented production line improvements is objective,

qualitative data.

• Subjective data originates from sound judgment and expert opinion. While objective data

is preferred, subjective data is often necessary. This speaks to the art of cost analysis being

every bit as important as the science. Acknowledging that the available objective data is

not useful or misleading might lead the analyst to rely upon subjective data to fill a void. A

SME opinion that the new product will be half the cost of the previous one is subjective,

quantitative data. A production manager predicting that planned upgrades to the facility

will deliver a moderate improvement in efficiency is an example of subjective, qualitative

data. Analysts should understand that subjective data has the potential for many forms of

bias. The 2014 JA CSRUH para. 2.5.2 “Elicitation of Subjective Bounds from Subject Matter

Experts” provides an overview of the most common biases and techniques to mitigate

them.

There is a distinction between primary and secondary data as shown in Table 10 of the 2009 GAO Cost

Estimating and Assessment Guide. Primary data is generally of higher pedigree than the secondary data,

as follows:

• Primary: Data collected from the original source such as the contractor accounting system.

• Secondary: Data derived, and possibly computed, from primary data e.g., $/lb.

Primary data is preferred during data collection so that the analyst does not inherit unknown derivations

or biases of a secondary data set. If only secondary data is available, then the analyst should ensure that

the cost team understands any derivations to the greatest extent possible.

Data Types

There is a variety of data types available to produce a quality cost estimate. Cost is just one type of data

the analyst must collect for a complete dataset. As with the program system description, the analyst

should obtain programmatic, performance, technical, and schedule data from the historical programs on

which many cost estimating methodologies are based. The remainder of this section describes the types

of data to be collected.

5.2.1 Cost Data

Cost data reflects monetary expenditures incurred on past or present systems. Cost data is best

explained in the context of life-cycle cost that includes the top level cost categories, or phases of the

system life cycle: R&D, Production, O&S, and Disposal. Each of these categories can be further

categorized as

For more detail, see DoDM 5000.04 CSDR Manual

https://cade.osd.mil/content/cade/files/csdr/guidance/DoDM%205000.04-M-1%20CSDR%20Manual.pdf

• Recurring: Repetitive elements of R&D, Production, O&S, or Disposal that generally vary

with the quantity being produced or maintained. Examples: fabrication, assembly, touch

labor, installation, check out, and preventative maintenance.

• Non-recurring: Non-repetitive elements of R&D, Production, O&S, or Disposal that do not

vary with the quantity being produced or maintained. Examples: definition, design,

acceptance testing, and establishing a facility.

Analysts further subdivide recurring and non-recurring costs into subcategories such as labor, material,

overhead, and fee. These subcategories are where the analyst is likely to find cost data.

It is also important to subdivide cost into time-sensitive and not-time-sensitive categories. Depending

on when the analysts performs the cost estimate, the estimate may include both the cost incurred to

date on the program and future costs. Costs incurred to date on the program are sunk cost and should

be part of the data collection effort.

5.2.2 Programmatic Data

Programmatic data describes overarching characteristics of the program. Examples of programmatic

data include: program WBS and/or O&S CES allocations (accounting), requirements growth, delay and

disruptions, accounting system changes (prior to or concurrent with production), different production

rates, and inflation/escalation. Each Component has developed cost guides that provide examples of

programmatic characteristics unique to their environment that an analyst should capture during data

collection to provide context and influence how to interpret the cost data. Programmatic data can have

a direct and significant influence on the recorded cost data.

Programmatic data can be quantitative or qualitative. Analysts, or more likely automated systems,

measure and record quantitative programmatic data (e.g., timekeeping systems, production line

instrumentation, integrated accounting systems, onboard measuring instruments) as numeric values

such as hours by labor category, quantities, production rates, purchasing, or fuel consumption.

Qualitative programmatic data is descriptive rather than numeric (e.g., contract type, competition

approach, heritage

, and maintenance concept). Though direct use of qualitative programmatic data in

a cost estimating model may not be immediately obvious, the context in which past costs have been

incurred is an essential part of the full picture.

5.2.3 Performance and Technical Data

Performance data describes what the systems can/must do. Technical data describes physical and

functional characteristics of the system. Speed, range, depth, survivability, and noise reduction are

examples of performance characteristic data. Size, weight, and power (SWaP) are examples of technical

characteristic data. Source lines of code (SLOC), function points, and story points are examples of

software technical data.

5.2.4 Schedule Data

Schedule data describes activities and activity interdependencies that control or influence the progress

on a program. Schedule dependencies and interactions between development, production, and

software modifications/upgrades are just a few of the issues that could significantly influence a system

schedule and therefore, the cost estimate. A well-developed schedule helps identify important handoffs

between participants in a program. It also provides a frame of reference for the analyst to work with

the scheduler to build resource loaded schedule. (See the 2015 GAO Schedule Assessment Guide, best

Examples of heritage data are percent new design, number of new or modified drawings, and Technology

Readiness Level (TRL).

practice 3 “Assigning Resources to Activities” for guidance on how to assign resources to a schedule.)

Durations of key processes (e.g., development, final design, production, trials) help add context to the

cost collected from the program. The top levels in the schedule should always be consistent with the

program WBS and the O&S CES to facilitate mapping schedule data to the cost model.

Data Sensitive to Duration or Quantity

An important distinction to understand when collecting data is if the data are sensitive to time or

quantity. This differs from the recurring and non-recurring data distinctions described in Section 5.2.1.

Cost can be sensitive to:

• Quantity: Where cost is a function of how many items are produced annually and in some

cases the rate at which they need to be produced.

• Duration: Where cost is a function of calendar or work days, weeks, months, years or some

other measure of time. For example, level-of-effort activity is sensitive to the number of

work weeks a given team is required to be on the program.

• Neither Duration nor Quantity: Where cost is influenced by neither duration nor quantity.

For example, the price of a facility may be the same regardless when the sale occurs.

Duration is a useful parameter to obtain in any data collection. Even if duration is not used directly in

the estimating method knowing that the estimating method was based on programs with an average

duration of X months and is to be applied to a program anticipated to run Y months provides a basis to

reconsider adjusting the estimating method for duration. (See Chapter 6.0 for estimating methods.)

Identify Data

There are a variety of sources that provide quality data on historical and current programs. Table 3

provides a generic summary of potential data sources.

Table 3: Data Types and Generic Sources (not exhaustive)

Data Type

Data Elements

Potential Sources

Cost

Historical Costs

Basic Accounting Records

Labor Costs

Cost Reports

Material Costs

CADE

Fee, Overhead

EVM Central Repository (EVM-CR)

Pricing Costs

Contracts and Proposals (Secondary)

Programmatic

Development and Production Schedules

CARD

Quantities Produced

Program Database

Production Rates, Breaks in Production

Functional Organizations

Significant Design Changes

Program Management Plan

Major Subcontractors

Performance/

Technical

Physical Characteristics

CARD

Performance Characteristics

Technical Databases

Performance Metrics

Engineering Specifications/Drawings

Technology Descriptors

Performance/Functional Specifications

Major Design Changes

Functional Specialist

Operational Environment

End User and Operators

Master Equipment Lists

Schedule

Start/End Dates

Integrated Master Schedule

Schedule Dependencies

CADE and EVM-CR

The remainder of this section discusses more specific data sources available to the analyst.

5.4.1 Data Repositories

DoD and the Components have established useful collections and databases where analysts can obtain

authoritative and curated data. These collections of documents and databases provide tremendous

potential for an analyst to identify the data required for a cost estimate. Many of these sources have

limited access in order to protect sensitive data. Analysts may need to apply for accounts to these

systems and declare their need for access to the data. Non-government personnel may need to go one

step further and prove they are supporting a government effort.

One of the largest data repositories in the DoD is CADE. CADE is a DoD initiative for collecting,

organizing, and displaying data in an integrated web-based application. CADE supports the search for

authoritative data by providing DoD employees access to a large amount of raw component/agency

acquisition and O&S data. This expanding compendium of data includes historical cost, technical,

programmatic, and contractual data across numerous ACAT I and II programs, information systems, and

some BCATs. Government analysts across the DoD are encouraged to take advantage of CADE by

obtaining accounts and accessing the system regularly to determine if data sources exist within CADE

that improve their cost estimates. Two of the primary data sources within CADE are:

• Contractor Cost Data Reporting (CCDR): Used by contractors to report all costs associated

with the contract. (See

https://cade.osd.mil/content/cade/files/csdr/guidance/DoDM%205000.04-M-

1%20CSDR%20Manual.pdf for more detail.)

• Software Resources Data Report (SRDR): Used by contractors to report all technical and

cost data on software development, software maintenance, and Enterprise Resource

Planning (ERP) development efforts. (See the SRDR Implementation Guidance

https://cade.osd.mil/content/cade/files/csdr/guidance/SRDR%20Implementation%20Guide_2019.02

.01.pdf

for more detail.)

The CCDRs and SRDRs are the primary means by which the DoD collects data on the costs that

contractors incur on DoD programs. Policies including DoDI 5000.73 and DoDM 5000.04 establish the

requirements for these two specific reports. The CADE website (https://cade.osd.mil/about/cade) provides

more information

. The FlexFile report and Quantity Data Report

are the default the cost reporting

requirements for new programs. The core of the FlexFile delivers time-phased dollars and hours at the

account level in contractor native categories. The Quantity Data Report ties the necessary quantity

information to the FlexFile. These files can be very large. (See

https://cade.osd.mil/policy/flexfile-quantity

for more detail.) Table 4 lists additional data sets and analysis options within CADE.

Another example of a collection of identified data sources is the EVM-CR, which the EVM division of AAP

manages. The EVM-CR establishes a source of authoritative EVM data for the DoD. ACAT programs with

contractual EVM reporting requirements submit their EVM data in the form of Integrated Program

Management Reports (IPMRs) to the EVM-CR. (See

https://www.acq.osd.mil/evm/#/home for more detail.)

Contracts that do not meet the EVM reporting thresholds submit EVM data as determined by their CAE,

typically reporting only directly to their program office or PEO.

On the legacy CCDR forms, quantity data were reported in tandem with cost data. Quantity data are now

reported separately from the cost data (FlexFile) as part of the Quantity Data Report.

Table 4: CADE Data

Name

URL

Synopsis

Defense Automated

Cost Information

Management System

https://service.cade.osd.mil/dacims

35/site/home.aspx

Second source of CSDRs. Contains historical

files back to 1966, and various 1921 forms

Data & Analytics

Program Search

https://reporting.cade.osd.mil/cade/

Site/FavoritePrograms.aspx

Search by Program, Contract, Plan, or

Submission for CSDRs

Data & Analytics

Cross Program Query

https://reporting.cade.osd.mil/cade/

Site/Queries/CrossProgramQueryHo

me.aspx

Allows search across multiple programs to

facilitate export of specific applicable data

Contract Database

https://service.cade.osd.mil/csdrsr/S

ite/Contracts/SearchContracts.aspx

Search by Service, Commodity, Contractor,

Plan Number, Program Manager, or

Submitter/Reviewer

1921-3 & Forward

Pricing Rates (FPR)

Browse

Submit-Review

https://reporting.cade.osd.mil/cade/

Site/FPRSRBrowse.aspx

Search by Submission ID, Contractor, Date

Range, and Reporting status to review

Contractor Business Base Data Reports (1921-

3) and Forward Pricing Rate Agreements

(FPRA) by Contractor

Enterprise Visibility and

Management of

Operating and Support

Cost (eVAMOSC)

https://service.cade.osd.mil/cade/Si

te/Tools.aspx

Provides a common user interface to search

each Component Visibility and Management

of Operating and Support Costs (VAMOSC)

system

CADE Cost Community

Library

https://reporting.cade.osd.mil/cade/

Site/Library.aspx

Various supporting documentation for

specific programs to include CARDs, ICEs,

technical data, etc.

Each Military Department has implemented robust data collection of O&S costs and related operational

data under the umbrella of the DoD VAMOSC program. The specific VAMOSC databases are:

• Department of the Army: The Operating and Support Management Information System

(OSMIS) contains reparable and consumable costs for selected tactical systems by major

command. The Army Military-Civilian Cost System (AMCOS) provides personnel cost factors

for estimating acquisition, installation operations, and force/unit requirements. AMCOS is

particularly useful for the development of the training mission. (See

https://www.osmisweb.army.mil/ for more detail.)

• DON: The Naval VAMOSC management information system collects and reports US Navy

and Marine Corps historical direct O&S costs of weapon systems, some linked indirect costs

(e.g., ship depot overhead), flying hour metrics, steaming hours, age of aircraft, etc. The

VAMOSC Military Personnel databases contain personnel costs and attribute data. (See

https://www.vamosc.navy.mil/ )

• Department of the Air Force: The Air Force Total Ownership Cost (AFTOC) database serves

to acquire, normalize, aggregate, allocate, and organize financial and logistic data. AFTOC

satisfies the need to provide a single source of authoritative, processed financial and

DCAPE intends to provide the ability to compare O&S data across DoD Components by using a common O&S

WBS and host the data as “eVAMOSC”.

logistics data organized by system or infrastructure. (See https://aftoc.hill.af.mil/ for more

detail.)

Additional Component-level repositories are available, but details of those repositories are left to

Component-level guidance.

A list of data repositories managed at the DoD level is shown in Table 5.

Table 5: DoD-level Data Repositories

Name

URL

Synopsis

ADVANA

https://audit.usmc.mil/#/landing/ab

out

Leverage leading edge analytics to deliver

business value

CADE https://cade.osd.mil

Authoritative source of all cost, software, and

technical data

Collaborative Cost

Research Library (CCRL)

System

https://www.ncca.navy.mil/library/li

brary.cfm

Cost analysis publications including technical

documentation, briefings, ICEs, CCEs, CARDs,

service cost positions, etc.

Contract Business

Analysis Repository

(CBAR)

https://www.dcma.mil/WBT/CBAR/

DoD Procurement Contracting Officer (PCO)

access to Defense Contract Management

Agency (DCMA) contract-related company

information

Defense Acquisition

Management

Information Retrieval

(DAMIR)

https://www.acq.osd.mil/damir/

Enterprise visibility to Acquisition program

information

Defense Acquisition

Visibility Environment

(DAVE)

https://dave.acq.osd.mil/login

Accurate, authoritative, and reliable data

supporting acquisition oversight, insight,

analysis, and decision-making

Defense Finance and

Accounting Service

(DFAS) Electronic

Document Access (EDA)

https://www.dfas.mil/contractorsve

ndors/irapt/eda.html

Secure online access, storage, and retrieval of

contracts, contract modifications,

Government Bills of Lading (GBLs), DFAS

Transactions for Others (E110), vouchers, and

Contract Deficiency Reports

Defense Technical

Information Center

(DTIC)

https://discover.dtic.mil/

Science and technology data to support

development of the next generation of

technologies for our Warfighters

EVM-CR

https://www.acq.osd.mil/evm/#/ab

out-evm-cr

Authoritative EVM data for DoD

Maintenance and

Availability Data

Warehouse (MADW)

https://madw.acq.osd.mil

Weapon system and readiness reportable

equipment availability, cost, inventory, and

transactional maintenance data

Analysts must fully understand the limitations of any data repository, including the intended purpose of

the repository and how the data was collected, normalized, and/or presented for user consumption.

The repository's data dictionary and/or user guide should provide this type of information.

5.4.2 Deliverables and Reports

DoD programs routinely prepare business and engineering products to organize information and guide

staff towards successful project completion. For cost estimating purposes, these artifacts are rich with

programmatic, performance, technical, and schedule data. There is a variety of specific government and

industry products that analysts can search for during data discovery.

Required acquisition documents can provide a wealth of information for an analyst. DoDI 5000.02T,

Enclosure 1, Table 3 lists all of the statutory and regulatory documents required for an acquisition

program, including the timing of the various documents. Table 6 uses that list to highlight possible data

sources. These documents may be a data source for both the system being estimated and historical

systems. Given the number and variety of reports Program Offices/Industry are required/contracted to

produce and deliver, analysts should research to determine whether desired data and information is

already available through established sources before initiating requests which duplicate existing

requirements.

Table 6: Potential Data Available in Required Acquisition Documents

Acquisition Documents

Cost

Programmatic

Performance

Technical

Schedule

2366a Written Determination

2366b Certification and Determination

Acquisition Decision Memorandum (ADM)

X X

Acquisition Program Baseline (APB)

X X X X

Acquisition Plan (AP)/Acquisition Strategy (AS)

Affordability Analysis

AoA

Bandwidth Requirements Review

Capability Development Document (CDD)

Capability Production Document (CPD)

CARD

CCE

CCP

Clinger-Cohen Act Compliance

X X

Concept of Operations (CONOPS)

Contract Data Requirements List (CDRL)

Core Logistics Determination/Sustaining

Workloads

X X X

Cybersecurity Strategy

Defense Intelligence Threat Library

Development RFP Release Cost Assessment

DoD Component Live Fire Test and Evaluation

Report

X X

Director, Operational Test and Evaluation

(DOT&E) Report on IOT&E

X X

X X X

Executive Order 12114 Compliance Schedule

Table 6: Required Acquisition Documents (continued)

Acquisition Documents (continued) Cost Programmatic Performance Technical Schedule

Exit Criteria

Frequency Allocation Application

Full Funding Certification Memorandum

ICE

Independent Logistics Assessment (ILA)

Information Support Plan (ISP)

IT and National Security Interoperability

Certification

Initial Capabilities Document (ICD)

X X

Item Unique Identification (IUID)

Implementation Plan

X X

Life-Cycle Mission Data Plan

Life-Cycle Sustainment Plan (LCSP)

Live Fire Test and Evaluation (LFT&E) Report

Low-Rate Initial Production (LRIP) Quantity

Operational Test Agency (OTA) Report of

Operational Test and Evaluation OT&E Results

X X

Operational Test Plan (OTP)

Post Implementation Review (PIR)

Preservation and Storage of Unique Tooling Plan

Program Protection Plan (PPP)

Replaced System Sustainment Plan

RFP

Should Cost Target

Spectrum Supportability Risk Assessment

Systems Engineering Plan (SEP)

Technology Readiness Assessment (TRA)

Technology Targeting Risk Assessment

Test & Evaluation Master Plan (TEMP)

Validated On-line Lifecycle Threat (VOLT) Report

Waveform Assessment Application

Table 7 lists additional government documents/reports that may provide data appropriate for

estimating. These are not required acquisition documents, but may support required acquisition

documents. They may be available for both the program being estimated and any identified analogous

systems.

Table 7: Potential Data Available in Identified Government Data Sources

Government Source Cost Programmatic Performance Technical Schedule

Contract Funds Status Report (CFSR)

Contracts

Contract History/Data (detailed)

CCDR

Defense Acquisition Executive Summary (DAES)

Deployment Plan/Beddown Plan

Depot Source of Repair (DSOR)

Detailed Test Execution Plans

EVM Reports

Failure Mode Effects and Criticality Analysis

(FMECA)

X X

Failure Reporting, Analysis, and Corrective

Action System (FRACAS)

X X

FPRAs

Integrated Logistics Support Plan (ILSP)

IPMR

Life-Cycle Management Plan (LCMP)

Manpower Estimates/Actuals

Performance Work Statement (PWS)

President’s Budget (PB)/Budget Estimate

Submission (BES)

Previous Cost Estimates

Resource Data Table (RDT) - Gov information

Risk Management Plan

Selected Acquisition Report (SAR)

Software Quality Report

SRDR

Spares Provisioning Report

Statement of Objectives/Work (SOO/SOW)

Store Technical and Mass Property Sheet

(STAMP)

Technical Requirements Description (TRD)

Table 8 lists documents/reports available from industry that may provide information relevant for a

analyst.

DON 2010 Cost Estimating Guide, para. 1.3.2.1; 2018 JA CER Handbook, para. 1.4; AFCAA Tabular Cost Analysis

Requirements Description (CARD) Sufficiency Review Handbook

Table 8: Industry Data Sources to Consider

Industry Source Cost Programmatic Performance Technical Schedule

Bill of Materials (BOM)/Parts List

Business Plans

Catalog Prices

Configuration Audit

Configuration Drawings

CCDR

Contract WBS (CWBS)

CSDR Technical Data Reports

Integrated Master Plan/Schedule (IMP/IMS)

Mass Properties (detailed)

Power Allocation Summary

Preliminary and Critical Design Review Reports

Proposals

RDT - contractor information

SWaP Reports

SRDR

Software Development/Sustainment Plan

Vendor Lists

The potential data sources listed and discussed in Section 5.4 are not an exhaustive list. Analysts should

always pursue additional sources appropriate for the specific subject matter being estimated.

Collect, Validate, Normalize, and Analyze Data

Although described as a logical sequence, an analyst is rarely able to perform the data collection,

validation, normalization, and analysis in a single pass. The process is typically ongoing and repeated

within the iterative estimating process. At any point, it can become apparent that the analyst needs to

revisit work performed in the previous step, or it could become clear that the data collected is unusable.

Consequently, it is common for an analyst to return to refine the cost estimate basis (Chapter 4.0) and

then search for other data sources. Sequential or not, the following sections describe the work to be

done to conduct this step of the estimating process.

5.5.1 Data Collection Plan

A data collection plan establishes the time and resources required specifically for data collection,

validation, normalization, and initial analysis. Analysts should recognize that the data collected provides

a primary source for modeling and/or analyses. Historical cost, technical, schedule, and other

programmatic data can/should be used to establish statistical parameters (e.g., measures of central

tendency, anticipated range of outcomes, parameter distribution, etc.) for modeling. The entire data

collection effort is a potentially difficult and time-consuming process. The analyst can make it more

efficient by thinking through and documenting a deliberate, systematic, and succinct plan to accomplish

the data collection goals. The analyst must adhere to the defined purpose of the collection effort and

exercise continuous discernment regarding data usefulness or it can quickly become unmanageable. For

DON 2010 CEG 1.3.2.1; 2018 JA CER Handbook, para. 1.4; AFCAA Tabular Cost Analysis Requirements Description

(CARD) Sufficiency Review Handbook

large programs with numerous cost elements and cost drivers, the amount of data to collect is

significant. Ensuring the data collected also supports the eventual estimate risk/opportunity and

uncertainty analysis adds to the complexity, effort, and amount of data to be collected. This leads back

to the importance of developing a data collection plan that maintains a focus on the largest cost

contributors and cost drivers. The plan should include alternative actions or paths for when data

collection and/or validation encounters dead-ends or useless data.

A data collection plan can treat these four levels of data collection sources sequentially: DoD level (e.g.,

CADE, EVM-CR), Component level (e.g., VAMOSC), program office, and industry. After each successive

data collection step, analysts are able to focus more narrowly on filling the holes. Therefore, an analyst

should start with CADE-housed and Component level data prior to approaching a program office.

Subsequently, the analyst should exhaust program office-housed data before approaching industry

partners. A clear and focused data request is extremely important because each party is busy fulfilling

their primary missions. At a minimum, a data collection plan:

• identifies the data required and where the focus should be, consistent with the purpose

and scope of the estimate,

• ensures that every cost element is covered,

• plans to capture time-phased data (e.g., monthly, quarterly), rather than just the total-at-

completion. Doing so will allow for more accurate inflation/escalation calculations and

analysis of the phasing profile,

• identifies the actions required to capture cost, programmatic, performance, technical, and

schedule data,

• recognizes that the types and quantity of data available evolve as a system progresses

through its life cycle,

• projects a data collection timeline to keep the estimating effort on track, and

• allows time for the inevitable need to iterate between the collection and validation phases.

5.5.2 Collecting Data

With a Data Collection Plan in place, an analyst can begin collecting the required data. Analysts handle

objective and subjective data collection in different ways, and analysts may need to conduct the

collection efforts more than once.

5.5.2.1 Objective Data Collection Activities

At a minimum, objective data collection activities include:

• Identify: Offices, organization, and points of contact.

• Collect Cost Data: Obtain costs (including labor hours), by cost account and accounting

period.

• Characterize Data: Identify which elements of the estimate structure are quantity and/or

time sensitive and which elements of the estimate structure are driven by one or more

other element(s). Characterization can also be related to situation (peace-time vs. war-

time) or other attributes that influence cost.

• Document the Phase and Recurring/Non-recurring: Identify the collected cost by life-cycle

phase and also as recurring or non-recurring.

• Allocate: For accounts that contribute to multiple products, allocate their costs to the

individual products. The program WBS and O&S CES dictionaries are key sources for

explanations of what is included and excluded. The 2018 MIL-STD-881D and the 2014 CAPE

O&S Cost-Estimating Guide provide definitions for individual elements. The program office

typically defines any elements in the dictionaries not included in these documents.

• Collect Cost Driver Data: Collect performance parameters (such as speed, range, depth,

stealth, and noise), technical parameters (such as size, weight, power, SLOC, frequency,

duration, quantities, production rates), and schedule parameters (such as start/finish dates

for phases and milestones), for each element of the estimate structure.

5.5.2.2 Subjective Data Collection

As mentioned in Section 5.1, the analyst may have to collect expert judgment from engineers,

managers, and other SMEs. Called elicitation, numerous biases influence this process. For instance, an

analyst may trace over-optimism both to cognitive biases, which are errors in the way the mind

processes information, and to organizational (motivational) pressures. SMEs base their predictions on

an assessment of their own capabilities, experiences, and expectations. The analyst can temper the

elicitation process by having a statistical analysis of relevant historical data on hand. Such data provides

a reality check that should have a positive influence on the SME’s intuitive view of the situation. An

analyst can often gauge SME input by asking for a range of answers vice a specific value. Section 5.6

recommends additional reading on elicitation and subjective biases. Appendix E is a sample form for

documenting SME information.

5.5.2.3 Data Collection Execution

Prior to any data collection, the analyst should understand and consider the proprietary, and possibly

classified, nature of the data to be collected. While individual data elements may themselves be

unclassified, at some level of aggregation they may become classified. It must be a priority to protect

the data and to handle it appropriately.

With data protection in mind, the analyst’s first round of data collection is the non-intrusive searching of

existing data housed within government repositories. These resources are preferred because the

analyst can query or browse them without imposing on others. Purposeful, efficient, and complete use

of these resources not only satisfies many of the data collection needs but also allows the analyst to

better focus on subsequent steps. While program office data is necessary and critical to the cost

estimate, the time required to respond to data requests can become burdensome.

Government analysts visiting program office sites should request and expect to obtain access to relevant

data. If the office internally manages execution data on shared drives or something similar with little to

no outside connectivity, it is important for the analyst to work closely with the program to gain access to

that data. As the program office delivers and/or the analyst retrieves necessary data, it may become

quickly apparent that certain pieces of data are not available from the program office. This leads the

analyst to propose discussions with the prime contractor, subcontractors, or other government offices.

Given that the defense industry manufacturers are a primary source of much of the data related to the

program of interest, the analyst should make an effort to arrange site visits to enhance the

understanding of the program and any relevant data. These site visits may involve participants from the

program office, the appropriate Component Cost Agency, and/or the CAPE to provide for simultaneous

participation rather than several individual visits. Analysts’ requests for program office and/or

contractor information and visits should include a list of data collection priorities well in advance. Many

times, analysts can combine their required visits with other programmatic meetings.

5.5.2.4 Data Collection is an Iterative Process

Once the analyst has completed an initial round of searching government and contractor sources, the

data collection picture is clearer. All expected cost elements and potential cost drivers should have an

initial data capture that at least partially, preferably mostly, addresses them. The analyst should

schedule repeat visits only after he/she has exhausted all other sources and clearly identified the

remaining data requirements.

Gaps in clean, objective data might still exist after the analyst collects data from the sources mentioned

in this chapter. If this happens, then the analyst should consider SME level guidance from other analysts

and literature. Subjective data and SME guidance is often necessary.

In the context of the cost estimating process, data collection is not finished until the cost estimate is

complete and approved. It begins again with the next estimate task. For data owners, data collection is

an ongoing process, which could cause a change in cost estimate results. Data updates can establish

trends and support key, fundamental findings within a cost estimate. Consequently, it is a good practice

to query data sources more than once over the course of the cost estimate development.

5.5.3 Validate Data

Closely following the collection of data is the validation of the data. The analyst should not confuse this

with validating the cost model or any other portion of the cost estimating process. Each of the

Component handbooks and guides provide some guidance for data validation. The 2010 DON Cost

Estimating Guide provides a good description of and basis for validating data. It explains the important

distinction between verification and validation in the context of a cost estimate when it states

“validation ensures ‘doing the right estimate’ while verification ensures ‘doing the estimate right’.” In

the context of data validation, one can restate this as: collecting the right data. Typical validation checks

include:

• Currency: Identify the most recent, up-to-date data on analogous programs.

• Applicability: The most useful data originates from sources consistent with the program

mission, operating environment, and platform type. As the analyst seeks analogous or

related data, he/she must take specific care to ensure the analogy or related data

appropriately represents the system being estimated.

• Accuracy: Import processes, manual entry, and interpretation of units are some of the

issues that need careful attention to ensure accuracy of the data. Accuracy is established

from evidence the data is correct, complete, and current for the item measured. Precision

is not a measure of accuracy. For example, capturing a data element value to 10 decimal

places is a measure of precision, but does not guarantee that the value is correct.

• Veracity: Try to obtain corroborating pieces of information from various sources.

Concurrence or divergence sheds important light on the quality of the data.

5.5.4 Normalize Data

The purpose of data normalization is to convert the collected data into a form consistent with and

comparable to other data used for the estimate. Normalization of data to support a particular estimate

requires attentiveness to anything that influences how the analyst interprets and reduces the data to a

form consistent with the cost estimate purpose. It is not just the cost data itself that requires attention.

The following is a summary view of data normalization:

• Cost Data: An analyst must address many influences on cost to render the data in a

consistent form. Contract WBS arrangements/changes/revised definitions, requirements

creep, program durations, accounting system changes, prior or in-parallel quantities,

production rates, labor rates (hours vs. days), and escalation/inflation are all examples of

program characteristics that influence how to interpret the state of the cost data. The 2017

DoD Inflation and Escalation Best Practices for Cost Analysis includes details on how to

address inflation/escalation normalization (

https://cade.osd.mil/policy/inflationandescalation/).

The 2018 JA CER Handbook provides guidance on how to address many cost data

normalization procedures beyond inflation/escalation

(

https://www.ncca.navy.mil/references/CER_Dev_Handbook_Feb2018_Final.pdf).

• Programmatic Data: An analyst uses programmatic data to adjust cost data for the

quantitative and qualitative program characteristics introduced in Section 5.2.2. For

example, the analyst can calculate the per unit cost for use in comparing costs to a budget

or to other programs. Unit costs must be characterized by their lot or unit of production

(e.g., the unit cost of the 100

item (UC100)). It is equally important to account for the

production rate (e.g., UC100 at a production rate of 10 per month), otherwise the analyst

may reach misleading conclusions in comparing programs with dissimilar rates of

production. Adjusting for quantity and production rate effects is called adjusting for

learning effects, a topic covered extensively in the 2018 JA CER Handbook. The

normalization process may not address some qualitative programmatic features of the

data. Rather, these considerations may influence the cost method functional form

selection.

• Performance and Technical Data: An analyst uses normalization of performance and

technical data to convert data to a common set of units. Also, the values must be mapped

to an element of the estimate structure or prorated across several elements based on

either accounting or SME guidance. For instance, the analyst may have to prorate the total

weight of an item across two or more elements of the estimate structure.

• Schedule Data: Schedule data includes milestone dates, activity durations, and activity

dependencies (schedule impacts of one or more tasks on one or more others). Reducing

costs to a cost per unit of time (e.g., cost per hour, week, month, or year) is a useful way to

compare costs across or within programs. It provides a means to build cost models that are

realistically sensitive to time. The analyst must confirm definitions of schedule terminology

such as: FY, labor year, and holiday/vacation/sick leave adjustments. The federal FY starts

on October 1 and runs through September 30 but this is typically not the same throughout

industry. Similarly, time allowed for holidays, leave, and sick time is not consistent.

However, each company has a standard definition for a labor year that they use for

planning purposes.

5.5.5 Analyze Data

While collecting, validating, and normalizing data, it is appropriate to begin performing exploratory data

analysis (detailed statistical analysis to support methodology selections comes later). The primary

benefit of doing exploratory data analysis early is to discover patterns in data, holes in the data,

potential outliers, and to narrow the gap between the collection of data and the understanding of it.

This understanding, in turn, helps to:

• identify outliers (an observation that lies outside the overall pattern of the data)

• suggest hypotheses regarding the initial specification of regression equations for explaining

changes in dependent variables such as cost or person hours of effort,

• support the selection of appropriate statistical tools and techniques, and/or

• provide a basis for further data collection.

Outliers can become apparent by simply graphing the data. Analysts should study these observations

should to ensure the data is captured correctly and that the observation is relevant to the program. A

more detailed look for outliers, and how to address them, happens in the estimating methods step of

the cost estimating process. (See Section 6.3.4 for a discussion on outliers.)

A wide range of statistical techniques is available to execute exploratory data analysis. These include:

• visuals (e.g., scatter plots, influence diagrams, and classification trees),

• traditional statistics (e.g., univariate, regression, and outlier considerations), and

• modern techniques (e.g., data-mining algorithms and machine learning).

DAU course BCF 130 “Fundamentals of Cost Analysis” introduces some of these techniques.

Additionally, the commercial market has many software packages and visualization tools that are

specifically oriented towards exploratory data analysis. The introduction of FlexFiles for collecting

contractor data further motivates the desire to consider powerful data analysis tools, as the amount of

data in a FlexFile can strain the limitations of more traditional tools like Microsoft Excel. (See Section

5.4.2 for a discussion on government and contractor sources of data.) Free open-source programming

languages are becoming popular alternatives to perform statistical analysis (e.g., R) and data science

(e.g., Python®) of large data sets. The data collected via FlexFiles will provide new opportunities for

more detailed investigations into the way contractors perform their work.

Data References

• DoDI 5000.02T, Operation of the Defense Acquisition System, 2020, Enclosure 10, para. 4

“Data to Support Cost Estimating”, pg. 135

• DoDI 5000.73, Cost Analysis Guidance and Procedures, 2020, Section 4, “Data Collection”,

pg. 30

• CAPE, Operating and Support Cost-Estimating Guide, 2014, para. 5.31, “Collect, Validate,

and Adjust Data”, pg. 5-7

• Department of the Army, Cost Analysis Manual, 2020, Chap 3 “Cost-Estimating Process”, pg.

• NCCA, Joint Agency Cost Estimating Relationship (CER) Development Handbook, 2018, para.

1.4 “Sources of Data”, pg. 19 and para. 1.5 “Collect and Validate the Raw Data”, pg. 21

• NCCA, Joint Agency Cost Schedule Risk and Uncertainty Handbook, 2014, para. 2.5.2,

“Elicitation of Subjective Bounds from Subject Matter Experts (SMEs)”, pg. 29

• NCCA, Cost Estimating Guide, 2010, para. 1.3.2 “Collect, Validate, Normalize, and Analyze

Data”, pg. 24

• SPAWAR, Inst 7110.1 Cost Estimating and Analysis, 2016, Enclosure 1, para. 5.a(2) “Collect,

Validate, Normalize, and Analyze Data” pg. 7

• Missile Defense Agency Cost Estimating and Analysis Handbook, 2012, Chapter 5 “Data” pg.

• GAO, Cost Estimating and Assessment Guide, 2009, Chapter 10 “Data” pg. 89

• NASA, Cost Estimating Handbook, 2015, para. 2.2.4 “Task 7: Gather and Normalize Data”,

pg. 22

• RAND, Improving the Cost Estimation of Space Systems, 2008, Chapter 3, “Data Availability

and Quality Issues”, pg. 56

Data Training

The DAU Cost Estimating certification program for members of the Defense Acquisition Workforce offers

training relevant to the cost estimate data. Additional information on each course may be found in the

DAU iCatalog (

https://icatalog.dau.edu/).

• BCF 130 Fundamentals of Cost Analysis, Lessons 3, 4

Data science involves developing methods of recording, storing, and analyzing data.

• BCF 132 Applied Cost Analysis, Lessons 2, 3

• BCF 216 Applied Operating and Support Cost Analysis, Lesson 2

• BCF 230 Intermediate Cost Analysis, Lesson 2

• BCF 250 Applied Software Cost Estimating, Lesson 3

• BCF 331 Advanced Concepts in Cost Analysis, Lessons 4, 5

• CLB 030 Data Collection and Sources (introduces the basics of data sources and collection as

it relates to cost estimating)

• CLB 033 Databases for the Cost Estimate (introduces a cross section of DoD databases

)

• CLE 035 Introduction to Probability and Statistics (basic introduction and understanding of

probability and statistics)

The ICEAA publishes the CEBoK. The follow modules are relevant to data:

• CEBoK v1.2, 2013, Module 4 “Data Collection”

• CEBoK v1.2, 2013, Module 5 “Inflation”

• CEBoK v1.2, 2013, Module 6 “Data Analysis”

• CEBoK v1.2, 2013, Module 10 “Probability and Statistics”

The following course numbers starting with FMF or FML refer to the course number assigned by the FM

Certification process. Information on these courses (including eligibility requirements) can be found in

the FM myLearn system:

https://fmonline.ousdc.osd.mil/FMmyLearn/Default.aspx.

• FMF 1253 FMA 202 - Financial Management Concepts Course - Descriptive Statistics

• FMF 1124 FMA 204 - Financial Management Concepts Course - Trend Analysis

• FML 4110 Building Business Acumen

• FMF 4439 Air Force Total Ownership Cost (AFTOC) Decision Support System (DSS) 101

• FMF 4440 AFTOC Decision Support System (DSS) - Data Access Techniques

• FMF 4441 AFTOC Decision Support System (DSS) - Account Tool Basics

• FMF 4442 AFTOC Decision Support System (DSS) - Advanced Data Mining

• FMF 1546 Business Case Analysis

• FMF 6540 Analytic Cost Expert Distance Phase (ACE dL)

• FMF 7815 WKSP 0672 Data Analytics Tools and Techniques

• FMF 7816 WKSP 0673 Applied Concepts of Data Analytics Tools and Techniques

• FMF 7883 Data Analytics

• FMF 1551 QMT 490 - Current Topics in Cost Estimating

Training on specific data sources is available at:

• CADE training videos: designed as a handy reference for the first-time user or seasoned

analysts that just need a refresher. Topics include: user guidance for the CADE portal, data

and analytics, plus “how to” guidance on CCDR, SRDR and available libraries are available at

https://cade.osd.mil/support/videos (public)

• CADE Pivot Tables for Analysts: https://cade.bridgeapp.com/learner/library (requires a

CADE login)

• Naval VAMOSC Training Videos:

https://www.vamosc.navy.mil/

• Army OSMIS Training Videos:

https://www.osmisweb.army.mil/Osmis/Support/SupportVideos

and

https://www.osmisweb.army.mil/Osmis/Support/Tutorials

Access to most of the DoD databases is controlled and in some cases, is classified; both of these issues limit the

databases that can be openly discussed.

6.0 SELECT COST/SCHEDULE ESTIMATING METHODS

Analysts build cost estimates using a

combination of the estimating

methods introduced in this chapter.

The suitability of a specific method

largely depends on the maturity of the

program, the nature of a particular

element of the estimate structure, the

usefulness of available data, and the

time available to develop the estimate.

Like all the steps in the cost estimating

process, this one is also iterative. In

particular, the estimate basis and data

collection steps both influence and are

influenced by the progress made in

identifying viable estimating methods.

The identification, collection,

validation, and normalization of data

along with the information from the

program definition and cost estimate

basis help determine the best cost estimating method for a particular element of the estimate structure.

The data analysis described in the previous chapter primarily supports the data validation process, but

that analysis may reveal patterns in the data that point to a specific estimating method. Additionally,

analysts should review previous, similar estimates to identify estimating methods that worked well in

the past.

Many estimating methods apply to estimating cost or schedule durations. For simplicity, this guide

refers to both cost and schedule estimating methods as “estimating methods”.

The remainder of this chapter introduces the most common DoD cost estimating methods, how to

address outliers, and how to determine the estimating method uncertainty.

Basic Estimating Methods

Common estimating methods used in DoD cost estimates are analogy, build-up, extrapolation of actuals,

and parametric. Ideally, the analyst will base any estimating method on data from completed analogous

programs. While an analyst can draw data from systems still under development or in production, it

may be less defendable than drawing data from a completed program because significant, unforeseen

changes could still occur in unfinished programs.

The methods described below are intentionally presented alphabetically to avoid any perceived

preferences. Component guidelines, circumstance, and the analyst’s assessment drive the rank order of

preference. The following sections introduce each method, and Table 9 compares the advantages and

disadvantages of each.

6.1.1 Analogy Estimating Method

With the analogy estimating method, the analyst bases his/her estimate for the new system or effort

upon the known cost of a similar, completed system or effort. In its most basic form, the analogy

method states that if a historical system is known to cost $X, and the new system is similar to the

historical system, then the new system cost estimate is $X, subject to adjustments to account for

differences between the programs.

A primary advantage of using a fully developed and deployed analogous system is the ability to declare

that the analyst has captured the impact of risk/opportunity and uncertainty experienced by the

analogous program in the reported cost, schedule, and technical characteristics. This may be an over

simplification and is discussed further in Section 6.1.3. A criticism of cost estimating based on past

program(s) is that the risks that impacted the original program(s) will likely be avoided in the new

program, but the new cost estimate still reflects these risks if the historical data has not been adjusted.

A counter to this argument is that even if previous risks are avoidable, it is likely that new ones that

influence the estimate in a similar way exist. The onus is on the analyst to develop a defendable

approach.

It is unlikely that the analyst can find a perfect analogy for the system being estimated since

technologies and capabilities evolve over time. Even if the new system is a direct replacement for an

existing system, the new system likely has more capability. For example, computers have better

processors, engines may have more thrust, or materials may weigh less. The analogy method should

include adjustments to account for differences between the historical and new system. The analyst

develops adjustments as objectively as possible based upon data analysis where feasible. In some cases,

the adjustment might be an add or a subtract to account for differences in the systems. In other cases,

the analyst may use factors, sometimes called scaling parameters, to account for differences in size,

performance, technology, and/or complexity. The analyst should document how the analogous system

relates to the new system, identify the important cost drivers, and decide how each cost driver

influences the overall cost in the analogous and new system. The analyst can apply the analogy method

to the program overall or to a specific, lower level element of the estimate structure. The selected

analogy must have a strong parallel to the item being estimated, and any adjustments should be

straightforward and readily understandable.

For this method, it is important for the estimator to research and discuss with program experts the

reasonableness of the analogy, its technical program drivers, and any required adjustments for the new

system. This discussion should address whether the adjustments are simple additions to or subtractions

from the new system or if there is a need to employ scaling factors to the analogy. Scaling factors can

be linear, nonlinear, or some other form. Linear adjustments are the most common and easiest to

apply. Examples of nonlinear adjustments include using the cost improvement curve formula (Section

6.3.2) to adjust the analogy directly or to estimate the reference cost. The analyst should consider

previously developed estimating methods for potential scaling approaches. The analogy method is a

useful crosscheck when a different primary method is used.

6.1.2 Build-up Estimating Method

The build-up cost estimating method assembles the overall cost estimate by summing or rolling-up

detailed estimates created at the lower levels of elements of the estimate structure. Because the lower-

level approach associated with the build-up method uses industrial engineering principles, it is also

referred to as an engineering build-up or a bottom-up estimating method. When extensive high-quality

data exists from closely-related systems and/or from limited-rate or prior production, the build-up

method becomes a viable candidate methodology.

A build-up estimate is a detailed treatment of direct/indirect labor hours, direct/indirect labor rates,

materials, facilities, travel, and all other costs for each element of the estimate structure. The analyst

assigns costs at the lowest level elements of the estimate structure according to how the worker

accomplishes the task(s). Typically, analysts work with manufacturing or maintenance engineers to

develop the detailed estimates. The analyst’s focus is to obtain detailed information from the engineers

in a way that is reasonable, complete, and consistent with the program definition and its ground rules

and assumptions.

When an analyst uses a build-up method for a production estimate, he/she normally applies it when the

system’s configuration is stable and the required details are available. The high level of detail requires

the manufacturer to identify, measure, and track each step of the work flow so that the analyst can use

the results to refine the estimate. When used as a primary method, the analyst should corroborate the

results using one or more of the other methods identified in this chapter.

6.1.3 Extrapolation from Actuals Method

Extrapolation from actuals uses observed costs from earlier stages in the program to project a cost in

future stages of the same program. Arithmetic averages, moving averages, burn rates, cost

improvement curves, and EVM estimates at completion (EAC) are examples of extrapolating from actual

costs. (See Section 6.3.2 for a discussion on cost improvement curves.) These projections can occur at

any level of elements in the estimate structure, depending on the availability of data. An analyst can

consider the extrapolation of actuals method once an item’s actual production or O&S data become

available.

The analyst can generally account for changes in the product design, manufacturing process, or

operating and support concept of follow-on items in the same ways discussed under the analogy

estimating method. In this case, he/she simply treats the earlier items as the “analogy” instead of using

another program. If major changes have occurred, analysts may need to consider a different estimating

method since the actuals may not be relevant enough to extrapolate for future costs.

6.1.4 Parametric Estimating Method

The parametric estimating method centers around relating cost or duration to one or more

programmatic, performance, or technical characteristics via an algebraic equation. The strength of a

parametric estimate lies in the relevance/quality of the data and in the validity of the relationships

within that data. Unlike an analogy, parametric estimating relies on data from many programs rather

than just one and yields a relationship that is valid over a range of possible solutions. Also, unlike the

analogy method, the parametric analysis captures the realities of situations addressed by a number of

similar completed programs, rather than realities from just one program. The analyst should consider

the number of data points required to form a statistically useful data set. The 2018 JA CER Handbook

addresses this topic.

Analysts use parametric cost estimating models throughout the life cycle, but they are particularly useful

tools for preparing early conceptual estimates when performance and technical details are not fully

developed. They are also useful for quickly establishing cost and/or schedule impacts over a range of

alternatives.

Ultimately, the parametric method’s objective is to find the best functional form of an equation to fit

the available data. While there are many ways to construct the best curve through a series of data

points, regression is popular because it provides statistical inference and an assessment of the

uncertainty present in a curve. The regression analysis used in this method is a statistical process for

estimating the relationship between a dependent variable (the element estimated) and one or more

independent variables (variables that influence the estimate). The resulting equation is a parametric

CER (to estimate a cost) or a SER (to estimate schedule durations).

An analyst applies parametric regression analysis in an iterative process testing functional forms against

the available data sets many times prior to selecting the best equation. The best equation is one that:

• makes sense (i.e., the behavior between the independent and dependent variables is

logical),

• is based on data that is relevant to the estimate,

• is populated with independent variables that are within the source data set range,

• passes all the statistical tests for significance,

• generates the least uncertainty, and

• is the simplest of the equally accurate, statistically significant relationships.

The two most common functional forms are:

• Linear:  =  +  ∗  + ℰ

• Nonlinear

:  =  ∗ 



∗ ℰ

The functional forms only show one parameter for simplicity. Parametric equations often have more

than one independent variable (parameter). Independent refers to the relationship between multiple

parameters used in the same equation. Regression theory requires parameters to be independent of

each other.

The error term ℰ in the functional forms represents the difference between the data and the result

predicted by the equation. The objective of regression analysis is to solve for the coefficients (e.g., a

and b) that minimize ℰ. The 2018 JA CER Handbook provides more detail on these and other parametric

equations and the available regression techniques used to solve for the coefficient values.

When the analyst uses regression analysis to develop parametric equations, he/she needs to document

the conditions under which the relationships were established. This information is necessary to support

the validity of the estimate and influence how to address uncertainty. Each equation used in the

estimate should be documented with descriptive and regression statistics, assumptions, and data

sources.

The following subsections describe two specific parametric equation variations.

6.1.5 Comparing Basic Estimating Methods

Table 9 summarizes the advantages and disadvantages of the basic estimating methods.

Also known as a “log-linear” equation because it becomes linear when taking the logarithm of both sides.

Natural log (LN) is the standard practice.

Table 9: Summary of Advantages and Disadvantages of Basic Estimating Methods

Estimating

Method

Advantages Disadvantages

Analogy

• Applicable before detailed program

requirements are known

• Can be developed quickly

• Completed analogous program inherently

includes risk and uncertainty

• Based on objective historical data that can

be readily communicated and understood

• Relies on a single data source

• May require adjustments for

risks/opportunities and uncertainties

not present in the current program

• Technical data required for scaling may

be elusive and/or difficult to defend

• Subjectivity with technical parameter

adjustment factors likely to be

introduced

• Appropriate analogy may not be

available

Build-Up

• Fully documents and addresses exactly what

the cost estimate includes

• Captures the specific manufacturer’s

processes and rates

• Explicitly reveals the major cost contributors

• Provides a basis to check for duplicates and

omissions

• May be expensive to implement and

time consuming

• Less flexible and may not answer many

of the what-if questions

• New estimates must be built for each

alternative

• Product specification must be well

known and stable

• All product and process changes must

be reflected in the estimate

• Small errors can grow into larger errors

through the summation

• Elements can easily be omitted or

duplicated by accident in large models

Extrapola-

tion of

Actuals

• Uses the program actual data to develop the

estimate

• Access to sufficient and reliable cost

data may be challenging

• Changes in accounting, engineering, and

manufacturing processes have to be

identified and addressed

Parametric

• Versatile and can be derived at any level

where the data is available

• Supports what-if explorations of design

alternatives

• Supports cost driver sensitivity analysis

• Provides objective measures of statistical

significance of each coefficient and of the

model as a whole

• Provides objective measure of uncertainty

(standard error)

• Objective measure of the result’s probability

of exceedance

• Derived from objective historical data

• Source data must be consistent,

accurate, and properly normalized

• Often have to rely on a few data points

• Cannot use without fully understanding

how it was generated

• Must be updated to capture the current

cost, technical, and program data

• Populating with independent variable

values outside the range of the source

data leads to increased uncertainty and

may produce erroneous results

• Complicated relationships may be

difficult to defend

Figure 2 is an illustration of which cost estimating methods are often most appropriate at different

times through the system’s life cycle. Appendix E contains figures from several guides and handbooks

that illustrate where in the major capability acquisition process each of the basic estimating methods

may apply. Figure 2 is a generalized rule-of-thumb for analysts. However, it is always up to the analyst

to decide the most appropriate method for a particular element or phase, usually dependent on the

data that is available.

Figure 2: Estimating Method Applicability

Other Estimating Methods

In addition to the basic methods discussed in Section 6.1, the analyst has many other methods available

to use that are applicable under certain circumstances. They include (listed alphabetically):

• Expert opinion: Relies on SMEs to give their opinion on what an element might cost or how

the analyst should adjust its cost. Analysts often rely on SMEs during the early stages of a

program, when they are making less detailed estimates.

• Full-time Equivalents (FTEs): One FTE represents the total hours available in a full-time

schedule. The analyst estimates the total FTEs required and multiply by the FTE labor rate

to arrive at cost. FTE estimates may be derived from an analogy, parametric equation,

extrapolation of actuals, expert opinion, or other tools. A simple count of the number of

people employed on the task is not a meaningful measure of the team’s cost. Some

members of the team may be hourly-on-call, some part-time, and others full-time. The

required number of FTEs to perform a task

or to be constantly available (sometimes called

the standing army) is a much more precise way to estimate effort and cost.

• Industrial engineering standards: Applies a realization factor to engineered work unit labor

standard times. A unit time value for the accomplishment of a work task is determined

from a work measurement program. Standard time is how long it takes to perform a

particular task, based on time and motion studies done in controlled environments. Since

It is rarely true that doubling the size of the team will reduce duration by half. The bigger the team, the more

effort spent in communications (e.g., meetings) and adhering to a collaborative work environment.

the standards reflect an optimal production environment, the analyst calculates variance

factors (also known as realization factors) based on measures of a company’s actual

experience compared to the standard. This approach can be a part of a build-up method.

• Tools: Contains previously developed estimating structures and/or methods that the

analyst uses to facilitate the development of a cost estimate. The tool is generally a

software package and may come from internal government research or commercial sources

specializing in cost estimating. The tools predicate their effectiveness on the ability to

calibrate the tool to align with a given set of data or item. Since these models typically

incorporate proprietary data and equations, visibility into their methods may be lacking.

Therefore, their use as a primary estimating method is discouraged. However, there is

utility in using these tools to crosscheck the reasonableness of an estimate. They can also

serve as a last resort when no other method is viable.

• Univariate analysis: Is the statistical analysis of a single type of data, not a search for cause

and effect relationships. Univariate analysis includes both descriptive and inferential

statistics. Descriptive statistics yield measures of central tendency (mean, median, and

mode) and variability (e.g., range, standard deviation, skewness, and kurtosis) of the data.

The analyst can use inferential statistics to characterize the usefulness of the mean of the

sample as the estimate for a new system. The advantage of using a univariate approach is

that it is simple to apply and understand. It provides an objective basis to characterize the

uncertainty of the result. The disadvantage is that it does not relate cost to a cost-driver

and is therefore of no help in what-if or sensitivity analysis.

Additional Considerations

6.3.1 Correlation

Correlation is a measure of the degree two (or more) variables move together. It is a convenient tool to

use to identify potential cost drivers. A good practice to build a correlation matrix, as illustrated in

Figure 3

, to assess the correlation among the item estimated (dependent variable), potential cost

drivers (independent variables), and the correlation between cost drivers. The example in Figure 3

shows two types of correlation

• Pearson Product Moment correlation measures the linear relationship between variables.

Figure 3 shows a strong linear relationship between cost and aperture and between cost

and power.

• Spearman Rank correlation measures the correlation regardless of the relationship type

(e.g., linear, nonlinear, something else) between the variables.

It is good practice to measure both types of correlation, particularly if one of them suggests there is little

or no correlation.

The results in Figure 3 indicate a high correlation between power and aperture. This means power and

aperture are not independent of each other, a behavior called multicollinearity. If an analyst regresses

both power and aperture against cost, the CER/SER coefficients may change erratically in response to

small changes in the data due to the interplay between power and aperture inputs. There are various

Figure 2 is a combined and modified version of Table 8 and 9 from the 2018 JA CER Handbook, pg. 40 and 43

respectively.

The CORREL function in Microsoft Excel calculates the Pearson Product Moment correlation. Converting the

data to ranks and applying the CORREL function yields the Spearman Rank correlation.

ways to address multicollinearity if there is motivation to retain the influence of both parameters in the

CER/SER. One way is to combine the two parameters into one, in this case: intensity. (See the 2018 JA

CER Handbook for more detail on all aspects of this type of analysis.)

Guidance from and collaboration with applicable SMEs is particularly useful in the development of

parametric relationships. SMEs can help propose or validate functional forms that make sense. It is not

enough for independent variables to have a high correlation with cost, since correlation is not causation.

The relationship should be logical, and ideally, a known scientific fact should relate the two. The

assumption driving the parametric approach is that the same factors that affected cost in the past will

continue to affect costs in the future.

Figure 3: Notional Correlation Matrix Example

Correlation between uncertain variables and between estimating methods are important considerations

when estimating total cost uncertainty. (See Sections 6.4 and 7.4 for a discussion of estimating method

uncertainty and cost model uncertainty.) The 2014 JA CSRUH provides guidance on how to address

risk/opportunity and uncertainty correlation.

6.3.2 Cost Improvement Curve

The cost improvement curve (also known as a learning curve) is a technique to account for measured

efficiencies in production when the manufacturer produces many units. The premise behind the use of

the cost improvement curve is that people and organizations learn to do things better and more

efficiently when they perform repetitive tasks, and the analyst should address the impact of this

efficiency in the cost estimate. While the original research was rooted in manufacturing labor, the

absence of repeated manual effort does not preclude its use. The same phenomenon is observable in

successive production lots as the entire program enterprise incrementally learns and adapts to doing

things better. The analyst’s challenge is to define a reference cost and objectively identify the

incremental improvement appropriate for the estimate.

An estimate derived from a cost improvement curve using inflation-adjusted CY dollars will be biased

because RPC is neglected. The 2017 DoD Inflation and Escalation Best Practices for Cost Analysis,

Appendix D “Normalizing for Learning Curves: Estimating Using Actuals” provides a step by step example

demonstrating this bias. Consequently, the best practice is to base cost improvement curves on CP$.

The 2017 DoD Inflation and Escalation Best Practices for Cost Analysis provides the authoritative detail

on calculating CP$.

The premise of cost improvement curves is that each time the product quantity doubles the resources

or cost required to produce the product is reduced by a determined percentage

(slope). For example,

a 90% slope in unit cost improvement curve theory indicates an expectation that item 4 costs 10% less

than item 2, item 8 costs 10% less than item 4, and so on. There is ongoing research to determine the

application of cost improvement curves in the presence of digital production technology.

The equation is comprised of a theoretical first unit cost or hours (a), the applicable unit number (X) and

an exponent (b)

as illustrated below.

 =  ∗ 



The cost improvement curve method is presented with the parametric method because in its simplest

form, the cost improvement curve equation is consistent with the parametric nonlinear functional form

introduced in Section 6.1.4 where the parameter is unit number. While analysts often use parametric

equations to estimate total cost from one or more parameters, this particular form of the cost

improvement curve equation estimates a unit cost. A regression analysis of historical data from a similar

program, or the actual data from the program of interest, derives the values for the first unit cost and

the exponent

. There are many variations of the cost improvement curve equation to account for

different theories (unit, average unit, lot)

, impact of production rate, and breaks in improvement

(down time, engineering change, process change, etc.).

When using a cost improvement curve equation, the analyst needs to document if the selected form of

the equation produces a cost for the unit, the average unit, a series of units (lot), or some other

combination. In practice, an analyst may choose to use any estimating method to estimate the

reference cost and exponent separately. If the available data leads to a reference cost that is not

associated with the first unit or lot, the analyst can use the formula to estimate one (e.g., knowing the

reference cost, the reference unit or lot number, and the exponent, calculate the first unit cost). The

analyst can derive the exponent separately from historical data. For example, if the manufacturer has

produced several production units or lots of an item, the analyst can derive the exponent from

regression analysis of the actual unit or lot cost data. An analyst can also estimate the exponent by

using observed exponents from the company’s similar efforts on other programs.

Ideally, the exponent and “a” are determined together through a regression analysis. This has the

advantage of generating an objective basis for the reference cost, the exponent, and the uncertainty

associated with the cost improvement curve equation as a whole. However, there may be situations

where the analyst must estimate the reference cost from one source and the exponent from another

source or from expert opinion. In this case, the analyst may be tempted to apply uncertainty (if

performing a simulation) or what-if analysis (as an alternative to simulation) to each of the reference

Unit theory assumes the unit cost is reduced while cumulative average theory assumes the cumulative average is

reduced. The choice of theory to use is made by the analyst with the proviso that only one is used throughout the

model.

Where “b” is the logarithm of the slope (e.g., 90%) divided by the logarithm of 2. Logarithm base 10 or natural

logarithm can be used as long as it is used for both numerator and denominator.

If the analyst uses regression to estimate both T1 and the exponent, then changes to either one invalidates the

regression uncertainty results.

See 2003 “Statistical Methods for Learning Curves and Cost Analysis”, Matthew Goldberg et al. at:

https://www.cna.org/CNA_files/PDF/D0006870.A3.pdf

See CEBoK v1.2, 2013, Module 7 “Learning Curve”

cost and the exponent separately to estimate the uncertainty of the equation as a whole. Section 2.8.5

of the 2014 JA CSRUH provides guidance on how to address this situation.

6.3.3 Linear Without Intercept

A special form of the parametric linear relationship is one where the y-intercept is zero, meaning a

regression analysis has revealed that the relationship is statistically stronger without the y-intercept

coefficient. In a linear relationship between cost and one or more cost drivers the method becomes a

straightforward multiplier. Multipliers can be categorized as:

• Factor: One cost is expressed as a multiple of another cost such as estimating the cost of

data as a factor of PMP.

• Rate: One cost or duration is expressed as a factor of a non-cost parameter such as $/hour

or hours/ton.

• Ratio: One non-cost parameter is expressed as a factor of another such as kilowatts per

ton.

6.3.4 Outliers

An outlier is an observation (data point) that lies outside the overall pattern of the data. Detection and

treatment of outliers are part of any regression analysis. Methods to detect outliers include

scatterplots, residual analysis, and leave-one-out regression

. Treatment of outliers centers on

understanding why they are so different compared to the other observations. (See Section 5.5.5 for

data analysis performed during data collection.) Their detection provides an opportunity for the analyst

to further understand the behavior of cost (univariate analysis) or the behavior of a cost and cost driver

relationship (parametric analysis). In addition to detecting the outlier, assessing its influence on the

result is necessary. The analyst must identify and address outliers that have a significant influence on

results by:

• applying appropriate statistical methods,

• accepting them as part of the dataset and the influence they have, or

• discarding the outlier data point(s).

The latter choice is only acceptable with strong supporting evidence. Discarding an outlier because it

negatively influences the result is not a valid or acceptable reason because that point may reveal an

important aspect of the relationship. The 2018 JA CER Handbook contains more detail on detecting and

addressing outliers.

Introduction to Estimating Method Uncertainty

This section addresses the uncertainty associated with an individual estimating method. Section 7.4.2

addresses the uncertainty associated with the total estimate.

Most cost estimates use a mixture of estimating methods. Though analysts do identify and evaluate

multiple estimating methods in the cost estimating process, a final estimate uses only one estimating

method for a given element of the estimate structure. Regardless of how well the estimating method is

developed and how accurate the inputs are, the resulting cost is always a point estimate, which is just

one result from a range of possible outcomes. Section 7.2 further discusses interpreting the point

estimate within this range.

See 2018 JA CER Handbook, para. 4.3.1.4 “Leave-One-Out Metrics”

In the case of parametric and univariate estimating methods, statistical techniques can be used to

calculate a prediction interval (PI) that defines a range within which the cost estimate is expected to

appear. This is an objective basis for estimating the uncertainty of the univariate or parametric result

Understanding and accounting for the estimating method uncertainty is an essential part of the cost

estimating process. The analyst must properly characterize what the result of the selected estimating

method represents. The estimating method can produce a mean, mode, median, or some other

probability result. By characterizing each element of the estimate structure result in the

documentation, it becomes apparent that the total estimate is a sum of many different types of

methods. This is a key reason the analyst should not refer to the total estimate as the most likely. It is

but one possible outcome (i.e., a point) in a range of possible outcomes. This gives rise to the term

point estimate. Analysts must endeavor to ensure decision authorities are fully aware of the

implications of relying too heavily on any point prediction without any assessment of where it may land

in the range of possible outcomes. In the special case where the cost estimating method for every

element of the estimate structure produces the mean, the total is also the mean. This is very rare.

Section 7.4.2 discusses how to estimate the combined effect of all sources of uncertainty in order to

assess the probability of exceeding a given budget. From an estimating method point of view, the

analyst must address the uncertainty of:

• parametric CERs/SERs including factors and cost improvement curve equations,

• CER inputs, complexity factors for analogies, engineering judgment,

• any other uncertain cost drivers (e.g., man-hours, FTEs, rates, ratios, overhead, fee), and

• the planned schedule (durations).

In addition to uncertainty, the cost model needs to have methods to estimate the impact of discrete

risk/opportunity events, risk mitigation plans identified by the program office, and proposed

opportunity initiatives. Risk/opportunity events are situations that result in an impact to the project

performance, cost, or schedule if they occur. Therefore, a risk/opportunity event has three

characteristics: a definable situation, a probability that situation will occur, and a consequence should

the event occur. If the consequence is negative to the program it is a risk. If the impact is positive, it is

an opportunity. The program’s Risk Register is a formal document that identifies all known risk and

opportunity events. The challenge for the analyst is to determine what, if any, risk register elements

that have the attention of the program manager are not captured by the estimating methods directly.

Having identified them, the next challenge is to find a way to capture them in the cost estimate. If there

are only a few, the analyst can treat them as what-if cases. The 2014 JA CSRUH provides guidance on

how to capture the impact of many risk/opportunity events.

Thus far, uncertainty has been discussed in the context of one estimating method for one element of

the estimate structure. Characterizing what the total cost estimate represents and its total uncertainty

is a function of the source data, the estimating methods used, and how the estimate is modeled, which

Section 7.4.2 discusses.

Estimating Methods References

• DoD Independent Government Cost Estimate Handbook for Service Acquisition, 2018, “Cost

Estimation Methods”, pg. 12

The 2018 JA CER Handbook para. 5.3 “Generate Prediction Interval”, pg. 173 illustrates how much smaller a

parametric CER/SER PI can be compared to the PI of an average cost.

• CAPE, Operating and Support Cost-Estimating Guide, 2014, para. 5.3.2, “Select Methods or

Models”, pg. 5-8

• Department of the Army, Cost Analysis Manual, 2020, Chap 4 “Cost Estimating

Methodologies”, pg. 20

• NCCA, Joint Agency Cost Estimating Relationship (CER) Development Handbook, 2018, para.

2.2 “Cost Estimating Methods”, pg. 33, para. 2.3 “Choosing Between Analogy, Straight

Average or a CER”, pg. 34, para. 2.4 “Univariate Analysis”, pg. 38, and Chapter 3 “Step 3:

Generate CER” pg. 57

• NCCA, Joint Agency Cost Schedule Risk and Uncertainty Handbook, 2014, para. 2.5.2

“Elicitation of Subjective Bounds from Subject Matter Experts (SMEs)”, pg. 29, and para. 2.8

“Special Considerations”

• NCCA, Cost Estimating Guide, 2010 para. 1.3.3 “Develop CERs and Analyze Risks and

Uncertainties”, pg. 27

• SPAWAR, Inst 7110.1 Cost Estimating and Analysis, 2016, Enclosure 1, para. 5.a(3) “Develop

CERs and Analyze Risks and Uncertainties” pg. 9

• USMC, Cost Analysis Guidebook, 2017, para. 2.1 “Cost Estimating Methodologies”, pg. 22

• Missile Defense Agency Cost Estimating and Analysis Handbook, 2012, Chapter 6

“Methodology” pg. 69

• GAO, Cost Estimating and Assessment Guide, 2009, Chapter 11 “Developing a Point

Estimate” pg. 107, Chapter 14 “Risk and Uncertainty Analysis”, pg. 153

• NASA, Cost Estimating Handbook, 2015, para. 2.2.2 “Task 5: Select Cost Estimating

Methodology”, pg. 14

Estimating Methods Training

The DAU Cost Estimating certification program for members of the Defense Acquisition Workforce offers

training relevant to the cost estimating methods. Additional information on each course may be found

in the DAU iCatalog (

https://icatalog.dau.edu/).

• BCF 130 Fundamentals of Cost Analysis, Lesson 5

• BCF 132 Applied Cost Analysis, Lessons 4, 6- 9

• BCF 216 Applied Operating and Support Cost Analysis, Lesson 3

• BCF 230 Intermediate Cost Analysis, Lessons 5, 6, 8, 9, 10

• BCF 250 Applied Software Cost Estimating, Lesson 4

• BCF 331 Advanced Concepts in Cost Analysis, Lesson 6

• CLB 023 Software Cost Estimating (overview of the Software Cost Estimating process and

highlights key issues)

• CLB 029 Rates (introduces the basics of wrap rate development as it relates to cost

estimating)

• CLB 034 Probability Trees (focuses on probability or decision trees, as they are used in the

context of cost estimating)

• CLB 035 Statistical Analysis (covers parametric and nonparametric analysis to support the

cost estimating process)

• CLE 076 Introduction to Agile Software Acquisition (explain what agile software acquisition

is and how it works for DoD software development)

The ICEAA publishes the CEBoK. The follow modules are relevant to methods:

• CEBoK v1.2, 2013, Module 2 “Cost Estimating Techniques”

• CEBoK v1.2, 2013, Module 3 “Parametrics”

• CEBoK v1.2, 2013, Module 7 “Learning Curve”

• CEBoK v1.2, 2013, Module 8 “Regression”

• CEBoK v1.2, 2013, Module 11 “Manufacturing Cost”

• CEBoK v1.2, 2013, Module 12 “Software Cost Estimating”

• CEBoK v1.2, 2013, Module 13 “Economic Analysis”

The following course numbers starting with FMF refer to the course number assigned by the FM

Certification process. Information on these courses (including eligibility requirements) can be found in

the FM myLearn system:

https://fmonline.ousdc.osd.mil/FMmyLearn/Default.aspx.

• FMF 1124 FMA 204 - Financial Management Concepts Course - Trend Analysis

• FMF 1551 QMT 490 - Current Topics in Cost Estimating

• FMF 1253 FMA 202 - Financial Management Concepts Course - Descriptive Statistics

• FMF 1550 QMT 290 - Integrated Cost Analysis

• FMF 1503 FMA 201 - Financial Management Concepts Course - Cost Estimates for Support

• FMF 1560 DoD FM 101 - Cost Analysis

• FMF 2802 Army 1.5-Hour e-Cost Benefit Analysis (e-CBA) Training class

• FMF 6175 AFIT Cost 669 - Advanced Cost Analysis

• FMF 6540 Analytic Cost Expert Distance Phase (ACE dL)

• FMF 7536 Applied Financial Planning - Breakeven Analysis

• FMF 7883 Data Analytics

Training opportunities specific to CADE include:

• CADE training videos: designed as a handy reference for the first-time user or seasoned

analysts that just need a refresher. Topics include: user guidance for the CADE portal, data

and analytics, plus “how to” guidance on CCDR, SRDR and available libraries are available at

https://cade.osd.mil/support/videos (public)

• CADE Pivot Tables for Analysts: https://cade.bridgeapp.com/learner/library (requires a

CADE login)

7.0 BUILD COST ESTIMATE MODEL

Cost estimating model development

should begin during program

definition and continue through the

cost estimate basis, data processes,

and the estimating methods

investigations. Starting the model

building process as early as possible

leads to a superior model design,

helps focus discussions, inspires timely

questions, and uncovers holes in the

data requirements early in the

process. This chapter summarizes the

cost estimate model characteristics

the analyst should be mindful of

throughout the model building

process.

Anatomy of a Cost

Estimate Model

The estimate structure is the skeleton that holds the estimate together and establishes the cost model

framework. The analyst populates elements of the estimate structure with estimating methods

supported by the data collection process. The core estimate structure of a life-cycle cost estimate

includes R&D, Production, O&S, and Disposal sections. The analyst assigns estimating methods at the

lowest level elements of the estimate structure, drawing on input values and intermediate calculations

performed elsewhere in the cost model. The analyst applies inflation, escalation, cost improvement

curves, scaling factors, and phasing methods as appropriate. The analyst must also capture sunk cost in

the estimating model.

Regardless of the tool used to create the cost estimate model, the structure should be centered around

the estimate structure which identifies all cost elements requiring a cost estimate. The analyst needs to

use careful judgment to settle on the necessary level of detail in the estimate structure. Greater detail

does not necessarily result in greater accuracy! The arrangement and level of detail needs to be

sufficient to perform the anticipated what-if and sensitivity analysis and provide all data necessary to

populate the required reports and charts. Building the model evolves in every step in the cost

estimating process.

As the estimate structure is developed, the analyst applies estimating methods at the lowest level of

detail and then sum the levels throughout the estimate. Each parent level is generally a simple sum of

subordinate elements. Exceptions to this include the case whereby the subordinates to a particular

parent element in the estimate structure do not capture all the anticipated cost and there is some valid

reason for not adding another subordinate element. In this case, the parent level formula will not be a

simple sum of its subordinate elements. This is just one possible exception to an otherwise simple

estimate structure hierarchy summation. An analyst should document any deviations from the simple

summation approach.

The analyst needs to apply cost estimating methods consistent with their derivation. For instance, the

analyst may need to adjust available CER/SER inputs and results based on the particulars of the CER/SER.

The analyst must make model adjustments when the:

• CER result produces a different unit of cost than other elements of the estimate structure

(e.g., $K vs. $M),

• CER result has fee, General and Administrative Expense (G&A), and/or overhead, but other

elements of the estimate structure in the cost model do not,

• CER source data comes from programs with significantly different framing assumptions,

schedules, or risks/opportunities than the program being estimated,

• input parameters have different units than the data used to create the CER,

• source rates apply to a different labor mix than used in the estimate,

• CER source data durations are significantly different than the item being estimated, and/or

• source data risks/opportunities do not address all the current project risks/opportunities.

When elements of the estimate structure relate directly to one another, the model should establish a

mathematical link whenever possible. For instance, if the quantity of one item is always an exact

multiple of another, then the one element should be mathematically dependent upon the other in the

model rather than having to manually change both quantities when performing what-if or sensitivity

analysis. Minimizing the number of overrides necessary to achieve a given what-if or sensitivity result

reduces the potential for manual entry errors, especially if many variations need to be explored. The

analyst must apply functional relationships wherever feasible to:

• help ensure consistency between trade studies (what-if cases),

• minimize overrides necessary to achieve a balanced estimate for a specific alternative (e.g.,

if production doubles, O&S follows automatically),

• improve the performance of simulation methods to address risk/opportunity, and

uncertainty, by helping to ensure the simulation behaves correctly. (See Section 7.4 for a

discussion of simulation methods.), and

• reduce errors.

The following sections address specific considerations for the cost model.

7.1.1 Characteristics to Simplify the Cost Estimate Model

Due to the complex nature of the programs being estimated, it is easy for a cost model to become very

complicated, very large, or both very quickly. Early in the development of the model, the analyst should

consider how to keep the model as simple as possible. Possible design considerations include:

• creating the simplest structure possible, consistent with the intended purpose and scope,

• building the cost model such that it is easy to add, remove, and modify elements necessary

to perform sensitivity and what-if analysis, and

• listing cost drivers and other parameters in a clean, systematic way and in a central location

to avoid duplication of data,

• developing concise, clear, and complete model documentation,

• developing a disciplined, concise, and easy to find way to record the history of significant

changes to the model, emphasizing changes from the previous version.

Model design suggestions that are more directly related to Microsoft Excel or other spreadsheet-based

models include:

• color-coding the model elements,

• making good use of the cell and cell range naming features (take care to delineate between

workbook and worksheet range names),

• exploiting array range names to reduce the number of unique formulae,

• creating conditional formatting and alerts that identify when impossible or irrelevant values

occur in the model,

• avoiding long, difficult, and/or complex formulae where possible,

• adding comments to help explain unusual formulae,

• avoiding the use of Microsoft Excel functions that cannot be traced through the

precedent/dependent feature,

• breaking down a complex section into its constituent parts,

• considering the use of a Data/Validation tool in Microsoft Excel to format cells so that

another user cannot input inappropriate values into the model,

• keeping links between sheets to a minimum,

• avoiding links to other workbooks, and/or

• avoiding writing macros.

A word of caution, some analysts have found that excessive use of conditional formatting, links, complex

formulae, and embedded features (e.g., cell validation) can severely impact performance. In particular,

large cost models that also make use of simulation methods can be overly stressed. It is up to the

analyst to find a balance between exploiting these features while retaining cost model stability and

acceptable calculation speed.

7.1.2 Phasing

Phasing is the allocation of a cost estimate over the program’s FYs to ensure adequate budget authority

is in place to achieve key program event dates. It should also be consistent with any constrained budget

realities. An analyst is required to forecast the spending profile across FYs in order to capture the

impact of inflation/escalation and other program unique considerations (discussed below) to develop

annual budget requests. It is essential that the model documentation explicitly defines the basis for the

chosen phasing profiles. There are two fundamentally different ways to develop a phasing profile from

historical data:

• Obligations: is where analyst bases the estimated obligation profile on historical or planned

obligation data. In this case, the profile may be applied directly to a properly

inflated/escalated cost estimate.

• Expenditures: is where the analyst bases the estimated spending profile on how the

program intends to spend money, then converts to obligation authority. Typical sources for

this method are CSDR or EVM data. In this case the time phased estimate (either a CY or CP

dollar profile) of resources must be converted to an obligation profile, which involves a

number of considerations, discussed next.

Converting a CY dollar expenditure profile using published appropriation indices is generally insufficient.

For instance, if an estimate identifies $100 CY$ in the first year, the appropriation indices may only

adjust this number by a few percent. In fact, given that many appropriations allow dollars to be

obligated and expended over a number of years, it may be necessary to substantially increase the first

year’s CY dollar estimate. Inflation/escalation adjustments need to be applied consistent with the 2017

CAPE Inflation and Escalation Best Practices for Cost Analysis: Analyst Handbook. Analysts are

encouraged to complement the CAPE guidance with Component-unique procedures, as applicable. In

general, the conversion of a CY dollar spend profile to a TY dollar should account for realities, such as:

• RPC (to convert to CP$), and

• an outlay profile that considers,

o termination liabilities,

o fee payment plan,

o invoicing cycles,

o long lead items, and

o supply chain commitments.

An analyst can estimate phasing at the program level or at any lower level of the Program WBS. He/she

should exercise caution when phasing at lower levels to ensure the total Program phasing profile is

consistent with the total resource (e.g., staffing) levels estimated. Analysts commonly use spreading

functions such as Uniform, Trapezoid

, Beta, Rayleigh, and Weibull because they provide some control

over how the model prorates costs across time

. Ideally, the analyst bases the selection of a spreading

function on relevant historical data. However, Components may provide guidance on selecting and

implementing preferred methods. In reality, these functions simply estimate the percent of total

spending within a given time frame. Consequently, the analyst can use a percent-per-time-period

directly to spread a total. The weakness of the percent-per-time-period spreading method is that it is

not dynamic and requires a greater degree of manual intervention to perform time-sensitive what-ifs.

An important, and often overlooked, phasing aspect is the need for dynamic phasing and estimate

structure linking:

• Dynamic Phasing: If baseline production quantities increase beyond the annual capacity,

the analyst must account for procuring additional quantities and any O&S implications. It

could mean increasing annual costs or extending production and/O&S durations. Ideally,

the selected method for spreading the new quantities or estimating O&S costs changes

dynamically to be consistent with annual capacity constraints.

• Estimate Structure Linking: In a schedule model

, the start and/or finish date of one

activity may influence the start or finish date of one or more other activities (called

dependencies). Analysts purposely build schedule tools to apply activity dependencies and

other scheduling attributes. Mimicking schedule model dependencies in a cost model is

extremely difficult. However, the 2014 JA CSRUH para. 2.2.5 “Duration Sensitive Cost

Estimating Methods” provides some guidance on where such linkages are feasible in a cost

model and how to implement them. Doing so will not replace the need for a schedule

model, but it does facilitate one of the most common cost estimating what-if drills:

schedule changes.

The analyst should automate dynamic phasing and linking elements of the estimate structure as much as

possible to minimize errors and to support any contemplated simulations. (See Section 7.4 for a

discussion on simulation methods.)

DoD is emphasizing the acceleration of program acquisition schedules by categorizing some programs as

MTA. (See Section 1.2.1, 10 USC Sec 2430 for an introduction to MTA). In order for a program to have a

reasonable chance to meet rapid prototyping / rapid fielding schedules, the typical time phasing profile

may not be sufficient. Early material purchases, hardware/software prototypes, and dual supplier

Trapezoid is a convenient way to combine a ramp-up, steady state and ramp-down spending profile.

They are also common distributions used to model the uncertainty of equations or parameters in a simulation

model.

The 2015 GAO Schedule Assessment Guide describes schedule modeling. The preface states that “A cost

estimate cannot be considered credible if it does not account for the cost effects of schedule slippage.”

activities required to accelerate program schedule may drive up front funding requirements over and

above durations for major capability programs. The cost analyst should consider making discrete

adjustments to phasing profiles drawn from major capability programs. He/she should exercise caution

with this phasing strategy because it may incentivize the program to maintain higher staffing levels for a

longer period of time in the event schedule delays occur. Figure 4 illustrates how an accelerated

program may impact the program budget and the potential consequences of subsequent schedule

delays.

Figure 4: Notional Major Capability Acquisition Budget Profile vs. a Notional MTA Program Schedule

Determining the impact on annual funding requirements from different production quantity phasing

profiles or OPTEMPOs are common what-if drills. Building a model that facilitates such investigations

should be a priority. The 2014 JA CSRUH recognizes the challenge of developing schedule features into a

spreadsheet based cost model. Chapter 2 of that handbook provides guidance on how to build a cost

model that automates changes in duration

that influence the cost estimate results. (See the 2015 GAO

Schedule Assessment Guide for schedule modeling best practices.)

7.1.3 Sunk Cost

A sunk cost is a cost that the program has already incurred and cannot be readily recovered by the

program. This is usually in the form of costs expended or obligated in the current year or prior years. If

the program being estimated is well into development or production, it may be necessary to incorporate

sunk costs and adjust estimating methods to address the remaining cost (cost to-go

). An analyst may

draw the sunk cost from actual early R&D and production costs (for acquisition costs) and fielded

systems (for O&S costs). In addition to capturing the sunk cost to build a complete cost estimate, the

analyst can use findings of the completed work to refine the estimate. For example, the analyst should

use test and evaluation results, including reliability and maintainability projections, to refine O&S cost

estimating methods.

The 2014 JA CSRUH focuses the concept of a “cost informed by schedule method” (CISM) suitable for

spreadsheet models. It also introduces the “fully integrated cost/schedule method” (FICSM), which require special

purpose tools. Variations on FICSM are embraced by NASA, the oil and gas industry, and others.

The cost estimate for specific elements of the estimate structure will be the sum of sunk costs and the cost

remaining, referred to in this guide as the “cost to-go”.

An analyst may deem authorized and obligated program funds from prior years as sunk costs even if the

program has not yet completely expended them. A life-cycle cost model should contain current and

prior-year sunk cost as part of a system’s total life-cycle cost. The cost estimating model should report

sunk costs and cost to-go in order to facilitate comparisons with the total cost of previous estimates.

Updating an estimate to include sunk cost is very challenging, particularly if the analyst needs to allocate

sunk costs across elements of the estimate structure. The process begins with a firm grasp on the

difference between costs produced by the estimating model and the collected sunk costs. The analyst

must consider if the sunk cost is in terms of obligation or expenditure in light of how the model has been

time-phased as described in Section 7.1.2. Typically, the analyst should trace the source of the sunk

cost back to the obligation year and apply that accordingly in the cost estimate. The 2014 JA CSRUH,

paragraph 2.8.2 “Sunk Costs” provides a detailed discussion of this process and an example.

Reports such as IPMRs or CSDRs represent actuals-to-date and forecasts for contracts and may not

include the detailed estimate structure information necessary to trace the cost back to the obligation

year. If using these data sources, the analyst makes adjustments so that the accruals are properly

entered as a sunk cost into an obligation estimate.

Addressing the impact of sunk costs on the estimating method can be complicated. The analyst

generally derives the estimating method from an analysis of total cost, not on cost to-go from some

point in the source program(s). Subtracting the sunk cost from the total estimate method to arrive at

cost to-go may make sense, but defining how much of the risk/opportunity, and uncertainty remains in

the cost to-go portion is more difficult to assess. Again, the 2014 JA CSRUH, paragraph 2.8.2 “Sunk

Costs” provides some guidance.

7.1.4 Cost Modeling Tools

Analysts build most DoD cost estimating models in Microsoft Excel or Automated Cost Estimating

Integrated Tools (ACEIT). Some organizations have built Microsoft Excel templates in an effort to bring

consistency to model building and facilitate their management. The Army requires the use of ACEIT on

all ACAT I and II programs

. There are also many tools available to support specific parts of the cost

estimating process such as statistical analysis, software cost estimating, data visualization, and

simulation. In addition to Microsoft Excel and ACEIT, system dynamics models and data science

applications like R

and Python are becoming popular for specific analysis, especially as data files get

larger. Analysts need to select tools to support the cost estimating process as outlined in this guide.

Analysts should not tailor the cost estimating process simply to accommodate the constraints of any

particular tool. Each Component promulgates their own guidance and preferences for the use of tools

and identifies the available training.

7.1.5 Multiple Cost Models for One Program

Large cost estimates are often broken into pieces to cope with very large programs, geographically

disperse analyst teams, and related realities. For example, an aircraft procurement cost model could be

broken into structure, propulsion, avionics, and then everything else. In such cases, the owners of each

cost model must collaborate to a high degree in order to combine the estimates and ensure a universal

Office of the Assistant Secretary of the Army Memorandum “Automated Cost Estimating Integrated Tools (ACE-

IT)”, 15 April 2004

R Core Team (2013). “R: A language and environment for statistical computing”, R Foundation for Statistical

Computing, Vienna, Austria. http://www.R-project.org/ .

understanding of common variables and results. The cost team should identify a single lead who is

made responsible for defining and integrating all the cost model pieces.

7.1.6 Common Cost Metrics

Although every cost estimate is unique, there are common metrics that the cost community uses to

discuss or compare estimates. Analysts should be aware of these metrics and build the cost model so

they are easily calculated.

The most common metrics are:

• Flyaway/Sailaway/Rollaway Cost: Sum of prime mission equipment, SEPM, system test

and evaluation, warranties, engineering changes, nonrecurring start-up production costs,

and other installed GFE.

• Weapon System Cost: Procurement cost of prime mission equipment plus the

procurement cost for support items.

• Procurement Cost: Cost of prime mission equipment, support items, and initial spares.

• Acquisition Cost: Sum of development costs for prime mission equipment and support

items plus the sum of the procurement costs for prime mission equipment, support items,

initial spares, and system-specific facilities.

• Life-Cycle Cost: Total cost of the program including development, procurement, O&S, and

disposal.

• Total Ownership Cost: Life-cycle cost plus related infrastructure or business process costs

not necessarily attributed to the program.

Figure 5 represents the general relationship between these six terms. Commodity specific versions of

this chart may exist at the Component level.

Figure 5: Total Ownership Cost Composition

Additional metrics include:

• APUC: Total program procurement cost divided by the production quantity.

• PAUC: Acquisition cost divided by the sum of development and production quantities.

• O&S $/year: Total O&S Cost

divided by number of years of sustainment.

• O&S $/operating metric/year: Total O&S Cost divided by the system’s usage metric

divided by the number of years of sustainment. The operating metric will vary by

commodity. Common operating metrics are flying hours (aircraft), steaming hours (ships

and submarines), and driving hours (vehicles).

Develop and Interpret the Baseline Cost Estimate

A systematic and well-documented process for the development of the baseline cost estimate simplifies

the interpretation and use of the estimate. This section offers best practices to create the baseline cost

estimate.

7.2.1 Develop the Baseline Cost Estimate

The analyst should relate the baseline cost estimate directly to the program definition. The what-if or

uncertainty analysis should address the degree to which the model may underestimate or overestimate

cost. Estimating method drivers (e.g., weight, code count, volume, power, hours, rates) should reflect

documented baseline values and not some lower or upper bound. Additionally, the baseline cost

estimate should not include extra dollars inserted to address risk/opportunity or uncertainty (unless

directed by the program manager) because they are handled separately. However, the cost of risk

mitigation plans that the program manager intends to execute as part of the program of record should

be included in the baseline cost estimate.

The cost estimate type, purpose, scope, and Component guidelines all influence how to develop the

baseline estimate. The analyst needs to ensure the model:

• is consistent with the program definition and the cost estimate basis,

• employs the best estimating method for every element of the estimate structure that

requires one,

• addresses any linkage between elements of the estimate structure and between input

variables where appropriate,

• applies inflation, escalation, phasing, cost improvement curves, and adjustments in a

defendable way,

• traces the cost drivers back to the CARD or other program definition documentation and

properly normalizes them,

• properly accounts for sunk cost and the affected estimating methods are adjusted to reflect

the cost to-go, rather than a total cost, and

• results at every level in the estimate structure are in a consistent dollar type (e.g., CY or TY),

year, and unit (e.g., $K, $M, $B).

After developing the baseline estimate, the analyst interprets the results at all model levels as discussed

in the next section.

7.2.2 Interpreting the Baseline Cost Estimate Results

Interpreting the cost estimate results begins with understanding where each estimating method’s result

is located within the range of possible outcomes. The total cost estimate is the sum of all cost elements

O&S cost is fully described in the 2014 CAPE Operating and Support Cost Estimating Guide.

and analysts often call it a point estimate because the result represents only one possible outcome.

Methods to estimate the bounds on the total estimate are discussed in Sections 7.3.2, 7.3.3, and 7.4.2.

Figure 6 illustrates how plotting the minimum, point estimate and maximum of subordinate elements

can improve the understanding why the point estimate at the total level falls where it does. In this case,

it is quickly evident that all the point estimates gravitate towards the minimum, in some cases

significantly. This may be cause for further investigation to verify the results are realistic.

Figure 6: Point Estimate Location Within a Range of Possible Outcomes

If an analyst uses a simulation method to estimate the point estimate bounds, then he/she should use

the point estimate for each of the lowest level results as a reference point to define the distribution of

possible outcomes. The location of the point estimate in the distribution

of the estimating method

result is a critical step in building a simulation model. Whether simulation is used or not, understanding

what the cost model is delivering (e.g., mean, median, mode, or something else) at each level of the

estimate structure is an important step towards interpreting and using the cost model results.

The analyst simplifies result interpretation if they use only estimating methods that produce a mean (or

average) cost. In that case, the result at each aggregate level is also the mean. However, since this is

rare, the following are a few cases where a result interpretation may differ across elements of the

estimate structure:

• Analogy: The analogy method adjusts an actual cost from the analogous program. The

estimating methods used to develop the adjustments to actual cost (e.g., additions,

subtractions, scaling factors) drive results interpretation.

• Build-Up: The build-up estimating method itself is exact. For example, hours times a labor

rate produces an exact cost. The uncertainty of a build-up result is a function of how the

analyst derives the inputs (hours and labor rates). Hours, for instance, could come from a

It is not always possible to anchor an uncertainty distribution to the point estimate result for a particular

element of the estimate structure, but it is an excellent way to help ensure the distribution scales and/or changes

shape properly when performing a simulation on a what-if drill. In most cases, the point estimate can serve as one

point (mean, median, upper bound, lower bound, something else) and the other distribution parameters required

to uniquely define the distribution can be scaled off of it. This is an effective way to help ensure distributions

remain meaningful when applied to what-if cases.

parametric estimating method. The labor rate could be a weighted average composite of

an assumed labor mix that may or may not match the program.

• Extrapolation from actuals: Extrapolation is often a specific type of univariate or

parametric estimating method. However, instead of using historical data from analogous

programs, the extrapolation method uses actual costs from the program being estimated.

This does not eliminate uncertainty in the estimate. The analyst needs to interpret the

result consistent with the mathematics used to perform the extrapolation.

• Parametric: Some parametric regression methods include an objective calculation of the

estimate error. The distribution of the error is an assumption, not necessarily a fact. For

the ordinary least squares (OLS) regression method, the assumption is that the method

produces the mean of a normal distribution. A log-linear form, however, yields the median

of the potential results (unless a correction factor is applied). True nonlinear regression

methods are not so straight forward to interpret, and the analyst may refer to the 2018 JA

CER Handbook for guidance.

• Univariate: The univariate method delivers several result types to choose from. For

example, if the analyst collects labor rates from a number of manufacturers (because the

performing company has not been selected), he/she could choose the mean or the median

value. If there are enough data, the analyst may choose to fit them to a distribution shape

and select the mode

• Tools: Some tools provide a framework to facilitate building, troubleshooting, and

generating documentation (e.g., Microsoft Excel, ACEIT). Other tools (e.g., commercial

parametric models) contain built in estimating methods to develop a point estimate. The

analyst must interpret the tool’s point estimate, which the tool may or may not have

documented. The analyst also needs to know how well the data supporting the tool results

compares to the program.

• Expert Opinion: Interviews with individuals or teams of experts invariably lead to estimates

identified as “most likely” or “most probable” or “on average”. That type of

characterization is never enough. The potential bounds of the estimate are essential for

the analyst to interpret the estimate meaning. There should be no comfort taken in

labeling an estimate as most likely or the average without also knowing the range of

possible outcomes. There could easily be compelling evidence that demonstrates a high

probability of an adverse outcome (e.g., the underlying spread of potential values is highly

skewed). Identifying the potential spread is an essential part of the expert opinion

interpretation.

Just because the analysis yields a mode, that is insufficient to characterize the estimate. A most likely value may

still have a high probability of overrun if it is the mode of a highly, right skewed distribution.

Review the Initial Results

At this point in the cost estimating

process, the analyst has a

preliminary cost estimate to review

and validate. This guide describes

validation as performed to ensure

the cost estimate is consistent with

the program definition and that it is

traceable, accurate, and reflects

realistic assumptions

. The

objective of the validation process is

to ensure the cost estimate is

credible, comprehensive, accurate,

and well documented. Iterating

through previous steps to refine and

correct initial results is a normal

part of the model building and

validation process. The estimate

achieves credibility and

comprehensiveness, in part, by

showing the estimate has captured all aspects of the program, including all excursions required by the

stakeholder. Validating cost estimate behavior also address credibility and accuracy. Chapter 8.0

discusses documentation in more detail.

Once the analyst builds the model, he/she validates its credibility and accuracy via crosschecks,

sensitivity analysis, and what-if analysis. This section discusses each of these topics.

7.3.1 Crosschecks

First-level crosschecks simply apply common sense (also known as sanity checks). For example, knowing

that the results should be in millions, but the results are in billions is evidence something is awry with

units in the estimate. Adding new elements with no discernable change to the total is similar evidence

of an error in the modeling logic.

Once past the sanity checks, an analyst can perform more detailed crosschecks by entering cost driver

data for analogous programs and verifying the model results reasonably match. For larger models, it

may not be feasible to do these at all levels. In such cases, the analyst needs to find ways to perform a

crosscheck for as many of the lower level elements of the estimate structure as possible. Sources for

crosschecks might include comparisons with similar historical programs and realism checks with SMEs.

It is good practice and often necessary to employ more than one cost estimating method for the more

contentious, complex, and/or expensive elements of the estimate structure to serve as crosschecks for

these particular elements. The analyst expects the chosen primary estimating method to yield the best

The 2010 NCCA Cost Estimating Guide, para. 1.5 “Verify and Validate Cost Estimate” and the 2008 AFCAA Cost

Analysis Handbook, para. 14-11 “Independent Verification and Validation” treat verification and validation

separately. The 2009 GAO Cost Assessment Guide “Chapter 15, “Validate the Estimate”, as does this guide, treats

the same concepts under one heading.

results in terms of realism, accuracy of the result

, completeness, and supportability of the estimate.

The analyst should use second and possibly third alternative crosscheck methods to corroborate the

primary method results. The crosscheck methods can serve as fallback positions in the event of data

non-availability, disappointing statistical results, or if the analyst anticipates significant controversy

among stakeholders. Additionally, incorporating several methodologies can help establish bounds for

the purposes of evaluating sensitivity and uncertainty.

The model can include crosschecks alongside the primary method but tagged in such a way that they do

not sum to the total. At a minimum, the analyst should perform crosschecks for the most important

cost drivers.

7.3.2 Sensitivity Analysis

Sensitivity analysis assesses the extent to which costs at various cost estimate levels react to changes in

cost drivers. If a specific cost driver change results in a relatively large change in an element of the

estimate structure, then the analyst can consider the cost estimate sensitive to that cost driver.

Analysts perform sensitivity analyses to test that the model delivers realistic results for cost driver

values over their potential range. In good sensitivity analyses, the analyst changes cost driver values

based on a careful assessment of their underlying uncertainties. If the model behaves correctly, then

the analyst should test the limits of the model by assigning cost driver values outside the expected

bounds to allow for unexpected values during what-if excursions and the application of simulation

methods.

Best practice cost models incorporate the ability to perform sensitivity analyses without altering the

model, other than changing through-puts

or cost driver values. This is where the analyst’s effort to

automate the cost model can pay off. The analyst conducts sensitivity analysis by changing a single cost

driver and holding all other model inputs constant. Automation should ensure that linked cost drivers

that must change with the one undergoing sensitivity analysis do so in an appropriate manner. For

example, if the program must procure three of item A for every one of item B, the model should

automatically account for this relationship. Additionally, if one element of the estimate structure is a

function of the total cost of one or more other elements of the estimate structure, the analyst should

build that link into the model. A well-automated model provides a more realistic assessment of cost

driver sensitivity. A systematic analysis yields those cost drivers that have the most impact on the

model. The estimating methods associated with the top cost drivers are the ones that are the most

important to refine.

The analyst documents the source (e.g., SMEs, historical information, contract documents), rationale

and results associated with the sensitivity analyses along with potential best and worst case values.

Analysts often use tornado charts (see Section 8.3.3) to present this type of information.

Sensitivity analysis helps identify where the analyst should focus risk/opportunity and uncertainty

analysis. It can identify areas in which design research (risk mitigation) may be warranted or areas in

In this guide, accuracy in the context of choosing between estimating methods is defined as the result with the

narrowest uncertainty range. The term realism is used to describe how closely the result compares to the correct

result. Accuracy of the collected data is discussed in Section 5.5.3.

Through-puts are cost or cost driver values entered directly into the model. Catalogs (Section 6.2) and Sunk cost

(Section 7.1.3) are an example of cost values that can be entered as a through-put.

which the program can improve performance without a significant impact on cost. The impact of

changing more than one cost driver in the model is the subject of the next section.

7.3.3 What-If Analysis

The analyst performs sensitivity analysis to verify that the model behaves realistically when a single cost

driver is changed and to identify those cost drivers that have the most influence on cost. A what-if

analysis assesses the impact of changing multiple cost drivers, and automation facilitates the modeling

of numerous what-if drills. The analyst must take care to ensure that changes in one or more cost

drivers do not invalidate estimating method inputs values that are not linked. For example, if quantities

and production rates are changed, the coefficients of any associated cost improvement curve may have

to change.

Many times, the program manager will ask the analyst to run excursions (often known as drills) on

different programmatic parameters like quantity, schedule, or fuel prices. This is also a type of what-if

analysis.

Addressing Risk/Opportunity, and Uncertainty

Section 1.5.8 defined risk/opportunity, and uncertainty. Analysts address risk/opportunity, and

uncertainty in different ways. Each approach has its place, and each Component provides specific

guidance on how to address them. A summary of the most common methods (listed alphabetically)

include:

• Case-based Risk

: The analyst develops one or more what-if cases from a detailed analysis

of what could go wrong or right in the program; the baseline estimate does not capture

these aspects. The focus is on determining how the schedule and the cost per unit duration

(dollars per hour, per month, etc.) changes should the risk/opportunity event occur. For

example, if a test fails, the analysis establishes the impact to the schedule and the resulting

impact to the model’s duration-sensitive estimating methods. Additionally, the analyst

must assess how the program might have to change to address the test result. The

strength of this process is that the analyst can directly link the cost change to one or more

specific events in a way that is easy to understand. It also provides the program office the

basis for devising effective risk mitigation plans. The CAPE prefers the case-based risk

method.

• Method of Moments

: This is an analytical approach to estimating total program

uncertainty. It relies on the fact that the sum of individual elements of the estimate

structure means and variances equals the mean and variance at the total level. A closed

form analytical method is also available to account for how correlation across elements of

the estimate structure impact the total variation.

The total mean and variation defines an

assumed distribution shape at the total level such as normal, lognormal, or beta. Method

of moments is useful when there is a need to sum large numbers of correlated uncertain

elements.

A different process with similar goals is documented in Garvey, Paul R. 2008. “A Scenario-Based Method for Cost

Risk Analysis” Journal of Cost Analysis and Parametrics.

Young, Philip H. 1992. “FRISK: Formal Risk Assessment of System Cost Estimates”

See 2014 JA CSRUH para. 3.3.3 “The Impact of Correlation on a Cost Model”

• Simulation

(Inputs-Based): Analysts use this problem solving technique to approximate

the probability of certain outcomes by executing multiple trial runs. The analyst assigns a

probability distribution to each uncertain cost driver and estimating method to describe its

possible values. The analyst either builds correlation across uncertain elements into the

functional arrangement of the model or applies it as required. Additionally, the analyst can

model events to address risk/opportunities that the uncertainty assessment does not

capture. He/she uses a tool (or Microsoft Excel) to randomly select values for all uncertain

variables to create and then calculate a what-if case. The tool repeats this process enough

times (hundreds or thousands) to generate statistically significant distributions of outcomes

for the cost elements of interest. Analysts must take care to ensure the simulation does

not generate trials where the combination of cost driver values represents an impractical

scenario. He/she can mitigate this by using functional (mathematically linking inputs) and

applied (user inputs into the simulation tool) correlation.

• Simulation (Outputs-Based): This variation of the simulation method applies uncertainty

directly to the cost model outputs rather than to the model’s estimating methods and

inputs. The analyst assigns uncertainty distributions to the outputs of elements in the

estimate structure to address the combined uncertainty of the cost method and the cost

method inputs

. He/she can also assign the impact of risk/opportunity events.

The need to address correlation in the method of moments and simulation methods cannot be over

emphasized. Aggregate uncertainties can be significantly understated if correlation in these methods is

ignored. There are techniques available to measure the correlation present in a simulation model to

identify where it may be under or overstated. Guidance on how to measure, interpret, and address

correlation in simulation methods is fully addressed in the 2014 JA CSRUH paragraph 3.3 “Measure Then

Apply Correlation”.

7.4.1 Risk/Opportunity

The program office is responsible for identifying risks/opportunities that may affect cost, schedule, and

performance. Program office documents provide starting points for determining what areas of risk and

opportunity to address. Additionally, framing assumptions, ground rules, and cost estimating

assumptions (see Section 4.2) may identify potential risks/opportunities. The program office usually

produces a risk register, which lists risk/opportunity events, the probability of the event occurring, and

the impact the event will have on the program should the event occur. The challenge for the analyst is

to determine which, if any, of the risk register events he/she has not already captured in the baseline

point estimate through the estimating methods directly or the process used to address estimating

method uncertainty. It begins with a thorough understanding of the risks/opportunities addressed in

the source data used to generate the estimating methods. This is a good example of when SME advice is

indispensable. Program managers need assurance that the cost model is not double or triple counting

risks/opportunities. Knowing the data, knowing the program risk register (which should also capture

opportunities), and pointing to advice from the appropriate SMEs is a good way to address this

challenge.

Simulation is often referred to as “Monte Carlo”. In fact, Monte Carlo is but one way to develop a string of

random numbers, the heart of the simulation method. There are many others; Latin Hypercube may be the most

popular.

One source for outputs based distributions is the 2010, AFCAA Cost Risk and Uncertainty Analysis Metrics

Manual (CRUAMM).

Risks that should not be captured in cost models includes the possibility of labor strikes, natural

disasters (e.g., hurricanes, earthquakes), industry collapses (e.g., bankruptcies, litigation), mission

changing events (e.g., space shuttle disaster), and world events (e.g., September 11th).

Capturing risk/opportunity impacts in the cost model can be simple if there are only a few such events.

If there are only a few, then the analyst builds what-if cases to assess the impact if the risk/opportunity

is realized. If there are many, it may be necessary to build a simulation. The 2014 JA CSRUH provides

guidance on how to capture risk/opportunity in a simulation model. The next section addresses

uncertainty.

7.4.2 Uncertainty

Program managers and stakeholders need to have a sense of the likelihood the budget (e.g., the cost

estimate) may be exceeded. An analyst can establish this probability by estimating the risk/opportunity,

and uncertainty resident in the estimate. To estimate the uncertainty of the results, the analyst first

needs to determine which elements of the estimate structure to assess for uncertainty. In general, the

analyst should assess the estimating methods and inputs of the elements of the estimate structure that

contribute the most to the total should be considered. Their estimating methods and their inputs need

to be assessed.

There are cost model data that an analyst can be treat as certain. They include:

• Statute and Policy: Values such as formally published discount rates and appropriation

inflation rates.

• A design fact: For example, for each item A, the system requires three of item B.

• Sunk cost.

• Unit of measure conversion factors: For example, yards to meters.

Data that can vary, but best treated as what-if cases when applying the simulation method, include:

• Quantities: It is uncommon to allow quantities to be flexible. Typically, they are either X or

Y amounts and as such, best treated as discrete what-if cases.

• Schedule: While there are methods available to cause cost models to be somewhat

reactive to uncertain schedules (see 2014 JA CSRUH), cost models tend to treat changes in

schedule as a what-if case. This can make it easier to explicitly identify the cost impacts

across the program for a schedule slip.

• Custom Inflation/Escalation: Both are highly uncertain, but there is no widely accepted

method to capture their uncertainty in a cost model.

The analyst can estimate uncertainty for the lowest level elements of the estimate structure through

what-if analysis. This is accomplished by estimating the results when inputs to the estimating method

are their most favorable, most likely, and most unfavorable. Total uncertainty can likewise be

investigated through the what-if analysis of specific scenarios (most favorable, most likely, most

unfavorable results) for a combination of elements of the estimate structure. The advantages of this

method include that it is straight forward to perform, the what-if cases are easily understood, and

potential model behavioral problems are more easily detected. A key disadvantage is that each

estimate is itself uncertain, representing just one possible result for a given set of conditions.

Method of moments is the next level of analytics to estimate total uncertainty. However, method of

moments can quickly become unmanageable as the complexity of the cost model increases. Even

simple estimating methods that rely on uncertain inputs to a method that itself is uncertain adds

complications that can be time consuming to address.

Simulation is a popular method to address uncertainty. The 2014 JA CSRUH provides detailed

instructions for building a simulation model that is independent of the tool used to perform the

simulation. The 2014 JA CSRUH applies the simulation method to a realistic cost model to show that the

uncertainty results throughout the model are effectively the same, regardless of the tool used. This is

demonstrated by building the model in three different simulation products and comparing results at any

level in the estimate structure.

Iterate as Necessary

At this point in the process, the cost model is almost complete and is producing results. There are many

reasons to circle back through the cost estimating process. While Figure 1 indicates iteration near the

end of the process, in reality it can happen at any point in the process. It may not be necessary to circle

back to program definition, but it is a good idea to do so to ensure the all aspects of the estimate remain

relevant and intact. Reasons to iterate include:

• Cost estimate basis change: Changes to the program requirement, framing assumptions,

ground rules, or cost estimate assumptions.

• Unexpected results or requirements: Unexpected results or the unexpected need for

results the model cannot deliver.

• Validation problems: When there is evidence the model is not behaving properly.

• Account for sunk costs: This is not a simple as it sounds. See Section 7.1.3.

• Automation: More automation may be required to facilitate what-if drills.

• New data: One or more of the estimating methods may need refining or replacing on the

discovery of new data.

• Superior estimating methods: The discovery of new and better ways to perform the

estimate can surface at any time.

Build Cost Model References

• CAPE, Operating and Support Cost-Estimating Guide, 2014, para. 5.3.4, “Estimate Costs”,

pg. 5-10 and para. 5.3.5, “Conduct Sensitivity Analysis”, pg. 5-11

• Department of the Army, Cost Analysis Manual, 2020, Chap 3 “Cost Estimating Process”, pg.

• NCCA, Joint Agency Cost Estimating Relationship (CER) Development Handbook, 2018,

Chapter 4 “Step 4: Validate CER” pg. 105

• NCCA, Joint Agency Cost Schedule Risk and Uncertainty Handbook, 2014, Chapter 2 “Cost

Informed By Schedule Method Model”, pg. 6 and Chapter 3, “Finish And Assess The CISM

Model”, pg. 45

• NCCA, Cost Estimating Guide, 2010, para. 1.3 “Develop a Baseline Cost Estimate”, pg. 20,

para. 1.4 “Conduct Risk and Uncertainty Analysis”, pg. 33, para. 1.5 “Verify and Validate the

Cost Estimate”, pg. 46

• SPAWAR, Inst 7110.1 Cost Estimating and Analysis, 2016, Enclosure 1, para. 5 “Develop

Baseline Cost Estimate” pg. 5, para. 6 “Conduct Risk and Uncertainty Analysis” pg. 12, para.

5 “Verify and Validate Cost Estimate” pg. 21,

• USMC, Cost Analysis Guidebook, 2017, para. 3.2 “Develop A Baseline Cost Estimate”, pg. 43;

para. 3.3 “Conduct Risk/Uncertainty Analysis”, pg. 46, and para. 3.4 “Verify and Validate the

Cost Estimate”, pg. 49

• Missile Defense Agency Cost Estimating and Analysis Handbook, 2012, Chapter 7

“Sensitivity Analysis” pg. 81 and Chapter 8 “Cost Risk and Uncertainty Analysis” pg. 89

• GAO, Cost Estimating and Assessment Guide, 2009, Chapter 11 “Developing a Point

Estimate” pg. 107 and Chapter 13 “Sensitivity Analysis” pg. 147, Chapter 14 “Risk and

Uncertainty Analysis”, pg. 153

• NASA, Cost Estimating Handbook, 2015, para. 2.3 “Part 3: Cost Estimate Tasks”, pg. 25 and

para. 4.1 “Sensitivity Analysis”, pg. 42

Build Cost Estimate Model Training

The DAU Cost Estimating certification program for members of the Defense Acquisition Workforce offers

training relevant to the cost estimating models. Additional information on each course may be found in

the DAU iCatalog (

https://icatalog.dau.edu/).

• BCF 130 Fundamentals of Cost Analysis, Lessons 11, 12, 13

• BCF 132 Applied Cost Analysis, Lessons 10, 11

• BCF 206 Cost/Risk Analysis, All Lessons

• BCF 216 Applied Operating and Support Cost Analysis, Lessons 4-6, 8-10

• BCF 230 Intermediate Cost Analysis, Lessons 9-11

• BCF 250 Applied Software Cost Estimating, Lessons 4-7

• BCF 331 Advanced Concepts in Cost Analysis, Lessons 6, 7

• CLB 031 Time Phasing Techniques (focuses on the methods that cost estimators can use to

time phase a cost estimate)

• CLB 038 Comparative Analysis (how various comparative analyses should be used to support

the cost estimating process)

• CLB 042 Cost Risk and Uncertainty Analysis (introductory framework for quantifying the risk

and uncertainty in cost estimates)

The ICEAA publishes the CEBoK. The follow modules are relevant to modeling:

• CEBoK v1.2, 2013, Module 9 “Risk”

• CEBoK v1.2, 2013, Module 13 “Economic Analysis”

• CEBoK v1.2, 2013, Module 14 “Contract Pricing”

The following course numbers starting with FMF refer to the course number assigned by the FM

Certification process. Information on these courses (including eligibility requirements) can be found in

the FM myLearn system:

https://fmonline.ousdc.osd.mil/FMmyLearn/Default.aspx.

• FMF 7883 Data Analytics

• FMF 7815 WKSP 0672 Data Analytics Tools and Techniques

• FMF 7816 WKSP 0673 Applied Concepts of Data Analytics Tools and Techniques

• FMF 1550 QMT 290 - Integrated Cost Analysis

• FMF 6175 AFIT Cost 669 - Advanced Cost Analysis

• FMF 6716 Risk and Risk Management

• FMF 3002 DCS 204 - Financial Management Concepts Course - Risk Management

• FMF 6540 Analytic Cost Expert Distance Phase (ACE dL)

• FMF 1503 FMA 201 - Financial Management Concepts Course - Cost Estimates for Support

Agreements

• FMF 1551 QMT 490 - Current Topics in Cost Estimating

8.0 FINAL RESULTS AND DOCUMENTATION

The cost estimate documentation is a

living document, and the analyst

should maintain and update it as the

program and cost estimate evolve.

Each of the Component cost

estimating guides and handbooks

includes instructions and best

practices on documentation. The

primary keyword used in these

reference documents with respect to

documentation is: understand.

Readers of the documentation should

be able to gain a full understanding of

the cost estimate, and another

analyst should be able to validate or

replicate the estimate. The estimate

documentation needs to clearly

identify:

• the organization that

performed it,

• when the estimate was performed,

• the reason for the estimate, and

• how was it developed.

Most of the estimate documentation should be devoted to how the estimate was developed. The

analyst shares the estimate documentation with stakeholders to ensure a complete and common

understanding of the results. The estimate documentation should portray a cost estimate that is

comprehensive, credible, and accurate. Finally, cost estimate documentation serves as a reference to

support future cost estimates.

Documentation varies in size depending on numerous factors, including the:

• size and complexity of the program,

• amount and level of data used in the development of estimate methodologies,

• number and type of different methodologies used in the estimate, and/or

• range of tabular and graphic reports required.

It is worth noting that analysts should not confuse the estimate documentation with a Basis of Estimate

(BOE). Although they contain much of the same information, a BOE is a formal term used by the DCMA

and the Defense Contract Audit Agency (DCAA). BOEs are formal deliverables provided by vendors

delivering products and services to the DoD. Estimate documentation normally includes this scope of

work in addition to the remaining program office activities beyond what the vendor provides.

Documentation Contents

The cost estimate documentation should include all thought processes and calculations used to develop

the results required by the stakeholders. Typical content includes the:

• purpose and scope of the estimate.

• description of the program definition,

• framing assumptions,

• program ground rules,

• cost estimating assumptions,

• estimate structure that expands on the program WBS to address purpose, scope and

anticipated what-if analysis,

• estimate structure index with dictionary,

• summary of the program IMS,

• cost, programmatic, performance, technical, and schedule data needed to support the

estimate,

• sources of data, explanation of veracity, and explanation of any exclusion and/or

adjustments,

• how data was normalized,

• identification of outliers and how they are handled,

• phasing of the project scope and deliverables,

• identification of potential risk/opportunities and uncertainty areas,

• proposed risk mitigation plans that impact the cost estimate,

• description of the estimating methods used to develop specific results,

• discussion of other estimating methods considered and why discarded,

• identification of estimating method limitations (e.g., viable range of inputs),

• recommendations for improving estimating methods and modeling approach in the next

iteration (e.g., identification of data which should/will become available, alternative

estimating methods that could not be investigated in this version)

• description of the inputs to define a baseline cost estimate,

• discussion of crosschecks, sensitivity, and what-If analysis (as required),

• cost estimate results including necessary charts and tables,

• cost estimate results match the final, post reconciliation numbers,

• changes to previous versions of the cost estimate, and

• description of how risk/opportunity and uncertainty is addressed.

Generate Final Documentation Report

The analyst should finalize and archive the list of documentation elements identified in Section 8.1 after

each estimate and maintain it throughout the life of the program. Ideally, overarching documentation is

consolidated into as few files as possible, preferably one, referencing all other documents supporting

the estimate. The analyst must retain all referenced documents.

There are several common elements in the cost estimate documentation. Table 10 provides a notional

organization for the cost estimate documentation content. These tables and figures serve as the focal

point for the reader as they provide a summary of the cost estimate. Although not every cost estimate

type requires each of the listed elements, most are applicable. Table 10 provides examples for content,

but the analyst should choose their documentation content on the specific cost estimate’s and

stakeholder’s needs. Additionally, the elements of Table 10 should indicate whether a cost estimate

result is reported in CY or budgeted TY dollars and identify the cost impacts associated with risk,

opportunity, and uncertainty.

Table 10: Common Cost Estimate Documentation Organization

Term

Definition

Summary Description

Key elements of project definition and the basis of

estimate to adequately explain the purpose, scope, and

structure of the cost estimate. Also includes framing

assumptions, ground rules, and cost estimate

assumptions.

Schedules

Programs with long and complex schedules should include

a summary level schedule that identifies key milestones,

quantities, and deliverable dates.

Estimate Structure Dictionary

Explains what is in (and, where appropriate, what is

excluded from) each element of the estimate structure to

help ensure the appropriate data and data types are

defined and categorized.

Cost Model and Results

Organized by Estimate Structure

A summary description of the cost model and results

organized by estimate structure. Results are normally

organized by life-cycle phase and dollar type (CY vs. TY).

Sand Chart

The total cost estimate by year and by phase or by year

and by appropriation. The chart illustrates the overlapping

of funds. A tabular form of the Sand Chart data often

includes prior approved values and current budget

controls for comparison.

Pareto Chart

A ranking of the top cost contributors (elements of the

estimate structure) to a target total cost.

Tornado Chart (Cost Contributors)

A ranking of cost contributors (elements of the estimate

structure) based upon their potential impact on a target

total cost estimate.

Tornado Chart (Cost Drivers)

A ranking of cost drivers based upon their potential impact

on a target total cost estimate.

What-If Analysis

Cost estimate of configurations other than the baseline

estimate. A thorough report on the scenario includes

sand, pareto and tornado charts for promising what-if

candidates.

CERs/SERs

A summary of the data sources, their normalization and

cost estimating methods employed to develop the

CERs/SERs for the top cost contributors along with

relevant validation results. Identification of outliers and

how handled. Identification of risk/opportunity events

and risk mitigation and how implemented in the model

The analyst must thoroughly document the estimating method, including the raw data set and the data

source. Subjective estimating methods must be documented with details on the source and the

estimate reasoning. The documentation of analogy adjustments and univariate estimating methods

should include applicable descriptive and inferential statistics. The 2018 JA CER Handbook fully

addresses how to document parametric CERs/SERs. Documentation of parametric CERs/SERs should

contain a succinct summary of the equation in a human readable form and include definitions for each

independent variable, their units of measure, and usage notes, such as the applicable range for each

independent variable. Analysts document parametric CERs/CERs developed from regression analysis by

explaining their derivation, the list of alternatives, and how the analyst evaluated the alternatives.

Parametric CER/SER documentation should summarize fit and predictive statistics along with the tools

or software used to calculate these statistics.

Fit statistics summaries should include t-statistics (significance of each coefficient) and the F-statistic

(significance of the CER/SER as a whole) to identify candidate CERs/SERs. If the CER/SER did not pass

any of the fit statistics, but is still used in the estimate, then the analyst should document the reasoning

for continuing with the CER/SER.

Analysts rely on predictive statistics to select the best CER/SER from the candidate CERs/SERs.

Predictive statistics (how well the CER/SER predicts the data and the estimate) include the coefficient of

determination (how well the CER/SER explains the variation in the data), standard error of the estimate,

confidence interval, prediction interval, and mean absolute deviation. These should be included in the

documentation. If any of the predictive statistics are unusual, the analyst should document the

justification for continuing with the CER/SER.

In situations where an estimate uses SME input(s) as its basis or for calibration, the documentation

should include the SME name, organization, and rationale for the input.

Present and Defend Results

In addition to detailed documentation, the cost team will prepare and present a cost estimate summary

for stakeholder consumption. The analysts tailor the presentation to meet the objectives of the review

and the needs of the decision makers and stakeholders. Clear, concise, and presented in a logical order,

these presentations normally begin with an overview of the key program definition and basis of

estimate elements that set the stage for the presentation objectives and the materials that follow. The

analyst is free to develop any tables and charts that are useful for telling the presentation story. The

analyst should have developed many of the tables and figures in the final results documentation (see

Section 8.2).

Any presentation should attempt to capture the entirety of the cost estimate documentation, only those

elements required to support the presentation objective. If stakeholders are fully aware of the program

definition, it may be appropriate for the presentation to begin with the relevant framing assumptions,

ground rules, and cost estimate assumptions. The estimating methods presentation should be limited to

the general approach, any specific difficulties in the process, and how the analysts overcame those

difficulties. The presentation should quickly get to the results for the stakeholders. Discussions of

estimating methods and mathematical calculations for the most important cost contributors and drivers

should be available, but presented on an as needed basis. Although the briefer(s) should be in position

to answer detailed questions regarding any aspect of the cost estimate, the presentation should provide

adequate information such that the audience gains an understanding of the estimate and provides

sufficient content to allow stakeholders to feel comfortable they are making decisions based on sound

and accurate results.

The remainder of this section provides an introduction to some commonly used charts. Components

generally provide specific guidance for presentation content. The sequence of the charts introduced in

the remainder of this session are loosely arranged to address: how much and when, what costs the

most, what is driving the cost, how are the funds allocated, and the program funding request.

8.3.1 Sand Chart

The sand chart displays values over time as areas. A common use is to illustrate the total cost estimate

by year and by phase or by year and by appropriation. This chart renders the different phased costs or

appropriations as layers (resembling layers of colored sand) or as stacked bar charts. The analyst should

use the layered version thoughtfully as it may be misleading in some use cases. For example, the data

supporting Figure 7 contains zero funding for FY18. Figure 7 however, suggests that funding is ramping

up during FY18 when it is not. A workaround is to begin the chart with FY19. However, by ending the

chart in FY35 (to avoid the appearance of dollars in FY36) leaves the question open: does funding end in

FY35 or did the x-axis end too early? The stacked bar chart, Figure 8, is less ambiguous, though perhaps

not as visually appealing as the sand chart.

Figure 7: Sand Chart (Layered) (notional)

Figure 8: Sand Chart (Stacked Bar) (notional)

8.3.2 Pareto Chart

The pareto chart displays the rank order of contributors to a selected item in descending order and a

line representing a cumulative total percentage. For example, Figure 9 presents the immediate cost

contributors to the production cost of a notional missile/ordnance system. In this example, payload is

the largest cost contributor immediately below production in the estimate structure. Typically, such

charts display the top elements that sum to 70-90% of the cost, depending on the number of elements

involved. The most left columns identify the biggest program cost contributors to the selected total cost

(in this case, production). However, they may not be the top potential contributors from a

risk/opportunity and uncertainty perspective. Tornado charts provide that insight and are discussed

next.

Figure 9: Pareto Chart (notional)

8.3.3 Tornado Charts

A tornado chart displays either sensitivity (Section 7.3.2), what-if (Section 7.3.3) or simulation (Section

7.4) results. The chart objective is to identify the cost drivers (sensitivity) and cost contributors (what-if)

that can have the most impact on the total program cost. The horizontal bar chart orders the widest

range in potential program cost at the top, with successive smaller impacts plotted below. The shape

resembles a tornado, giving rise to the chart’s name.

8.3.3.1 Cost Driver Tornado Chart

The cost driver tornado chart shows the results of a systematic sensitivity analysis. The analyst uses

three PEs to construct each horizontal bar

in Figure 10. The vertical line represents the program

baseline point estimate ($1,845 TY$M). The bar to the left represents the potential savings if the cost

driver takes on its most favorable value. The bar to the right is the most unfavorable value (from a cost

point of view). The bars in Figure 10 represent parameters and not elements of the estimate structure.

Tornado charts can also be produced from simulation results. (See the 2014 JA CSRUH, para. 4.1.5 for more

detail.)

The cost driver tornado chart is a useful tool for identifying parameters the program office may want to

consider for risk mitigation plans. In the case of Figure 10, speed is identified as the characteristic of the

missile that has the most impact on cost, and therefore worthy of attention.

Figure 10: Tornado for Cost Drivers Chart (notional)

8.3.3.2 Cost Contributor Tornado Chart

The analyst derives the cost contributor chart from what-if analysis. Each bar in Figure 11 represents

the cost impact after setting the cost drivers for one element of the estimate structure at a time to its

most favorable and unfavorable values. In this case, while the analysis identifies the propulsion

subsystem speed as the most important cost driver (see Figure 10), the combined uncertainty of the

guidance cost element inputs (accuracy and range) actually has a bigger potential impact. The bars in

the cost contributor chart are cost elements in contrast to parameters in the cost driver chart.

The analyst should not rely on one chart or one analysis to identify where the biggest impacts may occur

in the cost estimate. The pareto, cost driver tornado, and cost contributor tornado charts combined

may tell a more complete story than any one of them on its own.

Figure 11: Tornado for Cost Contributors Chart (notional)

8.3.4 Cost Element Chart

A cost element chart provides insight into how what-if cases or different estimates compare to each

other. Figure 12 compares a current O&S estimate (new program) with the ICE, a previous estimate,

and the legacy system. In this case, the chart should present the results in CY dollars if the legacy

program spans a vastly different timeframe. An analyst could produce similar charts for R&D,

production, or any lower level of the estimate structure. Supporting charts must explain any

differences.

Figure 12: O&S Cost Element Chart (notional)

8.3.5 Program Funding and Quantities Chart

Figure 13 provides an overview of key cost and quantity elements of a program cost estimate

, and the

analysts updates it throughout the acquisition process. The following is a brief summary of the

elements in the POM 2021 version of the chart. For detailed instructions, see the latest guidance from

USD(A&S). Variation to the chart are common (e.g. it may be organized by program phase or organized

appropriation) per Component requirements or best practices.

• Primary Line Items: List the primary budget line item(s) that fund the program. Footnotes

may be used for clarification/amplification.

• Prior: PB position submitted prior to the Current budget position.

• Current: Latest approved budget position.

• Required: Latest estimate of funds required to successfully execute program, e.g., support

the Warfighter and not simply match available budget TOAs. Typically, this would reflect

the Will-Cost

estimate, CCP, or POE that has not yet been validated by a Component Cost

Agency or the CAPE.

• System Operations and Maintenance (O&M): O&M-funded costs from initial system

deployment through end of system operations.

• Total Required Acquisition (BYXX$M): Current Estimate of total RDT&E, procurement,

military construction (MILCON) and acquisition-related O&M in BY dollars as reported in

the program's latest approved budget position. The percentage displayed is the portion of

the Acquisition cost out of the sum of Acquisition and O&S costs.

• Total Required O&S (BYXX$M): Current Estimate of total O&S costs in BY dollars. Disposal

costs should not be included in this value.

• Curr Est (APUC): Program manager’s current estimate of Average Procurement Unit Cost in

BY dollars (see Section 7.1.6).

• Curr Est (PAUC): Program manager’s current estimate of Program Acquisition Unit Cost, in

BY dollars (see Section 7.1.6).

• Δ Current: Program’s current APUC or PAUC divided by the program’s current APB Unit

Cost Reporting (UCR) baseline or equivalent, as applicable.

• Δ Original: Program’s current APUC or PAUC current estimate divided by the program’s

original APB UCR baseline, as applicable.

It is commonly known as the “Spruill chart”, named after Dr. Nancy Spruill a prominent figure in the Acquisition

community for many years and originator of this format.

See the should-cost, will-cost implementation memorandum at:

https://www.acq.osd.mil/fo/docs/USD(ATL)_Memorandum_on_Implementation_of_Will-Cost_and_Should-

Cost_Management_042211.pdf

Figure 13: Program Funding and Quantities (Spruill) Chart (notional)

8.3.6 S-Curve

An S-curve derives its name from its shape. It is one of the most common products of a simulation

model, and analysts use it to illustrate how cost changes with the probability. The analyst can also

produce it from the method of moments or applying a representative distribution from a source such as

the 2013, AFCAA CRUAMM. The CRUAMM can be found at:

https://www.ncca.navy.mil/tools/csruh/CRUAMM%20Version%2016Nov2011%20with%20Preface%2005April2013

.pdf

There are many ways to build and present an S-curve. Components are encouraged to establish

guidelines to promote a consistent and credible way to create them. Figure 14 is from Figure 4-8 of the

2014 JA CSRUH, which provides more detail on the content of and how to construct this particular

version of the S-curve. The CV in the subtitle stands for coefficient of variation. This is a useful metric

obtained by dividing the sample standard deviation by the average. Because the CV has no units, it can

be used to compare uncertainty across different elements in the estimate structure or across programs.

The 2014 JA CSRUH provides more detail on its use and interpretation.

Figure 14: S-Curve Example (notional)

GAO Cost Assessment Checklist

Congress often tasks the GAO to evaluate DoD programs to ensure that cost estimates are accurate,

credible, comprehensive, and well documented. The GAO has a standard series of questions they ask a

program office in order to establish the quality of the cost estimate. The questions are grouped by

estimate characteristic, based on best practices, and follows the 12-step cost estimating process defined

in the 2009 GAO Cost Estimating and Assessment Guide. Answers to these questions along with

program documentation serve as a basis for the GAO to make a quantitative assessment of the reliability

of the program’s cost estimate. DoD programs should understand each of these questions and be able

to provide documented answers and supporting documentation to each in preparation for a GAO audit.

This list of questions is included as Appendix G

to this guide. The checklist is mentioned here as a

means for the analyst to assess the completeness of his/her estimate documentation.

Lessons Learned

The analyst should formally document lessons learned that stem from developing, maintaining, and

updating a cost model and estimate. Lessons learned identify potential areas of risk/opportunities

and/or concerns that impacted a program’s cost estimate. Lessons learned databases document what

did and did not work in past programs, in the hopes that future programs can avoid the same pitfalls.

Lessons learned should be stored where the cost community can access them. The Community

Knowledge feature in CADE provides a resource to share lessons learned. This feature is accessible from

Acronyms used in Figure 12 include: cumulative distribution function (CDF), software (SW), month (Mth), and

Engineering and Manufacturing Development (EMD)

Appendix G is an updated list planned for the next version of the 2009 GAO Cost Estimating and Assessment

Guide.

each Program’s Dashboard. An analyst with CADE access may use this feature to store lessons learned

for use by future analysts. The analyst may also use the feature to research lessons learned by others.

Lessons learned may include any type of information that the estimator believes may be beneficial to a

future estimator that is updating the subject estimate or developing/updating a similar estimate.

Generally, lessons learned are only remembered for a short time, or by a select group of people.

Documenting lessons learned enhances the longevity of the lessons and increases the breadth of those

who are given a chance to learn from them.

The primary criterion for including a lesson learned is: does the analyst believe that knowing it in

advance it would have been beneficial. For example, a lesson learned might be that the planned

analogy required an adjustment to remove the effects of a year-long contractor labor strike that

occurred at the start of the analogous product’s manufacturing. Since events such as labor strikes

should not be accounted for in a cost estimate forecast, the analyst would explain the known labor

strike and its effects on the analogy, and how he/she adjusted the analogy to exclude these effects. The

analyst might want to include when the labor strike took place, as well as source documentation on the

labor strike. In this case, documentation might show that the analyst searched the CADE Community

Knowledge feature for the program of interest, and downloaded lessons learned for the analogous

program. Ideally, the content will confirm the labor strike occurred and provide insight into its impact.

This information serves as the basis to adjust the analogy. In this case, the analyst learned that he/she

needed to remove the labor strike impact from the analogy. Other, more straight forward, lessons

learned include: where to look for data, efficient estimate structure structure(s), most promising

estimating methods, unique and unexpected findings, where attention should have been focused, and

anything else that had the analyst known earlier, would have made the job easier.

Although documenting lessons learned takes time, the entire cost community can benefit from the

effort.

Documentation and Results References

• CAPE, Operating and Support Cost-Estimating Guide, 2014, para. 5.3.6, “Document Results”,

pg. 5-11, and para. 5.3.7 “Present Results”, pg. 5.11

• Department of the Army, Cost Analysis Manual, 2020, Chap 3 “Cost Estimating Process”, pg.

18 and Appendix 7 “Example Documentation”, pg. 81

• NCCA, Joint Agency Cost Estimating Relationship (CER) Development Handbook, 2018,

Chapter 6 “Step 6: Document CER”, pg. 177

• NCCA, Initial Cost Review Board (CRB) Guidance, 2015, Slides 11-34 (Various briefing

contents)

• NCCA, Joint Agency Cost Schedule Risk and Uncertainty Handbook, 2014, para. 2.6

“Document Cost Method and Cost Driver Uncertainty”, pg. 33 and Chapter 4 “How to

Present the CISM Risk and Uncertainty Story”, pg. 66

• NCCA, Cost Estimating Documentation Guide, 2012

• NCCA, Cost Estimating Guide, 2010 para. 1.6 “Present and Defend the Cost Estimate”, pg. 51

• NCCA-AFCAA, Software Cost Estimating Guide, 2008, Appendix F “System-Level Estimate

Case Study”, pg. 169, and Appendix G “Component-Level Estimate Case Study”, pg. 179

• SPAWAR, Inst 7110.1 Cost Estimating and Analysis, 2016, Enclosure 1, Chapter 8 “Present

and Defend Cost Estimate” pg. 22

• USMC, Cost Analysis Guidebook, 2017, para. 3.7 “Defend Cost Estimate Results”, pg. 53,

Appendix B “LCCE Report Format”, pg. 73 and Appendix “CARD Briefing Templates”, pg. 91

• Missile Defense Agency Cost Estimating and Analysis Handbook, 2012, Chapter 2

“Documentation” pg. 22

• GAO, Cost Estimating and Assessment Guide, 2009, Chapter 16 “Documenting the Estimate”

pg. 191, and Chapter 17 “Presenting the Estimate to Management”, pg. 197

• NASA, Cost Estimating Handbook, 2015, para. 2.2.2 “Task 5: Select Cost Estimating

Methodology”, pg. 14

Documentation and Results Training

The DAU Cost Estimating certification program for members of the Defense Acquisition Workforce offers

training relevant to the cost estimating results and documentation. Additional information on each

course may be found in the DAU iCatalog (

https://icatalog.dau.edu/).

• BCF 130 Fundamentals of Cost Analysis, Lessons 13, 14

• BCF 206 Cost/Risk Analysis, Lesson 6

• BCF 216 Applied Operating and Support Cost Analysis, Lesson 13

• BCF 230 Intermediate Cost Analysis, Lesson 13

• BCF 331 Advanced Concepts in Cost Analysis, Lesson 8

• CLM 052 Developing Stakeholder Engagement (understand how effective stakeholder

relationships contribute to improved acquisition outcomes)

The following course numbers starting with FMF refer to the course number assigned by the FM

Certification process. Information on these courses (including eligibility requirements) can be found in

the FM myLearn system:

https://fmonline.ousdc.osd.mil/FMmyLearn/Default.aspx.

• FMF 6016 FMA 301 - Business Case Analysis

• FMF 1550 QMT 290 - Integrated Cost Analysis

• FMF 1551 QMT 490 - Current Topics in Cost Estimating

9.0 NEXT ANALYSIS

It would be impossible for any guide

to cover every possible scenario or

circumstance relevant to the

development of a DoD cost estimate,

but this guide does provide

foundational knowledge for the DoD

cost community. The CAPE intends to

update this guide as necessary to

reflect policy changes, new estimating

methods and techniques, better ways

to present findings, and to capture

evolving best practices within the

community. The authors welcome

suggestions from the cost estimating

community for additional content.

These suggestions may be emailed to

osd.pentagon.cape.mbx.cost-

assessment@mail.mil.

At the conclusion of the final results and documentation, the cost estimate team should begin

evaluating and preparing for the next analysis. This may be a continuation with the same program or an

entirely new project. In either case, the final results and documentation of the completed project

should be made available to the DoD cost estimating community.

APPENDIX

The following appendices are included:

• Appendix A Acronyms

• Appendix B Sample Cost Estimating Flowcharts

• Appendix C Sample Questions To Get Started

• Appendix D Department of the Air Force Cost Estimate Documentation Checklist For ACAT I,

II, and III Cost Estimates

• Appendix E Sample SME Interview Form

• Appendix F Sample Assessments of Estimating Method Application

• Appendix G GAO Best Practice List

100

APPENDIX A ACRONYMS

$K, $M, $B Thousands, Millions, and Billions of Dollars, respectively

AACEI Association for the Advancement of Cost Engineering International

AAP Acquisition, Analytics, and Policy

ACAT Acquisition Category

ACE dL Analytic Cost Expert Distributed Learning

ACEIT Automated Cost Estimating Integrated Tools

ADM Acquisition Decision Memorandum

AFCAA Air Force Cost Analysis Agency

AFI Air Force Instruction

AFIT Air Force Institute of Technology

AFTOC Air Force Total Ownership Cost

AMCOS Army Military-Civilian Cost System

AoA Analysis of Alternatives

AP Acquisition Plan

APB Acquisition Program Baseline

APUC Average Procurement Unit Cost

AS Acquisition Strategy

ATP Authority to Proceed

BCA Business Case Analysis

BCAC Business Capability Acquisition Cycle

BCAT Business System Category

BCF Business, Cost Estimating, and Financial Management

BES Budget Estimate Submission

BOE Basis of Estimate

BOM Bill of Material

BY Base Year

C/CFO Comptroller/Chief Financial Officer

CADE Cost Assessment Data Enterprise

CAE Component Acquisition Executive

CAPE Cost Assessment and Program Evaluation

101

CARD Cost Analysis Requirements Description

CBAR Contract Business Analysis Repository

CCA Cost Capability Analysis

CCDR Contractor Cost Data Report

CCE Component Cost Estimate

CCEA™ Certified Cost Estimator/Analyst (ICEAA)

CCP Certified Cost Professional (AACEI)

CCP Component Cost Position

CCRL Collaborative Cost Research Library

CDD Capability Development Document

CDF Cumulative Distribution Function

CDRL Contract Data Requirements List

CDSG Cost Data Support Group

CEBoK Cost Estimating Body of Knowledge

CEMM Cost Estimating Methodology Matrix

CEP Certified Estimating Professional (AACEI)

CER Cost Estimating Relationship

CES Cost Element Structure

CFSR Contract Funds Status Report

CIO Chief Information Officer

CISM Cost Informed by Schedule Method

CLB Continuous Learning, Business

CLE Continuous Learning, Engineering

CLM Continuous Learning, Management

CLS Contractor Logistics Support

CONOPS Concept of Operations

COTS Commercial-Off-The-Shelf

CP Constant Price

CPD Capability Production Document

CRB Cost Review Board

CRUAMM Cost Risk and Uncertainty Analysis Metrics Manual

CSDR Cost and Software Data Reports (CSDR = CCDR + SRDR)

CWBS Contract Work Breakdown Structure

102

CY Constant Year

DAE Defense Acquisition Executive

DAES Defense Acquisition Executive Summary

DAMIR Defense Acquisition Management Information Retrieval

DAU Defense Acquisition University

DAVE Defense Acquisition Visibility Environment

DAWIA Defense Acquisition Workforce Improvement Act

DBS Defense Business System

DCAA Defense Contract Audit Agency

DCAPE Director of Cost Assessment and Program Evaluation

DCMA Defense Contract Management Agency

DFAS Defense Finance and Accounting Service

DoD Department of Defense

DoDD Department of Defense Directive

DoDI Department of Defense Instruction

DoDM Department of Defense Manual

DON Department of the Navy

DOT&E Director, Operational Test and Evaluation

DPC Defense Pricing and Contracting

DSOR Depot Source of Repair

DSS Decision Support System

DTIC Defense Technical Information Center

EA Economic Analysis

EAC Estimate At Completion

EDA Electronic Document Access

EDM Engineering Development Model

EMD Engineering and Manufacturing Development

ERP Enterprise Resource Planning

ESWBS Expanded Ship WBS

ETAB Estimating Technical Assurance Board

eVAMOSC Enterprise Visibility and Management of Operating and Support Cost

EVM Earned Value Management

EVM-CR EVM Central Repository

103

FA Framing Assumptions

FICSM Fully Integrated Cost and Schedule Method

FM Financial Management

FMECA Failure Mode Effects and Criticality Analysis

FMS Foreign Military Sales

FOC Full Operating Capability

FPRA Forward Pricing Rate Agreement

FRACAS Failure Reporting, Analysis, and Corrective Action System

FRP Full Rate Production

FTE Full Time Equivalent

FY Fiscal Year

G&A General and Administrative

GAO Government Accountability Office

GBL Government Bills of Lading

GFE Government Furnished Equipment

GFI Government Furnished Information

ICD Initial Capabilities Document

ICE Independent Cost Estimate

ICEAA International Cost Estimating and Analysis Association

ICS Interim Contractor Support

IGCE Independent Government Cost Estimate

ILA Independent Logistics Assessment

ILSP Integrated Logistics Support Plan

IMP Integrated Master Plan

IMS Integrated Master Schedule

IOC Initial Operational Capability

IOT&E Initial Operational Test and Evaluation

IPMR Integrated Program Management Reports

ISP Integrated Support Plan

IT Information Technology

IUID Item Unique Identification

JA CER Handbook Join Agency Cost Estimating Relationship (CER) Development Handbook

JA CSRUH Joint Agency Cost Schedule Risk and Uncertainty Handbook

104

JSCC Joint Space Cost Council

JST Job Support Tools

LCMP Life-Cycle Management Plan

LCSP Life-Cycle Sustainment Plan

LFT&E Live Fire Test and Evaluation

LOE Level of Effort

LRIP Low-Rate Initial Production

MADW Maintenance and Availability Data Warehouse

MCEA Masters in Cost Estimating and Analysis

MDA Milestone Decision Authority

MDAP Major Defense Acquisition Program

MILCON Military Construction

MIL-STD Military Standard

MSA Materiel Solution Analysis

MTA Middle Tier of Acquisition

MTBF Mean Time Between Failure

MTTR Mean Time To Repair

MYP Multiyear Procurements

NACA Non-Advocate Cost Assessment

NASA National Aeronautics and Space Administration

NAVAIR Naval Air Systems Command

NAVSEA Naval Sea Systems Command

NAVWAR Naval Information Warfare Systems Command

NCCA Naval Center for Cost Analysis

NDA Non-Disclosure Agreement

NDAA National Defense Authorization Act

NPS Naval Postgraduate School

NPV Net Present Value

O&M Operations and Maintenance

O&S Operating and Support

ODC Other Direct Cost

OLS Ordinary Least Squares

OMB Office of Management and Budget

105

OSD Office of the Secretary of Defense

OPTEMPO Operating/Operational/Operations Tempo

OSMIS Operating and Support Management Information System

OT&E Operational Test and Evaluation

OTA Operational Test Agency

OTP Operational Test Plan

PARCA Program Assessment and Root Cause Analysis Office (now AAP)

PAUC Program Acquisition Unit Cost

PB President’s Budget

PCEA™ Professional Cost Estimator/Analyst Certification (ICEAA)

PCO Procurement Contracting Officer

PEO Program Executive Officer

PI Prediction Interval

PIR Post Implementation Review

PMO Program Management Office

PMP Prime Mission Product

POA&M Plan of Action and Milestones (also POAM)

POE Program Office Estimate

POM Program Objective Memorandum

PPBE Programming, Planning, Budgeting, and Execution

PPP Program Protection Plan

PS Product Support

PWS Performance Work Statement

R&D Research and Development

RDT Resource Distribution Table

RFP Request For Proposal

ROI Return on Investment

RPC Real Price Change

S-CAT Services Acquisition Category

SAR Selected Acquisition Report

SCE Should Cost Estimate

Sec Section (referencing statute)

SEP Systems Engineering Plan

106

SEPM Systems Engineering and Program Management

SER Schedule Estimating Relationship

SLOC Software Lines of Code

SME Subject Matter Expert

SOO Statement of Objectives

SOW Statement of Work

SPAWAR Space and Naval Warfare Systems Command (changed to NAVWAR June 03, 2019)

SRDR Software Resource Data Report

STAMP Store Technical and Mass Property

SWaP Size, Weight, and Power

SYSCOM Systems Command

TEMP Test and Evaluation Master Plan

TMRR Technology Maturation and Risk Reduction

TOA Total Obligation Authority

TRA Technology Readiness Assessment

TRD Technical Requirements Description

TRL Technology Readiness Level

TY Then Year

UC100 Unit Cost of the 100

Item

UCR Unit Cost Reporting

USC United States Code

USD(A&S)

Under Secretary of Defense for Acquisition and Sustainment

USD(AT&L) Under Secretary of Defense for Acquisition, Technology, and Logistics

USD(R&E) Under Secretary of Defense Research and Engineering

USMC United States Marine Corps

VAMOSC Visibility and Management of Operating and Support Costs

VOLT Validated On-line Life-Cycle Threat

WBS Work Breakdown Structure

The 2017 NDAA separated the USD(AT&L) into the USD(R&E) and the USD(A&S).

107

APPENDIX B SAMPLE COST ESTIMATING FLOWCHARTS

It is recognized that some of the language in these graphics and flowcharts might be out of date with

current terminology. However, they do illustrate how other organizations describe cost estimating.

B.1 Government Accountability Office

Figure 15: GAO Cost Estimating Process

GAO, Cost Estimating and Assessment Guide, 2009, Chapter 1 Figure 1, pg. 8

B.2 CAPE

Figure 16: CAPE Recommended Analytic Approach for O&S Cost Estimate

CAPE, O&S Cost-Estimating Guide, 2014, Chapter 5, Figure 5-1, pg. 5-1

Figure 16 is labeled “Recommended Analytic Approach for O&S Cost Estimate” because that is the figure

title in the CAPE O&S Guide. But it is also representative for any kind of cost estimate.

108

B.3 Department of the Army

Figure 17: Department of the Army Cost Estimating Process

Department of the Army, Army Cost Analysis Manual, 2020, Chapter 3 “Cost Estimating Process,” pg. 9.

B.4 Department of the Navy

Figure 18: DON Cost Estimating Process Flow

DON, Cost Estimating Guide, 2010, Figure 2, pg. 10

(CEMM: Cost Estimating Methodology Matrix; ETAB: Estimating Technical Assurance Board)

109

Figure 19: NAVAIR Life-Cycle Cost Estimating Process Flow (Sep 2019)

The NAVAIR Life-Cycle Cost Estimating Process Flow diagram was provided directly by NAVAIR.

B.5 Department of the Air Force

Figure 20: AF Basic Cost Estimating Process

AFCAA, Air Force Cost Analysis Handbook, 2008, Exhibit 3-2, pg. 3-5

110

Figure 21: AF Cost Estimating Overview

AFCAA, Air Force Cost Analysis Handbook, 2008, Exhibit 3-1, pg. 3-3

111

B.6 Joint Space Cost Council (JSCC)

Figure 22: Joint Space Cost Council (JSCC) Cost Estimating Process

Draft JSCC Cost Estimating Guidebook, October 08, 2019, Table 5.3.1, pg. 33 (as applied to a Basis of

Estimate) and Figure 6-1, pg. 40 (as applied to a Realistic Cost Estimate)

B.7 NASA

Figure 23: NASA Cost Estimating Process

NASA, Cost Estimating Handbook, 2015, Figure 2, pg. 3

Step 1

Plan the

Development of the

Estimate

Step 2

Develop the

Execution Plan

Step 3

Determine the Scope

of Effort to Estimate

Step 4

Select the

Appropriate

Estimating Method

Step 5

Develop the

Resource Estimate

Step 6

Timephase the

Estimate

Step 7

Document the

Process and Results

Effort and

Schedule

Span

Correlate?

YESNO

112

APPENDIX C SAMPLE QUESTIONS TO GET STARTED

C.1 Sample Kickoff Meeting Questions

• How did this estimate/analysis become a requirement? How was it originated?

• What is the purpose of this estimate? Is this MS A/B/C? Something else?

• Are there any predecessor programs (pedigree) to this system, e.g., this is Increment 2, or it

is using 50% of System XYZ?

• Are there any policy implications or drivers specifically impacting this estimate, e.g., out of

cycle estimate, Middle Tier Acquisition Program?

• Is any other system that relies upon the development of this system, e.g., System XYZ will

be delayed if this system schedule slips?

• Does this system rely upon the development of any other system? e.g., this system

schedule will be delayed if the System XYZ schedule is delayed?

• Is this a completely new cost estimate or can a prior cost estimate be

adapted/modified/used in some fashion?

o If this can be a modified cost estimate, e.g., a cost model exists and can be adapted,

who built the prior cost model?

o If a prior cost model/estimate exists how familiar are you with the prior model?

o If a prior cost model/estimate exists, what are the primary changes that have to be

made?

• What is the schedule for the cost estimate? Do you think there is sufficient time in the

schedule to complete it?

• Regarding the stakeholders for this cost estimate:

o Does the program manager/PEO have any cost estimate result expectation?

o Are the Prime/Sub contractors providing the information you need?

o What is the size and makeup of the program office? Are any areas understaffed?

• What are the prime and subcontractor relationships, their contract types, cost reporting,

and challenges with the program manager and with each other?

• Will I have the support needed for this cost estimate/briefing/product?

• Is there a checklist of items to be accomplished by this cost estimate?

• What Project Definition documentation is available?

• Who is the POC for arranging data gathering visit to the program manager and the

contractor/subcontractors?

C.2 Sample Program Definition Questions

Questions about the CARD and Performance/Technical Baseline.

• Do you have concerns with the CARD?

• What areas of the CARD are incomplete or do not have enough detail?

• Was a CARD Narrative and CARD Microsoft Excel tables developed/delivered?

o Who drafted the CARD?

o Has the program manager reviewed the CARD? Approved it?

o Does anybody in the program office think that any element of the CARD is inaccurate?

• Is there a program WBS in the CARD? If not, why?

• Does the CARD make clear what the program office is funding vs. what it is not funding,

e.g., GFE?

• How well defined are the program risk/opportunity areas in the CARD?

113

• Are any of the technical parameters in the CARD confusing or ill defined?

• Does the CARD indicate whether the software development process is Agile, Waterfall, or

some other process?

• Are the quantities development, (e.g., prototype/engineering development model (EDM)),

test, and production, (e.g., LRIP, FRP) well defined or still changing?

• If integration is required, is it adequately addressed in the CARD?

• Has the program manager conveyed or mentioned any apprehension about the integration

effort/cost?

Questions Related to Schedule:

• Does the program acquisition schedule seem appropriate?

• Do you think the development/production schedule will slip? Why?

• Has the program manager mentioned this is a compressed/accelerated schedule?

• What is on the critical path?

• What program item is the most likely element to cause a delay in the schedule?

Questions Related to Schedule:

• Where is the O&S strategy defined? Is it sufficient?

o Are sustainment review requirements sufficiently addressed in the CARD?

o Are Tech refresh requirements adequately addressed in the CARD?

o Are obsolescence issues adequately considered?

o Has disposal been defined?

o What else, if anything, should be included in O&S, but was omitted?

Ground Rules and Assumptions:

• Is there a clear distinction between ground rules (requires program manager approval to

change) and assumptions?

• What are the major, cost contributing ground rules and assumptions?

• Does everybody agree on all of the ground rules and assumptions, including the CY,

inflation/escalation, quantities, phasing, shared production lines, technical readiness levels,

equipment lifetime, etc.?

• Are any of the assumptions likely to change? If so, what is the impact if they change?

114

APPENDIX D DEPARTMENT OF THE AIR FORCE COST ESTIMATE

DOCUMENTATION CHECKLIST FOR ACAT I, II, AND III COST ESTIMATES

The following checklist is adapted from: AFI 65-508, Attachment 3, 12 June 2018

D.1 Introduction

1.2. Table of Contents.

1.3. Program title and Program Elements.

1.4. Reference to the current program decision, if applicable, and CARD.

1.5. Purpose and scope of the estimate.

1.6. Cost estimate team members listed by organization, phone number, and area or estimating

responsibility.

1.7. Description of system or effort being estimated, with program phases estimated and excluded

costs identified.

1.8. Program schedule; buy and delivery schedules.

1.9. Applicable contract information.

1.10. Cost estimate summary by fiscal year in CY and TY dollars.

1.11. Ground rules and assumptions.

D.2 Body

2.1. Basis of estimate, by phase and appropriation, by program WBS or O&S CES.

2.2. Detailed methods, sources, and calculations provided by the program WBS or O&S CES along with

fiscal year phasing and rationale for phasing.

2.3. Rationale for selecting a specific cost estimating method, by the program WBS or O&S CES.

2.4. Source of data used when referencing analogous systems.

2.5. Contractor Cost Data Report and Software Resources Data Report

2.6. Cross checks, reasonableness and consistency checks addressed by the program WBS or O&S

CES. Specific references to studies, analogous systems or other appropriate documented

references.

2.7. Track to prior estimate, and rationale for differences.

2.8. Reconciliation between the Non-Advocate Cost Assessment (NACA)/ICE and POE. The body of

the cost estimate documentation should provide information (e.g., source data, estimating

115

methods, and results) sufficient to make it possible for a qualified analyst to recreate the

estimate using only the written documentation.

D.3 Additional checklist considerations identify whether:

3.1 All life-cycle costs are included

3.2. Estimates are organized consistently and logically

3.3. Learning curve slopes and factors are reasonable, similar system slopes and factors are included

as cross checks.

3.4. Actual historical data at or near program completion was used, when available.

3.5. Current inflation rates were used, documented and properly applied.

3.6. Historical data used is presented in the documentation, with rationale given as to why that

data/program is applicable for use as an analogy and, where applicable, extrapolation is

applicable.

3.7. Where systems have previously produced development or production units, unit or lot quantity

and associated costs are provided.

3.8. Briefing charts reference program funding provided in the most current budget (President’s

Budget or POM). If shortfalls exist, a zero ―shortfall option is provided.

3.9. Acronyms are defined.

3.10. Personnel costs are consistent with the Manpower Estimate Report, or deviations are properly

explained.

3.11. Sensitivity analysis and risk/opportunity/uncertainty analysis is documented.

3.12. Wrap rates and Forward Pricing Rate Agreement / Forward Pricing Rate Recommendation

assumptions are included.

116

APPENDIX E SAMPLE SME INTERVIEW FORM

The following form is intended as an example. It was provided by the Missile Defense Agency.

Figure 24: Example SME Documentation (Provided by the Missile Defense Agency, 2019)

117

APPENDIX F SAMPLE ASSESSMENTS OF ESTIMATING METHOD

APPLICATION

The following figures demonstrate a rough consensus of when the basic estimating methodologies are

applicable. Figure 25 is Exhibit 3-11 from page 3-29 of the 2008 AFCAA Cost Analysis Handbook

Figure 25: AFCAA: Selection of Methods

Figure 26 is Figure 6.2 from page 70 the MDA, 2012 Cost Estimating and Analysis Handbook.

Figure 26: Missile Defense Agency: Selection of Methods

118

Figure 27 is Figure 5 from page 14 of the 2015 NASA 2015 Cost Estimating Handbook.

Figure 27: NASA: Use of Cost Estimating Methodologies by Phase

The NASA figure contained the following footnote: Defense Acquisition University, “Integrated Defense

Acquisition, Technology, and Logistics Life Cycle Management Framework chart (v5.2),” 2008, as reproduced in the

International Cost Estimating and Analysis Association’s “Cost Estimating Body of Knowledge Module 2”.

119

APPENDIX G GAO BEST PRACTICE LIST

The 2009 GAO Cost Estimating and Assessment Guide was under revision when developing this guide.

The following checklist is a draft planned for the next version.

120

121