Department of Defense Independent Technical Risk Assessment Execution Guidance December 2020 Office of the Under Secretary of Defense for Research and Engineering Washington, D.C. Department of Defense Independent Technical Risk Assessment Execution Guidance Office of the Deputy Director for Engineering 3030 Defense Pentagon Washington, DC 20301 [email protected] https://ac.cto.mil/engineering Distribution Statement A: Approved for public release. Distribution is unlimited. CONTENTS 1. OVERVIEW .............................................................................................................................. 5 2. RESPONSIBILITIES ................................................................................................................ 6 2.1 For All ITRAs .................................................................................................................... 6 2.2 For ITRAs Conducted and Approved by USD(R&E) ....................................................... 6 2.3 For ITRAs Conducted by the Military Departments and Defense Agencies, and Approved by USD(R&E).................................................................................................... 7 2.4 For ITRAs Conducted and Approved by Military Departments and Defense Agencies ... 7 3 ITRA PLANNING...................................................................................................................... 9 3.1 Developing an ITRA Plan .................................................................................................. 9 3.2 Establishing the ITRA Team ............................................................................................. 9 3.3 ITRA Planning Considerations ........................................................................................ 10 3.4 ITRA Plan Review ........................................................................................................... 12 4. CONDUCT OF THE ITRA ..................................................................................................... 13 5. DEVELOPMENT OF THE ITRA REPORT .......................................................................... 14 5.1 Characterization of Program Risks .................................................................................. 14 5.2 ITRA Preliminary Report ................................................................................................ 14 5.3 ITRA Final Report ........................................................................................................... 14 5.4 ITRA Submission............................................................................................................. 16 ACRONYMS ................................................................................................................................ 17 REFERENCES ............................................................................................................................. 18 Figures Figure 1. Notional ITRA Timeline .............................................................................................. 10 Figure 2. USD(R&E) DTRAM Structure .................................................................................... 13 Figure 3. Notional Assessment Scorecard ................................................................................... 15 1. OVERVIEW Department of Defense (DoD) Independent Technical Risk Assessments (ITRAs) provide Congress, the Milestone Decision Authority (MDA), the Program Manager (PM), and other program stakeholders with an independent assessment of a program’s technical risk. These assessments inform the PM and MDA of key risks ahead of milestone and production decisions, allowing for appropriate actions to reduce the risk. In addition, ITRAs: a. Facilitate the MDA’s establishment of program cost and fielding targets in support of 10 U.S.C. 2448a (reference A). b. Form the technical basis of the independent assessment of technical risk, critical technologies, and manufacturing required by 10 U.S.C. 2366a, 2366b, and 2366c (references B-D). c. Inform Requests for Proposals (RFPs), as appropriate. d. Provide a basis of data for mission area analysis and cross-program analysis of systemic risks. In accordance with the DoD Instruction (DoDI) 5000.88, Engineering of Defense Systems, the Under Secretary of Defense for Research and Engineering (USD(R&E)) will conduct and approve ITRAs for Acquisition Category (ACAT) ID programs. The applicable DoD Departments, Services, and Agencies will conduct ITRAs for ACAT IB and IC programs. The USD(R&E) will determine the approval authority for all ACAT IB and IC program ITRAs. ITRAs should be conducted by independent personnel with an appropriate level of expertise; will assess the full spectrum of technical, engineering, and integration risks; and will be based on current and relevant data with respect to the milestone or production decision. ITRAs should be submitted in time to support approval not later than 30 days before the milestone or production decision. This guidance provides procedures and standards to use in planning, conducting, and submitting an ITRA. In this document statements using “will” refer to statutory or DoD policy requirements relating to ITRAs. Statements using “expects” identify the USD(R&E)’s minimum expectations for ITRAs. For the purpose of this discussion, the term “risk” refers to both risks and issues, although a risk differs from an issue in that risk implies a potential or probability to occur whereas an issue is certain or has already occurred. Additional lessons learned for planning, conducting, and submitting an ITRA are available from the DoD Deputy Director for Engineering (DD, ENG) website (reference F). 2. RESPONSIBILITIES 2.1 FOR ALL ITRAS DD, ENG will maintain all applicable DoD ITRA references, templates, tools, and training for ITRA teams on the DD, ENG website. The Military Department or Defense Agency is responsible for ensuring that Services, Agencies, and PMs support ITRA execution. This support includes: a. Providing access to programmatic and technical information to the ITRA team. b. Enabling visits to the program offices, product centers, test centers, and contractor sites. 2.2 FOR ITRAS CONDUCTED AND APPROVED BY USD(R&E) The office of DD, ENG will conduct the ITRA. DD, ENG is responsible for: a. Ensuring ITRAs are conducted in accordance with established policy and this guidance. b. Identifying an ITRA team lead, who should be a Government official responsible for planning, conducting, developing, and submitting the ITRA. c. Ensuring the ITRA team lead and the ITRA team members are independent from the program in accordance with DoDI 5000.88 (reference E). Independence is defined as: (1) Not obligated to the program under assessment. (2) Not in the direct chain of command leading from the program to the MDA. (3) Free from organizational or personal conflicts of interest. Non-independent personnel may participate in the ITRA in a limited role or in an advisory capacity but should not unduly influence the outcome of the assessment. d. Ensuring the team lead coordinates with the Services, Agencies, and PMs to support the ITRA execution. This support includes providing access to programmatic and technical information and enabling visits to the program offices, product centers, test centers, and contractor sites. e. Reviewing the ITRA plan with the team lead before the initiation of the ITRA and providing any recommended changes to the plan. f. Ensuring the ITRA final report is provided to the USD(R&E) in time to support approval not later than 30 days before the milestone or production decision. 2.3 FOR ITRAS CONDUCTED BY THE MILITARY DEPARTMENTS AND DEFENSE AGENCIES, AND APPROVED BY USD(R&E) The Department or Agency is responsible for: a. Identifying the organization responsible for conducting the ITRA. The organization is responsible for identifying the ITRA team lead, who should be a Government official responsible for planning, conducting, developing, and submitting the ITRA. b. Ensuring ITRAs are conducted in accordance with established policy and this guidance. c. Ensuring the organization conducting the ITRA, the ITRA team lead, and the ITRA team members are independent from the program in accordance with the DoDI 5000.88 (reference E). Independence is defined as: (1) Not obligated to the program under assessment. (2) Not in the direct chain of command leading from the program to the MDA. (3) Free from organizational or personal conflicts of interest. Non-independent personnel may participate in the ITRA in a limited role or in an advisory capacity but should not unduly influence the outcome of the assessment. d. Ensuring the ITRA final report is provided in time to support approval not later than 30 days before the milestone or production decision. USD(R&E) expects the Department or Agency to include representatives from DD, ENG to participate in the ITRA. The role of the DD, ENG representation will primarily be in an advisory capacity and will facilitate insight and USD(R&E) approval. DD, ENG may also provide personnel to serve as subject matter experts if requested. The role for the DD, ENG representation should be clearly defined in the ITRA plan. DD, ENG is responsible for: a. Reviewing the ITRA plan with the team lead before the initiation of the ITRA, and providing any recommended changes to the plan. b. Providing resources for DD, ENG participation. 2.4 FOR ITRAS CONDUCTED AND APPROVED BY MILITARY DEPARTMENTS AND DEFENSE AGENCIES The Department or Agency is responsible for: a. Identifying the technical authority responsible for approving ITRAs. b. Identifying the organization responsible for conducting the ITRA. The organization is responsible for identifying the ITRA team lead, who should be a Government official responsible for planning, conducting, developing, and submitting the ITRA. c. Ensuring ITRAs are conducted in accordance with established policy and this guidance. d. Ensuring the organization conducting the ITRA, the ITRA team lead, and the ITRA team members are independent from the program in accordance with the DoDI 5000.88 (reference E). Independence is defined as: (1) Not obligated to the program under assessment. (2) Not in the direct chain of command leading from the program to the MDA. (3) Free from organizational or personal conflicts of interest. Non-independent personnel may participate in the ITRA in a limited role or in an advisory capacity but should not unduly influence the outcome of the assessment. e. Ensuring the ITRA final report is provided in time to support approval not later than 30 days before the milestone or production decision. 3 ITRA PLANNING 3.1 DEVELOPING AN ITRA PLAN The ITRA team lead should develop a plan for conducting the ITRA. An ITRA plan is essential for the smooth execution and timely submission of the ITRA. An ITRA plan identifies how the ITRA team will ensure coverage of the full spectrum of technical risk, confirms consensus on the ITRA execution strategy, and identifies the schedule and resources needed to support timely submission of the ITRA. The ITRA plan identifies: a. Key technical drivers affecting the program’s risk posture. b. The ITRA team members’ expertise aligned with key technical drivers. c. Program documentation and program events the ITRA team plans to leverage to reduce the burden on the program. d. Any additional engagements necessary to assess technical risk. e. The schedule of key program engagements, preliminary and final report development, and submission of the ITRA preliminary and final reports in time to support approval and timely submission of the ITRA before the milestone or production decision. A sample ITRA plan is available on the DD, ENG website (reference F). 3.2 ESTABLISHING THE ITRA TEAM Selecting a quality team is key to achieving a quality ITRA. The ITRA team lead should be well versed in DoD’s Adaptive Acquisition Framework and should be able to deliberate with Service or Department leadership and PMs. Team members should have experience in the area they are to evaluate and be able to conduct independent analysis of data provided during the ITRA. The team lead is responsible for the overall conduct of the ITRA and should be a Government employee. The team lead establishes the ITRA team, develops an ITRA plan, conducts the ITRA, develops the ITRA report, and submits the ITRA report in time to support approval not later than 30 days before the milestone or production decision. Items to consider in identifying a team lead include: a. Strong leadership and organizational skills. The team lead should have a demonstrated ability to organize and manage a diverse group of individuals from different organizations. b. Demonstrated ability to compile, write, and edit technical reports. c. Experience with the domain of the program under review. Knowledge of the mission sets and similar systems is fundamental to understanding risk impacts. d. Familiarity with the program under review since knowledge of past program issues, performance, and requirements is invaluable in tailoring the assessment and accelerating the ITRA process. If the team lead is not familiar with the program, the ITRA plan should include time for the team lead to review past assessments and query stakeholders in order to obtain familiarity with the program under review. The team lead is responsible for coordinating with applicable organizations to identify ITRA team members. Items to consider in selecting team members include: a. Ensure team members possess the appropriate technical and domain expertise to assess the specific technical areas identified in the plan. The members should be able to assess the quality of program plans and actual performance data and not just assess the process or document compliance. b. Ensure ITRA team members’ organizations can provide appropriate resources (e.g., time for the team member to attend program engagements and ITRA team meetings, access to applicable assessment tools). c. Consider the need for team members to attend program engagements to reduce the burden on the program. Determine if a team member can adequately assess their specific technical area through document review and participation in program engagements through teleconferences. 3.3 ITRA PLANNING CONSIDERATIONS The team lead should consider the following during the development of the ITRA plan: a. Initiate planning at least 12 to 14 months before the milestone or production decision to allow time to plan, conduct, and submit an ITRA for approval to support MDA reporting requirements. Figure 1 shows a notional ITRA timeline. [Figure 1. Notional ITRA Timeline: Depicts phases across 12-14 months including ITRA Planning (Establish Team, Draft Plan, Review Plan), Conduct of the ITRA (Document & Model Review, Team Meetings, Program Scheduled Events, ITRA Specific Events), ITRA Development (Characterization of Risks, Draft Assessment, Preliminary Report, PMO Review, Final Report Staffing, Final Report Approved), leading to Milestone or Prod Decision.] b. Identify key technical drivers by assessing the program’s past performance, known risks, programmatic and technical dependencies, and lessons learned from previous, similar systems. c. Coordinate the ITRA plan early with the Program Management Office to leverage planned program technical reviews, design activities, and program events such as technology and manufacturing assessments. d. Consider the program’s life cycle. For example: (1) ITRAs to inform Milestone A decisions often involve nascent program offices with immature or non-existent documents, artifacts, and data. Having ITRA team members engaged throughout the concept development, industry days, and Analysis of Alternatives will enable the team to better understand the risks. (2) For programs that will be in a competitive, source selection environment during the conduct of an ITRA, the ITRA team lead should develop an execution plan that enables the ITRA to inform both the Request for Proposal (RFP) release, and the milestone or production decision. The team lead should plan to complete as much of the ITRA as possible in order to inform the RFP. Subsequently, the ITRA team should update the report to address the selected solution(s) in order to provide as accurate and current an assessment as practicable to the MDA and to Congress. The team lead should allot additional time for advanced planning and coordination with the program, Government Contracting Officer, and ITRA approval authority in order to ensure the ITRA accurately reflects the technical risk of the selected solution(s), supports the program’s planned milestone or production decision, and minimizes the risk of influencing the program’s source selection activities. The team lead should consider the following possible courses of action to determine the best approach for the ITRA. (A) Conduct the ITRA based on information that does not require access to source selection or vendor proposal information, and update the ITRA for the selected solution(s) after the Source Selection Authority (SSA) decision. Information on the selected solution(s) may be released by the SSA or obtained by conducting ITRA-specific events after the SSA decision. (B) Conduct the ITRA using source selection or vendor proposal information, obtained from the Government, to augment information obtained from other sources. During source selection, any contacts, inquiries, or clarifications from the competing vendors should be made through the Government Contracting Officer. This course of action should be used only if the previous course of action is inadequate to assess the risks of the selected design(s) in time to support the program’s milestone. e. Tailor the ITRA to the program. Clearly identify any technical areas or factors tailored out of the ITRA, and include the reasoning for tailoring. f. Ensure the ITRA approval authority approves the selected ITRA approach and any tailoring as documented in the ITRA plan. g. Submit the ITRA preliminary report in time to allow the PM to execute appropriate risk mitigation activities. For milestones or production decisions that align with new contracts, provide the ITRA preliminary findings in time to inform the RFP. h. Submit the ITRA final report with sufficient time for staffing, approval, and forwarding to the MDA to support the MDA’s statutory and regulatory reporting requirements. See the DD, ENG website (reference F) for additional areas to consider and lessons learned. 3.4 ITRA PLAN REVIEW The ITRA team lead should review the plan with the organization conducting the assessment, the PM, and the ITRA approval authority or his/her designated representative, before the initiation of an ITRA. This review will ensure coverage of the full spectrum of technical risk, confirm consensus on the assessment strategy, and ensure the schedule and resources support timely submission of the final report. For ITRAs approved by USD(R&E), USD(R&E) expects the ITRA team lead to review the plan with the office of DD, ENG, before initiating the ITRA. 4. CONDUCT OF THE ITRA Fundamentally, ITRAs rely on the team members’ individual expertise and critical thinking to identify and assess program risks. ITRAs typically consist of document and model reviews, participation in program scheduled events, and ITRA-specific events conducted at the Program Management Office, primary contractor sites, and as appropriate, key subcontractor sites. The ITRA team should assess the full spectrum of technology, manufacturing, engineering, and integration risk while reducing the burden to the program where practicable. To ensure thoroughness and consistency in assessments across all of DoD, USD(R&E) expects all ITRAs, regardless of the approval authority, to use the USD(R&E) Defense Technical Risk Assessment Methodology (DTRAM) during the assessment. The DTRAM, available at the DD, ENG website (reference F), is a guide for the ITRA team to ensure the full coverage of technical risk. The DTRAM does not replace critical thinking and should not be sent to the program or contractor as a questionnaire. The team lead may share the DTRAM with meeting participants to facilitate discussions. The DTRAM is organized into eight technical risk areas (mission capability, technology, system development and integration, modular open systems approach (MOSA), software, security and cybersecurity, manufacturing, and reliability, availability, and maintainability (RAM)/sustainment) across seven factors (performance and quality, scope and requirements, design and architecture, evaluation, schedule, decision and control, and resources) as shown in Figure 2. Figure 2. USD(R&E) DTRAM Structure [Matrix mapping 8 Risk Areas (Mission Capability, Technology, System Develop & Integrat., MOSA, Software, Security & Cybersecurity, Manufacturing, RAM & Sustainment) against 7 Factors (Performance & Quality, Scope & Requirements, Design & Architecture, Evaluation, Schedule, Decision & Control, Resources).] The DTRAM provides a top-level description for each area-factor combination, providing team members with an understanding of what to consider for that combination along with several detailed criteria for consideration. The DTRAM criteria are written to require assessment of program and product performance. Insufficient processes and artifacts may be identified during the ITRA process but generally should be identified as factors in the risks rather than primary risks themselves. 5. DEVELOPMENT OF THE ITRA REPORT To ensure thoroughness and consistency in assessments across DoD, USD(R&E) expects all ITRAs, regardless of the approval authority, to use a standard approach in characterizing and reporting program risks. This standard approach provides a structured means for consistent evaluation and description of risks across programs; enables accurate risk posture reporting to Congress, DoD, and MDA leadership; supports Department-level mission area analysis; and supports cross-program analysis of systemic risks. 5.1 CHARACTERIZATION OF PROGRAM RISKS USD(R&E) expects all ITRAs to characterize program risks, to include the likelihood and consequence criteria, consistent with the USD(R&E) Framework for Risk Categorization available at the DD, ENG website (reference F). Consistent with the framework, risk descriptions will identify the consequence to the program, and to any interdependent programs, and the likelihood the risk will occur. If known, the cause of the risk also should be described. USD(R&E) expects ITRA risks to be clearly documented and supported by data collected as part of the ITRA process. 5.2 ITRA PRELIMINARY REPORT An ITRA preliminary report summarizes the risks, provides actionable recommendations, and is supported by appropriate documentation and analysis. The preliminary report provides the PM with an early opportunity to review the ITRA team’s identified risks, address any factual inaccuracies, and initiate any risk mitigation activities the PM deems appropriate. The ITRA preliminary report also provides the approval authority and the MDA with early notification of any risks that may require outside support or elevation before the milestone or production decision. The format for the ITRA preliminary report is at the discretion of the ITRA team lead but should be a written report or briefing with adequate substantiation. The ITRA team lead should review the preliminary report with the PM and then provide the preliminary report to the ITRA approval authority. For ITRAs approved by USD(R&E), USD(R&E) expects the ITRA team lead to provide preliminary reports to DD, ENG after the PM’s review. 5.3 ITRA FINAL REPORT The final report provides the ITRA approval authority, MDA, Congress, and other stakeholders with an independent analysis of the program’s risk posture and provides the MDA with data to support statutory reporting responsibilities. In order to support the various levels of leadership, and still clearly document the team’s findings, the ITRA final report should consist of an executive summary and a detailed report including the requisite data and analysis needed to support the team’s findings, assertions, and recommendations. a. The executive summary provides an overview of the program’s technical risk posture, to include critical technologies and manufacturing processes. It identifies risks to be brought to the MDA’s attention and provides recommended mitigation strategies for high-risk areas. b. The detailed report provides greater detail and expands on the risks identified in the executive summary. The detailed report should include enough supporting data to substantiate the ITRA team’s assessment of the program’s risks and ensure the report can be understood without referencing external documents. c. The underlying supporting documentation and analysis include the data, briefs, and documentation the ITRA team collected during the conduct of the ITRA and the analysis the team conducted that form the basis of the ITRA risks and observations. This data should be archived for future reference and congressional inquiries as appropriate. A DTRAM scorecard similar to the one shown in Figure 3 is highly encouraged as an element of the report. This visual tool provides leadership with a quick cross-reference to the team’s assessment of program risks and key strengths, aligned to the DTRAM reporting areas and factors. The colors on the scorecard should be aligned with the risk likelihood and consequence ratings in the Framework for Risk Categorization. Additional information on the DTRAM and DTRAM scorecard is available on the DD, ENG website (reference F). Figure 3. Notional Assessment Scorecard [Matrix showing overall and category ratings (Performance, Schedule, Resources) across DTRAM areas and factors with color-coded risk levels: Low risk, Moderate risk, High risk, Positive, Assessed - No Significant Findings, Not Assessed.] For ITRAs approved by USD(R&E), USD(R&E) expects the ITRA final report to follow the USD(R&E) ITRA template available from the DD, ENG website (reference F). USD(R&E) expects any significant deviations from this template to be discussed with the DD, ENG, concurrent with the ITRA plan review. 5.4 ITRA SUBMISSION In accordance with the Policy Memorandum for Independent Technical Risk Assessments for Major Defense Acquisition Programs (2020) available on the DD, ENG website (reference F), the ITRA team lead will submit the ITRA final report to the approval authority in time to support approval not later than 30 days before the milestone or production decision. For ITRAs approved by USD(R&E), USD(R&E) expects the ITRA team lead to submit the ITRA final report, consisting of the ITRA executive summary and a detailed report including the underlying documentation and analysis supporting the assessment risks, findings, and assertions to the office of DD, ENG not later than 45 days before the program milestone or production decision. The approval authority will provide copies of approved ITRA final reports, to include the executive summary and the detailed report, and underlying documentation and analysis supporting the assessment risks, findings, and assertions, to the office of USD(R&E) to support the archiving of ITRAs. ACRONYMS ACAT Acquisition Category DD, ENG Deputy Director for Engineering DDR&E(AC) Director of Defense Research and Engineering, Advanced Capabilities DTRAM Defense Technical Risk Assessment Methodology DoD Department of Defense DoDI Department of Defense Instruction ITRA Independent Technical Risk Assessment MDA Milestone Decision Authority PM Program Manager RFP Request for Proposals U.S.C. United States Code USD(A&S) Under Secretary of Defense for Acquisition and Sustainment USD(R&E) Under Secretary of Defense for Research and Engineering REFERENCES A. United States Code, Title 10, Section 2448a, “Program Cost, Fielding, and Performance Goals in Planning Major Defense Acquisition Programs” B. United States Code, Title 10, Section 2366a, “Major Defense Acquisition Programs: Determination Required before Milestone A Approval” C. United States Code, Title 10, Section 2366b, “Major Defense Acquisition Programs: Certification Required before Milestone B Approval” D. United States Code, Title 10, Section 2366c, “Major Defense Acquisition Programs: Submissions to Congress on Milestone C” E. DoD Instruction 5000.88, “Engineering of Defense Systems,” November 18, 2020 F. Department of Defense Deputy Director for Engineering website files https://ac.cto.mil/engineering/ Defense Technical Risk Assessment Methodology (DTRAM) Framework for Risk Categorization, June 2018 Department of Defense Technology Readiness Assessment Guidance, April 2011 Department of Defense Independent Technical Risk Assessment Execution Guidance Office of the Deputy Director for Engineering 3030 Defense Pentagon Washington, DC 20301 [email protected] https://ac.cto.mil/engineering Distribution Statement A: Approved for public release. Distribution is unlimited.