AEROSPACE REPORT NO. TOR-2016-02121 Process Change Assessment Techniques Product Overview May 5, 2016 Nelson J. Ho Communication Systems Implementation Subdivision Communications and Cyber Division Prepared for: National Reconnaissance Office 14675 Lee Road Chantilly, VA 20151-1715 Contract No. FA8802-14-C-0001 Authorized by: National Systems Group Developed in conjunction with Government and Industry contributors as part of the U.S. Space Programs Mission Assurance Improvement Workshop. Distribution Statement A: Approved for public release; distribution unlimited. THE AEROSPACE CORPORATION Assuring Mission Success Abstract Preventing process changes from causing unintended consequences is important to achieving mission success. Process changes, often in the name of improvement, continue to be identified as contributors to hard and expensive lessons. These lessons can be linked to inadequately assessed processes or changes in upstream processes. This presentation was given at the 2016 Mission Assurance Improvement Workshop and provides an overview of the full report, Process Change Assessment Techniques (Aerospace Report No. TOR-2016-02187). Acknowledgments This document was created by multiple authors throughout the government, the aerospace industry, and academia. We thank the following contributing authors for making this collaborative effort possible: - Mary Addison (Northrop Grumman Aerospace Systems) - Anthony Eary (Northrop Grumman Aerospace Systems) - George Gitchev (SSL) - Nelson Ho (The Aerospace Corporation) - Johnny Jones (Harris Corporation) - Steve Killman (The Boeing Company) - Scott Miller (Ball Aerospace and Technologies Corporation) - Thomas Reinsel (Raytheon Space and Airborne Systems) A special thank you for co-leading this team and efforts to ensure completeness and quality of this document goes to Thomas Reinsel, Raytheon Space and Airborne Systems. The authors deeply appreciate the contributions of the subject matter experts who reviewed the document: - Rob Bauer (Moog) - Matthew Binondo (The Aerospace Corporation) - Emanuel Bucur (The Aerospace Corporation) - Ira Johnson (Moog) - Alfred Jung (The Aerospace Corporation) - Ed Kestel (Orbital ATK) - Mark King (Micropac Industries) - Katherine Losoncy (The Aerospace Corporation) - Art McClellan (The Aerospace Corporation) - James Parke (The Aerospace Corporation) - Frank Pastizzo (SSL) - Jason Rife (Harris Corporation) - Tom Seitz (Northrop Grumman) - Michelle Settles (NOAA) - Beth Springsteen (The Aerospace Corporation) - Anthony Taconi (Lockheed Martin) - David Tong (The Boeing Company) - Colin Wilson (Ball Aerospace) U.S. Space Program Mission Assurance Improvement Workshop Process Change Assessment Techniques Product Overview Tom Reinsel, Raytheon Space and Airborne Systems Nelson Ho, The Aerospace Corporation May 5, 2016 U.S. SPACE PROGRAM MISSION ASSURANCE IMPROVEMENT WORKSHOP HARRIS CORPORATION | MELBOURNE, FL | MAY 3–5, 2016 Motivation for Process Change Assessment Techniques • Process change, often in the name of “improvements,” can introduce unintended consequences for qualified hardware – Design changes traditionally vet through engineering change boards • Process changes are not consistently assessed for high risk – The less a supplier knows about the end use of a product, the less likely that supplier is to consider that a proposed process change could cause unwanted product impacts – Some companies have developed their own policies to manage process change, but there is no industry consensus on approach • “Late” escapes are infrequent, but can have high impact on mission success Problem Process changes may happen at outside specialty processors or in-house process “Centers of Excellence.” Problems occur when these changes manifest as unintended consequences at higher module- or unit-level builds, risking mission success. Structure/Flow Analysis: - Prime or Tier 1 Supplier (e.g. MAIW Member): Program level (Problem discovered here) - Supplier/Subtier (e.g. SSC level): Module or Unit (Build-to-Spec: Electronics, Reaction wheels, Telescopes, Antennas, Propulsion units) / Product Line (Build-to-Spec: Electronics, Propulsion subsystem) [Scope Boundary] - Processor: Specialty processor (Coatings, Composites, Heat treat, Soldering, Welding) / "Center of Excellence" process (Composites, Soldering, Welding) (Escape occurs here) Team Charter • Survey data from integration and test (I&T) (failure review board [FRB], material review board [MRB], etc.) for examples of failures resulting from supplier or in-house process changes • Identify and categorize causal themes and drivers from data – e.g., facility relocations, equipment or workforce changes, breaks in procurement or manufacturing, etc. • Benchmark trade associations for methods that have addressed control of process changes • Survey/assess effectiveness of existing customer process change notification (PCN) and control requirements • Compile best practice methods and tools for effective identification and evaluation of supplier PCNs • Prioritize guidance on when and how to use existing methods and tools versus introduction of new or unfamiliar methods or tools Process Escape Data Analysis • The team analyzed data from various sources (lessons learned and root cause corrective action databases, FRB, MRB, etc.) for process changes that created hardware failures at subsequent levels of assembly, integration, test, or customer use • Each escape was assigned to a potential cause on a fishbone • Potential causes were ranked and the top three were investigated further Included Charts: - Pareto of Potential Process Escape Causes (Counts and cumulative percentages across failure modes, requirement flow-downs, process changes, and verification) - Fish Bone Diagram for Failure to Notify Customer of Process Change (Categories: Human Factors, Methodology, System) Escape Examples Example 1: • Process Change: Change of gold plating supplier for connector pins resulted in intermittent connectivity failures at cold temperatures (found in vehicle-level thermal vacuum [TVAC] testing). • Result: Reach-across to multiple programs and customers at all phases of development and integration. • Recognition of potential impact from change of source could have been mitigated earlier by perceptive screening test. Example 2: • Process Change: A dynamic load (i.e., wiggle) test was implemented to screen for failures during manufacturing. The new screen induced unexpected latent stress failures in adjacent chip components. • Result: New failures found at higher-level printed wiring board (PWB) assemblies, necessitating more investigation and remove-and-replace (R&R). • Recognition of potential impact from “improved” screening could have been prevented with a thorough risk analysis. Trade Association Summaries • Conducted review of six trade associations’ policies and publications – For guidance to their practitioners to manage and control process changes • before the change was initiated • after an escape had occurred to assess the risk • These associations were selected because together they comprise the space system industry’s most well-developed sources for procedural guidance on the design and manufacture of hardware used in space applications Reviewed Associations: - AIAA (Shaping the Future of Aerospace) - IPC - I.A.Q.G. (International Aerospace Quality Group) - JEDEC - Nadcap - SAMPE International Aerospace Quality Group (IAQG)—Process Failure Modes and Effects Analysis (PFMEA) • PFMEA was seen as the most effective available technique for preventing a process change-related escape • Implementation of a PFMEA (which is one element of the new AS9145 standard) can be applied specifically to evaluate a proposed process change • This methodology is particularly valuable before a process change is implemented PFMEA Worksheet Summary Example: - Process Being Reviewed: Add glass to RTV - Potential Failure Modes: air inclusions into RTV (at ~40% by surface area) - Potential Effect(s) of Failure: reduce bond line and thermal control (Severity: 5) - Potential Cause(s) / Mechanism(s): Poor Operator Controls, no deairate operation - Current Controls (Prevention): de-airate process, Low viscosity RTV (Occurrence: 2) - Current Controls (Detection): variation in IR signature at end customer (Detection: #, RPN: 100) - Recommended Action(s): 1.) Determine acceptable void % RF thermal analysis (Resp: LaKomski); 2.) Provide supplier with IR camera system details to determine likelihood of inclusion into ATP process (Resp: B. Burk) - Action Results: Action 1 reduces severity (New Sev: 4, New Occ: 2, New Det: 10, Improved RPN: 80); Action 2 reduces detection (New Sev: 5, New Occ: 2, New Det: 5, Improved RPN: 50) Process Change Notification Benchmark • Reviewed several companies’ flowdown requirements for PCN • One was selected as a benchmark example – Benchmark PCN policy was selectively provided to key suppliers by the contractual inclusion of a Quality Assurance Provision (QAP), also called Q-note – Program and Supplier Quality conduct a review of the Supplier Change Request Notification (SCRN) with supply chain and technical SMEs – The review considers further actions such as conducting a new first article or a re-qualification plan for the part [Includes image preview of Supplier Change Request/Notification (SCRN) form] Process Change “Improvement” Questions • A set of 27 questions was developed to aid a supplier or in-house processor to identify changes that might affect the end user of the product • The questions are meant to invite the review of changes that go beyond typical form-fit-function constraints – Questions are not exhaustive • The questions help a supplier recognize that a change to an existing process could represent a risk to a product in its end use – Even if the supplier was unaware of all of the conditions of the end use of a product Question Categories Preview: - Methods: Questions involving methods (waste reduction, alternative paths, assembly aids, shifts, execution methods) - Machinery: Labor-reducing equipment, material handling devices, standard/improved inspection or packaging equipment - Material: Expense items reduction (adhesives, tapes, wires, lubricants, coolants, inks), obsolete/unusable/alternate expense items - Environment: Cleanliness, facility rearrangement, relocation within site, different site, additional manufacturing site Intended Product Use • The purpose is – to provide awareness to the general space community for the benefit of mitigating the risks of process changes – to benchmark the most useful tools and policies currently in use • Intended for program management, procurement, and technical disciplines, including engineering, production, and quality organizations responsible for the procurement, design, manufacturing, and quality assurance of space-qualified hardware • Some of the professional associations or educational organizations that would reach the target audience are AIAA, IPC, IAQG, JEDEC, Nadcap, and SAMPE Topic Follow-on Recommendations • Specific recommendations for the space industry: – Enhance existing reference documents (e.g., EIA-649-1, Definition of Major (Class I) Engineering Change Proposal, and similar sources) to include language that goes beyond Class I/Class II or Major/Minor to include unintended consequences from process changes – Develop a space industry common guideline for mitigating process changes and a common structure for PCN policy that could be selectively applied between Primes -> Tier 1 and also from Tier 1 -> outside processors • Specific recommendations for government: – Review the new AS 9145 and SAE J1739 processes from IAQG and develop a plan to flow these techniques down to suppliers depending on circumstances. May be appropriate to research a larger set of history for AS 9145 and SAE J1739 Team Introductions - Core Team Company | Participant - The Aerospace Corporation | Nelson Ho* - Ball Aerospace and Technologies Corporation | Scott Miller - The Boeing Company | Steve Killman - Harris | Johnny Jones* - Northrop Grumman Aerospace Systems | Mary Addison*, Anthony Eary - Raytheon Space and Airborne Systems | Thomas Reinsel* - SSL | George Gitchev* - MAIW Steering Committee/Sponsor (Raytheon Space and Airborne Systems) | Mark Baldwin* - MAIW Steering Committee (Ball Aerospace and Technologies Corporation) | Eli Minson* - MAIW Steering Committee (Northrop Grumman) | Ed Jopson* * Attending MAIW Team Introductions - Subject Matter Experts Company | Participant - The Aerospace Corporation | Emanuel Bucur* - The Aerospace Corporation | Matthew Binondo - The Aerospace Corporation | Alfred Jung - The Aerospace Corporation | Katherine Losoncy* - The Aerospace Corporation | Art McClellan* - The Aerospace Corporation | James Parke* - The Aerospace Corporation | Beth Springsteen - Ball Aerospace | Colin Wilson - The Boeing Company | David Tong - Harris Corporation | Jason Rife - Lockheed Martin | Anthony Taconi - Micropac Industries | Mark King* - Moog | Rob Bauer - Moog | Ira Johnson - NOAA | Michelle Settles - Northrop Grumman | Tom Seitz - Orbital ATK | Ed Kestel* - SSL | Frank Pastizzo* * Attending MAIW AEROSPACE REPORT NO. TOR-2016-02121 Process Change Assessment Techniques Product Overview Approved Electronically by: - Todd M. Nygren, GENERAL MANAGER, SYSTEMS ENGINEERING DIVISION, ENGINEERING & TECHNOLOGY GROUP - Jacqueline M. Wyrwitzke, PRINC DIRECTOR, MISSION ASSURANCE SUBDIVISION, SYSTEMS ENGINEERING DIVISION, ENGINEERING & TECHNOLOGY GROUP Aerospace Corporate Officer Approval: - Catherine J. Steele, SR VP NATL SYS, NATIONAL SYSTEMS GROUP © The Aerospace Corporation, 2016. All trademarks, service marks, and trade names are the property of their respective owners. SY0094 AEROSPACE REPORT NO. TOR-2016-02121 Process Change Assessment Techniques Product Overview Technical Peer Review Performed by: - Jacqueline M. Wyrwitzke, PRINC DIRECTOR, MISSION ASSURANCE SUBDIVISION, SYSTEMS ENGINEERING DIVISION, ENGINEERING & TECHNOLOGY GROUP Special Programs Security Approval Granted Electronically by: - Alvania W. Thompson, SECURITY STAFF IV, CHANTILLY SPECIAL PROGRAMS SECURITY OPERATIONS & SUPPORT GROUP © The Aerospace Corporation, 2016. All trademarks, service marks, and trade names are the property of their respective owners. SY0094