Human Factors & Aviation Safety Testimony to the Unites States House of Representatives Hearing on Boeing 737-Max8 Crashes — December 11, 2019 Mica R. Endsley, PhD – Human Factors and Ergonomics Society Chairman DeFazio, Ranking Member Graves, Members of the Committee, thank you for the opportunity to testify today, on behalf of the Human Factors and Ergonomics Society (HFES). With over 4,600 members, HFES is the world’s largest nonprofit association for Human Factors and Ergonomics professionals. HFES members including researchers, practitioners, and federal agency officials, all of whom have a common interest in working to develop safe, effective, and practical human use of technology, particularly in challenging settings. HFES has a particularly strong record of expertise in aviation over its 70-year history. There is a long history of blaming the pilots when aviation accidents occur. This however does nothing towards fixing the systemic problems that underlie aviation accidents that must be addressed to enhance the safety of air travel. Often accidents are caused by design flaws that do not take the human operator's capabilities and limitations into account. Bad design encourages accidents; good design prevents accidents. Solving these systematic design challenges is the primary calling of the field of Human Factors Engineering, which applies scientific research on human abilities, characteristics, and limitations to the design of equipment, jobs, systems and operational environments in order to promote safe and effective human performance. Its goal is to support the ability of people to perform their jobs safely and efficiently, thereby improving the overall performance of the combined human-technology system. Recent investigations of the Lion Air and Ethiopian Airlines crashes of the Boeing 737-Max8 aircraft have highlighted the importance of Human Factors in the design, testing and certification of aircraft. 1; 2 Neglect of attention to Human Factors was also cited by both the National Transportation Safety Board (NTSB) and the FAA Joint Authorities Technical Review (JATR) in their reviews of the contributors to these accidents. 3; 4 I will discuss the field of Human Factors Engineering and its role in supporting high levels of human performance and reducing accidents in safety critical systems such as aviation, particularly as it relates to the use of automation and these tragic aircraft accidents. This includes (1) a discussion of key Human Factors research on the ways that automation directly affects human performance, (2) Human Factors design problems associated with the 737-Max8 flight deck pilot interface, (3) Human Factors design process shortcomings, and (4) organizational and safety culture issues that are implicated in these accidents. The practice of Human Factors Engineering is based on scientifically derived data on how people perceive, think, move and act, particularly when interacting with technology. The way in which any technology is designed significantly affects the performance of the people who interact with it. The user interface of the technology can make human performance much more efficient and human error significantly less likely when it is designed to be compatible with basic human capabilities. When the system is easy to use, guards against typical human frailties and errors (i.e. error tolerance and error resistance), and helps people to rapidly understand key information about what is happening, high levels of human performance in operating the system can be achieved. Conversely, if the technology design is complex, it’s displays are difficult to perceive or understand, it is easy to make errors, and significant effort is required to piece together needed information in order to stay abreast of a complicated and dynamic situation, the likelihood of human error increases greatly. Human Factors supplants a misplaced emphasis on blaming the pilot or over-reliance on training, and instead creates systematic improvements in human performance through improved system design. While training is important, it cannot overcome poor system designs in the long run. 5 People are still likely to make the same types of errors if the system design is not consistent with human capabilities and limitations. For example, when researchers recreated one automation-related aviation accident, they found that 10 out of 12 pilots made the same error as the pilots in the accident when confronted with the same conditions.6 Further, a well-designed system that is consistent with user’s needs and is easier to operate is also easier to train; thus, potentially reducing training requirements as well as improving human performance. The Human Factors profession can be traced back to early work in aviation when it was discovered that a large number of crashes occurred due to human errors that resulted from aircraft cockpits that were inconsistent with basic human capabilities and limitations. This spurred research on the perception, movement, and reaction time of aviators that was used to significantly reduce the frequency of aviation accidents over the following decades by redesigning the controls and displays of the aircraft to be more consistent with pilot characteristics. 7 Since this beginning, Human Factors Engineering has expanded considerably to address human performance challenges across a wide range of industries including aviation, transportation, manufacturing, military operations, power systems, space, health care, consumer products, and many more. Today, Human Factors and Ergonomics Society members are involved in conducting research on how people interact with new technologies, and are actively engaged in applying Human Factors design processes and knowledge across government and industry organizations. The Human Factors field is multi-disciplinary; it includes primarily engineers and psychologists, as well as physiologists and other professionals. Over 50% of the Society's members have PhD’s and 32% have Masters degrees. This blend of backgrounds lends itself well to addressing the wide range of considerations needed to optimize human performance in any system. Automation has increasingly become a part of modern systems in a wide variety of domains, including aviation systems, power systems and automobiles. Across the past 50 years, considerable evidence has mounted demonstrating many benefits from automation, but also many challenges involving human interaction with automation that can contribute to catastrophic failures. 8-10 Just as no man is an island, so too, no automation is an island. Automation must be able to work successfully with human users or, ultimately, it will fail. While automation has many benefits, it also creates new types of errors that must be addressed through careful system design to prevent these new and often catastrophic errors. 8; 11 A long list of automation-related aviation accidents precedes the recent accidents involving the Boeing 737-Max8 that provide significant lessons learned. A recent study listed 26 automation-related accidents among major air carriers between 1972 and 2013 where the pilots were significantly challenged in understanding what the automation was doing and interacting with it correctly to avoid the resulting accident. 12 Several key challenges for human performance can be identified with automation use in aircraft flight decks. Insufficient Pilot Training and the Loss of Manual Skills Human operators have an important role in complex technological systems because of their ability to be flexible, learn, and adapt to unexpected situations. 13 To do this, however, pilots must be highly trained and experienced in managing the aircraft and its systems across a wide variety of flight conditions. 14; 15 Today's airline training environments have been criticized as providing insufficient attention to practice and exposure to the wide variety of alerts and non-normal situations that may be encountered in flight.16 Inadequate training on automation has been found to be a critical problem in many automation accidents. 17 Pilots are often encouraged to use automation and use it frequently. 18 As pilots use automation more often, however, they become reliant on it, 19 and skills needed for manual performance and decision-making can deteriorate. 9; 20 This includes both fine-motor skills associated with aircraft flight control and cognitive skills associated with cross-check and carrying out flight operations. 21; 22 Further, newer pilots, trained primarily to operate via automation, may never create well-learned skills for manual aircraft operations. Poor manual flight skills were implicated in the fatal crash of Colgan Air in 2009, for example. 23 Automation Creates High Workload Spikes and Long Periods of Boredom While automation has frequently been implemented with the goal of reducing manual workload, it can actually increase pilot workload during already high workload periods, such as when a route change is needed or when a problem occurs. This renders it difficult to use, and often pilots must quickly take over manual control in such circumstances which can be quite challenging. 24 It also can make already low workload periods even less engaging, creating new problems associated with lack of vigilance and poor monitoring. 25; 26 This has been called the irony of automation. 8 Automation Confusion is Common Poor operator understanding of system functioning is a common problem with automation, leading to inaccurate expectations of system behavior and inappropriate interactions with the automation. 27; 28 This is largely due to the fact that automation is inherently complex, and its operations are often not fully understood even by pilots with extensive experience using it. 29; 30 A study of the factors underlying automation accidents and incidents found that two of the biggest problems were inadequate understanding of automation and poor transparency of the behavior of the automation.17 Pilots report being highly challenged in determining what the plane is doing and why, and predicting what it will do next, creating a problem of automation surprise. 31 Very often the misalignment between pilots’ understanding of how the aircraft will behave and its actual behavior is only discovered when the aircraft acts unexpectedly. At that point there may be too little time available to discover the problem, properly understand it, and take appropriate action before an accident occurs. 32 Automation confusion is most likely to occur when three main factors are present 33: • The automation acts on its own without immediately preceding directions from the pilot, • The pilot has gaps in knowledge of how the automation will work in different situations, and • Weak feedback is provided to the pilot on the activities of the automation and its future activities relative to the state of the world. Low Situation Awareness to Support Automation Oversight and Intervention Automation is often brittle34, unable to operate outside of the situations that it is programmed for, and subject to inappropriate performance due to faulty sensors or limited knowledge about the current situation. Therefore, the ability of the pilot to supervise the automation and correct for its deficiencies is critical. While some engineers assume that pilots need less information about what is happening when automation is involved, the reverse is actually true. Situation awareness of both the state of the automation and of the systems the automation is controlling is critical to the ability of the pilot to effectively oversee it and make appropriate interventions and control inputs as needed35. Pilots need to keep track of the state of the aircraft and its operation in the flight environment, the state of the automation that is controlling some portion of the job, and information that will allow them to check the reliability and performance of the automation. Achieving a high level of situation awareness, however, has been found to be much more difficult when automation is involved. A key challenge associated with automated systems is that it tends to reduce the situation awareness of the human operator. 35 Pilots with low situation awareness are said to be “out-of-the loop”. Low situation awareness when working with automated systems stems from three main sources 35: • Displays - Poor information presentation is a significant problem with many automated systems. The system developers may either accidentally or intentionally remove key cues that pilots rely on to determine that the system is operating successfully, as was the case with the implementation of fly-by-wire aircraft. 35 The difficulty of determining that automation is not working correctly is a key challenge with automation use. The inadequacy of the displays provided has been found to be a frequent cause of aircraft accidents and incidents involving automation. 17 • Vigilance - Automation often puts people into the role of passive monitor, however, in general, people are poor monitors of automation 36. Vigilance decrements can be significant, occurring both because of over-trust in automation, 17; 19; 37 and because people are in general poor at maintaining vigilance when passively monitoring. 38; 39 • Engagement - A person’s level of engagement decreases when they move from actively performing a task to passively watching another entity performing the task. 35; 40; 41 With low engagement, it has been found that people have a much lower understanding of what is happening than when they are performing tasks themselves. A review of automation research was summarized by a fundamental automation conundrum: “The more automation is added to a system, and the more reliable and robust that automation is, the less likely that human operators overseeing the automation will be aware of critical information and able to take over manual control when needed”. 42 As automation becomes more technologically capable, with increasing levels of reliability and robustness for performing an ever-widening range of tasks, people will become even more hampered by low situation awareness and fall short in the requirement to oversee the automation and interact with it effectively. Even when system designs are improved and people are vigilant, the degrading effects of reduced engagement are difficult to overcome. 42 The two recent crashes involving the Boeing 737-Max8 aircraft involve several of these known automation challenges. These accidents resulted from inaccurate data provided by the aircraft's angle-of-attack (AOA) sensor and its cascading effects on the Maneuvering Characteristics Augmentation System (MCAS) that was developed to automatically provide pitch stability following the addition of new, larger engines on this version of the aircraft. The following analysis of the Human Factors and Safety problems contributing to these accidents is based on the accident reports released by the relevant investigation boards1; 2, reviews by the NTSB4 and the FAA JATR3 in the United States, and other publicly available information on the accidents and events leading up to it. Insufficient Reliability of MCAS Automation Several critical design decisions created an automated system that was inherently brittle and not resilient to the inevitable problems that can happen in the real world. First, the MCAS system on the 737-Max8 was designed to operate from the inputs of only one AOA sensor, unlike a version of the MCAS developed for the United States Air Force KC-46 that measured and compared the inputs from two sensors. 43 When the single AOA sensor provided inaccurate inputs, it created an automated system that performed repeated, erroneous trim actions. The automation had an erratic effect on the stability of the vehicle that was at odds with pilot goals and actions. Redundancy is fundamental to the design of a safe and resilient system; in this case a redundant sensor that could have provided the indications needed for alerting the automation and the pilot of an anomaly. Simple maintenance errors, as occurred in the Lion Air accident can have catastrophic consequences, and should be guarded against though the use of Human Factors design principles in the design of maintenance tasks, procedures and training. Further, the 737-Max8 MCAS was designed to engage and then reengage repeatedly, rather than only the single engagement allowed by the version designed for the United States Air Force. 43; 44 This created a situation in which the automation continued to perform inappropriate and unsafe actions (based on erroneous input data), and that the pilots could not seem to over-ride manually. The basic design of the MCAS automation contained built-in assumptions regarding automation reliability that proved to be unfounded, and that left the pilots highly challenged in managing the aircraft safely. Automation Confusion, Lack of Training and Inadequate Automation Transparency Automation confusion was high as the pilots struggled to understand what the aircraft was doing. The pilots had no previous knowledge of MCAS and the aircraft provided no displays to indicate that MCAS was acting on the aircraft trim, nor any displays to help them understand that it was getting erroneous data. They were in the dark regarding the functioning of MCAS in these accidents. Further, the pilots were not aware of or trained on MCAS, and it was not included in their flight manuals, leaving them confused as to why the plane was behaving erratically. They could not develop a correct understanding of the situation they were facing because they had no mental model to support this process. Effective training on how to overcome automation failures involves not only a written notice or description of the automation, but also actual experience in detecting, diagnosing and responding in such events 18 which was not provided on the Boeing 737-Max8. High Pilot Workload In both accidents the pilots were heavily overloaded in trying to manually control the airplane, needing to exert considerable physical pressure on the control column to compensate for the repeated out-of-limit trim problems. They were simultaneously faced with multiple competing alerts provided by the aircraft. Alerts associated with indicated airspeed (IAS) disagree and altitude disagree were inadequate to help them to understand the fundamental problem they were facing as a result of a faulty AOA sensor. The alerts further created extra workload as the Lion Air pilots attempted to run indicated checklists and work with Air Traffic Control to check their instrument readings. The NTSB’s preliminary report on these accidents highlights the significant mental workload caused by multiple alerts and their role in further distracting the pilots. 4 The alerts provided were insufficient to help the pilots properly understand and diagnose the situation they were in, or to direct them to the appropriate checklists for managing it. The NTSB recommends that “the FAA develop design standards, with the input of industry and human factors experts, for aircraft system diagnostic tools that improve the prioritization and clarity of failure indications (direct and indirect) presented to pilots to improve the timeliness and effectiveness of their response.”4 The Human Factors and Ergonomics Society strongly agrees with this recommendation. Lack of Support for Situation Awareness In these accidents, the pilots were unable to gain the needed situation awareness for accurate decision making. They were faced with an aircraft that repeatedly made uncommanded pitch changes while they received multiple alerts on airspeed disagreements and altitude disagreements which they attempted to address. However, these alerts primarily served to add workload and distractions. Displays of the MCAS operation actions (e.g. trim up or down), and displays that would have helped the pilots to understand the state of the aircraft as affected by the MCAS system were not provided. For example, the inclusion of the AOA sensor display on the primary flight display (PFD) was sold as a system upgrade option. It was not included as a part of the addition of MCAS to the 737-Max8. However, neither the airline customers nor the pilots may have been aware of the need for AOA displays to support proper diagnosis of MCAS behaviors when making such a purchasing decision, due to the lack of information provided about MCAS and how it functioned. While Boeing had previously classified the AOA indicator display and AOA disagree lights as supplemental information and not necessary for the operation of the aircraft, the development of the MCAS system, and its reliance on the AOA sensors, should have created a re-evaluation of this decision. The pilots in these accidents were not provided with the needed displays for understanding the functioning of MCAS, nor of information needed to oversee its performance. There is also some evidence that the pilots may have lost situation awareness of other automated systems whilst dealing with the problems generated by MCAS. While there is only a preliminary accident report available on the Ethiopian Airline accident, it indicates that the crew did correctly set the STAB TRIM to CUTOUT and turned off the auto-pilot. Subsequently, however, their high airspeed made it much more difficult to manually trim the aircraft and maintain the desired pitch. The aircraft throttle remained at 94% N1 throughout and the aircraft did not stop at the input speed of 238 knots, but continued to around 340 knots (vmo). While it is possible the pilots did not understand the impact of the airspeed on the control problems they were facing, it is likely they simply lost situation awareness of their airspeed due to over-reliance on the auto-throttle system. The Boeing's flight crew manual recommends use of auto-throttles in take out and climb and all other phases of flight. Problems with loss of situation awareness of the state of automation and the systems they control are known to be more frequent when people are under higher workload and when working on competing tasks 45; 46 Task fixation is more likely to occur under high workload. In this accident, the captain was highly loaded with trying to fly the aircraft manually, needing to exert considerable manual force, and with only a very inexperienced first officer for help. While alarms and alerts are a key method for helping pilots to detect and diagnose system failures, they were of little help in these accidents. Response to system alerts is not always automatic and immediate, contrary to the stated design assumption of 3 seconds. Responses to alarms and alerts are affected by many factors including the salience of the alert for gaining attention, form of presentation, agreement/disagreement with other indicators, and prior experience with the alert. 47; 48 People must also interpret the meaning of alarms, which depends on context, their mental model of what is happening, and expectations. 49; 50 Often people seek to confirm alarms, and need additional time to properly diagnose the meaning of the alarms in order to select appropriate actions. For example, Boeing’s own data on controlled flight into terrain accidents over a 17-year period show that 26% of these cases involved no response, a slow response, or an incorrect response by the pilot to the GPWS alarm. 51 When multiple alerts across multiple systems are involved, as was the case in these accidents, considerable workload is added and much more time may be required to determine the root cause of the problems so as to select the appropriate response. 47 Multiple failures can cause contradictions between procedures or even prevent their complete execution. The time to respond to the alerts was further delayed due to the fact that the pilots had not been trained to recognize the events and alerts they were presented with, nor to understand MCAS, its reliance on the AOA sensor, and its impact on aircraft control and other flight systems.18 A NASA Study found that the probability of responding correctly for non-trained aircraft emergencies was only 7%, as compared to highly trained "text-book" emergencies at 86%.52 In summary, information that would have informed pilots about the activation of the MCAS or the faulty data inputs to it were lacking on the 737-Max8. The absence of prior training on the MCAS led to a lack of understanding of what was happening to aircraft control. The various alerts that were provided were non-diagnostic and confusing, adding to workload and leading away from a correct understanding of the pitch trim problem, rather than contributing towards a correct resolution in the available time frame. Pilot responses to the alerts were significantly delayed and inadequate due to these deficiencies. Inability to Successfully Assume Manual Control Boeing has a widely publicized Cockpit Automation Philosophy that has guided its aircraft development over the past several decades. 53 Its key tenants are that the pilot can always over-ride the automation, and that it should be an aid to the pilot but not replace the pilot. In keeping with this guiding principle, in most other Boeing aircraft the pilot can always easily resume control by shutting off the autopilot system. However, the MCAS operated outside of this autopilot system and operated at odds with commanded pilot inputs. It is unclear why the design of the 737-Max8 MCAS departed from this consistent automation design philosophy and how the pilots were to know that the MCAS automation was continuing to operate, even after the autopilot was disengaged. Normally, pulling back on the control column will interrupt electronic stabilizer nose down commands in the 737. However, this was not effective in the 737-Max8 as it was set to repeat its actions if it continued to detect an out of trim problem.1 Further, the first officer's side was modified to inhibit pilot column cut-out functions while the MCAS was functioning.1 Thus, the simple, and normal responses that normally worked did not. Proper decision making and performance in the aircraft is highly dependent on accurate situation awareness. While it has been noted that the STAB TRIM CUTOUT switch could have been used to resolve the MCAS problem, this procedure was not used by the crew of Lion Air due to the many factors discussed that lead to their lack of situation awareness. Procedures are only useful when the correct procedure can be selected and applied in a given situation. In the case of the Ethiopian Airlines accident, which occurred after the FAA issued an Emergency Airworthiness Directive on the MCAS54, the pilots set the STAB TRIM to CUTOUT as directed, however, the flight crew continued to experience flight control problems and concluded that the trim was not working. It appears that their loss of situation awareness of the airspeed may have confounded their efforts to manually trim the aircraft, due to the high speeds generated. Once the pilots became involved in trying to overcome the MCAS trim activations, they were required to exert considerable manual force (in excess of 100 pounds according to the Lion Air accident investigation) to combat the actions of the system. Concerns have arisen as to the levels of physical force required and the ability of pilots to combat the strong forces associated with MCAS and the 737-Max8’s engines under the conditions involved in these accidents. Although the Ethiopian Airline crew was able to turn off MCAS via the STAB TRIM CUTOUT switch, they subsequently flew for some two and half minutes while needing to exert manual forces on the control column to compensate for the mis-trim. The FAA Code of Federal Regulations (CFR 25.143) requirement is to not exceed 75 pounds for one-handed or 50 pounds for two-handed short-term control of pitch, and 10 pounds of force for any long-term control of pitch (more than 3 seconds), due to the effects of manual fatigue. These pilots eventually stated "pitch up together" and "pitch is not enough" before turning the electric trim system back on, presumably because they could no longer perform this task manually. This led to the reactivation of MCAS and loss of aircraft control. A determination is needed as to the ability of pilots (both male and female) to exert sufficient manual force to counteract the forces exerted by MCAS at the pitches and speeds in the operational envelope, and to operate manually in the case of a need to deactivate the system due to failures such as were experienced by these aircrew. A recent study by the FAA found that over 60% of females and between 15 and 65% of males (depending on age) were unable to meet current FAA code requirements for short term force application.55 Further, 10 pounds of force for yoke pitch and stick pitch (the long term requirement) could be maintained for less than 5 minutes by between 42% and 60% of females and 12% of males. 55 These results should be extended to address international populations and used to update CFR 25.143 and to update aircraft cockpits to support actual pilot capabilities. A number of good design principles for improving people’s ability to successfully oversee and interact with automated systems have been developed that could have prevented these accidents had they been applied. 48; 56; 57 1. Provide automation reliability. A key tenant of safety is the design of highly reliable systems. Automation needs to be resilient to bad data and avoid single point failures by cross checking across multiple inputs. Further, graceful degradation should be supported such that if the automation is not getting good data, it can provide automatic self-checking behaviors, with an accompanying message to the pilot. In this case, the MCAS should have been designed to read and compare inputs from both AOA sensors, with significant AOA sensor disagreements being used to disable MCAS and support pilot understanding of its operation. 2. The user should be in command. Automation should not interfere with manual operations and manual override should always be possible. Because people have the ultimate responsibility for system safety, because they are more able to adapt to novel, unforeseen situations, and because they may have information about the situation that an automated system does not, they should always be able to easily and simply over-ride the automation and take control. Pilots should be able to easily override activation by the MCAS, rather than having the system fight the user for control. Overcoming the MCAS actions on the trim system should have been as easy as overcoming other electronic trim actions via the control column. 3. Provide automation transparency. The state of the autonomy and its intended actions must be made highly transparent to the pilots. The current goals and assumptions of the autonomy, its current and projected actions, and how much confidence should be placed in its data and algorithms should be clearly represented.48 The system should provide sufficient information to (1) keep pilots informed of its operating mode, intent, function and output, (2) inform pilots of automation failure or degradation, and (3) inform pilots if potentially unsafe modes are manually selected.56 It is critical that the automation mode and status be clearly and saliently displayed. In this case a display showing that the MCAS was on and each time it engaged, as well as its effect on aircraft trim, would have provided key input to the pilots as to what the system was doing. If the MCAS is over-ridden by the pilot and turned off, this should be displayed as well to provide clear feedback to the pilots on its state. Secondly, the state of the world that the automation is basing its actions on, such as the AOA sensors in this case, need to be clearly displayed so the pilot can cross check the reliability of the automation to decide whether to trust it or override it. 4. Provide training to users on automation to ensure adequate understanding and appropriate levels of trust. New automation should be introduced with training to allow pilots to develop accurate mental models of how it works, an understanding of its limitations and reliability in different situations, and information on how to detect and recover from abnormal events and failure conditions. As a significantly new piece of automation that had a direct effect on aircraft control, experiential training (e.g. via simulations) should have been provided that would allow pilots to experience MCAS operations, its failure conditions, and to perform the tasks needed to recover from and effectively overcome abnormal conditions. 5. Avoid increasing cognitive demands, workload and distractions and make tasks easy to perform. The need to sort through multiple competing alerts provided a significant distraction and added workload. Systems should be intelligent enough to filter out extraneous, incorrect, and misleading alerts in order to eliminate both nuisance alarms and reduce unnecessary workload and distraction. 6. Make alarms unambiguous. A failure of the MCAS system due to poor sensor data input should be displayed with a clear unambiguous message. Attempting to diagnose a problem with messages or displays that also have other meanings (e.g. the altitude disagree and airspeed disagree warnings), is an invitation to error and significant delays in responding appropriately to emergent events. Any abnormal behavior of MCAS (as affected by degraded sensors or other factors), should be displayed with an MCAS alert warning that is distinct from other alerts. 7. Support the diagnosis, management, and assessment of multiple alarms. System displays need to support pilots in determining the relationship between multiple alarms, so as to better understand the root cause of any warnings. If root causes are not independent, this needs to be understood, otherwise individually addressing them may not resolve a problem or make it worse. Pilots need support in responding to and handling multiple alerts which can cause contradictions between procedures or even prevent their complete execution, and degrade the utility of the alerts. Alarm management systems for aircraft need to be redesigned to support pilot understanding of how alarms across systems interact, which actions are a priority, and what actions should actually be taken to resolve the underlying problem. The lessons learned from these devastating accidents are important for the design, development and testing of automated systems for not only aviation, but also many other industries where automation is being implemented including military systems, power systems and automobiles. Assumptions of perfect automation are unwarranted and unless great care is taken in supporting the needs of the human operators to have good situation awareness of both the automation and the systems they are controlling, the resulting effect will be repeated tragedies of this nature. A FAA Human Factors team conducted a detailed study of automation-related aviation accidents in 1996. They found that "problems with automation were not limited to any one aircraft type, manufacturer, or air carrier, but were systemic, pointing to much larger problems with the design of the pilot interfaces to the automation, as well as the processes used for design, training, testing, and regulation that were inadequate for addressing the inherent challenges associated with automation." 30 Consistent with their findings, a number of issues pertaining to the Human Factors processes used for the design and certification of aircraft are highly relevant to the 737-MAX8 accidents that will be discussed in more detail. Addressing them across the aviation industry is critical to preventing future accidents. Compliance with Human Factors Design Standards A number of detailed design standards exist relevant to Human Factors and automation that should be adhered to in order to promote good performance and accident prevention. This includes the FAA Human Factors Design Standard, 56, DOD MIL-STD 1472G Design Criteria Standard: Human Engineering, 58 and SAE 6909 Standard Practice for Human Systems Integration. 59 In the case of the 737-Max8, adherence to design principles for human-automation interaction and alarms would have significantly reduced the likelihood of these accidents, as has been discussed. Incorporation of Human Factors Engineering in the Design Process Early incorporation of Human Factors analysis, design and testing during the design process must be emphasized in order to build-in safe, efficient operability. The importance of designing in a consideration of human capabilities and limitations throughout the design process is well established. 60 The design of the operator interface cannot occur at the end of the design process; it is integral to the system design and must occur early during system design to ensure that the combined human-machine system will operate safely and effectively. It is unknown whether Boeing included Human Factors Engineers in its analysis, design and testing activities, and, if so, whether they were sufficiently empowered to affect the 737-Max8 design. Given the many Human Factors deficiencies reported on in the accident analyses, NTSB and JATR studies, it is highly unlikely that Human Factors considerations received sufficient attention or prioritization in the design and development of the 737-Max8 MCAS system. Professionals trained in Human Factors Engineering should be included on the design team and engaged throughout the design process in: (1) conducting analyses of requirements to support human performance, (2) determining system functionality and information needs, (3) designing displays needed to support human performance in both normal and non-normal conditions, and (4) conducting tests of the ability of operators to perform in both normal and non-normal conditions. Conduct and Validate Safety Analyses The value of any safety analysis rests on its thoroughness and its assumptions. A number of poor assumptions regarding MCAS were made during its development: (1) that uncommanded system inputs would be readily recognizable and acted upon by the flight crew with no additional training, (2) action to counter the failure would not require exceptional skill or strength, (3) the pilot would take immediate action counter the problem, and (4) trained flight crew memory procedures would be followed to mitigate the failure. These assumptions proved to be unwarranted in the accidents. The JATR also found that “the system safety assessment and the functional hazard assessment, were not consistently updated.”3 This set the stage for a failure of the safety analyses conducted to adequately capture the real risks involved in the system design. Any assumptions made during safety analyses should be thoroughly vetted and evaluated to ensure that overly optimistic assumptions do not invalidate the benefits of such efforts. When automated systems are involved it is important that safety analyses always consider the potential for invalid inputs to the system, encountering unexpected situations outside of system design limitations, the need for human oversight and intervention, and recovery from automation failures of any kind. Further, safety analyses need to ensure that accurate assumptions are made about human performance, based on human performance data collected in realistic operational conditions when using the system as designed. Conduct Robust Human User Testing to Validate System Designs The careful testing of any new safety critical system is imperative, particularly when automation is involved. In that various types of real-world events occur that may not have been anticipated during the design process, automation's behavior may often prove unexpected. The NTSB’s recent Safety Recommendation Report 4 points out that specific failure modes that could lead to uncommanded MCAS activation were not simulated as a part of Boeing’s function hazard assessment validation tests. Therefore, resultant flight deck problems, such as misleading warning messages and erroneous information displays, were not unearthed during the testing process or assessed for their safety implications. The NTSB recommends that “the FAA develop robust tools and methods, with the input of industry and human factors experts, for use in validating assumptions about pilot recognition and response to safety-significant failure conditions as a part of the design certification process”.4 The Human Factors and Ergonomics Society strongly agrees with this recommendation. It is critical that testing of automation and operator interfaces include: 1) both normal and non-normal events, including automation failures and recovery, 2) a representative sample typical of operators who are external to the system design process, and 3) objective measures of human performance, including actions taken, errors, performance times, workload and situation awareness. Support for Human Factors Assessments in Aircraft Certification The FAA also has a significant role in the design and development process for aircraft technology due to its responsibility as the certifying body. In that it is always possible for design teams to make errors in their assumptions and processes, or for cost and schedule goals to subtly degrade safety decisions, there is great value in having an external certification body who can provide a second review and an independent assessment of the safety of the system. The JATR report indicates that: (1) the FAA certification team did not fully understand the overall impact of the new MCAS system design, (2) the MCAS was not evaluated as a complete and integrated system on the new aircraft, and (3) Boeing failed to inform the FAA of significant design changes over the design process complicating their task.3 It appears that the FAA was unable to perform its important safety role due to the use of delegated authority, or “self-certification”, in which Boeing was able to provide many of its own tests and analyses without independent verification and validation. This process misses the point of the value provided by an independent certification process. Critical to this situation is that the FAA may have inadequate numbers of Human Factors Engineers involved in aircraft certification in addition to the pilots who are often serve in this role. Further, the JATR found that the FAA “sometimes didn’t follow their own rules, used out-of-date procedures and lacked the resources and expertise to fully vet the design changes implicated in two fatal crashes.”3 The JATR recommends that: “the FAA integrate and emphasize human systems integration throughout its certification process. Human factors relevant policies and guidance should be expanded and clarified and compliance with regulatory requirements as 14 CFR 25.1302 (Installed systems and equipment for use by aircrew), 25.1309 (Equipment, Systems, and Installations), and 25.1322 (Flight crew Alerting) should be thoroughly verified and documented. To enable the thorough analysis and verification of compliance, the FAA should expand its aircraft certification resources in human factors and in human systems integration.” 3 The Human Factors and Ergonomics Society strongly agrees with this conclusion and recommendation. Development of Safety Culture A strong safety culture is widely recognized as critical in high-consequence organizations such as aviation, power systems, and ground transportation. Studies have found that workplace-related disasters are often a result of a breakdown in an organization’s policies and procedures that were established to deal with safety, and from inadequate attention being paid to safety issues. 61-66 Many of the effects of a poor or broken safety culture may be subtle and subconscious. For example, a large body of research shows that decisions about what is a problem or not a problem can be easily influenced by reward structures, time pressures, or instructions. 67; 68 Actions and communications of senior management, or rewards for cost and schedule performance (which can be easily measured), can act to subtly shift people towards more risky decisions. While some organizations may tout “safety first” without backing it up, strong safety cultures are effective in promoting safety over short term profit objectives, overcoming fear of reporting that can keep problems hidden, countering non-compliance with standards, rules and procedures, and avoiding miscommunication on critical design and operational factors. Boeing has historically had a strong commitment to safety and human factors in its flight deck programs, however, there are concerns about the possible degradation of the safety culture at Boeing. For example, it was reported in the media that a survey found that one in three Boeing employees reported they felt undue pressure from managers regarding safety-related approvals due to time and schedule concerns, and some 15% reported encountering such problems several times or frequently.69 Further, over the past 15 years Boeing has reportedly lost many of its employees involved in Human Factors and it may not have involved them in the development of MCAS. A renewed focus on developing and maintaining an effective safety culture sets the stage for avoiding the deterioration of design processes that led to these accidents, and should be encouraged in all safety critical organizations. This includes aircraft manufacturers, airlines operations and maintenance, and certifying bodies. The Human Factors and Ergonomics Society applauds recent statements by the FAA Administrator and Boeing Senior Management to reinforce their commitment to safety as the highest priority, and hopes they continue to support this message through their actions. For example, Boeing should examine the degree to which Human Factors and Safety experts are involved in their design and development programs across the enterprise and ensure that they are fully empowered to support safety. Organizational Structure to Support Emphasis on Human Factors and Safety In that program managers can consciously or subconsciously compromise safety due to organizational pressures to meet cost and schedule goals, it is imperative that safety-critical organizations put in place organizational structures to counter such tendencies and avoid safety breakdowns. For this reason, a best practice is to appoint a Vice-President level Manager of Human Factors and Safety who oversees safety at the organization and who can raise safety concerns to the highest level of management. Qualified Human Factors and Safety professionals should be assigned to all technology development programs who can identify potential problems to program managers and recommend design solutions to enhance human performance and avoid serious accidents. They should also have a direct line to the VP of Human Factors and Safety so that unaddressed problems do not remain hidden from top-level management. Use of Qualified Human Factors Professionals In many organizations, people assigned to address user interface design, user experience, human factors, training and safety may have no formal training in these fields, or only cursory knowledge. This unfortunately seriously degrades the effectiveness of their efforts. Just like every other field of engineering, Human Factors Engineering is based on a significant body of formal education that is paramount to its successful practice. Because it is inherently interdisciplinary, however, its practitioners may have different degree titles (often Industrial Engineering or Cognitive/Experimental Psychology, as well as Human Factors). While there are a few bachelors level educational programs in Human Factors, most qualified practitioners will have Masters or PhD degrees in the field. The Human Factors and Ergonomics Society maintains a Directory of accredited academic programs in the field. The Board of Certification of Professional Ergonomists (BCPE) is the recognized body for certifying that individuals are qualified to practice Human Factors, ergonomics and user experience through a combination of testing, experience and academic qualifications. Currently there is no federal job code for Human Factors professionals, significantly complicating the ability of the FAA and other agencies from hiring personnel with the appropriate expertise. The objective of Human Factors Engineering is not to assign blame after accidents occur, but rather to prevent accidents from occurring by improving the design of technologies and systems in advance. In this light, the Human Factors and Ergonomics Society recommends that the FAA: 1) Encourage Boeing, and other aviation manufacturers, to incorporate Human Factors processes and personnel into the analysis, design, development, testing, manufacturing, and maintenance of aircraft systems in order to comply with certification requirements. (Supports FAA JATR recommendation 4) 2) Promote a safety culture in aviation that drives a primary focus on the creation of safe products, which in turn comply with certification requirements. (Supports FAA JATR recommendation 6) 3) Expand its aircraft certification resources in human factors and in human system integration, and integrate and emphasize human factors and human system integration throughout its certification process. (Supports FAA JATR recommendation 7) 4) Review training programs for automated systems and recommend improvements to ensure flight crew are adequately trained in automation processes and dependencies. 5) Conduct a study to determine the adequacy of policy, guidance, and assumptions related to maintenance and ground handling training requirements, and needed human factors improvements to reduce maintenance errors. (Supports FAA JATR recommendation 11) 6) Conduct system safety assessments for all manufacturers of type rated aircraft to demonstrate the adequacy of assumptions regarding human performance, particularly as it relates to pilot understanding of and response to alerts, and ability to perform control functions manually when needed. (Supports NTSB recommendations) In addition to these recommendations, Congress and the federal government can enact policies that will build up the FAA’s and industry’s capacity and expertise to better understand and address issues pertaining to alerting systems and human-automation interaction that are crucial to avoiding future catastrophic accidents in the nation's transportation and infrastructure. HFES recommends that Congress enact the following policies: 7) Direct the National Academies of Science Board on Human Systems Integration (NAS BOHSI) to conduct a study on human interaction with artificial intelligence, autonomy, and advanced automation technologies: enhancing safety and effectiveness. Such a study could bolster the knowledge and understanding of the many complex issues involved and provide important directions for the nation as it develops and implements these technologies across the coming decade. 8) Increase funding for Human Factors research at NASA and FAA on: a. Understanding the effects of multiple alerts and the "design of aircraft system diagnostic tools that improve the prioritization and clarity of failure indications (direct and indirect) presented to pilots to improve the timeliness and effectiveness of their response". (supporting NTSB recommendations) b. Developing "robust tools and methods for use in validating assumptions about pilot recognition and response to safety-significant failure conditions as part of the design certification process". (supporting NTSB recommendations) c. The development of effective methods, displays and training for supporting human oversight and interaction with automated systems. This could include support for the FAA’s NextGen – Air Ground Integration Human Factors program’s efforts around Human Error mitigation research, as well as the Flightdeck/Maintenance/System Integration Human Factors program, and NASA's Crew Systems and Aviation Operations program. d. The development of tools to support human factors and safety assessments in the certification process. (supporting FAA JATR recommendations) 9) Direct the FAA to develop programs for educating management and engineers on Human Factors, the effects of automation on human performance in safety critical systems, and the incorporation of Human Factors Engineering processes, as well as the development of improved human-automation interaction approaches. 10) Direct the Office of Personnel Management to add job codes for Human Factors Engineer and Human Factors Psychologist to its list of occupational positions and establish appropriate qualifications in order to ensure that FAA and other federal agencies have access to and employ qualified Human Factors professionals Summary The science and practice of Human Factors Engineering is well established, with roots going back to the earliest days of aviation. Aviation is highly dependent on the design and development of safe, effective flight decks for pilot control. Achieving this goal is highly dependent on the early incorporation of Human Factors in the analysis, design, testing and certification processes. While this is true in general, it is even more important with automated systems and as use of artificial intelligence and system autonomy increases. The lessons learned from the tragic accidents of the 737-Max8 should be leveraged to improve the safety of our aviation system, and to guard against similar problems in other safety critical systems. References 1. Republic of Indonesia Komite Nasional Keselamatan Transportasi. (2019). Final Aircraft Accident Investigation Report. PT Lion Mentari Airlines, Boeing 737-8 (max); PK-LQP, Tanjung, Karawang, West Java Republic of Indonesia: Author. 2. Federal Democratic Republic of Ethiopia Ministry of Transport Aircraft Accident Investigation Bureau. (2019). Aircraft Accident Investigation Preliminary Report, Ethiopian Airlines Group, B737-8 (Max) Registered ET-AVJ, 28 NM South East of Addis Ababa, Bole International Airport Federal Democratic Repubic of Ethiopia: Author. 3. Federal Aviation Administration. (2019). Joint Authorities Technical Review: Boeing 737 Max Flight Control System Observations, Findings and Recommendations Washington, DC: Author. 4. National Transportation Safety Board. (2019). Safety Recommendation Report: Assumptions used in the safety assessment process and the effects of multiple alerts and indications on pilot performance Washington, DC: Author. 5. Wickens, C. D. (1992). Engineering psychology and human performance (2nd ed.). New York: Harper Collins. 6. Johnson, E. N., & Pritchett, A. R. (1995). Experimental study of vertical flight path mode awareness. Proceedings of the 6th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design and Evaluation of Man-Machine Systems (pp. 185-190). Cambridge, MA: MIT. 7. Meister, D. (1965). Human factors evaluation in system development. New York: Wiley and Sons. 8. Bainbridge, L. (1983). Ironies of automation. Automatica, 19, 775-779. 9. Wiener, E. L., & Curry, R. E. (1980). Flight deck automation: Promises and problems. Ergonomics, 23(10), 995-1011. 10. Wiener, E. L. (1993). Life in the second decade of the glass cockpit. Proceedings of the Seventh International Symposium on Aviation Psychology (pp. 1-11). Columbus, OH: Department of Aviation, The Ohio State University. 11. Strauch, B. (2017). Ironies of automation: Still unresolved after all these years. IEEE Transactions on Human-Machine Systems, 48(5), 419-433. 12. Gawron, V. (2019). Automation in aviation accidents: Accident analyses McLean, VA: MITRE Corporation. 13. Rasmussen, J. (1980). What can be learned from human error reports? Change in the Working Life. 14. Ericsson, K. A., & Lehmann, A. C. (1996). Expert and exceptional performance: Evidence of maximal adaptation to task constraints. Annual Review of Psychology, 47, 273-305. 15. Meshkati, N., & Khashe, Y. (2015). Operators' improvisation in complex technological systems: Successfully tackling ambiguity, enhancing resiliency and the last resort to averting disaster. Journal of Contingencies and Crisis Management, 23(2), 90-96. 16. DeCrespigny, R. C. (2015). Resilience - Recoveing pilots’ lost flying skills. Air Transport(June), 32-37. 17. Funk, K., Lyall, B., Wilson, J., Vint, R., Niemczyk, M., Suroteguh, C., & Owen, G. (1999). Flight deck automation issues. The International Journal of Aviation Psychology, 9(2), 109-123. 18. Orlady, L. M. (2010). Airline pilot training today and tomorrow Crew resource management (pp. 469-491): Elsevier. 19. Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50-80. 20. Jacobson, S. (2010). Aircraft loss of control causal factors and mitigation challenges. Proceedings of the AIAA Guidance, navigation, and control conference (pp. 8007). 21. Haslbeck, A., & Hoermann, H.-J. (2016). Flying the needles: flight deck automation erodes fine-motor flying skills among airline pilots. Human factors, 58(4), 533-545. 22. Casner, S. M., Geven, R. W., Recker, M. P., & Schooler, J. W. (2014). The retention of manual flying skills in the automated cockpit. Human factors, 56(8), 1506-1516. 23. Board, N. T. S. (2010). Aviation accident report: Loss of control on approach Colgan Air, Inc. operating as Continental Connection Flight 3407 Bombardier DHC-8-400, N200WQ Clarence Center New York, February 12, 2009. (Tech. Rep. No. NTSB/AAR-10/01 PB2010-910401. 24. Wiener, E. L., & Nagel, D. C. (Eds.). (1988). Human Factors in Aviation. San Diego: Academic Press. 25. Warm, J. S., Dember, W. N., & Hancock, P. A. (1996). Vigilance and workload in automated systems. Automation and human performance: Theory and applications, 183-200. 26. Molloy, R., & Parasuraman, R. (1996). Monitoring an automated system for a single failure: Vigilance and task complexity effects. Human Factors, 38(2), 311-322. 27. Sarter, N. B., & Woods, D. D. (1992). Pilot interaction with cockpit automation: Operational experiences with the flight management system. The International Journal of Aviation Psychology, 2(4), 303-321. 28. Sarter, N. B., & Woods, D. D. (1994). "How in the world did I ever get into that mode": Mode error and awareness in supervisory control. In R. D. Gilson, D. J. Garland & J. M. Koonce (Eds.), Situational awareness in complex systems (pp. 111-124). Daytona Beach, FL: Embry-Riddle Aeronautical University Press. 29. McClumpha, A., & James, M. (1994). Understanding automated aircraft. In M. Mouloua & R. Parasuraman (Eds.), Human performance in automated systems: Current research and trends (pp. 183-190). Hillsdale, NJ: LEA. 30. Federal Aviation Administration Human Factors Team. (1996). The interfaces between flightcrews and modern flight deck systems Washington, DC: FAA. 31. Wiener, E. L. (1988). Cockpit automation. In E. L. Wiener & D. C. Nagel (Eds.), Human Factors in Aviation (pp. 433-461). San Diego: Academic Press. 32. Billings, C. E. (1997). Aviation automation: The search for a human-centered approach. Mahwah, NJ: Lawrence Erlbaum. 33. Woods, D. D., & Sarter, N. B. (2000). Learning from automation surprises and going sour accidents. Cognitive engineering in the aviation domain, 327-353. 34. Woods, D. D., & Cook, R. I. (2017). Incidents–markers of resilience or brittleness? Resilience Engineering (pp. 69-76): CRC Press. 35. Endsley, M. R., & Kiris, E. O. (1995). The out-of-the-loop performance problem and level of control in automation. Human Factors, 37(2), 381-394. 36. Moray, N. (1986). Monitoring behavior and supervisory control. In K. Boff (Ed.), Handbook of perception and human performance (Vol. II, pp. 40/41-40/51). New York: Wiley. 37. Muir, B. M., & Moray, N. (1996). Trust in automation: Part 2. Experimental studies of trust and human intervention in a process control simulation. Ergonomics, 39, 429-460. 38. Davies, D. R., & Parasuraman, R. (1980). The psychology of vigilance. London: Academic Press. 39. Hancock, P. (2013). In search of vigilance: The problem of iatrogenically created psychological phenomena. American Psychologist, 68(2), 97-109. 40. Manzey, D., Reichenbach, J., & Onnasch, L. (2012). Human performance consequences of automated decision aids: The impact of degree of automation and system experience. Journal of Cognitive Engineering and Decision Making, 6, 57-87. 41. Metzger, U., & Parasuraman, R. (2001). Automation-related "complacency": Theory, empirical data and design implications. Proceedings of the Human Factors and Ergonomics Society 45th Annual Meeting (pp. 463-467). Santa Monica, CA: Human Factors and Ergonomics Society. 42. Endsley, M. R. (2017). From here to autonomy: Lessons learned from human-automation research. Human Factors, 59(1), 5-27. 43. Everstine, B., & Tirpack, J. A. (2019, 3/22/2019). USAF reviewing training after MAX 8 crashes; KC-46 uses similar MCAS. Air Force Magazine. 44. Sider, A., & Tangel, A. (2019, September 29, 2019). Before 737 MAX, Boeing’s flight control system included key safeguards. The Wall Street Journal. 45. Kaber, D. B., & Endsley, M. R. (2004). The Effects of Level of Automation and Adaptive Automation on Human Performance, Situation Awareness and Workload in a Dynamic Control Task. Theoretical Issues in Ergonomic Science, 5(2), 113-153. 46. Ma, R., Sheik-Nainar, M. A., & Kaber, D. B. (2005). Situation awareness in driving while using adaptive cruise control and a cell phone. Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting (pp. 381-385). Santa Monica, CA Human Factors and Ergonomics Society. 47. Gilson, R. D., Deaton, J. E., & Mouloua, M. (1996). Coping with Complex Alarms: Sophisticated Aircraft Cockpit Alarm Systems Demand a Shift in Training Strategies. Ergonomics in Design, 4(4), 12-18. 48. Endsley, M. R., & Jones, D. G. (2012). Designing for situation awareness: An approach to human-centered design (2nd ed.). London: Taylor & Francis. 49. Gilson, R. D., Mouloua, M., Graft, A. S., & McDonald, D. P. (2001). Behavioral influences of proximal alarms. Human Factors, 4(4), 595-610. 50. Seagull, F. J., & Sanderson, P. M. (2001). Anesthesia alarms in context: An observational study. Human Factors, 43(1), 66-78. 51. Graeber, R. C. (1996). Integrating human factors and safety into airplane design and operations. In B. J. Hayward & A. R. Lowe (Eds.), Applied aviation psychology: Achievement, change and challenge (pp. 27-38). Aldershot, UK: Avebury Aviation. 52. Burian, B. K., Barshi, I., & Dismukes, K. (2005). The challenge of aviation emergency and abnormal situations (NASA/TM-2005-213462). Moffett Field, CA: NASA. 53. Braune, R. J., & Graeber, R. C. (1992). Human-centered designs in commercial transport aircraft. Proceedings of the Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 1118-1122). SAGE Publications Sage CA: Los Angeles, CA. 54. Federal Aviation Administration. (2018). Emergency Airworthiness Directive AD #2018-23-51: Author. 55. Beringer, D. B. (2019). NextGen Final Report: Data for updating 14 CFR Part 25.143 and potential reference standards for part 23, 27 and 29 aircraft: An evaluation of muscular force that can be applied to flight controls (DOT/FAA/AM-19/5). Oklahoma City: FAA Civil Aerospace Medical Institute. 56. Federal Aviation Administration. (2016). The Human Factors Design Standard (HF-STD-001). Washington, DC: Author. 57. Yeh, M., Swider, C., Jo, Y. J., & Donovan, C. (2016). Human factors considerations in the design and evaluation of flight deck displays and controls: version 2.0: John A. Volpe National Transportation Systems Center (US). 58. Department of Defense. (2012). MIL-STD-1472G Design Criteria Standard: Human Engineering Washington, DC: Author. 59. SAE International. (2019). SAE 6906 Standard Practice for Human Systems Integration Warrendale, PA: Author. 60. Pew, R. W., & Mavor, A. S. (Eds.). (2007). Human system integration in the system development process: A new look. Washington, DC: National Academic Press. 61. Cooper, M. D. (2000). Towards a model of safety culture. Safety science, 36(2), 111-136. 62. Pidgeon, N., & O’Leary, M. (1994). Organizational safety culture: Implications for aviation practice. Aviation psychology in practice, 21-43. 63. Meshkati, N. (2002). Macroergonomics and aviation safety: The importance of cultural factors in technology transfer Macroergonomics (pp. 337-344): CRC Press. 64. Meshkati, N. (2007). Lessons of the Chernobyl nuclear accident for sustainable energy generation: Creation of the safety culture in nuclear power plants around the world. Energy Sources, Part A, 29(9), 807-815. 65. Reason, J. (2016). Managing the risks of organizational accidents: Routledge. 66. Wiegmann, D. A., Zhang, H., Von Thaden, T. L., Sharma, G., & Gibbons, A. M. (2004). Safety culture: An integrative review. The International Journal of Aviation Psychology, 14(2), 117-134. 67. Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics (Vol. 1): Wiley New York. 68. Barkan, R. (2002). Using a signal detection safety model to simulate managerial expectations and supervisory feedback. Organizational Behavior and Human Decision Processes, 89(2), 1005-1031. 69. Tangel, A., & Pasztor, A. (2019, October 10, 2019). Congress ramps up scrutiny of Boeing executives, board, The Wall Street Journal.