FDAI logo   ::  Site Map  ::   
Home  |  About this Website  |  Contact Us
About this Website » Phase 2 Methodology

Phase 2 Methodology
Overview

Introduction

In Phase 2 of the study, we found and recorded evidence related to each issue identified as a problem or concern in Phase 1 (see Phase 1 Report for more information). Unlike in Phase 1, where we considered any assertion that a problem or concern exists as satisfactory grounds for recording a citation, in Phase 2 we looked for clear, documented evidence supporting or contradicting the issue statement.

For example, consider the issue statement:

Pilots may not understand the structure and function of automation or the interaction of automation devices well enough to safely perform their duties (Issue #105)

This statement suggests that a problem exists. In Phase 1, when we found an unsubstantiated claim that pilots do not adequately understand the automation they use, we recorded the text of that claim as a citation for Issue 105. Without some form of substantial justification, however, that claim could not be considered evidence related to Issue 105 in Phase 2. Rather, the criterion for information about Issue 105 to be considered evidence was that a reasoned, careful study of the issue by an individual or group with demonstrable expertise in aircraft automation and flight safety yielded results supporting or contradicting the Issue 105 statement. That evidence supporting the side of the issue suggested by its issue statement we call supportive evidence. Evidence supporting the other side of the issue we call contradictory evidence. For example, the following could be considered evidence related to Issue 105:
  • data from an experiment showing that a significant number of line pilot subjects demonstrated inadequate knowledge to successfully perform tasks in a simulator scenario (supportive evidence), or
  • data from an experiment showing that a significant number of line pilot subjects demonstrated adequate knowledge to successfully perform tasks in a simulator scenario (contradictory evidence).
  • results of a survey in which a significant number of line pilots claimed adequate knowledge of automation (contradictory evidence), or
  • the conclusions of an accident investigation board that the flightcrew of the accident aircraft did not understand the automation and that lack of understanding was contributory to the accident (supportive evidence).

Top of Page  


Types of Studies

Our analysis consisted of reviewing documented studies potentially yielding evidence related to one or more issues:

We identified these studies in several ways. First, our Phase 1 review yielded a large bibliography of documents related to flight deck automation. Since we were familiar with these documents, having reviewed them for citations of problems and concerns, we knew which ones described studies containing evidence. These documents in turn led us to other documents describing other studies. Our own survey of experts yielded evidence, plus the experts we surveyed in many cases identified documented studies on which they based their responses. Our Phase 1 incident analysis yielded many Aviation Safety Reporting System incident reports potentially containing evidence for issues. Reviews of news reports and accident report abstracts helped us identify accident reports that potentially contained evidence. Finally, by monitoring the current literature, attending professional conferences, and consulting our colleagues in the field, we supplemented our list of documented studies.

Top of Page  


Document Review and Evidence Approval

We carefully read the documentation from each study, looking for information meeting the criterion for evidence described above. When we found it, we recorded the following in our database:

  • the issue supported or contradicted by the evidence
  • the study which yielded the evidence
  • the excerpt from the document which described the evidence
  • the aircraft to which the evidence applied, if specified, such as
    • Airbus A320
    • Boeing B737-300
    • McDonnell Douglas MD11
  • the equipment to which the evidence applied, if specified, including
    • autoflight:
      • autothrust
      • autopilot
    • EICAS (Engine Indication and Crew Alerting System - Boeing)
    • ECAM (Electronic Centralized Aircraft Monitor - Airbus)
    • EFIS (Electronic Flight Instrument System)
    • FMS (Flight Management System)
    • flight control (e.g. fly-by-wire)
    • automation (general or not specified)
  • a numeric value indicating the strength of the evidence supporting or contradicting the asserted issue (see Evidence Strengths section)
  • the name of the analyst

Each evidence record in the database also contained fields for analyst's notes and two approval fields. Both principal investigators (Lyall and Funk) reviewed each evidence record, checking the excerpt against the criterion for evidence, checking the excerpt and other data for completeness and correctness, correcting the record where necessary, and marking the approval fields accordingly. Consequently, each evidence record was reviewed at least three times: once by the original analyst and once each by both principal investigators. Through this process we documented what we consider to be a sound body of evidence related to the issues.

Top of Page  


Evidence Strengths

We found both supportive and contradictory evidence related to the flight deck automation issues, and some evidence was stronger than others. Although comparing the strength of two instances of evidence related to an issue from the same pilot survey, for example, was relatively straightforward, the same cannot be said for comparing the strength of evidence from, say, a pilot survey and an accident report. To try to make the assignment of strengths as objective and consistent as possible, we developed a set of strength assignment rules for each type of study. Evidence strength can range from -5, for strongly contradictory evidence, to +5, for strongly supportive evidence. Although a strength of 0 could be used for evidence that both supported and contradicted an issue, no such evidence was recorded. Rather, for studies with results supporting both sides of an issue, we recorded two instances of evidence, one supportive and one contradictory. The following table summarizes our strength assignment rules. Details are provided on the pages describing our studies.

Evidence Strengths Experiments, Observation Studies, Surveys, and Incident Studies Surveys Experiments to Test Hypotheses Accident Reports
subjects' performance, respondent responses, or report content mean response as a % of max possible (i.e., full agreement with issue statement) rating subjects tasks results statement of investigating board's findings, as compared to problem suggested by issue statement contribution of problem to accident
+5 90-100% supportive 91-100% (strongly agree with issue) line pilots trained on equipment line operations support issue equivalent probable
+4 75-89% supportive 81-90% line pilots trained on equipment simulated line operations support issue equivalent possible
+3 50-74% (or 'most') supportive 71-80% (agree with issue) other line pilots simulated line operations support issue similar probable
line pilots trained on equipment other simulated flight tasks
+2 25-49% supportive 61-70% other line pilots other simulated flight tasks support issue similar possible
GA/student pilots simulated line operations
+1 1-24% (or 'some') supportive 50-60% (slightly agree with issue) line pilots generic automation tasks support issue analyst unsure probable or possible
GA/student pilots other flight automation tasks any of above analyst unsure
-1 1-24% (or 'some') contradictory 40-50% (slightly disagree with issue) line pilots generic automation tasks contradict issue
GA/student pilots other flight automation task
-2 25-49% contradictory 30-39% other line pilots other simulated flight tasks contradict issue
GA/student pilots simulated line operations
-3 50-74% (or 'most') contradictory 20-29% (disagree with issue) other line pilots simulated line operations contradict issue
line pilots trained on equipment other simulated flight tasks
-4 75-89% contradictory 10-19% line pilots trained on equipment simulated line operations contradict issue
-5 90-100% contradictory 0-9% (strongly disagree with issue) line pilots trained on equipment line operations contradict issue

 The reader should exercise some discretion in interpreting strength values. These values reflect our assessments of the extent to which an instance of evidence supports one side or the other of an issue and, as such, are relative assessments. While we found this process extremely useful in analyzing, comparing, and organizing evidence related to the issues, we make no claim that these strengths have universal validity. We urge the reader to make his/her own assessment of how much significance to attach to a particular instance of evidence and to use our numbers merely as a relative guide.

Top of Page  

  Last update: 4 June 2003 Flight Deck Automation Issues Website  
© 1997-2013 Research Integrations, Inc.