HARD/SOFT INFORMATION FUSION IN THE CONDITION MONITORING OF AIRCRAFT

ABSTRACT

The synergistic integration of information from electronic sensors and human sources is called hard/soft information fusion. In the condition monitoring of aircraft, the addition of the multisensory capability of human cognition to traditional condition monitoring may create a more complete picture of aircraft condition. A large data set from Naval Air Systems Command (NAVAIR) on maintenance of multi-mission vertical takeoff and landing (VTOL) 22 series B (MV-22B)  aircraft provided the opportunity to explore the value of hard/soft information fusion in aviation maintenance.

First, cognitive and functional frameworks were applied to hard/soft information fusion in the condition monitoring of aircraft. The steps of the Orasanu decision process model were applied to the macrocognitive functions and processes of the aviation maintainer. Emerging literature on hard/soft information fusion in condition monitoring was organized into the levels of the Joint Directors of Laboratories (JDL) data fusion process model, and the levels were applied to the process functions of aviation maintenance.

Second, a research design was created for a retrospective analysis of sensor readings, human observations, and choices made in the maintenance of MV-22B aircraft.

The data set from Decision Knowledge Programming for Logistics Analysis and Technical Evaluation (DECKPLATE), a NAVAIR database, provided information collected without the interference of interviewer bias.

Third, a research methodology was created for studying hard/soft information fusion in aviation maintenance. Content analysis of the descriptive and corrective action narratives showed faults and aircraft components chosen for repair, replacement, fabrication, or calibration. Problem complexity was found to be an important factor. Additionally, expertise level also had an effect, and it was described through longitudinal trending.

 

Chapter 1

Introduction

The problem of aircraft maintenance and operation involves multiple challenges in understanding and processing sensor data, accessing and applying information from humans including pilots, maintenance personnel, engineers, fleet support teams (FSTs), baseline managers (BLMs), and logisticians. Monitoring the mechanical condition of aircraft is ultimately a critical requirement for the safety of passengers and pilots. While increasing opportunities for advanced sensors are available to support condition-based maintenance plus (CBM⁺) and monitoring of aircraft, human observations, including assessments of relevant contextual information would appear to be important for success.

Numerous advances have been made in the past few years as follows: i) multi-sensor data fusion, including integrating information from physical “hard” sensors and from human observations “soft” data [1] , ii) understanding of cognitive models for human decisionmaking and situation awareness [2] , and iii) human-centric design of human-computer systems [3] . This dissertation focused on hard/soft information fusion and its use in decision-making in the context of the condition monitoring of aircraft.

Definitions

Information fusionis the synergistic integration of information from multiple sources [4] .

Hard fusionis the integration of data from electronic sensors [5] .

Soft fusionis the integration of observations from human sources [5] .

CBM⁺is the application and integration of processes, technologies, and knowledge-based capabilities to improve the availability, reliability, and operation of systems [6] .

Sociotechnical Systems

Until the mid-1970s, most information behavior research was focused on the system rather than on user behavior [7] . Since that time, the dominant focus has shifted to the behavior of the user [8] [9] [10] [11] . Today, researchers attempt to conceptualize the interaction between the system and user.

According to Suchman, the machine-centric approach to system design does not result in effective cognition and collaboration [12] . Her study of the relations of human and machine at Xerox Palo Alto Research Center (PARC) in 1981 illustrates the limitations of electronic sensors, even so-called intelligent ones [12] . Specifically, her study of artificially intelligent photocopiers, designed to sense the user and to autonomously offer the most appropriate help, revealed that the machine was oblivious to human difficulty, and thus utterly failed at its task. She reported, “It was as if the machine were watching the user’s actions through a very small keyhole.” Apparently, electronic sensors have their limitations.

Suchman advocated systems design through the analysis of situated actions that arise in the work context [12] . Specifically, she viewed cognition to be heavily influenced by social processes and situational contingencies [13] . Her view is called distributed cognition, which, like naturalistic decision making (NDM), places importance on naturalistic, situated components [14] . NDM is a framework that is built upon cognitive engineering principles in which the design is informed with knowledge elicited from culture-sharing group members in the exercise of their work in its natural setting [15] . The study of sociotechnical systems led to a specialized branch of systems engineering—cognitive systems engineering [16] .

Systems design efforts have been problematic in two ways. First, the traditional approach toward systems design has been machine-centric [17] : the systems designer utilized technology or data without knowing the demands and constraints of the work domain. The resulting designs were often flawed with a device’s clumsy automation surprises [18] .

The lesson from these mistakes is that design for human interaction with machines, particularly for aviation, requires improved understanding of the context in which machines are operated. For example, the introduction of gas-plasma displays decreased human performance in aviation command and control (C2) in the U.S. Navy because the displays were not specifically designed for human use (e.g., inadequate refresh rate for real-time video)  [10] . Hence, displays that “look good” in a lab environment may be unsuitable for actual operational environments.

Second, systems designed from task listings and data flow analyses tend to be brittle, and perform poorly when pushed to the limit. Clumsy automation can become an obstacle during nonstandard events [19] . For example, Obradovich and Woods  reported that when hospital patients were given a system to self-administer medication, many errors occurred due to lack of feedback on success or failure of the patient’s attempts because the system designers had failed to consider the patient’s cognitive demands [20] . In other cases, operators have “just turned off the system rather than bother wrestling with it” [21] , as with Electronic Support Measure (ESM) systems that produce a number of false alarms [22] .

In contrast, a human-centric design is a more effective design approach. It uses the Subject Matter Expert’s (SME’s) knowledge of the needs, requirements, and constraints of the work environment to inform the design [23] . In this approach, knowledge elicitation, and design storyboarding can be employed to encode the SME’s knowledge into an effective design. One human-centric method is decision-centered design (DCD). It exploits difficult, key decisions in the work context to focus design efforts where it will have the highest impact [24] , resulting in a robust system design [21] .

Sociotechnical systems involve human information behavior. According to Wilson, information seeking behavior “is the purposive seeking for information as a consequence of a need to satisfy some goal” [7] . In hard/soft fusion, in addition to sensors, humans provide information for input to the fusion process, and information is transformed into usable knowledge through cognitive processes [25] .

There are several motivations for the utilization of information fusion for CBM⁺.

First, sensors to detect impending mechanical problems are not infallible [4] . Second, the “mapping” between a sensor measurement and possible mechanical failure precursors to failure are not one-to-one (e.g., many types of mechanical problems cause excessive vibration) [26] . Third, humans can often provide contextual information that improves the potential to understand the sensor measurements or possible failure conditions [27] [28] [3] .

Examples of contextual factors in aviation maintenance are as follows:

1. failed diagnostic equipment
2. improper maintenance, such as over-torqueing a bolt, incorrect information in the maintenance manuals, such as an incorrect torque value

• unexpected environment conditions, such as sand, causing additional wear on components

1. higher than expected operational tempo (i.e., decrease time available)
2. material deficiency, such as improper parts manufacture and or quality assurance, which may be attributed to a different manufacturer using substandard materials or manufacturing processes
3. pilot error, such as hard landings, gravitational force (g-force) exceedances, and engine overspeed conditions

 

In an additional example, a pilot might make the following observation, “At an engine speed of 2,100 RPM, I felt moderately heavy vibration through the airframe having a two-to-one beat frequency.” This is important because time-variant beat frequencies measured with sensors are difficult to analyze. The potential value of a correct diagnosis is the prevention of catastrophic mechanical failure and therefore, prevention of loss of life. Driven by the information need described by Spink and Cole [29] to discover why that vibration occurred, the maintainer seeks additional information from the crew and from interactive electronic technical manuals (IETMs). Additionally, the maintainer takes test measurements recorded by the built-in test equipment. He or she pores over IETMs, and seeks information from the original equipment manufacturer (OEM) field service representative (FSR). He or she diagnoses the cause of the vibration to a scratch on a rotor blade causing it to become out of balance. Finally, he or she traces the root cause to a particular mission during which gravel was unexpectedly present at an off-airport landing zone.

Failure Modes and Effects Analysis (FMEA) and Failure Modes, Effects, and Criticality Analysis (FMECA)

The severity and likelihood of failures can be determined by performing FMEA [30] [31] . FMEA is a formal, bottom-up, inductive process that has been extensively employed by The Department of Defense (DoD) since 1949 [32] . National Aeronautics and Space Administration (NASA) adopted FMEA for the Apollo program in 1966 [33] , and it has been applied in commercial aviation as well [34] .

In Figure 1, Subramaniam gives a good summary of FMEA [35] . Data are collected either by electronic sensors or by human observation about impending failures of a particular component. Based upon this data, a FMEA analysis is performed as follows:

1. The possible failure modes (e.g., corrosion, deformation, and open circuit) are listed.
2. Based upon impact to the user, the severity of each failure is determined. The severity is dependent on the context of use (e.g., flight regime, mission).
3. A probability of occurrence is assigned to each failure mode. Data for probabilities of occurrence are obtained initially from a test bed, and then from historical data.
4. A probability of detection is assigned to each failure mode. A probability of detection is different from probability of occurrence, because some failure modes are not easily detectible (e.g., corrosion in concealed places). Note that the probability of detection is certainly not static, but it is a function of things such as, the progression of the failure or condition, the flight regime of the aircraft, the condition of the sensor, etc.
5. Risk priority = (Severity) x (Probability of occurrence) x (probability of detection)
6. Corrective action is performed, and participants learn from their mistakes as they begin the next cycle.

 

 

Figure 1 – FMEA analysis cycle

Courtesy of Subramaniam [35]

 

An example of the use of FMEA analysis in CBM⁺ for aircraft is Milner and

Ochieng’s use of FMEA for the diagnosis of faults occurring on aircraft’s global positioning system (GPS) navigation equipment [36] . Using FMEA, they created a new failure model of the GPS navigation equipment that accurately considers the effects of receiver autonomous integrity monitoring (RAIM) performance.

During the 1990s, the medical community embraced FMEA [37] as a complement to root cause analysis (RCA) in diagnosing patients’ illnesses [38] . According to Senders,

FMEA and RCA are inseparable complements: FMEA seeks the effects of root causes, and FCA seeks the root causes of effects [39] . The medical community’s growing use of FMEA is a tribute to transdisciplinary efforts of physicians, information scientists, and engineers.

There is an interesting parallel between condition-based maintenance and the use of similar techniques in the medical community. One motivation for CBM⁺ rather than timebased or use-based maintenance, is that even when preventive maintenance procedures are performed correctly, there is a significant (30-40%) chance of inducing a mechanical failure simply due to disturbing the structural integrity of a machine during the maintenance process [40] . Similarly, in medical procedures, there is a significant chance of inducing problems simply due to hospitalization or undergoing the procedure. A prime example is the 1 million people (approximately 500,000 die) per year who contract sepsis [41] . This problem of hospital or procedure-based illness is called iatropic illness and is an entire field of study in medicine.

FMEA and the JDL Model

In the Joint Directors of Laboratories (JDL) data fusion process model, FMEA is diagnostic in nature, and it is a level-2 function: situation refinement [42] . In the condition monitoring of aircraft, the situation consists of equipment state parameters, equipment faults, abnormal conditions, flight regimes, environmental conditions, and mission type. The JDL data fusion process model is discussed more fully in Chapter 3.

FMECA is an extension to FMEA that assesses the evolution, trajectory, effects, and impact of faults [33] . FMECA is predictive in nature, and it is a level-3 function in the JDL data fusion process model.

Hard/soft fusion utilizes FMEA, particularly steps two and four. At FMEA step two, judgments are made about context of use and context (e.g., flight regime, mission). At step four, the human cognitive function of adaptability is useful to observe problems not easily detected by electronic sensors (e.g., the odor of a slow fuel leak inside the cabin).

Motivation

Aircraft maintenance is crucial to flight safety—low-quality maintenance has been a leading factor of aviation accidents, flight delays, and flight diversions [43] . Additionally, maintainers, face unique stresses [44] , such as knowing that the work that they perform today will affect the safety of the crew for years in the future—an emotional burden that is largely unrecognized outside the maintenance community.

A major trend in information fusion is to include human observation as a source of information [1] , often referred to as “soft sensing” [4] . According to Hobbs, “From a human factors perspective, maintenance personnel have more in common with doctors than with pilots” [44] . As doctors involved in medical treatment may unintentionally cause iatrogenic injury, a threat to patient health induced by the act of treatment [45] . Likewise, the disassembly of aircraft components required for routine inspection  may cause  aircraft mechanical problems [44] . These maintenance-induced problems cause almost 15% of commercial aviation accidents [46] . Maintenance of a single civil aircraft used as an air carrier costs an average of $300,000 per year [47] . A single flight cancellation costs an airline $140,000, and flight delays cost an airline $17,000 per hour on average [48] . In the safety-critical domain of aviation, the avoidance of iatrogenic (i.e., maintenance-induced) problems constitutes a strong rationale for condition-based maintenance, which is a maintenance strategy that relies on evidence that indicates the state of deterioration. According to this strategy, the decision to disassemble an aircraft component is based upon evidence rather than a specified time interval (e.g., 1 year) or use interval (e.g., 2000 engine hours).

Research Questions

Aviation maintenance is a complex domain [49] that is multidisciplinary [44] , high stakes [50] , and safety-critical [51] . Therefore, it is crucial to understand how the addition of human observation to sensor data affects not only the aircraft components chosen for action in diagnostic maintenance, but also the outcome of those actions. This leads to two research questions.

 

RQ1: Is there a significant association between the aircraft components selected for action and the addition of human observations to sensor data in independent, diagnostic maintenance actions on MV-22B aircraft?

 

RQ2: Is there a significant association between outcomes and the addition of human observation to sensor data significantly associated in independent, diagnostic maintenance actions on MV-22B aircraft?

 

RQ1 inquires about the association of sources of information and its effect on aircraft components chosen for repair, replacement, fabrication, or calibration (see Figure 2).

 

 

Figure 2 – Associations between sources of information, aircraft components chosen for repair; replacement, fabrication, or calibration; and outcomes

 

Hard/soft information fusion offers an interdisciplinary perspective that is appropriate for the problem of hard/soft fusion in the context of the condition monitoring of aircraft. A human-machine collaborative approach to decision-making for fusion exploits the relative strengths of both human and machine, resulting in successful outcomes. Therefore, RQ2 inquires about outcomes of diagnostic maintenance.

Findings and Implications

In this dissertation, the addition of human observation to sensor data was highly associated with the aircraft components chosen for repair, replacement, fabrication, or calibration. Additionally, for complex problems, the addition of human observation to sensor data was significantly associated with improved outcomes. Descriptive statistics showed reduced diagnostic effort with human observation in complex problems. Maintainers’ decision-making may have benefitted from peoples’ innate strengths in context awareness

[52] and  adaptability [21] , which led to the gathering of additional information in the form of human observation. The more complete, accurate assessment of aircraft condition led to better diagnostics. The improved outcomes and reduced diagnostic effort with human observation in complex problems may reduce operational maintenance cost, increase mission readiness, and increase flight safety.

Research Goal and Contributions

Research in hard/soft fusion creates a foundation for improving flight safety, increasing mission readiness, and reducing the cost of maintenance operations. Therefore, doing so may save lives, money, and time. The fusion of data from electronic sensors with human observation provides an opportunity for better outcomes because of humans’ cognitive strength in context awareness [52] .

The current effort is the first to study how hard/soft information fusion could support the condition monitoring of aircraft. This proposed research studies the fusion of pilot, crew, and maintainer observations with sensor data, aircraft components chosen for repair, replacement, fabrication, or calibration, and resulting outcomes. This may provide a foundation for improving flight safety, increasing mission readiness, and reducing the cost of maintenance operations.

The following six contributions regarding hard/soft information fusion’s potential improvement of the condition monitoring of aircraft are realized in this study:

1. Literature review (see Chapter 3) on the topic
2. Application of functional and cognitive frameworks (see Chapter 4) to hard/soft information fusion for aviation maintenance
3. Research design (see Chapter 5) for studying hard/soft information fusion for

CBM⁺ in aviation

4. Methodology (see Chapter 6) for studying hard/soft information fusion that uses content analysis on maintenance narratives in support of CBM⁺ in aviation
5. Summary of results
6. Suggestions for further research

Roadmap for this Dissertation

Figure 3  details the work completed in nine chapters. Chapter 1 introduces CBM⁺ in aviation and describes the shortcomings of traditional sensor fusion, as well as the potential benefits of adding human sources into the information fusion process. It explains the type of research, research questions, research goal, and contributions.

 

 

Figure 3 – Roadmap for this dissertation

 

Chapter 2 describes the background surrounding the context of maintenance work on MV-22B aircraft. Starting with an introduction to naval aviation and the MV-22B Osprey, it explains how aircraft work. The chapter describes the crew, including the pilots who fly the aircraft, the warrant officers who provide technical assistance, and the maintainers who repair the aircraft. Furthermore, the structure and operation of aviation maintenance organizations are described.

Chapter 3 is a transdisciplinary review of the literature on macrocognition, cognitive systems engineering, CBM⁺, hard/soft information fusion, the JDL data fusion process model, and alternative information fusion frameworks.

Chapter 4 discusses a gap that exists in previous research: the application of cognitive and functional frameworks for hard/soft information fusion for condition monitoring of aircraft. These frameworks serve to frame the discussion.

Chapter 5 describes the research design. It explains the research philosophy, study population, assumption, group, and hypotheses. Chapter 6 explains the research methodology that was used, including the statistical methods that were used to test the hypotheses.

Chapter 7 contains a summary of results. Chapter 8 includes a discussion of findings and suggestions for further research. Finally, in Chapter 9 are the conclusions of the study.

HARD/SOFT INFORMATION FUSION IN THE CONDITION MONITORING OF AIRCRAFT

Sharing is caring!