AN INVESTIGATION OF DATA OVERLOAD IN TEAM-BASED DISTRIBUTED COGNITION SYSTEMS

  • : Ms Word, Ms Word Format
  • : 70 Pages
  • : ₦3,000 | $25 | ₵60 | Ksh 2720
  • : 1-5 Chapters
  •  
  • Click to DOWNLOAD Materials

AN INVESTIGATION OF DATA OVERLOAD IN TEAM-BASED DISTRIBUTED COGNITION SYSTEMS

ABSTRACT

The modern military command center is a hybrid system of computer automated surveillance and human oriented decision making. In these distributed cognition systems, data overload refers simultaneously to the glut of raw data processed by information technology systems and the dearth of actionable knowledge useful to human decision makers.  Designing new systems that mitigates this paradox of data availability is a challenge that faces multiple scientific and engineering disciplines. This thesis surveyed the literature from the following fields and disciplines in order to comprehensively evaluate their characterizations and solutions to the problem of data overload:

  1. Multi Sensor Data Fusion
  2. Cognitive Systems Engineering
  3. Human Computer Interaction
  4. Decision Making
  5. Computer Supported Collaborative Work

 

The outcome of the literature review is a new taxonomy for describing data overload problems in distributed cognition systems. This new taxonomy synergizes the various unique perspectives offered from the literature to provide an interdisciplinary tool that can be employed to diagnose reported problems of data overload in distributed cognition systems. This dissertation demonstrates the application of the taxonomy to the NeoCITIES simulation.  NeoCITIES is a simulated command and control environment designed to mimic the sensemaking and decision making processes of analysts in 9-1-1 dispatch centers.

The NeoCITIES task environment was selected for the experimental task in this dissertation. Experimental manipulations were designed to recreate data overload conditions in NeoCITIES and explore the impact of new team user interface elements. New team user interface elements were designed to mitigate communication aspects of data overload as defined by the taxonomy.

This dissertation summarizes the results of 28 team experiments conducted with 84 participants in the spring semester of 2009.  The results were applied to the main research question investigating the relationship between data overload and team user interface elements as it pertains to the team’s task performance, communication behavior, and perception.

 

 

TABLE OF CONTENTS

LIST OF FIGURES …………………………………………………………………………………………………..vii

LIST OF TABLES…………………………………………………………………………………………………….viii

ACKNOWLEDGEMENTS………………………………………………………………………………………..ix

Chapter 1  Introduction to Data Overload in Distributed Cognition Systems …………………….1

Motivation…………………………………………………………………………………………………………3

The Transformation Process ………………………………………………………………………………..5

Challenges…………………………………………………………………………………………………………7

Chapter 2  Review of Data Overload Perspectives…………………………………………………………9

Cognitive Systems Engineering (CSE) Perspectives ……………………………………………….16

Human Computer Interaction (HCI) Perspectives…………………………………………………..22

Decision Making Perspectives……………………………………………………………………………..29

Computer Supported Collaborative Work (CSCW) Perspectives………………………………40

Summary…………………………………………………………………………………………………………..50

Chapter 3  Materials & Methods………………………………………………………………………………….53

Research Approach …………………………………………………………………………………………….53

The Living Laboratory Framework………………………………………………………………..54

Applying the Data Overload Taxonomy to NeoCITIES……………………………………57

Simulation Structure …………………………………………………………………………………………..60

The Human Performance Scoring Model………………………………………………………..62

The NeoCITIES Client User Interface……………………………………………………………65

Pilot Testing ……………………………………………………………………………………………….69

Experimentation…………………………………………………………………………………………………71

Independent Variable Manipulation……………………………………………………………….72

Experiment Design………………………………………………………………………………………75

Research Design…………………………………………………………………………………………………78

Participants…………………………………………………………………………………………………82

Equipment ………………………………………………………………………………………………….83

Task Procedure……………………………………………………………………………………………84

Data Collection……………………………………………………………………………………………86

Chapter 4  Results……………………………………………………………………………………………………..89

Participant Demographics……………………………………………………………………………………89

NeoCITIES Team Scores…………………………………………………………………………………….91Survey Scales…………………………………………………………………………………………………….93

Coded Chat Logs………………………………………………………………………………………………..96

NeoCITIES Interface Evaluations…………………………………………………………………………97

Summary…………………………………………………………………………………………………………..99

Chapter 5  Discussion ………………………………………………………………………………………………..101

Experiment Analysis…………………………………………………………………………………………..101

Data Overload Scenarios………………………………………………………………………………104

Team User Interface Elements………………………………………………………………………108

Team Communication Analysis…………………………………………………………………….110

Contributions……………………………………………………………………………………………………..118

Limitations………………………………………………………………………………………………………..124

Future Work………………………………………………………………………………………………………129

Chapter 6  Conclusion………………………………………………………………………………………………..135

References………………………………………………………………………………………………………………..140

Appendix A  Survey of Data Fusion Process Models……………………………………………………..148

Appendix B  Survey of Cognitive Process Models…………………………………………………………158

Appendix C  IRB Materials ………………………………………………………………………………………..171

Informed Consent Form for Social Science Research ……………………………………………..172

Post Condition Survey Instrument………………………………………………………………………..174

Post Experiment Questionnaire…………………………………………………………………………….176

NeoCITIES Recruitment Slide……………………………………………………………………………..180

Appendix D  Coding Procedure for NeoCITIES Chat Logs…………………………………………….181

 

LIST OF FIGURES

Figure 1-1:  The transformation process from raw data to knowledge………………………………6

Figure 2-1:  Top level view of the JDL data fusion process model…………………………………..11

Figure 2-2:  Network visualization clutter…………………………………………………………………….25

Figure 2-3:  The transition from Microsoft Word 2003 to Word 2007………………………………27

Figure 2-4:  The optimum point of information load………………………………………………………32

Figure 3-1:  The Living Laboratory Framework…………………………………………………………….55

Figure 3-2:  The NeoCITIES system architecture diagram ……………………………………………..60

Figure 3-3:  The NeoCITIES Scoring Model & Equation……………………………………………….63

Figure 3-4:  Screenshot of the NeoCITIES user interface ……………………………………………….66

Figure 3-5:  The Team Panel User Interface Adjacent to the Chat……………………………………76

Figure 3-6: The Team Event Activity Window……………………………………………………………..77

Figure 5-1:  The NeoCITIES Scoring Summary Screen………………………………………………….102

Figure 5-2:  Histogram of the Normal Scores………………………………………………………………..106

Figure 5-3:  Histogram of Total Communications………………………………………………………….112

 

 

 

LIST OF TABLES

Table 1-1:  Examples of Problem Domains affected by Data Overload…………………………….1

Table 2-1:  Summary of human roles and tasks……………………………………………………………..38

Table 2-2:  Literature Perspectives on Data Overload…………………………………………………….50

Table 2-3:  The Data Overload Taxonomy……………………………………………………………………52

Table 3-1:  Survey of Data Overload Symptoms from previous NeoCITIES studies ………….57

Table 3-2:  Pilot Testing Summary………………………………………………………………………………69

Table 3-3:  Generated Dependent Variables & Measurements in NeoCITIES……………………71

Table 3-4:  Potential Independent Variable Manipulation in NeoCITIES………………………….73

Table 3-5:  2×2 Factorial Design. ………………………………………………………………………………..79

Table 3-6:  Latin Squares Experiment Design……………………………………………………………….80

Table 3-7:  The Experiment Subject Pool Course Listing. ………………………………………………82

Table 3-8:  Participant Task List and Time Spent. …………………………………………………………85

Table 3-9:  The NeoCITIES chat log coding scheme……………………………………………………..87

Table 4-1:  Factor Analysis for Scenario Assessment Surveys…………………………………………93

Table 4-2:  Interface Evaluation Questions and Mean Responses…………………………………….97

Table 4-3:  Experiment Hypotheses Results Summary……………………………………………………99

Table 5-1:  Survey Responses across Team Performance Levels……………………………………..107

Table 5-2:  Chat Types across Team Communication Frequency…………………………………….113

Table 5-3:  Chat Types across Prior Team Acquaintance………………………………………………..116

 

 

Chapter 1

 

Introduction to Data Overload in Distributed Cognition Systems

The modern command center is a rich source of research that studies human decision making and machine operations. This system is a type of distributed cognition system, requiring both human and machine interaction to achieve a common goal (Hutchins, 1996). As a distributed cognition researcher, we are particularly interested in studying how information can be applied using technology to enhance a person’s ability to assess a situation and make a decision. Although we can never guarantee that people will make better decisions, we assume that given better information, technology, and training, we can make their work as a decision maker easier and more effective.

This thesis is focused on the domain of problems where data and information overload hinders the human analyst in their ability to assess a situation and enact effective decisions. This thesis critically examines the problem of data overload in the context of operation centers. Specifically, we study primarily operation centers at the level of analysis of a command and control dispatch center.

Table 1-1 lists three example domains in which data or information overload is a problem for a human analyst. This table also lists the types of data that feed the problem domain and the types of decision inferences that must be made by an analyst. These problems are summarily described in the paragraphs below.

Table 1-1 Examples of Problem Domains affected by Data Overload

Domain Problem Description Data Types Decision Inferences
Military command and control Coordination of  military operations in order to achieve battlefield awareness Geospatial, human observations, remote sensing Threat assessment, impact assessment, logistical operations
Domain Problem Description Data Types Decision Inferences
Emergency crisis management Emergency

dispatchers coordinate local agencies to respond to natural and man-made disasters

Geospatial data, human observations, media reports Monitoring, coordination, logistical operations, response
Cyber security Detecting and diagnosing potential

threats on virtual networks

Network sensor data, network topology Detection, threat assessment, monitoring, response,

 

 

The military command center has changed from the cold-war era image of the North American Air Defense (NORAD) Command (http://www.norad.mil/) where enemy combatants were well known in advance and remained relatively static. Today, modern command centers must be agile to switch from peacekeeping operations, to humanitarian assistance, to fighting terrorism, often in on the same day, in the same location, referred to as the “Three-Block War” (Surdu, Parsons, & Tran, 2005).  The data provided to analysts and their commanders are gathered from various sources including human observations of the situation, geospatial maps of the terrain, and other data collected through remote satellites and unmanned aerial vehicles

(UAV). In this domain, the goal of the operational commander is to achieve a state of “Battlefield Awareness” (Jameson, 2001) in which the commander is aware of the locations and movements of friendly and hostile troops on the battlefield. The decisions associated with analysts and their commanders include the monitoring of war zone situations, coordination between military branches, logistical planning of fleet resources, and finally determining the appropriate course of action to respond to hostile threats.

Emergency crisis management is a domain focused on responding to disasters both natural and man-made. Similar to military command and control centers, emergency crisis management centers need to coordinate response efforts between multiple agencies while continually monitoring and assessing the severity of the situation. Data for these decisions is gathered from remote satellites including geospatial maps of afflicted areas and human observations including individual agency reports and public information received from the news media. In this domain, the goal of the emergency dispatcher is to understand the “Common Operational Picture (COP).” (M. McNeese & Hall, 2003).  The COP is a state of awareness in which the dispatcher understands the locations and status of local agency assets across other working dispatchers.  As recent disasters have shown, it is important for dispatchers to have an understanding of the resources available between the different civilian agencies such as fire, police, and medical services in order to communicate an effective response (R. E. T. Jones, McNeese, Connors, Jefferson, & Hall, 2004).

Cyber Security is a domain that is focused on detecting and responding to network intrusions.  Observations of data are gathered not from physical sensors, but from network sensors that digitally record network traffic.  In the world of networks, millions of packets of information can pass by at any given minute. The massive amount of data gathered from these networks’ sensors necessitates the use of data fusion algorithms and mathematical techniques (Bass, 2000). Once the raw data is processed into metadata using data fusion, the information can be accessed by network analysts to monitor and evaluate any potential network threats. The responsibility of the analyst is to effectively monitor a network and then subsequently detect and respond to network intrusions.

Motivation

In the context of command centers, human analysts suffer from the consequences of data overload. The military commander experiences data overload when multiple battles occur simultaneously, each demanding his attention and requiring his decisions to act. The emergency dispatcher experiences data overload when multiple emergency events occur, each event requiring the coordination of resources between local civil authorities. The network analyst experiences data overload when multiple network intrusions are detected, requiring the analyst to act quickly before data is compromised.

The problem of data overload is important to study because the consequences of human failure are extreme.  Lives of military troops and civilians, can be lost if the military commander is overwhelmed. Buildings can crumble and lives can be lost if the emergency dispatcher is not able to send the local fire, police, and medical authorities in time. Networks can be compromised, and secure data can be stolen if the network analyst can not detect the intrusion and respond accordingly. While the consequences can be less extreme in more daily activities, they are no less real. Lessons learned from studying data overload in extreme conditions and domains can be applied towards common applications such as monitoring logistics for business operations, monitoring the operation and health of machines and manufacturing plants, etc.

Outside of the listed domains, the study of data overload becomes increasingly relevant with the advent of the information technology trends. As a result of Moore’s law, the doubling of computing processor power every 12 – 18 months, powerful home computing is ubiquitous (Schaller, 1997). The growth of home computing and development of the World Wide Web has lead directly to a growth in the consumption and generation of information.  Communities of information consumers continue to grow around such popular sites such as Youtube, Flickr, and Digg. Yet the quality of information is not necessarily increasing. Greater volumes of information being made accessible through the world wide web has not lead to greater guarantees in its accuracy, reliability or credibility (Berghel, 1997). At the same time, the ability to store the information has greatly reduced in cost, less than a hundred dollars for a Terabyte of data.  Thus not only is society moving towards generating more information, both useless and relevant, it is also archiving this information indefinitely.

All of these IT trends serve to increase the problem of information overload. Cognitive

Science researcher David Woods calls this the data availability paradox. While technology has made data more accessible and readable, it has also increasingly challenged the human ability to make sense of the data (Woods, 2002).  For the average individual, while there are more choices for any given decision, their ability to select which choice is the most relevant becomes more difficult. What is needed is a deeper understanding of the causes and characterizations of data overload such that technology can be designed to aid human analysts and decision makers, mitigating the negative effects associated with data overload.

The Transformation Process

A survey of the domains described previously, indicate that at an abstract level of information flow, each of the operation centers share a common operational process.  Through analysis of this shared process, we can detect and characterize areas where data overload occurs. This process, illustrated in figure 1-1, shows the common elements of a command, control, communication (C3) system that are responsible for the transformation of raw data into actionable human knowledge. The goal of this distributed cognition system is to analyze intelligence and enact decisions.

 

 

Figure 1-1: The transformation process from raw data to knowledge

 

The transformation process starts with the collection of raw data from the domain via an electronic sensor. The type of sensor varies across the domain and depends on the phenomena that are observed. A list of sensors and their usage can be found in the Handbook of Data Fusion (Hall, 2004).  Often more than one sensor type is used to collect data from the environment.  For example in maritime sensing, satellite imagery, radar, and sonar are used in conjunction to pinpoint detected objects.

The use of multiple sensors of multiple types necessitates the application of data fusion system. Data fusion systems process the multiple streams of raw data and transform it into one coherent stream of metadata. Mathematical models are used to extract significant features from the data, perform pattern recognition, and filter out extraneous data. The metadata, the assimilated collection of raw data generated from the data fusion system, is then presented visually in some form to a human user.

The human, in the loop between data fusion and decision support, is responsible for two main functions represented in the cloud bubbles in figure 1: sense making and decision making.  Sense making, is the act of making ‘sense’ of the metadata and recognizing the situation represents the domain (Weick, 1995). The responsibility of the human in the loop while performing sense making is to detect anomalies and decide if it constitutes a problem.

Once a problem is detected, the human in the loop begins the process of decision making. At this point, data is semantically referred to as information.  Information is data with a context; the context in this case is dependent on the decision of the human in the loop. With information of the problem in hand, the human in the loop is responsible for generating potential solutions to the problem, evaluating their potential effectiveness, and arriving at a conclusion.

The responsibility of the human in the loop does not necessarily end there. In these domains, the human in the loop is often expected to enact their decision through a decision support system or a system for Computer Supported Collaborative Work (CSCW).  The purpose of the CSCW system is twofold: First, to aid the user in evaluating potential courses of action, and second to record their actions for purposes of accountability and dissemination.  At this point of the transformation process, information becomes knowledge, as it provides experiential value to other users and systems.

Finally, the transformation process serves its purpose and produces an action – an action that was generated from the automated collection and aggregation of data by a machine, and the cognitive sense making and decision making process of a human. It is important to note that actions generated as an outcome of this system directly impact the same domain that was observed in the first place.  Thus the impact of the actions, of the decisions generated by the human in the loop, can be verified, validated, and measured by the same human sense maker.

Challenges

The transformation process and the complex distributed cognition systems that are built around it comprise many different fields of academic research. The fields identified within the transformation process are:

  1. Multi Sensor Data Fusion
  2. Cognitive Systems Engineering
  3. Human Computer Interaction
  4. Decision Making
  5. Computer Supported Collaborative Work

 

In each of these scientific fields, the concept of data overload has been identified as a contemporary research problem. Papers were selected and grouped in each field based on the characterization of the problem of data overload in a unique way. An interdisciplinary approach was taken to survey the many disparate areas of research on the problem of data overload.  In other words, a study of the problem of data overload, its causes and characterizations, is thus an analysis conducted within each of the scientific fields contained by the transformation process.

Over the next few chapters this thesis will:

  • Review the literature for each scientific discipline contained in the transformation process and identify the characterization of the problem of data overload.
  • Create a taxonomy of data overload types as informed by the literature review
  • Describe the use of the NeoCITIES simulated task environment as a platform for conducting research into data overload and command centers.
  • Review the design of an experiment based on the NeoCITIES task meant to recreate data overload conditions and explore the impact of new team user interface elements.
  • Summarize the results of the controlled laboratory experiment in which 28 teams, 84 participants, performed the NeoCITIES task.
  • Interpret and discuss the results, applying the analysis to the respective fields identified in the transformation process

AN INVESTIGATION OF DATA OVERLOAD IN TEAM-BASED DISTRIBUTED COGNITION SYSTEMS

 

 

Sharing is caring!

Leave a Reply