HYBRID HUMAN-COMPUTING DISTRIBUTED SENSE-MAKING: EXTENDING THE SOA PARADIGM FOR DYNAMIC ADJUDICATION AND OPTIMIZATION OF HUMAN AND COMPUTER ROLES

HYBRID HUMAN-COMPUTING DISTRIBUTED SENSE-MAKING: EXTENDING THE SOA PARADIGM FOR DYNAMIC ADJUDICATION AND OPTIMIZATION OF HUMAN AND COMPUTER ROLES

ABSTRACT

In many evolving systems, inputs can be derived from both human observations and physical sensors.  Additionally, many computation and analysis tasks can be performed by either human beings or artificial intelligence (AI) applications.  For example, weather prediction, emergency event response, assistive technology for various human sensory and cognitive impairments, individual and community medical systems, energy efficient buildings/processes, and a host of other complex management and sense-making applications have the potential to be implemented as hybrid human/computing systems in which: (1) observational data can be provided by either physical sensors or humans acting as observers (or a combination of such input), and (2) sense-making can be performed by either automated inference algorithms (computer automated reasoning/pattern recognition) or by human cognition (or both).  This category of hybrid system (referred to as “hard and soft information fusion”) has wide-ranging promise for analysis of both physical data and abstract concepts.  However, there are many challenges related to the effective storage, representation, and transmission of the vastly heterogeneous data necessary for scalable, loosely-coupled service-based communication between physical sensors, human observers, AI-based machine cognition tools, and human analysts.  Additionally, there is currently a lack of techniques for adjudicating which tasks should be assigned to humans and which should be assigned to machine/ computer systems.

This research explores the current state of the art in distributed hard and soft information fusion and seeks to address the above-mentioned gaps and challenges through a novel integration of paradigms and techniques such as service oriented architecture (SOA), multi-agent software systems (MAS), complex event processing (CEP), sonification (auditory display), message oriented middleware (MOM), and community standard data representation.  Additionally, it provides a prototype system implementation and a simulation experiment to evaluate the efficacy of the proposed techniques.

 

Chapter 1

Introduction

Information is currently undergoing a paradigm shift that is radically changing how it is sensed, transmitted, processed, and utilized.  One of the primary driving forces in this shift is the transformation of mobile device usage.  The new mobile device user is a an amazingly capable hybrid system of human senses, cognitive powers, and physical capabilities along with a suite of powerful physical sensors including high definition (HD) video/still camera, global positioning satellite (GPS) positioning, and multi-axis accelerometers.  Additionally, these devices can be linked to the “hive mind” (Kelly 1996) of various social networks and the distributed power of ever-growing open-source information provided by nearly countless applications that have the potential to funnel specific geospatial and temporally appropriate information to a user on a highresolution display or through high-fidelity audio.  The potential information gathering and processing power of massive social networks connecting these human/machine hybrids that we call “mobile device users” is unprecedented.  However, advances in architecture and infrastructure are required to fully recognize this potential.

In (O’Reilly and Battelle 2009), Tim O’Reilly discusses the potential for “collective intelligence” that exists in the World Wide Web, but states that the current Web is somewhat like a newborn baby – having the basic facilities necessary to grow into an intelligent and conscious entity yet still “awash in sensations, few of which she understands.”  This analogy can also be applied to human-centric information fusion, which in many ways is a part of the secondgeneration “Web 2.0” that O’Reilly discusses.

To address these needs, the information fusion community has recently been shifting its emphasis toward network-centric and human-centric operations (see (Keisler 2008), (Fan et al. 2010), (Castanedo et al. 2008), (Kipp 2006), (Hall and Jordan 2010)).  Distributed human-centric information fusion is proving indispensible in a broad variety of civilian and military applications.  Team-based tasks that were formerly limited by geographic distance, siloed information, and inability to share mental models present great opportunities for a hybridsensing/hybrid-cognition model.  There has been extensive research into “participatory sensing” (Burke et al. 2006) campaigns that facilitate decentralized collaboration for disaster response, environmental conservation efforts, and “citizen science” (Kamel Boulos et al. 2011).  Additionally, there is a long history of human vs. machine allocation of functions such as the popular “Fitt’s List” (Reason 1987) approach from the 1960s which attempted to quantify which tasks are best suited to humans and which to machines.  However, the potential for true collective intelligence (O’Reilly and Battelle 2009) remains largely unrealized.  Addressing this need requires a re-evaluation of the nature of both the network and the concept of services in order to accomplish the effective tasking and utilization of both technological resources and human analysts/participants.  The primary factors driving and facilitating these changes can be divided into the following three categories:

 

1. Emerging information technologies – Rapid changes in information technology in sensing, communications and computing enable the extension of human senses (at both the micro and macro scales), collaboration across large groups of people, and enhanced cognition and decision-making. Key emerging information technology trends include: a) an increasing proliferation and capability of mobile sensing, computing and communications devices (a.k.a. “smart phones”); b) software/infrastructure advances including Service Oriented Architecture (SOA), Message Oriented Middleware (MOM), Complex Event Processing (CEP), and multi-agent software systems; c) the proliferation of embedded smart sensors in ordinary and wearable devices (viz., the emerging “Internet of Things”).
2. Evolving cognitive and process models – New and evolving process and cognitive models provide improved guidelines for situational awareness, sense-making, and information fusion. Additionally, new models are emerging in the area of hard and soft information fusion. Finally, new models are emerging for distributed team cognition.

• Emerging social trends and human factors – Changes have occurred in the everyday relationships between humans and computing devices which include routine sharing of information and observations, collaboration with strangers, participating in global social networks, and engaging in interactions with an increasing number of digitally enhanced “smart” items that previously had no sensing or processing capability. Enabling social factors include: a) the emergence of the “net-generation” or “digital natives” who are increasingly familiar with and reliant upon mobile computing and communications, b) increased use and popularity of social media, c) an increasing propensity by the net generation to gather and share information, and d) the emergence of routine distributed collaboration (the “hive-mind” phenomena).

 

As illustrated in Figure 1, these factors lead to the opportunity for a new era in application areas such as citizen science; human-assisted monitoring of environmental/severe weather conditions; assistive technologies; linking of individual, local and global health concerns; and a new era of human participation in “green” technologies to reduce energy use.

 

Distributed, information-augmented humans have the capability to:

1. i) Observe local and global phenomena and report those observations (the concept of “participatory sensing”) – in this case the mobile device provides the enabling technology for a human observer to report and distribute his or her observations and interpretations;
2. ii) Use mobile devices to collect observations (via phone, video, etc.) – in this

case the human acts as a sort of “sensor platform”, directing and using the sensing capabilities of the mobile device to capture information about the environment;

iii)  Act in concert with an increasingly sophisticated mobile computer to perform

pattern recognition and context-based reasoning  – the human user “in and on the loop” for semi-automated situation assessment;

 

1. Collaborate with distributed colleagues to understand emerging phenomena.

 

 

1.1 Hybrid Human-Computing, Distributed Sense Making (HHCDSM)

 

This creates the opportunity for Hybrid Human-Computing, Distributed Sense-Making (HHCDSM) conceptually illustrated in Figure 1.    As sensors and computers become embedded in everyday devices with distributed communications, and as humans become more “connected” and routinely utilize their mobile devices for an increasing number of sensing and processing tasks, then each new human observation, pattern recognition, cognition and decision-making event becomes part of an surrounding environment containing large numbers of additional human-device hybrid teams.

 

Figure 1: Hybrid Human-Computing, Distributed Sense-Making

 

Despite the rapid technological and social trends, there are a number of gaps or challenges remaining to make this routine and effective.  Challenges and gaps include:

1. Lack of a Unifying Information Framework – Currently, a general information architecture or framework does not exist to represent the functions associated with hybrid human-computing, distributed sense-making.

 

1. Lack of a Robust Cyber-Infrastructure – While there are emerging standards and elements of SOA, these have arisen in different fields and applications and have not been established into a general cyber-infrastructure framework.

 

• Lack of Dynamic Human-Computing Adjudication – There is a lack of understanding of how to dynamically adjudicate between what a human can and should do (related to observations, collaboration and cognition) versus what a computer should do in these roles.

1.2 Research Contributions

 

This dissertation addresses the general topic of hybrid human-computing distributed sense-making (HHCDSM), and makes the following contributions:

1. i) Literature review – a literature search and analysis is conducted in the areas related to the emerging enabling technologies, evolving cognitive and process models, enabling social factors, and human cognitive and perceptual factors; and identifies several opportunities for the use of HHCDSM; ii) Information framework – an information framework is developed to identify the relevant functions and factors required for HHCDSM.   This framework generalizes recent information fusion models for hard and soft fusion and integrates advances in cognitive models such as the recognition-primed decision model.  Included in this model is the advent of a novel combination of Complex Event Processing and Multi-Agent Systems (CEP+MAS) to accomplish both “top down” and “bottom up” processing of data and facilitate human-in-the-loop interactions via sonification and other HCI techniques.  Section 1.3 briefly introduces the novel combination of CEP, MAS, and sonification in order to provide context for the literature review and offer a preview to more detailed discussion of this new paradigm in later chapters of the dissertation; iii) Cyber infrastructure – a cyber- framework and model is introduced which incorporates recent emerging standards related to sensors, data, SOA, and CEP+MAS; iv)          Prototype implementation – The StormSense prototype application is developed and demonstrated as an example of HHCDSM and CEP+MAS for human-assisted severe weather monitoring;
2. v) Simulation experiment – A simulation experiment is conducted to evaluate the efficacy of HHCDSM and CEP+MAS for fusing hard and soft data weather data for tornado threat assessment; vi)        Data analysis – The experimental results are analyzed and assessed, and finally vii)      Recommendations – Recommendations are provided for follow-on research.

 

1.3 Complex Event Processing with Multi-Agent Software (CEP+MAS) and Sonification

 

During the course of this research, many technologies, models, and paradigms were explored for their applicability to the task at hand.  This review of existing technology and literature (detailed in chapter 2) led to the discovery of three technologies that are particularly well-suited to be composed into a new framework for approaching the category of complex challenges discussed herein.  These three technologies are Complex Event Processing (CEP), Multi-Agent Software (MAS), and sonification.  Chapter 4 provides a thorough discussion of the strengths and weakness of each of these tools within the context of a HHCDSM system, and introduces a new paradigm for combining them.  In order to explore the potential of these technologies, and to focus the experiments described subsequently in this dissertation, we selected a specific application involving monitoring a dynamic weather-related event – namely assessing the potential of tornado activity during an extreme weather condition, using a combination of sensor data and human observations.  The details of this prototype and the numerical experiment that it supports are provided in chapters 4 and 5 respectively.

[1] Excerpts adapted from J. Rimland, “Service-Oriented Architecture for Human-Centric Information Fusion,” chapter 13 in Distributed Data Fusion for Network-Centric Operations, CRC Press, August, 2012, edited by D. Hall, J. Llinas, C. Chong and K. C. Chang

HYBRID HUMAN-COMPUTING DISTRIBUTED SENSE-MAKING: EXTENDING THE SOA PARADIGM FOR DYNAMIC ADJUDICATION AND OPTIMIZATION OF HUMAN AND COMPUTER ROLES

Sharing is caring!