AWARENESS AND KNOWLEDGE SHARING IN COLLABORATIVE COMPUTING: EXPERIMENTAL METHODS

  • : Ms Word, Ms Word Format
  • : 70 Pages
  • : ₦3,000 | $25 | ₵60 | Ksh 2720
  • : 1-5 Chapters
  •  
  • Click to DOWNLOAD Materials

AWARENESS AND KNOWLEDGE SHARING IN COLLABORATIVE COMPUTING: EXPERIMENTAL METHODS

ABSTRACT

Research methods in Computer-Supported Cooperative Work (CSCW) are mostly imported from other disciplines, primarily psychology, social sciences, and computer science. The imported methods are generally used with little or no grounding in explicit theoretical models about collaboration; the measures considered are very heterogeneous (ranging from performance of individuals to practices of organizations) and not explicitly mapped onto investigated concepts; also, the overall methodology of the specific studies tends to be unilaterally oriented either toward naturalistic field methods or controlled laboratory methods.  A number of CSCW researchers have pointed to some of these deficiencies, however, to date there has been no attempt to address these methodological deficiencies within an empirical program that incrementally investigates the same research construct.

This thesis addresses the problem of constructing appropriate research methods for studying awareness and knowledge sharing (common ground) in CSCW. It presents a research program: a sequence of studies on awareness and knowledge sharing intended as an instantiation of a new methodological approach.  The approach has three characteristics: (1) Model-based: provides a mapping between conceptual models and methods; (2) Centered on group-level processes: the group is the unit of analysis and specific group processes are the focus of investigation; (3) Comprehensive in measurement: field and laboratory results are integrated and multiple measures of the same constructs are used.

The first half of the research program focused on activity awareness in CSCW.  Drawing on the findings of a prior field study, a laboratory method was developed.  A first lab study was devoted to validate the laboratory method and a second lab study provided detailed measurements of activity awareness.  This study measured different aspects of the activity awareness construct, examined its relationship with known variables, and compared the effects of two CSCW systems, BRIDGE and Groove.  The findings confirmed that many events tend to remain unnoticed in current systems. Key classes of factors affecting activity awareness included the properties of the event itself (e.g. distribution in time), the properties of the workspace (e.g. integration of content across tools, flexibility in navigation among the tools), the properties of the coworkers (e.g. metacognition, teamwork attitudes), and the properties of the group over time (e.g. amount of shared experience, increasing over time).

The second half of the program focused on common ground, a sub-process of activity awareness. Two laboratory expeirments investigated the knowledge sharing process, respectively, in collocated teams using a paper prototype and in distributed teams using a software prototype.  Subjective and objective measures in both studies showed that the amount of common ground increased as the shared experience increased (repeated task runs).  The dialog patterns of the teams were also analyzed to understand the ways in which the increment in common ground occurred.  While working together, the teammates developed not only shared knowledge about the content but also about the process and team strategies (i.e. how to do the task).  As such process common ground was established, the teammates needed fewer explicit acts to regulate the process.  As a result, the efforts were turned to building ‘content common ground,’ which led to greater efficiency. By comparing the results from the two experiments, specific effects of medium and setting were also identified.

After presenting the results of the studies, the thesis discusses the proposed approach and specific experimental techniques developed and used in the program.  Finally, the thesis draws  some implications for future investigation and support of activity awareness and common ground.

 

TABLE OF CONTENTS

LIST OF FIGURES                         viii

LIST OF TABLES                            xi

ACKNOWLEDGEMENTS                   xiii

CHAPTER 1………………………………………………………………………………………………………………………………………………… 1

QUESTION ………………………………………………………………………………………………………………………………………………… 1

PROBLEM ………………………………………………………………………………………………………………………………………………….. 1

PROPOSED APPROACH & RESEARCH PROGRAM ………………………………………………………………………………..2

SUMMARY OF FINDINGS…………………………………………………………………………………………………………………………… 3

THESIS STRUCTURE……………………………………………………………………………………………………………………………………8

EMBEDDED PUBLICATIONS……………………………………………………………………………………………………………………… 9

THE AUDIENCE ………………………………………………………………………………………………………………………………………….10

WHY RESEARCH METHODS MATTER …………………………………………………………………………………………………..11

CHAPTER 2………………………………………………………………………………………………………………………………………………….13

BACKGROUND……………………………………………………………………………………………………………………………………………13

RESEARCH METHODS IN HCI AND HF…………………………………………………………………………………………………………..13

RESEARCH METHODS IN CSCW……………………………………………………………………………………………………………………14

Prior analyses of CSCW methods…………………………………………………………………………………………………………….15

A meta-analysis of ACM CSCW 1986-2006 papers…………………………………………………………………………………..18

Methods in ACM CSCW 1986 & 1988………………………………………………………………………………………………………………… 20

Methods in ACM CSCW 2006…………………………………………………………………………………………………………………………….. 22

Participants and Setting…………………………………………………………………………………………………………………………… 23

Degree of Manipulation or Control in the Studies……………………………………………………………………………………….. 24

Data Collection and Analysis Techniques …………………………………………………………………………………………………….. 24

Goal of the study………………………………………………………………………………………………………………………………………. 25

Findings from the Meta-Analysis …………………………………………………………………………………………………………….26

General Deficiencies of CSCW Methodology……………………………………………………………………………………………27

METHODS FOR STUDYING AWARENESS AND KNOWLEDGE SHARING ……………………………………………..28

Prior Methods for Studying Awareness ……………………………………………………………………………………………………29

Measuring activity Awareness and its Sub-Processes ……………………………………………………………………………….30

Long-term ………………………………………………………………………………………………………………………………………………………….. 31

Distributed …………………………………………………………………………………………………………………………………………………………. 31

Systemic: emergent…………………………………………………………………………………………………………………………………………….. 31

Common Ground ……………………………………………………………………………………………………………………………………32

Communities of Practice…………………………………………………………………………………………………………………………33

Social Capital ………………………………………………………………………………………………………………………………………..34

Human Development………………………………………………………………………………………………………………………………35

CHAPTER 3………………………………………………………………………………………………………………………………………………….37

EMPIRICAL RESEARCH ON ACTIVITY AWARENESS …………………………………………………………………………37

DEFINING ACTIVITY AWARENESS …………………………………………………………………………………………………………………37

MEASURING ACTIVITY AWARENESS……………………………………………………………………………………………………………..39

The Research Program……………………………………………………………………………………………………………………………39

The field study………………………………………………………………………………………………………………………………………..40

First Lab Study……………………………………………………………………………………………………………………………………….43

Rationale and Research Questions ……………………………………………………………………………………………………………………….. 43

Participants and laboratory setting……………………………………………………………………………………………………………………….. 44

Tool and tasks…………………………………………………………………………………………………………………………………………………….. 44

Experimental procedure………………………………………………………………………………………………………………………………………. 46

Data collection and analysis ………………………………………………………………………………………………………………………………… 48

First Lab Study Results……………………………………………………………………………………………………………………………………….. 49

Second Lab Study……………………………………………………………………………………………………………………………………51

Research Questions…………………………………………………………………………………………………………………………………………….. 51

Conceptual Model of Group Process ……………………………………………………………………………………………………………………. 52

Experimental Design and Measures……………………………………………………………………………………………………………………… 52

Group Task …………………………………………………………………………………………………………………….. 54

Participants and Setting……………………………………………………………………………………………………………………………………….. 55

Collaborative Systems…………………………………………………………………………………………………………………………………………. 55

Scenarios……………………………………………………………………………………………………………………………………………….58

Preparation, Procedure and Analysis ……………………………………………………………………………………………………………………. 59

Results…………………………………………………………………………………………………………………………………………. 61

Measured AA ……………………………………………………………………………………………………………………………… 61

Controlling Person Variables: questionnaire measures……………………………………………………………………………… 63

Measured effects of System, Session and Scenarios on AA………………………………………………………………………………. 64

Criterion………………………………………………………………………………………………………………………………………………..68

Table 3-10: Testing Global Null Hypothesis: BETA=0 …………………………………………………………………………………………. 68

Table 3-11: Type 3 Analysis of Effects………………………………………………………………………………………………………………… 68

Measured Consequences of AA………………………………………………………………………………………………………………………. 70

Measured work performance…………………………………………………………………………………………………………………………… 71

Overall Discussion …………………………………………………………………………………………………………………………………73

First Lab Study: Method validity and measurement………………………………………………………………………………………………. 74

First Lab Study: The Activity Awareness Phenomenon………………………………………………………………………………. 75

Second Lab Study: Discussion…………………………………………………………………………………………………………………………….. 76

Measured AA …………………………………………………………………………………………………………………………………………. 76

Session, System and Scenario Type effects……………………………………………………………………………………………………… 77

Person variables effects……………………………………………………………………………………………………………………………… 78

Consequences and correlates of AA………………………………………………………………………………………………………………… 79

Summary of Results on AA ………………………………………………………………………………………………………………………… 80

CHAPTER 4………………………………………………………………………………………………………………………………………………….82

EMPIRICAL RESEARCH ON COMMON GROUND…………………………………………………………………………………82

DEFINING COMMON GROUND……………………………………………………………………………………………………………………….82

Two Theoretical Sources…………………………………………………………………………………………………………………………82

First Source: The Group Process Model……………………………………………………………………………………………………………….. 82

Second Source: The Activity System model……………………………………………………………………………………………… 84

The Integrated Model……………………………………………………………………………………………………………………………..85

The conceptual model…………………………………………………………………………………………………………………………………………. 86

MEASURING COMMON GROUND …………………………………………………………………………………………………………………..88

Research Program………………………………………………………………………………………………………………………………….88

Empirical model: reference task, manipulations, and measures………………………………………………………………………………. 90

Research Questions ……………………………………………………………………………………………………………………………93

PAPER PROTOTYPE STUDY……………………………………………………………………………………………………………………………94

Method…………………………………………………………………………………………………………………………………………………..94

Study Design………………………………………………………………………………………………………………………………………………………. 94

Participants……………………………………………………………………………………………………………………………………………. 94

Collaborative Task and Roles………………………………………………………………………………………………………………………………. 95

Procedures and Data Collection …………………………………………………………………………………………………………………………… 96

Analysis of Questionnaire Data……………………………………………………………………………………………………………………………. 97

Statistical Analysis of effects………………………………………………………………………………………………………………… 98

Results …………………………………………………………………………………………………………………………………………………..98 Evidence of Increase in Common Ground…………………………………………………………………………………………………………….. 98

Task Performance………………………………………………………………………………………………………………………………………. 98

Questionnaire Results: Perceptions of Group Process and Performance………………………………………………………….. 99

Communication Structure……………………………………………………………………………………………………………………………. 99

Recall of Task Information ……………………………………………………………………………………………………………………………101

Results on Dialog Patterns………………………………………………………………………………………………………………………………….102

Checks for Understanding……………………………………………………………………………………………………………………………..102

Transfer of Information…………………………………………………………………………………………………………………………………104

Management of Process…………………………………………………………………………………………………………………………………105

Discussion on Dialog Patterns…………………………………………………………………………………………………………………………….105

Checks for Understanding……………………………………………………………………………………………………………………………..106

Transfer of Information…………………………………………………………………………………………………………………………………107

Management of Process…………………………………………………………………………………………………………………………………107

SOFTWARE PROTOTYPE STUDY …………………………………………………………………………………………………………………..110

Method…………………………………………………………………………………………………………………………………………………110

Study Design…………………………………………………………………………………………………………………………………………………….. 110

Participants……………………………………………………………………………………………………………………………………………..110

Collaborative Task and Roles……………………………………………………………………………………………………………………………..110

Procedures………………………………………………………………………………………………………………………………………………..111

FIGURE 4-12: COLLABORATION IN A DISTRIBUTED, COMPUTER-MEDIATED SETTING. ………………………………………111

Software Prototype…………………………………………………………………………………………………………………………………………….112

Results …………………………………………………………………………………………………………………………………………………113

Task Performance…………………………………………………………………………………………………………………………………………113

Questionnaire Results: Perceptions of Group Process and Performance…………………………………………………………… 114

Communication Structure………………………………………………………………………………………………………………………………115

Recall of Task Information ……………………………………………………………………………………………………………………………116

Results on Dialog Patterns………………………………………………………………………………………………………………………………….117

Transfer of Information…………………………………………………………………………………………………………………………………118

Agreement and Judgment Acts……………………………………………………………………………………………………………….. 119

COMPARING RESULTS: PAPER VS. SOFTWARE STUDIES…………………………………………………………………………………120

Comparing Questionnaire Results between Paper and Software studies…………………………………………………..120

Figure 4-17: Ease of Reference and Planning………………………………………………………………………………………………………122

Figure 4-18: Satisfaction by Run and Medium/Setting ………………………………………………………………………………… 122

Figure 4-20: Satisfaction by Run and Pre-Briefing……………………………………………………………………………………………….123

Comparing Member Recall between Paper and Software studies …………………………………………………………….123

Member Post-Task Recall of cons discussed by the team……………………………………………………………………………….. 123

Member Post-Task Recall of shelter choices by the team……………………………………………………………………………….. 124

Comparing Dialog Patterns between Paper and Software Studies……………………………………………………………126

Comparing Medium and Setting effects………………………………………………………………………………………………….128

CHAPTER 5………………………………………………………………………………………………………………………………………………..132

IMPLICATIONS FOR THE FIELD……………………………………………………………………………………………………………132

METHODOLOGICAL IMPLICATIONS………………………………………………………………………………………………………………132

A CSCW-native Approach to Methods……………………………………………………………………………………………………132

Model-based approach………………………………………………………………………………………………………………………….133

Centered on group process ……………………………………………………………………………………………………………………135

A comprehensive set of methods…………………………………………………………………………………………………………….137

Methodological Lessons From Empirical Studies……………………………………………………………………………………139

Measurement techniques…………………………………………………………………………………………………………………………………….140

Toward Valid and Informative measures………………………………………………………………………………………………………..140

Process plus Outcome measures……………………………………………………………………………………………………………….. 141

An embedded measure of efficiency ………………………………………………………………………………………………………………141

Manipulation techniques…………………………………………………………………………………………………………………………………….142

Control techniques…………………………………………………………………………………………………………………………………………….. 143

Why a research program? …………………………………………………………………………………………………………………………………..143

THEORETICAL IMPLICATIONS………………………………………………………………………………………………………………………145

Activity Awareness………………………………………………………………………………………………………………………………..145

Common Ground ………………………………………………………………………………………………………………………………….147 DESIGN IMPLICATIONS AND FUTURE WORK…………………………………………………………………………………………………149

Design Implications………………………………………………………………………………………………………………………………149

Supporting Activity Awareness…………………………………………………………………………………………………………………………..149

Supporting Common Ground in Geo-Collaboration……………………………………………………………………………………………..151

Studying Common Ground in CSCW ……………………………………………………………………………………………………..153

CONCLUSION …………………………………………………………………………………………………………………………………………….154

BIBLIOGRAPHY………………………………………………………………………………………………………………………………………..156

APPENDIX A: ACTIVITY AWARENESS QUESTIONNAIRE…………………………………………………………………168

APPENDIX B………………………………………………………………………………………………………………………………………………170

APPENDIX C: ACM CSCW 2004 DOCTORAL CONSORTIUM POSTER………………………………………………171

APPENDIX D: MEASURES OF COMMUNICATION STRUCTURE: TURNS AND SIMULTANEOUS

SPEECH MEASURES IN BOTH STUDIES……………………………………………………………………………………………….172

APPENDIX E: COMPARING QUESTIONNAIRE RESPONSES ……………………………………………………………..173

APPENDIX F: DESCRIPTIVE RESULTS. AVERAGE RATINGS BY MEDIUM/SETTING, PRE-

BRIEFING, AND RUN………………………………………………………………………………………………………………………………..174

 

LIST OF FIGURES

Figure 2-1: Level of analysis (%) of contribution by CSCW conference venue……………18

Figure 2-2: Type of contribution (%) by CSCW conference venue. …………………………….19

Figure 2-3: Function of contribution (%) by CSCW conference venue. ………………………19

Figure 2-4: Paper Contribution……………………………………………………………………………………20

Figure 2-5: Variable Manipulation………………………………………………………………………………21

Figure 2-6: Paper Contribution……………………………………………………………………………………22

Figure 2-7: Data collection techniques. ………………………………………………………………………..24

Figure 2-8: Data analysis techniques……………………………………………………………………………25

Figure 3-1: Groove Workspace…………………………………………………………………………………….44

Figure 3-2: Group Process Model ………………………………………………………………………………..53

Figure 3-3: Laboratory Set-up……………………………………………………………………………………..55

Figure 3-4: Groove Workspace…………………………………………………………………………………….57

Figure 3-5: BRIDGE Workspace. ………………………………………………………………………………..57

Figure 3-6: Scenarios distributed by collaborative Session and Breakdown Factor…….60

Figure 3-7: AA Process Factor……………………………………………………………………………………..65

Figure 3-8: AA Process by Meta-cognition level…………………………………………………………..66

Figure 3-9: AA Process by Meta-cognition (Low vs. High) and Session (1-4) ………………67

Figure 3-10: Awareness responses (%) to all scenarios introduced by System (BRIDGE vs. Groove), Type of Scenario (Multi-session vs. Single-session), System BY Type of

Scenario………………………………………………………………………………………………………………..67

Figure 3-11: Number of task switches (transitions to or from each task) by Session and

System…………………………………………………………………………………………………………………..72

Figure 3-12: Quiz task………………………………………………………………………………………………….73

Figure 4-1: Group Process Model ………………………………………………………………………………..84

Figure 4-2: Activity System Model……………………………………………………………………………….86

Figure 4-3: Integrated Conceptual Model ……………………………………………………………………87

Figure 4-4: Empirical Model………………………………………………………………………………………..92

Figure 4-5: Collaboration on maps around a tabletop………………………………………………….96

Figure 4-6: Clarify, Check & Align Dialog Acts…………………………………………………………..104

Figure 4-7: Add Info vs. Reply……………………………………………………………………………………..105

Figure 4-8: Checking Acts by Run and Performance Level …………………………………………106

Figure 4-9: Add Info and Reply by Run and Performance Level…………………………………107

Figure 4-10: Management by Run and Performance …………………………………………………..108

Figure 4-11: Management Sub-Types by Run ……………………………………………………………..109

Figure 4-12: Collaboration in a distributed, computer-mediated setting……………………..111

Figure 4-13: Software Prototype User Interface…………………………………………………………..112

Figure 4-14: Add Info vs. Reply……………………………………………………………………………………118

Figure 4-15: Add Info vs. Reply by Performance Level……………………………………………….119

Figure 4-16: Judgment Acts by Performance Level……………………………………………………..120

Figure 4-17: Ease of Reference and Planning ………………………………………………………………121

Figure 4-18: Satisfaction by Run and Medium/Setting ………………………………………………..122

Figure 4-19: Amount of Shared Knowledge by Run and Medium/Setting …………………..122

Figure 4-20: Satisfaction by Run and Pre-Briefing………………………………………………………123

Figure 4-21: Member recall of the first shelter choice between Paper (left) and Software

(right) studies ……………………………………………………………………………………………………….125

Figure 4-22: Member recall of the second shelter choice between Paper (left) and Software (right) studies ……………………………………………………………………………………………………….125

Figure 4-23: Member recall of the third shelter choice between Paper (left) and Software

(right) studies ……………………………………………………………………………………………………….125

Figure 4-24: Push vs. Pull in Paper Study……………………………………………………………………..127

Figure 4-25: Push vs. Pull in SW Study…………………………………………………………………………127

Figure 4-26: Check Understanding in Paper Study ………………………………………………………127

Figure 4-27: Check Understanding in SW Study…………………………………………………………..127

Figure 4-28: Manage Acts in paper and SW studies…………………………………………………….128

Figure 4-29: Management Acts for Paper Study by Performance…………………………………128

Figure 4-30: Management Acts for Software Study Low and High Performers ……………128

Figure 4-31: Agreement and Judgment Acts for Paper Study ……………………………………..129

Figure 4-32: Agreement and Judgment Acts for SW Study …………………………………………129

Figure 4-33: Judgment Acts for Paper Study……………………………………………………………….130

Figure 4-34: Judgment Acts for SW Study…………………………………………………………………..130

Figure 4-35: Check, Align, Clarify, and Acknowledge Acts for Paper Study by Performance

Level……………………………………………………………………………………………………………………..130

Figure 4-36: Check, Align, Clarify, and Acknowledge Acts for SW Study by Performance Level……………………………………………………………………………………………………………………..130

Figure 5-1: Multi-method research program……………………………………………………………….138

Figure 5-2: BRIDGE Workspace …………………………………………………………………………………149

Figure 5-3: Annotation tool………………………………………………………………………………………….150

Figure 5-4: Spatial location of an annotation on the map…………………………………………….150

Figure 5-5: Timeline-by-role to show annotations, their times, and authors………………..151

Figure 5-6: Tagging an annotation……………………………………………………………………………….151

Figure 5-7: Bar chart views………………………………………………………………………………………….151

LIST OF TABLES

Table 1-1: Contribution ……………………………………………………………………………………………….3

Table 2-1: Characteristics of AA and implications for research methods…………………….30

Table 2-2: The four sub-processes of activity awareness and method implications………32

Table 3-1: Confederate’s Script for a Scenario…………………………………………………………….45

Table 3-2: Scripted Scenarios used in the First Lab Study…………………………………………..45

Table 3-3: Collaborative sessions and scenarios schedule…………………………………………….47

Table 3-4: Mapping of features between Groove and BRIDGE …………………………………..56

Table 3-5: Scenarios: Types, Factors and Manipulations introduced…………………………..58

Table 3-6: Confederate’s scenario scripts: single-session (SS) and multi-session (MS) ..59

Table 3-7: Sessions, Work Performed and Scenarios…………………………………………………..61

Table 3-8: Reliability by Cluster and Session: Alpha values………………………………………..62

Table 3-9: Model Fit Statistics……………………………………………………………………………………..68

Table 3-10: Testing Global Null Hypothesis: BETA=0…………………………………………………68

Table 3-11: Type 3 Analysis of Effects …………………………………………………………………………68

Table 3-12: Baseline Logistic Regression Results…………………………………………………………69

Table 4-1: Examples of manipulations and measures by model component…………………91

Table 4-2: Summary of group performance…………………………………………………………………97

Table 4-3: Reliability by Cluster and Session: Alpha Values ……………………………………….100

Table 4-4: Ratios of Information Retention………………………………………………………………….101

Table 4-5: Dialog Act Codes…………………………………………………………………………………………103

Table 4-6: Management Act Sub-Types……………………………………………………………………….108

Table 4-7: Average completion times and percentages of optimal plans (performance by

Run and Study………………………………………………………………………………………………………113 Table 4-8: Team Performance Measures……………………………………………………………………..113 Table 4-9: Reliability by Cluster and Session: Alpha values………………………………………..114

Table 4-10: Retention (%) of cons regarding the Chosen Shelter ………………………………..116

Table 4-11: Descriptive and t-test recall of Cons in Run 1 and Run 3………………………….117

Table 4-12: Dialog Act Codes……………………………………………………………………………………….117

Table 4-13: Testing Main Effects …………………………………………………………………………………121

Table 4-14: Retention of cons regarding the Chosen Shelter in the Software Study…….124

Table 4-15: Retention of cons regarding the Chosen Shelter in the Paper Study …………124

Table 4-16: Member Recall of first, second, and third shelter team choices in the Paper (left) and the Software (right) studies. Each teams chose among four shelters ……………..124

Table 4-17: Dialog Act Codes in Paper and Software Studies………………………………………126

 

 

 

Chapter 1

 

Question

The general question to which this dissertation aims to respond is how can we investigate complex CSCW phenomena such as activity awareness and common ground in ways that (a) capture these factors realistically, at the right level, and accurately (i.e. validity, focus, reliability); (b) allow leveraging and then improving existing theory of collaboration (i.e. accumulation of results); and, more generally, (c) address concerns that are specific to CSCW researchers and not to psychologists or computer scientists.  The present study will response to this question by specifying the problem, proposing an approach, and presenting an empirical program for studying activity awareness and common ground in CSCW.

 

Problem

This thesis addresses the problem of developing experimental research methods in ComputerSupported Cooperative Work for studying awareness and knowledge sharing. 

CSCW literature suffers from an imbalance in the kinds of contributions. While many efforts have been made to contribute tools for supporting awareness and knowledge sharing, we still lack clear conceptual models for explaining and appropriate research methods for investigating these factors.

Shmidt (2002) has already noted this deficiency in awareness research:

 

A significant effort was devoted to exploring the potential benefits of “media space” technologies […] Unfortunately, however, the expected benefits from these technologies never materialized.  Something was obviously amiss in the understanding of “awareness” underlying this line of research (p. 285).

 

To begin with, more specifically, research methods have been imported without adaptation although the object of study and the end goal of researchers have changed.  The selection of research methods in CSCW appears driven by “surprisingly orthodox strategies” (Randall et al. 1996, 16): research methods are mostly imported from other disciplines, primarily psychology, social sciences, and computer science.

Secondly, the imported research methods are rarely related to explicit theoretical models about collaboration and tend to be ad hoc (e.g., Inkpen et al. 2004). The measures selected are very etherogeneous (from performance of individuals to practices of organizations), not explicitly mapped onto concepts, and not clearly motivated.

Thirdly, the overall methodology of each research project tends to be unilaterally oriented either toward naturalistic field methods or controlled laboratory methods.

Finally, CSCW researchers have recognized that the evaluation of CSCW systems is difficult (Grudin 1988) and that it needs to occur in a realistic context (Twidale et al. 1994, Prinz 1998). Prior reviews of the methods for evaluating systems or investigating work suggested that the procedures adopted in field studies are often unsystematic and based on informal observations; or, when systematic and occurring in the lab, the conditions observed are often too simplistic (e.g. Twidale et al. 1994; Pinelle & Gutwin 2000, Randall et al. 1996).

The consequences of the lack of systematic (or rigorous but not valid) methods make it difficult to compare and accumulate results across studies.  In the long term, this leads to unsuccessful systems because designers may fail to consider the most relevant factors that will affect the success of a given system.  The approach presented in this thesis claims that fieldwork is a necessary but not sufficient research strategy in CSCW.  Experimental methods can be developed and used in conjunction with field methods.

A number of authors have pointed to some of the deficiencies listed above (e.g. Carroll et al. 1992, Carroll 2000, Neale et al. 2004, McNeese 1996, Steves and Scholtz 2005, Damianos et al. 1999), but to date there has been no attempt to address these methodological deficiencies within an empirical program that incrementally investigates the same research construct.  This thesis addresses those deficiencies by developing and implementing such an empirical program.

 

Proposed Approach & Research Program

The approach proposed for solving the problem has three features:

 

Model-based. The epistemology of the research is model based (or theory based): a conceptual model is used as a guide to plan the measurement and interpretation of results

Centered on group process. The level of analysis of the research is the group and the focus of investigation is on global variables characterizing group process in CSCW such as awareness and knowledge sharing.  The emphasis is placed on the process mediated by technology rather than on the resulting mental representations or levels (states) of performance.

Comprehensive in methods. The research uses a comprehensive measurement approach: results from field investigations are integrated with results from laboratory investigation.  Moreover, different types of data are collected to gain information about the same central concept (e.g.

awareness or common ground).

 

The thesis presents an instantiation of this approach through a research program, or sequence of studies, on awareness and knowledge sharing. The program structure and the specific contributions of each study are outlined in the table below.

 

 

 

Timeline Field Studies Lab Studies (main contribution)
Activity

Awareness

 

2003-04 Field study: school groups (*)

Analyzed data

Carroll et al. 2003; Neale et al. 2004

 

First study

Validated AA Method

Convertino et al. NordiCHI 04

2005-06  

 

Second study

Measured AA

Convertino et al. Submitted

Common Ground

 

2006-07 Field study: EM teams (*)

Modeled Task & FEMA roles

Schafer, Ganoe, and Carroll 2007

 

Paper study

Measured CG, FtF

Convertino et al. Group 07, CHI 08

2008  

 

SW study

Measure CG, Dis

Convertino et al. ISCRAM 08

Table 1-1: Contribution. The thesis reports on a research program. It focuses on two pairs of lab studies (rightmost column), both grounded in related field studies conducted by collaborators (central column), and pertaining to Activity Awareness and Common Ground, respectively (leftmost column). Specific contributions by the author in each study are underlined in each cell.

 

Summary of Findings

The first half of the research program focused on activity awareness in CSCW.  In implementing the proposed approach, the investigation was (1) oriented by a conceptual model of activity awareness, (2) centered on long-term group activity – where activity awareness consists of collaborators’ understanding and management of interdependencies among tools, tasks, people, and situations in a group project that includes synchronous and asynchronous work – and (3) comprehensive in methods by integrating findings from a prior field study in a school setting with two laboratory studies and by using, within each study, multiple measures for the same construct.

The two-year field study with distributed groups in a school setting revealed the need for a novel and broader perspective on awareness in order to properly account for the articulation work conducted by collaborators in distributed long-term projects (Schmidt & Bannon 1992).  Systematic observations of the articulation work performed by groups over several months led to the identification of common breakdown factors that affect these work conditions (Carroll et al. 2003).  The breakdowns were a negative indicator of collaborators’ activity awareness and, at the same time, an indicator of the extra coordination costs required in distributed and long-term work.

The first laboratory modeled and measured for the first time in a controlled setting the collaborators’ awareness for these breakdowns factors in conditions that were representative of those observed in field. Distributed pairs of collaborators performed a multi-session collaborative editing task. Several aspects of the work observed supported the validity of the lab method: the pairs of collaborators were clearly engaged in the work, performed the project with autonomous initiative, conducted lively discussions, solved problems collaboratively and creatively, and more importantly exhibited awareness difficulties that were representative of those observed in the field. In more than half the cases of the experimental manipulations, the participants did not become aware of the disruptive events (breakdowns) that were introduced (for example, by misplacing useful content in the wrong tool within the workspace), even after receiving a systematic prompt about the breakdown.  In general, the participants tended to lack project-level awareness of the plan, the actual work time available, and the current status of the activity.

While the first lab study supported the validity of the method and provided an initial systematic measurement of awareness, the second lab study, using a broader and refined version of lab method, focused specifically on measuring activity awareness and related variables.  Pairs of collaborators used one of two CSCW systems (BRIDGE or Groove) and completed a collaborative project including four sessions over a period of about three weeks.  The subjective measures of awareness suggested that activity awareness develops as a process over time: the level began to increase after two sessions and then grew steadily as the amount of shared experience in each pair increased.  The perceived activity awareness predicted the perceived quality of outcomes and correlated with a measure of collaborator efficiency (assessed at the level of a small repeated task).

The CSCW system used, the temporality of disruptive events introduced (i.e., events that typically caused breakdowns in the field), and the meta-cognitive skills of the participants all influenced their level of awareness.  Behavioral measures of AA suggested that the participants using BRIDGE exhibited higher awareness, higher collaboration efficiency, and lower coordination costs than participants using Groove.  These differences were visible through the behavioral measures but not through the perceived measures of AA.  The products generated in BRIDGE also appeared more integrated and variable than those generated in Groove.  Regarding the temporality of events, those introduced across multiple sessions were noticed less often than noticeable events occurring within the same session.  Finally, high meta-cognition participants exhibited a higher level of awareness, which points to meta-cognition as a relevant variable in relation to awareness.

In order to help build CSCW systems that more adequately support activity awareness, researchers need to build and test explanatory models that allow for the clear identification of factors that affect the visibility of critical events that can cause disruptions and require reparatory actions.  The results from this program suggest that important classes of factors include the properties of the event itself (e.g. distribution in time, granularity, concreteness, relation to prior evens), the properties of the workspace (level of integration of content across tools, flexibility in navigation among the tools), the properties of the coworker (e.g. meta-cognition, teamwork attitudes), and the properties of the pair or group (amount of shared experience at each point in time – see the ‘Standing Group’ in McGrath 1984).

The second half of the program focused on common ground, a sub-process of activity awareness. Two laboratory studies were conducted to study the knowledge sharing process, respectively, in collocated teams using a paper prototype and in distributed teams using a software prototype. The investigation in this second half of the program was also (1) led by an explicit conceptual model, (2) centered on group process – grounding is operationalized and measured in teams that perform an emergency management planning task on shared maps – and (3) comprehensive in methods by using findings of a prior field study with real emergency management teams and, by using, within each lab study, multiple measures of common ground.  The two laboratory studies were conducted in comparable conditions with teams performing a geo-collaborative task using maps.

Both perceived and objective measures from both studies showed that the amount of common ground increased as the shared experience increased (significant effect of repeated runs).  The perceived and objective measures of performance also increased as shared experience and amount of common ground increased.  The collaborative medium and setting affected team performance: paper-based, collocated teams rated the quality of the process and outcome higher than the software-based, distributed teams (e.g. ease of referencing and overall satisfaction ratings).  However, the ratings of the perceived gain of shared knowledge did not differ but grew similarly and significantly in both conditions.

After establishing that common ground increased, we investigated the ways in which such increment occurred by systematically analyzing verbal interactions in both studies.  In both studies, the team increased their efficiency in transferring information over time by using less query-reply (pull) dialog acts and more direct information add (push) dialog acts, which require implicit understanding of when content is needed.  They also decreased the number of explicit moves devoted to managing the group process (fewer management acts), which suggests an increasing number of shared assumptions about ‘how to’ run the task over time – such increment in common ground about the led to more efficient work (see the improvement in the performance measures).

Software-based teams exhibited an enhanced effect of increasing push over pull dialog acts. Compared to the paper medium, the software tool made the process of sharing information more explicit, keeping users focused on the content.  Dialog patterns were also affected by medium and setting in that the teams who worked in a distributed setting (via the software medium) tended to increase rather than decrease the proportion of explicit agreement and judgment acts.  This is probably due to the lack of visual cues in the distributed condition: participants were consistently more vocal in confirming and discussing judgments in the distributed condition than the collocated or face-to-face condition.

A key general finding from the analysis of dialog patterns was that the proportion of acts devoted to checking for understanding of transferred information tended to increase or remain stable rather than decrease over time, as is typically expected and observed in experiments with simple tangram-type communication tasks.  This suggests that the conceptual models developed in research on communication do not properly explain the process of knowledge sharing within teamwork.  For example, in order to predict the costs of grounding in this new context, it is critical to factor in not only the properties of the medium but also the properties of the collaborative task, such as the task complexity (e.g., Straus 1999, Straus and McGrath 1994).

Overall the results on common ground suggest that as the members of a team keep working together they develop not only common knowledge about the content or facts but also about the process and the team strategies (i.e., how to best complete the task). As ‘process common ground’ is established, the teammates need fewer explicit acts to regulate their process because shared assumptions can be made. As a result, the efforts are gradually turned on building ‘content common ground’, which leads to greater efficiency, as observed in our studies. A general implication for CSCW design is that collaborative systems give us the unprecedented opportunity for catalyzing a faster development of process common ground through specific tools for managing of actions and judgments in teams. As mentioned for the support of activity awareness, I argue that empirically-grounded explanatory models of common ground formation in specific collaboration domains can inform the design of useful technological support. Another general implication for CSCW theory is that the development of common ground about shared content (i.e., relevant facts) and shared process (i.e., useful strategies) may follow different laws and depend on different parameters. This is a problem that future theoretical work on common ground in CSCW needs to address.

Different kinds of implications for research methods in CSCW can be drawn from the empirical work in this research program.  The program provided a concrete implementation of the proposed approach: the guidance of an explicit theoretical model (e.g. McGrath’s (1984) group process model, Endsley et al.’s (2000) model of situation awareness) helped to specify hypotheses, select measures, and make sense of results.  Given that awareness and common ground are collective processes, all the studies focused on measures of group process, but at the same time related these to performance measures.  For example, common ground was assessed using data about turn structure, dialog acts, member recall, and psychometric scales, and these findings were then related to the overall quality and timeliness of the team plans.  Finally, the empirical work in the lab was conducted on realistic tasks and conditions that directly modeled results from closely related field studies thereby ensuring proper scope and validity.  Then the laboratory experiment, as research strategy, added the ability to focus on and control specific variables of interest.

Specific implications can be drawn about the techniques manipulation, control, and measurement. In the laboratory experiments presented, novel techniques were developed and adopted for within-group manipulations (i.e. experimental confederate technique, repeated sessions or task runs) and betweengroup manipulations (i.e. comparing alternative systems or settings for collaborations, changing amount of initial shared knowledge via pre-briefing on roles).

As the phenomena investigated in CSCW experiments become more complex, the need for control of relevant factors increases.  In the laboratory experiments, variables pertaining to task, environment and collaborator (partner, gender) were held constant.  Specifically, various parameters of the collaborative task (i.e. structure and content) were kept constant by constructing specific reference tasks to study awareness or knowledge sharing (e.g. Carroll et al. 2007, Whittaker et al. 2000).  Other relevant variables were measured and treated as co-variates in the analysis, cognitive skills and personality factors; this helped to better understand the effects on the dependent variables (see second awareness lab study).

If the goal is to develop experimental methods appropriate for studying CSCW phenomena in valid conditions, an important lesson to keep in mind is the key tradeoff between validity and statistical power.  The sample size will be drastically reduced for a number of useful reasons: the use of group variables (N = participants/group size), the study of group process in long-term collaboration via repeated measures (1 case = k sessions), and the systematic comparison among alternative experimental conditions (e.g. independent samples use alternative collaborative tools or collaborative settings). The empirical results from the program suggest that the strategies for measurement and the criteria for assessing the quality of the findings (p-value of 0.5 or 0.1), which are currently imported from behavioral and social sciences, need to be adapted to the specific needs of CSCW researchers.  For example, multiple measures can be used to compensate for a small N.  In fact, the integration of different measures in the program helped to increase the reliability and sensitivity of results derived from specific measures of awareness and common ground; in some cases, the discrepancies between measures were also informative (e.g., measures in the second awareness lab study some effects on activity awareness were measured through behaviors but through subjective).  The combination of the multiple process measures with performance measures increases the sensitivity with respect to experimental effects (see Monk et al. 1996 on process measures).  Moreover, the second awareness lab study demonstrated that researchers can introduce control at the level of a small repeated task with a low cost for the overall task validity. This requires the experimenter to control only the conditions of a small task in order to extract useful low-level measures but at the same time enables the preservation of natural conditions for the overall collaborative activity (ecologic validity).

There is a strategic need for research programs in CSCW research motivated by the fact that studying the effects of CSCW systems in realistic conditions is difficult (e.g. Grudin 1988).  In general, as the complexity of the phenomena investigated increases, the usefulness of incremental, step-by-step research programs becomes more evident.  In new research areas such as CSCW, running sequences of studies that are methodologically and theoretically related can help the researcher to progressively define a corpus of reusable conceptual models and measures native to CSCW.  This approach is inspired by the concept of “Progressive Research Programmes” by Lakatos (1978, 1995).

Thesis Structure

 

Chapter 2 surveys research methods for CSCW research in general and then turns specifically to research on awareness in collaboration.

In Chapters 3 and 4, as an instantiation of this approach, I present a research program (i.e.

sequence of empirical studies) on awareness and knowledge sharing.

The first part of the program focuses on activity awareness: a two-year field study was conducted to outline the theoretical framework on activity awareness.  In 2003, during the second year of the field study, I contributed to the analysis of the field data and developed a laboratory method for studying activity awareness in long-term collaborative editing projects.  The method was validated in an initial lab study at Virginia Tech.  Between 2004 and 2006, an extended version of the method was implemented in an experiment at Penn State designed to measure activity awareness and subsequently a theoretical articulation of the concept of activity awareness was proposed.  The second part of the program focused on common ground – a sub-process of activity awareness.  During the last three years of the program (2005-2007), a lab method was developed for studying knowledge sharing and two related experiments investigated the common ground process with teams that performed a complex planning task using a geocollaborative prototype.

Finally, in Chapter 5, I discuss the proposed approach, draw lessons from my empirical studies, and suggest implications for designing better support for awareness and knowledge sharing.

Embedded publications

The dissertation integrates and refines contributions from my previous and forthcoming published research, with the aim of proposing a more comprehensive contribution on experimental methods for collaborative computing; and, specifically, on methods for studying awareness and knowledge sharing. While the details of these publications are not included in the present study, the research and its specific relationship with the following portions of this thesis are here indicated for further inquiry.

 

Chapters 1 and Chapters 2: The problem

Integrates content from CSCW 2004 poster, CSCW 2006 Activity Workshop presentation, CSCW 2006 poster, and a manuscript on meta-analysis of ACM CSCW (1986-2006) papers.

 

Chapter 3: Empirical Research on Activity Awareness

Integrates content from of NordiCHI 2004 paper, coauthored IWC 2006 journal paper, and the “Measuring Activity Awareness” manuscript under revision.

 

Chapter 4: Empirical Research on Common Ground

Integrates content from CMV 2005 paper, coauthored CKI book chapter, HCII 2007 poster, conference paper submitted in Fall07, future journal submission on both common ground experiments

 

Chapter 5: Implications for the field

Integrates content from discussion sections of the publications and manuscripts mentioned above plus additional unpublished content.

 

 

The audience

CSCW researchers

The primary audience of this work is CSCW researchers. The expected contribution to this research community is a methodological approach for studying complex collaborative phenomena such as awareness, common ground, and group decision-making in CSCW.

Different groups of CSCW researchers will benefit from this work in different ways:

Researchers interested in methods, the methodologists, could be interested in how field and lab evaluation are related and how we model and measure phenomena in the lab that were previously observed only in the field.

Researchers interested in theory, the theorists, might consider our approach to evaluation a useful strategy for developing and validating theories. Our approach is based on the theoretical assumption that CSCW artifacts embody specific hypotheses and theories, which are indirectly tested as the artifacts are tested in the real world (Carroll and Campbell 1989).

Researchers investigating the design practice, scientists of design, in order to make this practice more efficient and successful, may benefit from relating theory-based evaluation to recent technology design proposals with a rigorous theoretical grounding (e.g. Briggs 2006; Carroll et al. 2006).

Cutting across all three groups listed above, researchers focusing on how to support awareness, knowledge sharing, or more general group decision-making could benefit from my review and development of research methods.

 

CSCW community

The overall CSCW community may be interested in tackling foundational issues such as how to assess the quality of empirical studies in CSCW (i.e. reliability, validity, and theoretical foundations); how to increase benefits vs. reduce costs; how to develop rigorous theory (or models) to promote sound evaluation and efficient design; how to enable practitioners to learn and better implement methods for usability evaluation by integrating the findings of researchers, who explore and compare new strategies for evaluation.

Why Research Methods Matter

A method is a general procedure that we can use to solve a class of problems: those related to understanding phenomena in the world. Thus the goal of research methods is the production of new accurate knowledge about world phenomena.  In 1637, in his Discourse on Method, the philosopher, mathematician and scientist René Descartes outlines why method is critical to science and design in general:

 

Good sense is, of all things among men, the most equally distributed; for every one thinks himself so abundantly provided with it, that those even who are the most difficult to satisfy in everything else, do not usually desire a larger measure of this quality than they already possess. And in this it is not likely that all are mistaken the conviction is rather to be held as testifying that the power of judging aright and of distinguishing truth from error, which is properly what is called good sense or reason, is by nature equal in all men; and that the diversity of our opinions, consequently, does not arise from some being endowed with a larger share of reason than others, but solely from this, that we conduct our thoughts along different ways, and do not fix our attention on the same objects.  For to be possessed of a vigorous mind is not enough; the prime requisite is rightly to apply it.  The greatest minds, as they are capable of the highest excellences, are open likewise to the greatest aberrations; and those who travel very slowly may yet make far greater progress, provided they keep always to the straight road, than those who, while they run, forsake it (Part I, emphasis added; Descartes 1637 in Laurence J. Lafleur (trans.)1960).

I begin the present study introducing the problem of research methods historically and conceptually.  Below I will summarize the trajectory that has led to the current state of methods in human-computer interaction and then, more specifically, in collaborative computing.

New disciplines such as human-computer interaction, human factors (ergonomics), computersupported cooperative work (CSCW) and information systems are relatively young research areas that followed with various timing the invention of computers (1946) and are still defining their own theories and corpora of research methods.  However, a unique trait that these technology-related disciplines have in common is that researchers aim at impacting science and engineering as well as technology design.  The researchers (or their interdisciplinary teams) in some cases pursue primarily the goal of producing new knowledge; in other cases they primarily engineer new tools for practical use.  Mackay and Fayard (1997) suggest that the human-computer interaction community introduces an new model of research, where researchers from different disciplines move dialectically between the distinct practices of collecting observations, designing new artifacts and revising theories or conceptual models.

This is the broad context in which I pose the problem of lack of native methods in CSCW: the lack of a rigorous treatment of methods in HCI and CSCW has affected the intellectual debate on methods and manifests itself in the undifferentiated treatment of methods from two different categories.  On the one hand, there are the techniques used to assess tools, where the methods inform the development of commercial products (e.g. heuristic evaluation for detecting usability problems). On the other hand, there are the methods used by scientists to experiment with proof-of-concepts prototypes and to understand the principles of interaction (e.g. laboratory experiment with a prototype).  Often research methods are developed in the context of scientific research and are then adapted, made cost-effective, and used for informing tool development and testing by product groups (e.g. heuristic evaluation, cognitive walkthrough). This is a useful contribution from research to design that should be encouraged, however the discussion about the two types of research methods should be kept distinct because the goals are different. Therefore, the criteria (or priorities) for assessing the quality of the methods in the two contexts are also different (e.g. time-efficiency vs. accuracy).

In Chapter 2, I focus on the methods currently used in research. I will specifically describe the development and use of experimental methods for studying knowledge sharing and awareness in computer-supported working groups.  This chapter first summarizes the debate on methods in CSCW and parent disciplines. It will point to (but not exhaustively present) relevant contributions on research methods.

AWARENESS AND KNOWLEDGE SHARING IN COLLABORATIVE COMPUTING: EXPERIMENTAL METHODS

Sharing is caring!

Leave a Reply