LUDIC ELICITATION: USING GAMES FOR KNOWLEDGE ELICITATION

  • : Ms Word, Ms Word Format
  • : 70 Pages
  • : ₦3,000 | $25 | ₵60 | Ksh 2720
  • : 1-5 Chapters
  •  
  • Click to DOWNLOAD Materials

LUDIC ELICITATION: USING GAMES FOR KNOWLEDGE ELICITATION

Abstract

Knowledge elicitation from human beings is important for many fields, such as decision support systems, risk communication, and customer preference studying. Traditional approaches include observations, questionnaires, structured and semi-structured interviews, and group discussions. Many publications have been studying different techniques for a variety of data elicitation tasks as well. However, few of them have considered participants’ user experience in the process. One main drawback of these methods is their time consuming and labor intensive nature, because of which participants often lose their interest and attention quickly in data elicitation activities. Innovated by the success of games with a purpose in many fields such as participatory city exploration and community building, we propose to adopt a game approach for knowledge elicitation tasks.

We have developed two browser-based casual games, LinkIT and SortIT, and have applied them for three knowledge elicitation applications: relation elicitation, rank elicitation, and probability elicitation. The LinkIT game elicits relations between variables/concepts and facilitates the construction of relation network structures such as concept maps and Bayesian networks. The SortIT game presents puzzles in the form of multiple-choice questions. This format supports rank elicitation in a pairwise comparison approach. The second application of this game is probability elicitation by using probability intervals or verbal expressions. By comparing the two games with more traditional methods such as questionnaires, we have established the external validity of the games for the three knowledge elicitation tasks. Further, user experience studies conclude that the games improve user experience by forming the elicitation tasks as a play activity and making the activity more interesting, engaging, exciting, and fun. These findings provide positive support for the applications of GWAP for more knowledge elicitation tasks.

Chapter 1

Introduction

Knowledge elicitation has its formal beginnings in the 1980’s in the context of knowledge engineering and expert systems. With the realization of “knowledge is power” and technological advance, many areas grow a need for expert systems [1] . Questions regarding knowledge elicitation and knowledge acquisition become central to both applied and basic endeavors [2] . How can knowledge be effectively elicited from domain experts? A closer look at the literature reveals that more attention and effort are paid to the representation and conceptualizations of knowledge structure such as networks and schemas [3, 4] . Fewer publications focus on the actual knowledge elicitation process and applicable methods.

Researchers and practitioners have developed many knowledge elicitation methods, many of which are adapted from cognitive methods or methods in other disciplines such as counseling, education, and management [5, 6] . In recent years, knowledge elicitation has surfaced in more areas, including human-computer interaction, training systems, and cognitive engineering [7, 8] . Driven by the need, more work is devoted to developing additional methods.

Many elicitation techniques have been widely used in publications, such as structured and unstructured interviews, ranking, card sorting, and thinking aloud. One important note is that no single method dominates others and no single method is applicable for all knowledge elicitation tasks. The type of knowledge elicited is an important dimension to distinguish the numerous knowledge elicitation methods. For example, some methods (thinking-aloud and interviews) rely heavily on verbal reports, compared to others (similarity ratings and observations), and are thus more applicable for enumerating concepts and decomposing tasks into steps. Another important note is that for any of the methods, there is no single definitive procedure for the application in practices. A systematic review on empirical studies concerning the effectiveness of elicitation techniques concluded that interviews, especially structured interviews, appear to be the most effective approach [9] . Also, no significant effects on the elicitation of intermediate representations, such as visual hints, is found [9] . Below is a short summary of knowledge elicitation methods, divided into four groups.

Observations

Observations are usually the beginning of a knowledge elicitation process and provides an overview of the domain [6] . By observations, researchers and practitioners can generate an initial conceptualization of the task and identify potential constraints or issues to be dealt with during later phases. Depending on the nature of the task, observations can occur in a natural or controlled setting. Advantages of observations include (1) the interference is minimized, (2) observations generates a lot of data. On the other hand, one disadvantage lies on the uncertainty embedded in the interpretation of the elicited data. Many modern technologies could be used to facilitate observations, such as video recorders and video analyzing software.

Interviews

Interviews provide a direct approach to elicit knowledge by asking the corresponding people (experts). Two forms of interviews, unstructured and structured, are widely used in practice. Unstructured interviews are usually adopted for early stages of elicitation and help the elicitors understand the domain and prepare more structured interviews. Structured interviews have predetermined content and sequencing and thus provide more constraints on the experts’ responses and thus more systematic converge of the domain. Interviews are relatively easy to administer compared to other knowledge elicitation methods. However, they pose a higher pressure on the data analysis and interpretation phase. In recent years, some domains have developed highly specific interview methodologies, such as the mental models approach for risk communication [10] .

Process Tracing

Process tracing is most often used to elicit procedural information such as sequential behavioral events, including both verbal events and non-verbal events. When and how the data is collected are two important elements for determining the quality of collected information.

Conceptual Methods

Conceptual methods are used to elicit concepts and their inter-relations in domains. These methods usually associate with several steps, including (1) the elicitation of domain-related concepts through interviews or document analysis, (2) the determination of relationships between concepts, and (3) the representation and interpretation of the relationships [6] . A number of relationship judgment methods have been developed, such as pairwise similarity ratings, sorting techniques, repertory grid, and frequency of co-occurrence. The similarity ratings approach involves presenting pairs of concepts to the expert and requesting a quantitative estimate of the similarity of the two concepts. It exists a scaling issue and becomes very time costly when the number of concepts exceeds 30. In the repertory grid approach, all concepts are rated against a set of dimensions and the similarity between concepts can thus be derived from the ratings.

Although many publications have been devoted to knowledge elicitation methods and applications, few of them have discussed participants’ user experience in this process and have not proposed any method to improve their experiences. Our previous studies in the elicitation of lay people’s risk mental models indicate that interviews and surveys with human subjects for structural model elicitation purposes are usually very time-consuming and repetitive in terms of the nature of the tasks. As a consequence, human participants get bored quite easily and are unable to maintain their interest and attention. This implies the necessity to study and improve user experience in the process of knowledge elicitation tasks.

Simulation and gaming scholar Richard Duke argued in his 1974 text that “gaming is the future’s language” [11] . The sentiment underlying this seemingly radical statement for the time was based on the author’s observations that game mechanics can be used to stimulate meaningful “multilogue” among players that other approaches would be hard-pressed to achieve. It is now apparent that after four decades of gaming research, modern gaming has exceeded Professor Duke’s original expectations. Gaming has become more than a language; it has become an approach to stimulate productivity of all sorts. Games are routinely used these days to complement classroom instruction [12] , facilitate group discussions about disaster preparedness [13] , and incentivize participation in data collection and human computation campaigns [14] , to name just a few applications. In fact, contemporary discourse on the future of gaming centers on the parallel notions of ubiquitous and pervasive gaming, where some philosophers suggest that gaming has the potential to change the world [15, 16] .

Among the more popular forms of contemporary gaming is the “casual game”. A casual game is game distinguished by simple rules and lack of commitment required to become proficient and competitive [17] . Many traditional board games fall under the category of casual game, including such popular titles asCONNECT FOUR, SCRABBLE and SORRY! as well as classic games such as BACKGAMMON, PACHISI, and MANCALA. The Nintendo Wii, a leading video game console, attributes its success through the appeal of casual gaming to non-traditional players [18] . The Internet offers myriad casual games that are free-of-charge to play and take advantage of standard web browsers and associated plugins or extensions (e.g., Adobe Flash) to create a widely accessible and lightweight gaming environment [19] . Such games are known as browser games.

Human computation is a field that studies the employment of human’s computing ability to solve complex problems that computers cannot easily solve in time. Games present one platform for human computation in an entertaining manner. Pioneered by Luis von Ahn and his colleagues, many games with a purpose (GWAP) systems have been developed in recent years, such as ESP [20] ,GWAP for the Semantic Web [21] , andCityExplorer [22] . Time-intensive or otherwise tedious tasks are “crowd-sourced” to the players and solved by human power. Thus, the “serious purpose” of a GWAP, that is, the designer’s intent for creating the game [23] , is to incentivize productive work via the allure of fun. Numerous publications have shown the successes of these systems in fulfilling the serious purpose while providing an enjoyable environment for participants [20, 21, 22] . This dissertation focuses on the application of GWAP on several knowledge elicitation tasks including relation elicitation, rank elicitation, and probability elicitation.

Motivated by the concepts of serious games [24] and Games With A Purpose [25, 20, 21, 22] , a gaming approach has the potential to improve user experience by making the activities more entertaining. The basic idea is to form the tasks as a play activity and to deploy players’ knowledge and computing power in their game play. For example, players provide high-quality image tags while playing theESP game [20] . GWAP has demonstrated its usefulness for many serious purposes, such as building the semantic Web [21] and creating domain-specific sentiment lexicons [26] .

The purpose of this research is to propose the application of GWAP for several different knowledge elicitation tasks, including relation elicitation, rank elicitation, and probability elicitation. This gaming system consists of two browser-based online casual games, LinkIT and SortIT. The LinkIT game is a relation game and game players draw the relations between variables to proceed in the game play. The SortIT game presents puzzles in the form of multiple-choice questions and is adaptable for pairwise comparison applications, which is a frequently used approach for rank elicitation. Meanwhile, utilizing the flexibility of the puzzle question prompt, we can use the game for probability elicitation. Combining these applications provide a potential platform for Bayesian network elicitation as well.

However, it is not guaranteed that the data collected in a game approach is as valid and as accurate as the data collected from a “more serious” environment (such as questionnaires and interviews). Thus, our study focuses on the external validity of the game approach for the three data elicitation tasks. In particular, the main research question is as follows.

Can a game based approach match the effectiveness of knowledge elicitation using more traditional questionnaire approach? In particular, does the data collected from the game approach maintain the validity?

Further, because the main reason that we introduce the game approach is to improve user experience in the traditionally tedious knowledge elicitation tasks, our second research question is

Can a game be designed to improve the user experience of knowledge elicitation compared to the more traditional questionnaire approach?

In terms of the three application fields, our research questions are as follows.

Can a game based approach be used to elicit probabilities in addition to Bayesian network structure?

How well does a game based approach elicit probability knowledge, compared to more traditional questionnaires?

We conduct experimental studies for the three knowledge elicitation tasks to explore the external validity and improvement on user experience of the two games. Significant similarity between the data collected from the game approach and the data from more traditional approaches is identified, supporting the external validity of the game approach. Also, participants have rated the game play activity to be more enjoyable and maintain their attention longer than traditional questionnaire activity does.

The contributions of this research are three-folds. First, it highlights the importance of studying user experience in knowledge elicitation processes. Second, we propose a game approach for several different knowledge elicitation tasks and call it “ludic elicitation”. We have developed two casual games for these applications, LinkIT and SortIT. Third, experimental studies are conducted to demonstrate the external validity of the game approach and their improvement on participants’ user experience.

The rest of this dissertation is organized as follows. Chapter 2 introduces the two games that have been developed, LinkIT and SortIT. The following three chapters summarizes our experimental studies on the three knowledge elicitation tasks: relation elicitation, rank elicitation, and probability elicitation, respectively. Each chapter starts with an introduction to the knowledge elicitation task, followed by the corresponding research question(s). We then describe the experiment design and participants in the method section. The results are presented and discussed afterwards. Each chapter also has a conclusion of the findings. In the last chapter, we further propose a game approach for eliciting Bayesian networks, on the basis of the three completed studies.

LUDIC ELICITATION: USING GAMES FOR KNOWLEDGE ELICITATION

Sharing is caring!

Leave a Reply