AIDING THE USER INPUT TO VIRTUAL TRAINING ENVIRONMENTS: VIRTUAL ROLE PLAYERS WITH SPEECH AND GESTURE RECOGNITION

  • : Ms Word, Ms Word Format
  • : 70 Pages
  • : ₦3,000 | $25 | ₵60 | Ksh 2720
  • : 1-5 Chapters
  •  
  • Click to DOWNLOAD Materials

AIDING THE USER INPUT TO VIRTUAL TRAINING ENVIRONMENTS: VIRTUAL ROLE PLAYERS WITH SPEECH AND GESTURE RECOGNITION

ABSTRACT

The purpose of this thesis is to address the fact that users’ input to training systems in virtual environments is not suited to their natural skills and abilities. These skills and abilities include speaking and gesturing with their bodies. This mismatch may have negative effects on their usage of the virtual environment. One assumption guiding this thesis is that it would increase immersion to allow the users to interact with the system in the same way they interact with real people. The second assumption is that multimodal input can increase users’ performance in the training scenario, especially regarding habitual and physical skills. While people use the mouse and keyboard inputs to computers all of the time, the third assumption is that natural speech and gestures would make military virtual training systems even easier to get acquainted with and use.  The fourth assumption is that more natural systems may increase the amount of training that trainees can transfer to the real world.

To show the potential of the approach of multimodal input, two prototype systems were created. The design and evaluation of the first prototype are described. It was intended to show the potential of gesture recognition and multimodal fusion under both ideal theoretical circumstances and controlled, but more realistic, ones. The primary problem with the first prototype was found to be the limitations with the hand recognition and tracking system. The design of the second prototype is then described. This prototype is a fully-operational virtual checkpoint training system with multimodal input and was created based on the hand tracking and other insights from the first prototype. Then the results of a demonstration at a conference are explained, including environmental factors on its usage. The thesis ends with a discussion of the insights from the last prototype and some future work, including implementation ideas, empirical studies, and general guidelines for multimodal system design.

TABLE OF CONTENTS

LIST OF FIGURES………………………………………………………………………………………………………………………………… v LIST OF TABLES………………………………………………………………………………………………………………………………….vi ACKNOWLEDGEMENTS …………………………………………………………………………………………………………………..vii

Section 1: Introduction ………………………………………………………………………………………………………………………1

Problems with Virtual Training Systems …………………………………………………………………………………………………… 1

Solution …………………………………………………………………………………………………………………………………………………………….. 2

Preview of this Thesis ……………………………………………………………………………………………………………………………………. 3

Section 2: Human Factors Problems in Virtual Training……………………………………………………………..3

Immersion………………………………………………………………………………………………………………………………………………………… 4

Human Performance………………………………………………………………………………………………………………………………………. 5

Ease of Use ……………………………………………………………………………………………………………………………………………………….. 6

Transfer of Learning………………………………………………………………………………………………………………………………………. 7

Summary…………………………………………………………………………………………………………………………………………………………… 9

Section 3: Gesture Recognition Overview ………………………………………………………………………………………9

Uses of Gesture Recognition…………………………………………………………………………………………………………………………. 9

Outline and General Approach to Gesture Recognition …………………………………………………………………………12

Static Recognition: Object Detection and Recognition …………………………………………………………………………..12

Dynamic Recognition…………………………………………………………………………………………………………………………………….13

Conclusion: Symbolic vs. Non-Symbolic……………………………………………………………………………………………………18

Section 4: First Gesture Recognition Prototype…………………………………………………………………………. 19

System Details ………………………………………………………………………………………………………………………………………………..20

Architecture and Design……………………………………………………………………………………………………………………………….20

System Improvements Before Evaluation ………………………………………………………………………………………………..25

Evaluation Method…………………………………………………………………………………………………………………………………………26

Evaluation Results …………………………………………………………………………………………………………………………………………27

Summary………………………………………………………………………………………………………………………………………………………….30

Section 5: Possible System Improvements …………………………………………………………………………………. 32

The Gesture Command Recognition Algorithms……………………………………………………………………………………..33

The Jahmm Toolkit………………………………………………………………………………………………………………………………………..34

Hand Tracking………………………………………………………………………………………………………………………………………………..35

Summary………………………………………………………………………………………………………………………………………………………….38

Section 6: The Final VRP System…………………………………………………………………………………………………… 38

System Details ………………………………………………………………………………………………………………………………………………..39

Informal Conference Evaluation…………………………………………………………………………………………………………………45

Summary………………………………………………………………………………………………………………………………………………………….47

Section 7: Discussion………………………………………………………………………………………………………………………. 48

Conclusions……………………………………………………………………………………………………………………………………………………..48

Summary of Contributions…………………………………………………………………………………………………………………………..49

Open Problems and Future work……………………………………………………………………………………………………………….50 References ………………………………………………………………………………………………………………………………………… 54

LIST OF FIGURES

Figure 1: The Architecture of the First Prototype’s Gesture Recognizer………………………………. 21

Figure 2: The “Closed” Hand Posture Being Recognized by HandVu………………………………….. 23

Figure 3: Architecture of the Multimodal Fusion Component in the First Prototype ……………… 25

Figure 4: First Prototype Experiment #2 Results……………………………………………………………….. 28

Figure 5: The Checkpoint Scenario in VBS2 ……………………………………………………………………. 40

Figure 6: The Architecture of the VRP System…………………………………………………………………. 41

Figure 7: The Gesture Recognizer Subsystem in VRP……………………………………………………….. 43

 

LIST OF TABLES

Table 1: Confusion Matrix of the Cross Validation in Experiment #1………………………………….. 32

 

Section 1: Introduction

Simulations used by the military have been characterized as live, virtual, or constructive

(LVC). Live training is using real people operating real devices, but under simulated circumstances. Constructive simulations are those in which trainees provide input to the simulations, but do not actively play roles to determine the outcome. Finally, virtual simulations are using trainees to actively control virtual characters in a virtual environment analogous to real environments (Frank, Helms, and Voor, 2000; Page and Smith, 1998).

Problems with Virtual Training Systems

While virtual environments hold much promise for training skills, their application to such a domain still poses many challenges. According to Caird (1996), features that should be part of any training simulation system that hopes to be effective should include cost effectiveness, good interface usability, transfer of the training to the real world, feedback on the training, a match to theoretical constructs like fidelity and presence, and a system design based on task analysis. This research addresses the issues of usability and transfer of training, in addition to other benefits of natural user input modalities.

This research makes the primary assumption that the standard mode of computer input— a mouse and keyboard—is an unnatural way to train real-life skills. Furthermore, it is assumed that the resulting lack of realistic simulation may make the users feel less immersed in the virtual environment, may reduce their ability to perform well and improve in the simulations, and may increase the time and effort to learn to use the system to its full capacity. As a result, it is assumed that these effects on the users may cause inefficient transfer of the trained skills and knowledge to the real world upon deployment, which is the whole reason for the training exercise.

Solution

In light of the problems facing virtual training environments, the solution attempted in this study is the idea of a “virtual role player,” which plays roles such as civilians or insurgents and reacts in meaningful ways to actions performed by the user, while at the same time being controlled by the computer. Non-player characters (NPCs) are usually scripted to react in predictable ways and to focus on counter-attacking the trainees. Conversely, in the Virtual Role Players (VRP) training system, the NPCs’ reactions are based on both the trainee’s speech and hand gestures, simulating real-world interaction. Because these virtual role players respond to automatically-recognized speech and gestures from the user, the input is more natural than a keyboard and mouse. This, in turn, makes the training experience more realistic and immersive and allows the trainees to transfer what they learn from the training to the real world. Also, by using the trainees’ natural knowledge and skills, they should be able to achieve higher performance and this makes the system easier to use.

The VRP project is aimed at making a prototype training simulation in Virtual Battlespace 2 (VBS2)—the current US Army standard game-based training platform—in the domain of checkpoint security. In a checkpoint security scenario, the user is tasked with asking people to stop their cars, approach the checkpoint, continue through the checkpoint, get out of the car, and move to various places.

Preview of this Thesis

This thesis starts with an overview of the current human factors problems with the usage of virtual environments for training, followed by an overview of gesture-based input mechanisms and the recognition of these gestures by a computer system. The subsequent section outlines the resulting theoretical foundations of a first prototype, its design and implementation, and the results of its theoretical and pilot study evaluations. Following that is a section on what the evaluation results meant for a more robust second prototype. Details of the second prototype’s design and conference evaluation are then mentioned. The thesis ends with a discussion of the implications and lessons learned from these prototypes and future work for the successful application of multimodal input for training in virtual environments. The contributions of this thesis therefore are the aggregation of theoretical foundations for multimodal input related to virtual environments, creation and evaluation of the two prototypes, and the insights drawn from these evaluations, including their promise of accurate recognition and their requirements of environmental precautions required for deployment.

AIDING THE USER INPUT TO VIRTUAL TRAINING ENVIRONMENTS: VIRTUAL ROLE PLAYERS WITH SPEECH AND GESTURE RECOGNITION

Sharing is caring!

Leave a Reply