EVOLUTIONARY-BASED FEATURE EXTRACTION FOR GESTURE RECOGNITION USING A MOTION CAMERA

ABSTRACT

Gesture recognition systems have garnered increasing interest for their potential to support more natural human-computer interactions. However, compared to other human computer interaction technologies such as speech recognition, gesture recognition has not been actively applied to personal devices such as mobile phones or laptops due to the spatial requirements when performing gestures as well as sensitivity to background noise. My research first devises a problem of recognizing speed sensitive finger gestures using a novel camera called Dynamic Vision Sensor camera, which detects the temporal luminance difference for each pixel at microsecond-level granularity and outputs a stream of on-events (brighter) and off-events (darker) to the hardware. As with other machine learning problems, the performance of a gesture classification task depends on how well the representative features are extracted. Thus the feature extraction process must consider device-specific data properties to maximize the feature recognition abilities while minimizing computational cost. My research studies two feature extraction methods, namely local and global feature extractions, which are designed to maximize the performance of the DVS camera-based gesture recognition system.

First, the local feature extraction method aims to extract a smaller number of representative features from a long sequence of the raw gesture events detected by the DVS camera using segmentation. This approach is called the local feature extraction, since the features are extracted by considering neighboring events only. Specifically, I propose bottom-up segmentation methods, where the sequence of events are first divided into segments having the same time interval, called the time-based, or the same number of events, called the event-based, and the segments are repeatedly augmented based on the event distributions of the neighboring segments. The experimental results show that the event-based initial segmentation outperforms the time-based across different classifiers, and is more robust to noise. I also found that Bayesian network classifier is more accurate than hidden Markov model when features are well extracted using the event-based segmentation.

Second, the global feature extraction method aims to construct higher level compound features by transforming the locally extracted features. Specifically, an evolutionary algorithm is employed to find a good set of simple and compound features. This is a challenging task due to the large search space and the risks of overfitting. I define problem-specific representation, genetic operators, and evaluation methods, and analyze how the specified mutation and crossover operator controls the individual‟s search space. The experimental results show that the proposed EA can extract a good set of compound features that can enhance the performance accuracy with a smaller number of features. Finally, I show how my evolutionary-based feature extraction approach can serve as a knowledge discovery process in the context of gesture recognition.

 

Chapter 1  

Introduction

Gesture recognition is an emerging human-computer interaction technology, and has recently been applied to games such as Kinect and Wii. However, compared to other humancomputer interaction technologies such as speech recognition, the vision-based gesture recognition has not been actively applied to personal mobile phones or laptops. This might be due to the spatial requirements when performing gestures as well as sensitivity to background noise.

Recently, a research group at ETH has developed a motion camera called Dynamic Vision Sensor Camera (DVS), which responds to pixels with temporal luminance differences [1] , but its realworld applications have not been developed yet. In my research, I devised a new problem called speed-sensitive finger gesture recognition using a DVS camera aimed at exploring potential realworld applications to mobile devices.

As with other machine learning problems, extracting representative features is critical to the performance of a speed sensitive finger gesture recognition system. Especially when a new device, i.e., the DVS camera, is employed, the properties of this new device should be well considered during the feature extraction process. Gesture recognition using a general frame-based camera detects movement by comparing consecutive frames and uses detected changes as classification features. In contrast, when the DVS camera is used for gesture recognition, a framebased comparison is not necessary since the movements of an object are detected at the hardware level. Instead, since the DVS camera outputs a long sequence of events sampled at a microsecond-level granularity, winnowing the number of events while not losing  the overall gesture pattern is key for good performance.

 

The first part of my dissertation aims to extract a few representative features while preserving information about the patterns of the gesture through segmentation. To capture small but important movement changes, the local approach extracts features using similarity metrics between the neighboring events. Extracted features from this approach are called simple features. However, I expect that better performance can be achieved if higher-level abstract features can be constructed by transforming the locally extracted simple features to the difference space (e.g., from the Cartesian coordinate space to the polar space) or by finding the interactions between locally extracted simple features (e.g., the average speed of a gesture). In addition, higher level compound features may reduce the number of features, and thereby reduce the computational cost for learning and classification. Furthermore, they can serve as a knowledge discovery process by finding the underlying relationships between the features. The second part of my dissertation considers how to find a set of good simple and compound features.

1. Research Questions

My research aims to tackle two main research questions: (Q1) how to extract smaller number of representative local features from a long sequence of events and (Q2) how to construct abstract compound features from the locally extracted features (i.e., the simple features). Since the first research questions directly deals with the raw data from a DVS camera, the properties of the DVS camera must be carefully considered.

My approach to Q1 uses segmentation to extract local features; a sequence of gesture events are divided into segments from which simple features are locally extracted. Thus, the related sub-questions are (Q1-1) how to segment a sequence of events so that the patterns of gestures are well represented while being robust to noise and (Q1-2) how to extract features from those segments.

To construct high level compound features and to find a set of good simple and compound features, I use an evolutionary algorithm (EA) as a search algorithm. To use evolutionary algorithms, the representation method, reproduction operators, and fitness evaluation must first be well defined (Q2-1). The difficulties of using an EA for feature transformation include exploring the large search space and the risks of overfitting to the training data. Thus, I address the following sub-questions: (Q2-2) how to efficiently search for a good set of simple and compound features, and (Q2-3) how to reduce the overfitting problem. Finally, my research aims to provide qualitative analysis on the abstract features found in the EA (Q2-4). The main research questions and the related sub-questions are summarized in Table 1-1.

 

Table 1-1. The Research Questions

	RESEARCH QUESTIONS	RELATED QUESTION
Q1	How to extract simple features so that spatial and temporal information is well preserved?	Q1-1. How to perform segmentation? Q1-2. How to extract features from segments?
Q2	How to construct higher-level compound features?	Q2-1. How to design EA to find a good set of simple and compound features? Q2-2. How to make the search process more efficient? Q2-3. How to reduce overfitting? Q2-4. Can compound features detect useful underlying relationships between the features?

2. Dissertation Organization

Chapter 2 describes the properties of the DVS camera in detail, and summarizes related work on gesture recognition; segmentation; feature selection and construction; and the evolutionary-based feature extraction methods. Chapter 3 defines my problem domain of the speed-sensitive finger gesture recognition using a DVS camera and the two feature extraction problems, namely local and global feature extractions. In Chapter 4, I propose a segmentation method to locally extract features, and two initial segmentation schemes (time-based and eventbased) are evaluated. I quantitatively evaluate the two schemes in terms of performance accuracy of classifiers and qualitatively analyze the segmented gesture data. Chapter 5 proposes an evolutionary algorithm for feature construction and selection to extract features from a more global perspective. The representation, reproduction operators, and fitness evaluation methods specified to my gesture recognition problem are proposed and evaluated in terms of prediction accuracy and the complexity of the extracted features. I also analyze contextual meaning of the extracted features. Finally, Chapter 6 discusses the contributions of my research, and concludes the paper with an overview of future work.

EVOLUTIONARY-BASED FEATURE EXTRACTION FOR GESTURE RECOGNITION USING A MOTION CAMERA

Sharing is caring!