Cursor Movement On Object Motion

Cursor Movement On Object Motion

Abstract

Human-Computer Interaction (HCI) is a necessary part of everyday live. With over 350 million computers sold globally every year, computers are now the corner stone of technological development. The main device for HCI for the past decade is the computer mouse, which is not suitable in some instances such as Human-Robot Interaction. This project proposes a novel vision based cursor control system, using hand gestures captured from a webcam. The system will allow the user to navigate the computer cursor using their hand and cursor functions, such as right and left clicks, will be performed using different hand gestures. The proposed system uses nothing more than a low resolution webcam and it is able to track the users hand in two dimensions and can recognize up to seven hand gestures, which are interpreted as mouse functions. The input frame from the webcam is first converted to the HSV colour space, where skin detection and background subtraction are performed. Edge detection is then performed, to obtain the hand contour, which would be used for hand tracking and gesture recognition. Evaluation were also done on two common skin detection methods, Histogram based and Explicit threshold, as to determine which is more accurate and should be used in the proposed system. It was found that the Explicit threshold method performed poorly, with an average TPR of 63% and FPR of 22%, while the Histogram method performed much better with a TPR of 94% and FPR of 12%. The results showed that the system performs best in a well lit room. This is due to the fact that the accuracy of skin detection, by extension the entire system, is greatly dependent on the surrounding lighting condition. The major constraint on the system is that it must be operated in a well lit room. The accuracy of the proposed cursor control system was found as 59%, while the accuracy of a computer mouse was 99%. Although the accuracy of the proposed system is a lot less than the computer mouse, there are improvements that can be made to the system to increase its accuracy. These improvements are discussed in the report.

CHAPTER 1–Introduction

1.1  Background to Project

Computer technology has tremendously grown over the past decade and has become a necessary part of everyday live. The primary computer accessory for human computer interaction (HCI) is the mouse. The mouse is not suitable for HCI in some real life situations, such as with human robot interaction (HRI). There have been many researches on alternative methods to the computer mouse for HCI (Quam and David 1990; Zhu et al 2005). The most natural and intuitive technique for HCI, that is a viable replacement for the computer mouse is with the use of hand gestures (Yeo et al 2013). This project is therefore aimed at investigating and developing a CC system using hand gestures.

1.2  Project Justification

Most laptops today are equipped with webcams, which have recently been used in security applications utilizing face recognition. In order to harness the full potential of a webcam, it can be used for vision based CC, which would effectively eliminate the need for a computer mouse or mouse pad. The usefulness of a webcam can also be greatly extended to other HCI application such as a sign language database (Starner et al 1998) or motion controller (Rehg and Kanade 1994). Over the past decades there have been significant advancements in HCI technologies for gaming purposes, such as the Microsoft Kinect and Nintendo Wii. These gaming technologies provide a more natural and interactive means of playing videogames. According to Benedetti (2009), motion controls is the future of gaming and it have tremendously boosted the sales of video games, such as the Nintendo Wii which sold over 50 million consoles within a year of its release.

HCI using hand gestures is very intuitive and effective for one to one interaction with computers and it provides a Natural User Interface (NUI). There have been extensive research towards novel devices and techniques for cursor control using hand gestures (Yeo et al 2013; Jophin et al 2012).Besides HCI, hand gesture recognition are also used in sign language recognition (Guan et al 2008), which makes hand gesture recognition even more significant.

1.3  Project Objective

1. Research existing methods and accessories for cursor control and the suitability of visually based methods for cursor control.
2. Investigate tracking algorithms utilized for cursor control.
3. Develop and implement a computer application that utilizes alternate methods for cursor control.
4. Compare the accuracy and precision of the application with alternative accessories (such as the Microsoft Kinect/Computer Mouse).

1.4  Scope of Project

There are generally two approaches for hand gesture recognition, which are hardware based (Quam 1990; Zhu et al 2006), where the user must wear a device, and the other is vision based (Shrivastava 2013; Wang and Popović 2009), which uses image processing techniques with inputs from a camera. The proposed system is vision based, which uses image processing techniques and inputs from a computer webcam. Vision based gesture recognition systems are generally broken down into four stages, skin detection, hand contour extraction, hand tracking and gesture recognition. The input frame would be captured from the webcam and the skin region would be detected using skin detection. The hand contour would then be found and used for hand tracking and gesture recognition. Hand tracking would be used to navigate the computer cursor and hand gestures would be used to perform mouse functions such as right click, left click, scroll up and scroll down. The scope of the project would therefore be to design a vision based CC system, which can perform the mouse function previously stated.

1.5  Background Theory

In this section an overview of the system would be given and the fundamental theories used in developing the system would be explained. This section is separated in the following subsection:

1. Overview of the system

2. Skin Detection using HSV colour space

3. Hand Contour Extraction

4. Hand Tracking

5. Gesture Recognition

6. Cursor Control

1.5.1    Overview of System

In the overview of the system, the suggested environmental setup of the system is described as well as the architecture of the system. The environmental setup for the system is shown in the Figure 1 below.

Figure 1Environmental Setup of System

The user’s hand should be at least 24cm from the webcam and the room should be well lit. The source of the light in the room must be white, since the appearance of skin changes under different colours of light. The webcam must not be moving and if it is shifted the program should be restarted. In Figure 1, the user is only using one hand to operate the program and the other hand must not be in front of the webcam.

The cursor control system consists of two main parts, the back end system and the front end application. In the back end system, image processing techniques are used to allow hand gesture recognition, while the front end application uses the interpreted hand gestures to allow CC. The architecture for the CC system is shown in Figure 2 below.

Figure 2Achitecture of system

The steps involved in the back-end system, shown in Figure 2 above, are similar for most vision based, marker less hand gesture recognition system. The outputs of the back-end system are the hand gesture and the hand position. Although the front-end application for this project is cursor control, there are many different front end applications that can be used with the back-end system, such as sign language recognition.

1.5.2     Skin Detection

Skin detection can be defined as detecting the skin colour pixels in an image. It is a fundamental step a wide range of image processing application such as face detection (Deng et al 2005;Ishii et al 2004), hand tracking (Barczak and Dadgostar 2005) and hand gesture recognition(Chen 2007). Skin detection using colour information has recently gained a lot of attention, since it is computationally effective and provides robust information against scaling, rotation and partial occlusion. Skin detection using colour information can be a challenging task, since skin appearance in images is affected by illumination, camera characteristics, background and ethnicity (Kakumanu et al 2007).

In order to reduce the effects of illumination, the image can be converted to a chrominance colour space, which is less sensitive to illumination changes. A chrominance colour space is one where the intensity information (luminance), is separated from the colour information (chromaticity). In the proposed method, the HSV colour space was used with the Histogram-based skin detection method. The HSV colour space has three channels, Hue(H), Saturation(S) and Value(V). The H and S channels hold the colour information, while the V channel holds the intensity information. The input image from the webcam would be in the RGB colour space, thus it would have to be converted to the HSV colour space using the formulas given below.

The Histogram-based skin detection method proposed by (Jones and Rehg 1999) uses 32 bins H and S histograms to achieve skin detection. Using a small skin region, the colour of this region is converted to a chrominance colour space. A 32 bin histogram for the region is then found and is used as the histogram model. Each pixel in the image is then evaluated on how much probability it has to a histogram model (Jones and Rehg 1999). This method is also called Histogram Back Projection. Back projection can be defined as recording how well pixels or patches of pixels fit the distribution of pixels in a histogram model (Bradski and Kaehler 2008). The result would be a grayscale image (back projected image), where the intensity indicates the likelihood that the pixel is a skin colour pixel. This method is adaptive since the histogram model is obtained from the user’s skin, under the present lighting condition.

1.5.3     Hand Contour Extraction

A fundamental step in image processing feature extraction application is contour extraction or edge detection. Edges in an image can be considered as the points where the image brightness changes sharply. A contour can be defined as a sequence of points that outlines a shape or region. The contours or edges in the image were found using the technique developed by (Satoshi 1985). OpenCV implements this method in the function named “cvFindContours”, which finds the contours of the image and stores them in an array. According to (Yeo et al 2013), in a hand gesture recognition system, the hand contour can be assumed to be the largest contour in the image. Thus this assumption was also made in the proposed design, to identify the hand contour from the array of contours in the image.

1.5.4    Hand Tacking

Hand tracking can be considered as identifying the hand or a point on the hand in each input frame. In the proposed design, hand tracking was achieved by finding the tip of the index finger and tracking it over successive frames (Yeo et al 2013). In order to identify the tip of the index finger, the hand centre must first be found. Using the hand centre and convexity defect, the tip of the index finger can easily be identified using the method proposed by (Balazs 2012). Figure 3 below shows a conceptual illustration of the palm with its convex hull and convexity defects.

Figure 3Hand shape with convexity defects (Bradski and Kaehler 2008)

The hand centre was found using the method proposed by (Yeo et al 2013). The shortest distance of each point inside the contour, to the contour perimeter is calculated. The point with the largest distance is taken as the hand centre and the corresponding distance is taken as the hand radius. From Figure 3 above, the hand centre would be the point labelled as C_a and the radius of the hand would be taken as

1. r_a. The method used to identify the tip of the index finger is described in the following subsection.

1.5.5     Gesture Recognition

An effective technique for analysing the shape of an object is to find the convex hull of the object and then compute its convexity defects (Bradski, and Kaehler 2008). From Figure 3, the black line outlining the palm is the convex hull of the hand and each of the gridded regions (A,B,C..G), are the convexity defect of the palm. For each convexity defect, there is a start point ( ), end point ( ), depth point ( ) and depth length ( ). The start point is the point where the contour and convex hull intersects. The depth point is the point farthest away from the convex hull and the depth length is the distance between the start point and depth point. The end point is the end of the defect. All these points would be found and stored in an array for further use. The start point is the point at which the contour intersects with the convex hull, thus each fingertip would have a starting point as seen in Figure 3 above.

In order for a start point to be classified as a fingertip, if it passes the following tests:

1. The distance between a start point and the hand centre must be greater than 1.2*hand radius but less than 3*hand radius (Yeo et al 2013).
2. The angle between the start point (p_S) and end point must be less than 85 degrees (Yeo et al 2013).

• The angle of the start point from the hand centre must be less than 200 degrees (Balazs 2012).

To further increase accuracy, we can limit the ROI to a circle with centre point at the hand centre and set the radius of the circle to 3*hand radius. This minimum enclosing circle was shown in Figure 3 as the blue circle. We can then only consider convexity points inside this circle.

In the method proposed by (Balazs 2012), hand gesture recognition is achieved by identifying the thumb, index and pinkie fingers in the hand. Each finger is identified based on the angle it makes with the hand centre. This is illustrated in Figure 4 below.

Figure 4Angle of fingers from hand centre (Balazs 2012).

The thumb finger can be identified as the fingertip with an angle between 200 and 120 degrees from the hand centre. The index finger can be identified as a fingertip with an angle between 120 and 60 degrees and the pinkie finger is a fingertip with an angle less than 60 degrees. This range is for the right hand, for the left hand the range would have to be reversed. If the hand is rotated, it can cause misclassification of the fingers which is a major source of error in this method. By identifying the thumb, index and pinkie fingers only, we can recognize hand gestures as shown in Figure 13-19 of the Result and Analysis Section. In this gesture recognition method, all the gestures are made with these three fingers or less, thus a maximum of eight different hand gestures are possible. This is acceptable though, since the proposed application only requires seven different hand gestures.