CONTEXT MODELING FOR SEMANTIC TEXT MATCHING AND SCENE TEXT DETECTION

Abstract

Context is the information that surrounds and defines the target information it encapsulates. Without context, the most related target information could be misinterpreted. Most existing models utilize context by encoding it as a set of human-crafted heuristic features for machine learning, which may not fully capture many connections between the context and the target. We contend that, in the setting of big data, context information should be modeled in a more principled way that is tightly coupled with learning algorithms. We present several machine learning models that learn the relations between context and related target information for two fundamental tasks in natural language processing and computer vision: semantic text matching and scene text detection. In particular, this dissertation addresses two different applications with context modeling: citation recommendation for scientific papers and localizing text in the wild.

Citations are crucial in academic attribution. A good citation recommendation engine can help both researchers and reviewers check the completeness of citations. Existing models for citation recommendation were mostly built on general recommendation models. Such methods usually project context into high dimensional feature vectors without directly modeling the relation between the citation context and the citation. Here, we propose two context-based models which learn the semantic relations between the citation contexts and the cited documents. Both models achieve state-of-the-art recommendation results on the CiteSeerX dataset.

Detecting text in an unconstrained natural scene environment is a challenging task because of the many fonts, sizes, backgrounds, and alignments of the characters. Most existing models for scene text detection focus on small image patches of character areas. However, text in natural scenes is surrounded with informational context which can help locate the wanted text. We present a novel context-based attention model for detecting arbitrary oriented and curved scene text. Combining the model with an off-the-shelf text region proposal method, Extremal Regions, the text detection pipeline achieves the state-of-the-art performance on the ICDAR 2013 dataset and the MSRA Text Detection 500 dataset.

 

Chapter 1 |

Introduction

Context is the information that surrounds and defines the target information it encapsulates. Without context, the most related target information could be misinterpreted. The goal of this dissertation is to develop various machine learning frameworks that model the relations between context and related target. The models proposed in this dissertation tackle two fundamental problems from the field of Natural Language Processing (NLP) and the field of Computer Vision (CV): semantic text matching and scene text detection.

In recent years, there has been a surge of interest in incorporating context information in machine learning models for numerous NLP and CV tasks. However, most existing approaches utilize context information by encoding it as a set of humancrafted heuristic features for general machine learning models. Such approaches may not fully capture the potential connection between the context and the target. We contend that, in the setting of big data, context information should be modeled in a more principled way that is tightly coupled with the learning algorithms. In this dissertation, my research has been focused on two applications with context modeling: (1) citation recommendation for scientific papers, and (2) detecting arbitrary oriented text in natural scenes. I show that, by modeling the context properly, we can achieve better performances in these two applications when compared with other state-of-the-art methods.

1.1 Citation Recommendation

Citations are important in academic dissemination in at least two ways. First, correct citations demonstrate intellectual honesty by giving credit to the work of others; second, proper citations help readers trace the source and evaluate whether the referenced works support authors’ claims. So as to attribute completely the work of previous researchers, authors must be very careful when creating the literature review to avoid missing significant references.

Researchers start to compile a literature review by retrieving relevant documents, based upon a certain pre-selected set of keywords, from search engines such as Google Scholar ^[1] , CiteSeer^X [1] or Microsoft Academic Search^[2] . Then they review the retrieved documents manually to find out works that need to be cited. Afterwards, researchers have to trace down the citation chains from the papers selected from the first step to get a larger set of candidate papers and filter out the irrelevant ones. This two-step process is a hard task for both the junior as well as experienced research practitioners mainly because of the two reasons: (1) the tremendous growth of research articles in the past decade, and (2) introduction of new terminology as the science progresses and new knowledge accumulates in any research field.

Most current literature search engines concentrate on short queries. However, when dealing with long queries ranging from sentences to a paragraph, or even a paper manuscript, keyword-matching based search engines perform far from satisfactorily. Unlike these traditional literature search engines, a citation recommendation system deals with the cases when the users are able to provide a longer query from their paper manuscript. It will automatically suggest a list of candidate references based on the query input. Such a citation recommendation engine can be used to check the completeness of citations while authoring a paper. Researchers can use our citation recommendation engine to find prior works related to the problem they seek to investigate. It can also be used by reviewers to check whether a paper cites all relevant papers. In addition, having a complete list of citations can improve applications that use citations, e.g., to identify the influence of one paper on another, or of an author on another [2] , identify the flow of ideas across authors and documents, find experts, suggest collaborators [3] , etc.

Existing works on citation recommendation can be divided into two categories: non-context based and context based. Non-context based models [4–9] mostly utilize the user profile information, the citation network, or the content of a paper manuscript. These information is usually encoded as feature vectors and input to classic recommendation models such as: collaborative filtering [10] , matrix factorization, network analysis, classification models for citation recommendation. Context-based models [11–14] utilize the citation contexts to recommend which papers to cite. A citation context is defined as a sequence of words that appear around a particular citation. Usually a citation context contains words that describe and summarize the cited papers. Figure 1.1 shows several examples of citation contexts of a same paper. The words in red, such as “PageRank,” “hyperlink,” and “node ranking,” provide information to predict which paper should be cited in these given contexts.

Figure 1.1. An example of a paper gets cited using different citation contexts.

The first part of this dissertation will focus on developing context-based models for citation recommendation. We present context-based machine learning models which learn the semantic relationships between the citation contexts and the cited documents. The translation model based method learns the probability of aligning a cited document with a word/phrase from citation context. Whereas the neural probabilistic model learns to match the semantic embeddings of a citation context and its cited document(s). Both models achieve the state-of-the-art recommendation results on the CiteSeer dataset with the benefit of context modeling.

1.2 Scene Text Detection

Text in natural scenes usually provides important semantic information about the scene and its content. For example, text on the package of a typical food product often specifies the brand, the ingredients, nutrition facts, expiration date, and maybe usages (data that is usually not found in most bar codes). Text on street and shop signs embeds not only navigational information but also semantic information about what is there. A scene text detection system that automatically localize text in the wild will enable numerous multimedia applications such as improving automatic object recognition and image categorization [15,16] , multimedia document indexing and retrieval, assisting the visually challenged and impaired, automatic navigation system, and many robotic vision systems.

Although there have been numerous work on reading text in the wild [17–21] , the problem remains unsolved because of the insufficient accuracy in real-world applications. Especially, localizing text in natural scenes is extremely difficult because of the unconstrained fonts, sizes, backgrounds, inconsistent illumination, and occlusions. In addition, text in the wild is usually captured with different orientations, perspective distortions and curved shapes. Most existing works on text detection have focused on horizontal or near-horizontal texts. While a few works target localizing text with arbitrary orientations, perspectives, and skews [22,23] , these approaches still rely on hand-crafted features or rules for grouping oriented or skewed text.

There are at least two major approaches in text localization. The first is sliding window based methods [17,18,20,21,24] which slide a classifier over the entire image. The second is connected components methods, which group pixels into character regions using local properties such as color, gradient, intensity, stroke-width, etc. Components are group by algorithms such as Stroke Width Transform [22,25] , Extremal Regions [19,26–28] , and Gradient Vector Flow [29] . However, most existing work on text detection has been focused on classifying the text areas in isolation of any surrounding context. Little work has considered detecting text with the present of the surrounding context. We argue that modeling text detection problem with context information is dangerous, especially with the unconstrained background information. Figure 1.2 shows two examples of text in the wild where the surrounding contexts of the image patches play an important role in the classification process. The image patches on the left are usually considered as letters in isolation of the context. However, the same image patches (in red boxes) shown within the surrounding context on the right can be easily recognized as no-text patches.

Figure 1.2. Examples of image patches which could be classified as text patches in isolation (left), while they are no-text patches when presented within context scenes.

The second part of this dissertation will focus on modeling the context information for scene text detection. We define two types of context information in text detection task: the spatial context, and the sequential context. The spatial context is defined as surrounding pixel information of a text area; whereas the sequential context is defined as the sequence of previous spatial context information. The proposed context-based model is inspired by the human visual system. When humans recognize an object, the visual system presents the attention mechanism [30,31] where an objects is detected by focusing on a small area, maintaining a blurry sense of the surrounding context [32,33] , and moving attention to the next relevant parts of the object. We propose to use a glimpse network to extract features not only from the text area, but also the surrounding context. A recurrent neural network is implemented to learn from the sequential context. Experiment on the two most cited text detection dataset: the ICDAR 2013 dataset and the MSRA Text Detection 500 dataset, shows that the context-based attention model is capable of accurately localizing text with orientation and perspective in the natural scenes.

1.3 Contribution and Outline

This dissertation presents several context-based machine learning models that address two practical applications in both NLP and CV areas: (1) citation recommendation for scientific papers, and (2) detecting arbitrary oriented text in natural scenes. The major contributions of this dissertation are :

• In the citation recommendation application, we show that by modeling the context information in a more principled way that is tightly coupled with the learning algorithms, our models achieve significant improvement over other state-of-the-art methods on various performance metrics.
• The proposed context based models for citation recommendation are efficient in both training and recommending stages, which makes it possible to build a scalable citation recommendation system, RefSeer, for public uses.
• In the scene text detection problem, our context-aware attention model provides a natural solution for localizing arbitrary oriented and skewed text. Comparing with other state-of-the-art methods which do not model the context information, our model not only achieves the best detection results, but also generates more accurate bounding boxes for arbitrary oriented text.
• In these two practical applications in both NLP and CV areas, we show that incorporating context information in the learning algorithms can benefit the overall performance.

The outline of this dissertation will be as follows. I will first review the previous works on citation recommendation and scene text detection in Chapter 2. In Chapter 3 and Chapter 4, I will introduce the works on developing the context-based models for citation recommendation. Chapter 3 proposes the work which utilized the statistical machine translation model to bridge the alignment between the citation context words and the cited documents. Chapter 4 further addresses the semantic matching problem between the citation context words and the cited documents with a neural probabilistic model. The neural network model jointly learns the semantic representations of words and documents. The learnt semantic representations not only bridge the semantic relationships between different words, but also model the semantic relationships between context words and cited documents. Chapter 5 presents a citation recommendation system build on top of the proposed two models. We show that such a recommendation system can recommend citations with good quality and it is very efficient and scalable. Chapter 6 aims at building an attention model which utilizes the context information to detect text in the wild.

Chapter 7 concludes this dissertation.

CONTEXT MODELING FOR SEMANTIC TEXT MATCHING AND SCENE TEXT DETECTION

Sharing is caring!