- : Format
- : Pages
- :
- : Chapters
- Click to DOWNLOAD Materials
AN ARCHITECTURE FOR MULTIMODAL INFORMATION EXTRACTION FROM SCHOLARLY DOCUMENTS
Abstract
A scholarly paper (journal article, conference proceeding) has both unstructured(text) and semi-structured data sources (tables and figures). An experimental figure such as a line graph is generated from a data table that stores the results of an experiment. Typically that data table is not reported in the paper, hence can not be queried directly. Similarly, a scholarly table reports the results of an experiment but is not structured enough to support anything more than a keyword query.
This dissertation has two contributions. First, we show methods to reduce these semi-structured data sources to structured content that can support factoid queries such as “What is the best precision for Image net classification task?” or “What is the best BLEU score for English to Arabic translation?”
For the scholarly figures, we report an end to end system. First, we report a batch extractor to extract all figures (including vector graphics) and associated metadata from a document with 81% and 87% accuracy. Next, we report image processing algorithms to detect compound figures with 82% accuracy and classify non-compound figures as line graphs or bar charts with 84% average accuracy. We improve the accuracy for text extraction from raster graphics by 39% and show algorithms to classify the text inside the plots with an average accuracy of 90%. The majority of figures in computer science papers are embedded as vector graphics. While previous work has always extracted them as raster graphics, we show methods to extract them in a vector graphics format, which allows us to scalably separate curves in line graphs with 75% average accuracy. This reduces a line graph to the original data points from which it was generated, allowing the factoid queries. We report a similar architecture for scholarly tables that can reduce the tables to data based triples supporting similar queries.
Finally, we show supervised methods to extract scholarly entities from the text of the paper. Specifically, we show that a non-sequential classifier learning the informativeness of a phrase globally and a sequential classifier learning the same utilizing the local context can be combined to improve the accuracy of the process.
Table of Contents
List of Figures viii
List of Tables x
Acknowledgments xii
Chapter 1
Introduction 1
1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Knowledge Extraction from Scholarly Figures . . . . . . . . . . . . 1
1.3 Knowledge Extraction from Scholarly Tables . . . . . . . . . . . . . 5
1.4 Extraction of Scientific Domain Knowledge Entities . . . . . . . . . 6
Chapter 2
Related Work 7
2.1 Developing an Architecture for Analysis of Scholarly Plots . . . . . 7
2.1.1 Extraction of Figure Metadata . . . . . . . . . . . . . . . . . 7
2.1.2 Extraction of Figures from PDF Documents . . . . . . . . . 8
2.1.3 Raster Image Processing for Scholarly Plot Classification
and Text Extraction . . . . . . . . . . . . . . . . . . . . . . 10
2.1.4 Data Extraction from Line Graphs . . . . . . . . . . . . . . 11
2.2 Knowledge Extraction from Scholarly Tables . . . . . . . . . . . . . 13
2.2.1 Extraction & Semantics of Scholarly Tables . . . . . . . . . 13
2.2.2 Semantics of Web Tables . . . . . . . . . . . . . . . . . . . . 14
2.3 Extraction of Scholarly Entities . . . . . . . . . . . . . . . . . . . . 15
Chapter 3
Extraction of Figures and Associated Metadata from PDF Doc-
uments 17
3.1 A Rule Based System for Extraction of Figures and Metadata . . . 18
3.1.1 Description of the System . . . . . . . . . . . . . . . . . . . 18
3.1.1.1Preprocessing . . . . . . . . . . . . . . . . . . . . . 18
3.1.1.2Determination of figure id . . . . . . . . . . . . . . 19
3.1.1.3Caption extraction . . . . . . . . . . . . . . . . . . 19
3.1.1.4Identifying mentions . . . . . . . . . . . . . . . . . 21
3.1.2 Experiments and Results . . . . . . . . . . . . . . . . . . . . 21
3.1.2.1Dataset . . . . . . . . . . . . . . . . . . . . . . . . 21
3.1.2.2Experiment design . . . . . . . . . . . . . . . . . . 21
3.1.2.3Results. . . . . . . . . . . . . . . . . . . . . . . . 22
3.1.3 Discussion and Limitations . . . . . . . . . . . . . . . . . . . 22
3.2 Lexical features for Caption Extraction . . . . . . . . . . . . . . . . 23
3.2.0.1Features for caption beginning line identification . 23
3.2.0.2Features for caption ending line identification . . . 24
3.2.1 Experiments and Results . . . . . . . . . . . . . . . . . . . . 25
3.2.1.1Dataset . . . . . . . . . . . . . . . . . . . . . . . . 25
3.2.1.2Experiment design . . . . . . . . . . . . . . . . . . 25
3.2.1.3Results. . . . . . . . . . . . . . . . . . . . . . . . 26
3.3 Extraction of Figures from PDF Documents . . . . . . . . . . . . . 27
3.3.1 Image Processing Approach . . . . . . . . . . . . . . . . . . 27
3.3.2 Improving Text/Graphics Segmentation . . . . . . . . . . . 28
3.3.2.1Problem Description and Dataset . . . . . . . . . . 28
3.3.2.2A K-means Based Approach . . . . . . . . . . . . . 29
3.3.2.3Results and Discussions . . . . . . . . . . . . . . . 33
3.3.2.4Limitations . . . . . . . . . . . . . . . . . . . . . . 34
3.3.3 Using PDF Object Model for Figure Extraction . . . . . . . 34
3.3.3.1Classification of Raster Graphics and Paths . . . . 36
3.3.3.2Combining Paths into Figure Regions. . . . . . . 38
3.3.3.3Dataset . . . . . . . . . . . . . . . . . . . . . . . . 39
3.3.3.4Evaluation . . . . . . . . . . . . . . . . . . . . . . 39
3.3.3.5Classification: Experiments and Results . . . . . . 41
3.3.3.6Clustering: Experiments and Results . . . . . . . . 42
3.3.3.7Error Analysis . . . . . . . . . . . . . . . . . . . . 43
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Chapter 4
Image Processing for Scholarly Figures 46
4.1 Bag-of-Features Representation for Scholarly Plot Classification . . 47
4.1.1 Compound Figure Detection . . . . . . . . . . . . . . . . . . 48
4.1.2 Line Graph and Bar Chart Detection . . . . . . . . . . . . . 49
4.1.2.1Codebook Generation. . . . . . . . . . . . . . . . 49
4.1.2.2Feature Extraction and Scalability Improvements . 49
4.1.3 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.1.4 Experiments: Compound Figure Detection . . . . . . . . . . 51
4.1.5 Experiments: Classification of Non-Compound Figures . . . 52
4.1.6 Text Extraction from Raster Scholarly Plots . . . . . . . . . 53
4.1.6.1Experiments and Results . . . . . . . . . . . . . . . 55
4.1.7 Word Classification . . . . . . . . . . . . . . . . . . . . . . . 56
4.1.8 Heuristic Algorithm . . . . . . . . . . . . . . . . . . . . . . . 57
4.1.9 Machine Learning Algorithm . . . . . . . . . . . . . . . . . . 58
4.1.10 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.2 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Chapter 5
Vector Graphics Processing for Curve Extraction from Line
Graphs 61
5.1 Dataset Description . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
5.2 Analysis of Line Graphs for Data Extraction . . . . . . . . . . . . . 66
5.3 Description of the SVG Format . . . . . . . . . . . . . . . . . . . . 68
5.4 A Parser for Producing Atomic SVGs . . . . . . . . . . . . . . . . . 69
5.4.1 Parsing Path Commands . . . . . . . . . . . . . . . . . . . . 69
5.4.2 Text and Image Commands . . . . . . . . . . . . . . . . . . 71
5.4.3 Transform Commands . . . . . . . . . . . . . . . . . . . . . 72
5.4.4 Development . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5.5 Curve Extraction from Atomic SVGs . . . . . . . . . . . . . . . . . 72
5.6 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
5.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Chapter 6
Knowledge Extraction from
Scholarly Tables 80
6.1 Well Formed Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6.2 Algorithms to Extract n-Tuples from WFTs . . . . . . . . . . . . . 84
6.2.1 Combining Words in Table Cells . . . . . . . . . . . . . . . 84
6.2.2 Identifying the Table Substructure . . . . . . . . . . . . . . 86
6.2.3 Detecting the Row-Header/ Column-Header/ Data Cells . . 89
6.2.4 Creating n-tuples for the Data Cells . . . . . . . . . . . . . . 89
6.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
6.3.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
6.3.2 Experiments and Results . . . . . . . . . . . . . . . . . . . . 92
6.3.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
6.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
Chapter 7 A Hybrid Approach to Extract Scientific Domain Knowledge
Entities 96
7.1 Scholarly Key Phrases & Scientific Domain Knowledge Entities . . . 96
7.2 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
7.2.1 EKE: NP-Chunking Based Extractor . . . . . . . . . . . . . 98
7.2.2 Supervised EKE . . . . . . . . . . . . . . . . . . . . . . . . 99
7.2.3 Sequential Labeling of SDKEs with CRF . . . . . . . . . . . 100
7.2.4 Combining CRF and Supervised EKE . . . . . . . . . . . . 101
7.3 Experiments & Results . . . . . . . . . . . . . . . . . . . . . . . . . 101
7.3.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
7.3.2 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
7.3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
7.3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
7.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
Chapter 8
Conclusion & Future Work 105
Bibliography 108
| regions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 40 | |
| 3.7 | An example where some instances of the positive class (green) are classified as negative class (red). However, that doesn’t change the | |
| clustering quality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 41 | |
| 3.8 | Sample error cases from our dataset. . . . . . . . . . . . . . . . . . | 44 |
| 4.1 | Comparative results for text extraction: our method and Kataria et | |
| al. [58] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 56 | |
| 5.1 | The left sub-figure shows a color line graph extracted from Hasan et al. [52] in a vector graphics format (SVG). The middle sub-figures show two overlapping curves extracted perfectly. The right sub-figure showstwo SVG commands painting the curves at an overlapping | |
| point. In a raster graphics, we just have one pixel instead. . . . . . | 62 | |
| 5.2 | Chronological statistics for vector graphics. . . . . . . . . . . . . . . | 66 |
List of Figures
| 1.1 System architecture. . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Metadata generation for scholarly figures. Each step of our architec- |
3 |
| ture generates a metadata richer than the last one.. . . . . . . . . | 5 |
| 3.1 Manhattan distance between two bounding boxes. . . . . . . . . . .
3.2 Example situations where the outputs of Leptonica need/ need not |
30 |
| to be processed further. . . . . . . . . . . . . . . . . . . . . . . . . . | 31 |
| 3.3 A state diagram showing how graphics is drawn in PDF documents.
3.4 A page from a PDF document, with red lines showing the bounding boxes of vector elements. The boxes are always axes parallel. They often have very small width/height, therefore are rendered as straight |
36 |
| lines and not rectangles in this figure. . . . . . . . . . . . . . . . . . | 37 |
3.5 Steps in figure extraction from PDF pages, using PDF object model. 38
3.6 Distribution of input points and number of clusters for test data.
Input points are the paths/raster graphics. Clusters are the figure
| 5.3 | Some examples of perfect curve extraction. . . . . . . . . . . . . . . | 77 |
| 5.4 | Some examples of partial curve extraction. . . . . . . . . . . . . . . | 78 |
| 5.5 | Some examples where the algorithm fails to extract any curve com- | |
| pletely correctly. . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 79 | |
| 6.1 | A table showing values for the relationships “population”, “land area” and “population density” for the entities Manhattan, Bronx, | |
| Brooklyn, Queens and Staten Island. Collected from Wikipedia. . . | 81 | |
| 6.2 | A table extracted from Birch et al. [4] . It shows the BLEU scores | |
| and the phrase table sizes for a Spanish-English translation task. . . | 82 | |
| 6.3 | A table extracted from Perez et al. [73] . It also shows the BLEU | |
| scores for a Spanish-English translation task. . . . . . . . . . . . . . | 83 | |
| 6.4 | The system architecture. The top-left figure shows the original table. The words are then converted into cells (top-right). The rownumber and column-number for the cells are identified (bottom-left). Based on that, the cells are classified as column-header (brown), row-header (red) and data (blue) cells. The lower panel shows the n-tuples generated from the table by identifying the row-header and | |
| column-header paths. . . . . . . . . . . . . . . . . . . . . . . . . . . | 85 | |
| 6.5 | The left subfigure shows the row and column numbers we want to produce for each cell. The right sub-figure shows the algorithm | |
| output. The left sub-figure can be easily obtained from the right one. | 87 | |
| 6.6 | DAG representation of a table for column-number identification. Each node is a table cell, there exists an edge X → Y if X is to the left of Y. The red edges show two possible cases where we might not | |
| add an edge even though we should. . . . . . . . . . . . . . . . . . . | 88 | |
| 6.7 | DAG representation of a table for row-number identification. . . . . | 88 |
| 6.8 | Two tables with multi-level column-headers and row-headers. The | |
| critical cells are identified. . . . . . . . . . . . . . . . . . . . . . . . | 90 |
List of Tables
| 3.1 | Comparative results of extraction efficiency of our system (Epdbx) with another [65] (EXpdf). Row 1 reports fraction of (figure, caption) pairs that were retrieved from the dataset. Row 2 reports fraction of retrieved captions that were found correct. Row 3 reports fraction | |
| of correct captions that were extracted with ≥ 95% accuracy. . . . . | 22 | |
| 3.2 | Accuracy, sensitivity and specificity results are shown for two binary classification problems. Column 1 results correspond to the problem of caption beginning line identification and column 2 results | |
| correspond to the problem of caption ending line identification . . . | 27 | |
| 3.3 | Results for clustering of bounding boxes for different distance functions. Euclidean-center ( sum of euclidean distance between central points of bounding boxes in a cluster and the cluster center) | |
| is clearly better than others. . . . . . . . . . . . . . . . . . . . . . . | 33 | |
| 3.4 | Classification results for used classifiers. . . . . . . . . . . . . . . . . | 42 |
| 3.5 | Figure-precision,recall and F1-scores on test data. . . . . . . . . . . | 43 |
| 4.1 | Feature performance (5-fold cross validation accuracy) for compound | |
| figure detection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 52 | |
| 4.2 | Accuracy results for computer generated charts classification: random patch based unsupervised feature learning outperforms low-level | |
| image descriptors such as HoG and SIFT. . . . . . . . . . . . . . . | 53 | |
| 4.3 | Figure classification results for multiple feature extraction methods | |
| in unsupervised feature learning using a Random Forest classifier. . | 53 | |
| 4.4 | Heuristics for the classification of words in the figures. . . . . . . . . | 58 |
| 4.5 | Precision, Recall and F1-scores for the classification of words in the | |
| figures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 60 | |
| 5.1 | Curve Extraction Results: A.{P, R, F1}: Absolute precision, recall | |
| and F1 scores; R.{P, R, F1}: Relative precision, recall and F1 scores. | 75 | |
| 6.1 Statistics for the table dataset . . . . . . . . . . . . . . . . . . . . . | 92 | |
| 6.2 Precision, recall and F1-score for table cell extraction . . . . . . . . | 93 | |
| 6.3 Precision, recall and F1-score for row-header extraction . . . . . . . | 93 | |
| 6.4 Precision, recall and F1-score for column-header extraction . . . . . | 94 | |
| 6.5 Precision, recall and F1-score for data cell extraction . . . . . . . . | 94 | |
7.1 Comparison of HESKD against baselines. . . . . . . . . . . . . . . . 103
Chapter 1 |
Introduction
1.1 Overview
A scholarly paper (journal article, conference proceeding) has both unstructured (text) and semi-structured data sources (tables and figures). In this dissertation, we have two contributions: 1. We show supervised methods to extract scholarly entities from natural text and 2. We show methods to utilize these semi-structured sources with the goal of building a better information retrieval system that can support factoid scholarly queries such as “What is the average precision of Support Vector Machine on Imagenet data ” or “What is the highest BLEU score for English to Arabic machine translation”. Specifically, we report an architecture to extract, classify and reverse-engineer scholarly plots to the data tables from which they were generated. We report a similar architecture to reduce scholarly tables to data based triples. These structured data sources can support informative queries.
1.2 Knowledge Extraction from Scholarly Figures
Scholarly papers usually contain multiple figures or diagrams. These figures represent important findings and experimental results. Therefore, they are of great interest to the academic community and are rich resources of information [17] . These figures can be analyzed for multiple purposes, such as 1. Extraction of data, 2. Understanding the intended message of the figure or 3. Designing an information retrieval system focused on better retrieval of such figures. Researchers in document analysis and computer vision community have previously explored extraction of data from figures (specifically, 2d scatter plots and line graphs). Researchers in linguistic and information retrieval community have analyzed captions in images collected from the web to understand the intended message. However, these research activities have not yet converged. A complete workflow for such analysis especially focused on figures in academic documents, is not available.
We propose a complete architecture (figure 1.1) for data extraction from scholarly figures. The motivation of the architecture is to solve two problems. The first problem is of reproducibility. Consider the scenario where two researchers are working on the same problem: unsupervised key phrase extraction from documents. The first researcher reported the results of the methods he developed in the form of a precision-recall graph. This graph was generated from some data table, but the table was not reported in the paper. The second researcher develops some other methods for the same problem on the same dataset. To compare his methods, he would have to redo the experiments from the first researcher. Instead, if we could reproduce the data table from the first researcher’s paper, the second one can use that to compare his method against others. This would naturally save the time required to reproduce the previous work.
Another motivation for the work is to improve scholarly search. In scholarly documents, most researchers look at the figures first. However, very few search engines (except a few recent ones such as Viziomatrix and Semantic Scholar) allow users to search on the figures itself. Most of these search engines allow users to search on the caption and the mention of a figure. Consider a search intent where a researcher is trying to find figures that show “Support Vector Machine has higher
average precision than Random Forest”. To satisfy this query, the system must generate metadata for a figure with the following information:
Precision is mentioned as an evaluation metric.
Random Forest and Support Vector Machine are mentioned as methods.
Average precision for Random Forest is actually higher than Support Vector Machine.
Only the figures reporting some experimental results (typically, line graphs, bar charts, pie charts) can be processed to extract this information. Such experimental figures are generated from data tables. To answer the query, they would have
| Input PDF |
| Figure, |
| caption, |
| mention , |
| words |
| extraction |
| Raster |
| graphics |
| Vector graphics |
| > |
| Caption |
| < |
| > |
| Mention |
| < |
| Metadata 1 |
| Compound |
| figure |
| detection |
| single figure |
| Line graphs, |
| Bar Graphs, |
| Others |
| Classification |
| > |
| Caption |
| < |
| > |
| Mention |
| < |
| > |
| Words |
| < |
| Compound |
| figure |
| Metadata 2 |
| Others |
| Line and bar |
| Figure Text |
| Classification |
| axes value, |
| legends |
| etc. |
| < |
| Caption |
| > |
| < |
| Mention |
| > |
| Words |
| > |
| < |
| < |
| Word class |
| labels> |
| Bar Graphs |
| Metadata 3 |
| Line graphs |
| Curve Separation, |
| Curve Legend Association, |
| Natural language |
| summary |
| < |
| Caption |
| > |
| < |
| Mention |
| > |
| < |
| Words |
| > |
| < |
| Word class |
| labels> |
| < |
| Summary for |
| curves> |
| Metadata 4 |
Figure 1.1: System architecture.
to reduced to the same. It is extremely hard to create an end-to-end system supporting these queries on all experimental figures with reasonable accuracy and high throughput.
In this dissertation, we show that it is indeed possible to design such a system with acceptable accuracy and throughput, albeit on a smaller subset of figures. Figures are embedded in scholarly documents as raster graphics (PNG/JPEG) or vector graphics (PS/EPS/SVG). A surprising finding of this dissertation is that more than 70% of all figures in computer science (even higher percentage in physics and mathematics) are actually embedded as vector graphics and it can be extremely beneficial to extract and process them as such. While there has been plenty of work before on scholarly plot processing, this observation was largely unnoticed.
Our modular architecture generates metadata for the figures in increasing order of richness (see figure 1.2). The input to the system is a scholarly document in PDF format. For any figure in the document, we generate the caption, mention, and image file for the figure, which we will refer to as metadata 1. While this metadata will fail to provide an exact answer to the query mentioned before, it will be at least be able to show some relevant figures. All batch figure extractors prior to our work extracted only raster graphics from PDF documents because it is hard to extract vector graphics from them (see chapter 3), therefore had a very poor recall on computer science papers. The first contribution of this dissertation is to design a batch extractor for vector graphics (and associated metadata) that predicts the bounding box of a figure region on a PDF page. The page is then rasterized and the necessary regions are cropped out [25–27,75] .
While it is hard to extract the vector graphics, they are easier to process downstream. For example, the text inside these figures can be extracted accurately and scalably without any optical character recognition. Therefore, for these figures, we generate the information in metadata 1 and the text inside the figure. We will refer to this improved metadata as metadata 2, which can support the last query on the caption, mention and the figure text.
Line graphs and bar charts typically report experimental results. For these figures, a classification scheme exists for the text inside the figure: X and Y axis values, X and Y axis labels, figure label, legends and other text. Consider a typical experiment, specifically, in applied computer science whose results are reported in a line graph or a bar chart. Each curve or bar reports the result for a particular method and the name of the method can be found in the legend section. Axes labels denote the evaluation metrics. The dataset can be reported in the figure label, caption, or mention. Therefore, for these figures, we generate metadata 3 with all information in metadata 2 and the class labels for the text regions inside the figure. This metadata can provide a more accurate result to the last query than metadata 2. Also, the class labels are required for reducing the curves in line graphs to data points, which is the next step in our architecture.
The final module in our figure analysis architecture focuses on the line graphs only because they are the most prevalent among the experimental figures. Specifically, we intend to reduce each curve in a line graph to the original data values from which they were generated. This final metadata can support the data based queries such as “Show the figures where Support Vector Machine has higher average precision than Random Forest on ImageNet classification task”, which is the final level of a navigational query we intend to support.
The generic algorithm for a curve to data point conversion is simple: if we can identify two X-axis values and two Y-axis values, every point in the plotting region can be reduced to a “data point”. The hardest task is to separate the curves, especially when they overlap. Previous work in this problem used only raster graphics and either made simplifying assumptions [66] or reported results on synthetic data [11] , which is expected given prior to 2015 no batch extractor for vector graphics was available. Another major contribution of the dissertation is to show that if a vector figure is embedded in a PDF, we can extract it in another vector graphics format. Not only it is more scalable than PDF rasterization, it helps to extract the curves more accurately. We discuss the algorithms in 5 of the dissertation.
To summarize, we designed scalable and accurate algorithms for the following
Figure 1.2: Metadata generation for scholarly figures. Each step of our architecture generates a metadata richer than the last one.
problems:
Extraction of figures (as image files) and associated metadata (figure captions, mentions) from documents (chapter 3).
Extraction of text from raster graphics (chapter 4).
Classification of scholarly plots: binary classification such as compound/ noncompound or multi-class classification such as line graphs, bar charts, or others (chapter 4).
Classification of text inside line graphs and bar charts (chapter 4).
Curve extraction from line graphs embedded as vector graphics (chapter 5).
1.3 Knowledge Extraction from Scholarly Tables
In the first four chapters of this dissertation, we report algorithms to reduce line graphs to the data values from which they are generated. Scholarly tables are also important data sources: many experimental results are reported in tables. In this chapter, we ask the question, how to reduce a table to a more structured format so that we can support the factoid queries. We show that it is possible to do so by reducing a table to a set of triples of the form <“row-header path”,“column-header path”,“data cell”> such as <“Support Vector Machine”,“Precision”,“25.49”>). In chapter 5 of this dissertation, we report unsupervised algorithms for this problem.
1.4 Extraction of Scientific Domain Knowledge Entities
In the final chapter of this dissertation (chapter 6) we investigate a variant of the problem of automatic keyphrase extraction from scientific documents which we define as Scientific Domain Knowledge Entity (SDKE) extraction. Keyphrases are noun phrases important to the documents themselves. In contrast, an SDKE is a text that refers to a concept and can be classified as a process, material, task, dataset etc. An SDKE represents domain knowledge but is not necessarily important to the document it is in. Supervised keyphrase extraction algorithms using non-sequential classifiers and global measures of informativeness (PMI, tf-idf) have been used for this task. Another approach is to use sequential labeling algorithms with local context from a sentence, as done in the named entity recognition. We show that these two methods can complement each other and a simple merging can improve the extraction accuracy by 5-7 percentiles. We further propose several heuristics to improve the extraction accuracy. Our preliminary experiments suggest that it is possible to improve the accuracy of the sequential learner itself by utilizing the predictions of the non-sequential model.
AN ARCHITECTURE FOR MULTIMODAL INFORMATION EXTRACTION FROM SCHOLARLY DOCUMENTS
