CHARACTERIZING SCIENTIFIC CONTRIBUTIONS THROUGH AUTOMATIC ACKNOWLEDGEMENT INDEXING AND CITATION ANALYSIS

ABSTRACT

Acknowledgements in research publications, like citations, indicate influential contributions to scientific work.  However, acknowledgements are different from citations in an important regard; whereas citations are formal expressions of debt, acknowledgements are arguably more personal, singular, or private expressions of appreciation and contribution.  Furthermore, many institutional sponsors of science expect researchers to acknowledge support that contributed to the completion of published work.  Citation analysis has proved to be an important tool for evaluating research contributions; however, supplementing citation information with acknowledgements provides a more complete picture of communication and influence in science.

This dissertation reports the development of automated methods for acknowledgement identification and analysis in research publications.  The methods were implemented within the CiteSeer Digital Library in order to produce the largest acknowledgement analysis to date by an order of magnitude.  Acknowledgement data is supplemented by CiteSeer’s automatically derived citation index in order to characterize the previously “hidden” impact of acknowledged entities, including funding agencies, corporations, educational institutions, and individuals.  As the analysis of acknowledgements depends upon accurate and up-to-date citation indexing, a nextgeneration citation matching framework is presented which promises to increase the accuracy, precision, and timeliness of automatic citation indices.

 

Chapter 1

 

Introduction, Background, and Problem Statement

1.1 Introduction

Acknowledgements have been given little attention in analyses of scientific communication and contribution.  This holds true for measurements of the contributions individuals have made to the scientific process (i.e., acknowledgements are ignored during tenure decisions) as well as contributions made by organizational bodies such as funding agencies, corporations, and educational institutions.  The assessment of research contributions is instead measured in terms of raw output (or number of publications), the positioning of that output (which journal or conference published the work), and the number of formal citations the work has received from other published materials.  For individual researchers, the number of authored papers accounts for the raw output where organizations measure output as the number of papers authored by employees of the organization or papers generated under the auspices of funded projects.

Authorship and citations have been given special attention in the academic process and formal styles have been developed for the presentation of this information (e.g., APA, IEEE, and ACM styles, to name a few).  Authors of publications are included in all major bibliographic formats and special indices have been created for the storage and retrieval of citations, easing the task of generating impact assessments based on these data.  The reason for such special attention seems clear: authorship and citations, as for all labeled bibliographic data, possess uniform semantics, thus they can be used to generate unambiguous analyses (this is a simplification, but holds true in the practice of assessment).

Acknowledgements are different.  Acknowledgements can express appreciation for any contribution to the research at hand, no matter the type or “size” of the contribution.  From acknowledging the useful ideas of a peer researcher to the patience of a spouse to the guidance of a deity, there are no restrictions on the content or purpose of acknowledgements other than the broad semantics of the word itself.  One may suspect that acknowledgements have been left unadulterated by styles and formalisms precisely because of their open-endedness.  To prescribe a formal ontology for acknowledgements would be to limit their potential expressiveness.

The expressive power of acknowledgements, coupled with their extensive use as a convention, presents an opportunity for uncovering detail about communication and influence in the scientific process much richer than what is possible through analyses of formal authorship and citations.  The lack of a formal semantics of acknowledgements, however, has limited the extent to which they can be practically employed in formal research assessment.  Thus, the open-endedness of acknowledgements is both boon and bane: a boon in terms of raw information content and a bane since acknowledgements have been largely trivialized and ignored due to their ambiguity.

As not all acknowledgements are equal, it is prudent to link acknowledgements to more objective metrics in order to characterize the impact of acknowledged entities.  An obvious choice is to supplement acknowledgement data a traditional metrics for scientific impact, namely citation analysis.  Citations in research publications represent an important knowledge source regarding the context of scientific work.  Since the introduction of the Science Citation Index (Garfield, 1964) citations have been used to measure research impact in terms of authors, publications, and publication venues.  More recently, citations have been used to facilitate information search and retrieval in scientific digital libraries.  Citation relationships have been shown to be valuable for tasks such as ranking search results, identification of related research documents, trend analysis, and social network analysis.

In collections of academic publications, citations represent relationships between documents. These relationships form a data structure generally known as a “citation graph”, where documents are vertices and citations are directed edges between citing and cited documents.  Constructing this graph requires discovering which documents are referenced by individual citations, a task that can be achieved through matching citation and document metadata.  For each citation, the citation text must be parsed to find specific informational items such as authors, title, publication venue, publisher, editors, year of publication, and any other available information.  The parsed metadata can then be used to find documents with the same or similar metadata.  This process is complicated by frequent errors of information extraction as well as errors in the original citation.  Additionally, stylistic variation results in identity uncertainty – for instance, citing a paper in “Proceedings of the Fifth ACM/IEEE Joint Conference on Digital Libraries” or simply “Proc JCDL” may both be acceptable formats.

Large citation indices such as ISI have historically depended upon manual information extraction, requiring human effort to tag and correct information in citations and to facilitate relationship discovery.  This process is time-consuming and expensive such that citation indexing is typically beyond the capability of non-commercial digital research libraries.  In recent years, work has shown that it is possible to handle the task of citation indexing automatically through methods of artificial intelligence.  The CiteSeer Digital Library (Lawrence et al., 1999a) was created in 1997 to demonstrate autonomous citation indexing (ACI), and has since grown to a collection of over 725,000 documents with over 8 million citations.  More recently, Google has released The Google Scholar^[1] , which incorporates ACI to index over 433 million document and citation records^[2] .

ACI represents a challenging automated data management task.  The first step in this process is the extraction of citations from research papers and subsequent parsing of citation subfields to build accurate metadata for each citation.  The problem of citation parsing remains unsolved and the best parsers to date, built using machine learning tools such as hidden Markov models and maximum entropy models, are error-prone and often produce noisy results.  Errors at this level, along with errors in the citation text (such as typos), negatively influence subsequent development of the citation graph.

1.2 Problem Statement and Goals

This body of work aims to explore the information content of acknowledgements in a scale equivalent to standard impact assessment through citation and authorship indices.  This requires the development of automatic techniques for mining acknowledgements from document text in order to facilitate analysis.  Acknowledgement extraction is performed through a combination of artificial intelligence and machinelearning methods for text mining and classification.  The goals of this research are the following:

• Develop technology for automatic acknowledgement indexing. A fundamental algorithm is needed for accurately extracting acknowledgement information from the text of research publications.  This includes locating acknowledgements in labeled acknowledgement sections, document headers, and body text, and extracting the names of acknowledged entities.
• Develop assessment metrics for research impact. In order to characterize the contributions indicated by acknowledgements, it is necessary to develop objective metrics for contributions.  This may be done by supplementing acknowledgement data with traditional citation metrics.
• Create the largest acknowledgement data set yet studied. As a proof of concept for automatic acknowledgement indexing and to create a data set for further analysis, acknowledgements are extracted from a large corpus of computer science documents and linked to an existing automatically derived citation index for assessment.
• Analyze the contributions of the most acknowledged entities in the generated data set. This goal aims to produce rankings of the most influential acknowledged entities, broken down into the categories of entity, such as funding agencies, corporations, educational institutions, and individuals.  In addition, the resulting graphical structures produced by

large-scale acknowledgement analysis are characterized in order to better understand the context of acknowledgement and differences between the acknowledgement contexts according to the entity being acknowledged.

As the primary supplemental metric to acknowledgements in this study is citation data, a novel framework for automatically building citation indices is sought that will alleviate some of the defects of current techniques.  The goals in designing the citation management system are the following:

• Provide better document metadata. Current indexing methods build canonical metadata by using the subfields of the most similar citation to a document from the document’s citation group. This is unsatisfactory since there is no guarantee that the most similar citation contains the best metadata, or even that any citation contains the best representations of all metadata fields.  This problem is addressed by the current work by using a data fusion approach based on Bayesian inference to combine data from citations into belief vectors in the values of document metadata.  Each metadata element in the document record is supported by observations across all citations.
• Reduce the cost of maintenance. Building a citation graph from a large set of citations is typically an offline task that takes several days to complete on low-tier enterprise hardware.  Due to batch clustering, the addition of a single citation requires rebuilding the entire citation graph to include the new instance.  The present work investigates the use of on-line citation matching such that the citation graph environment can be adjusted

immediately based on a single new citation, eliminating the need for expensive batch updates, providing more up-to-date data, and eliminating the need for batch synchronization processes across servers.

• Improve citation matching performance. Finally, a core weakness of current citation graph maintenance systems is addressed – that the process of building canonical metadata for a document cannot influence citation matches to that document in a principled, declarative manner. Once canonical metadata is determined for a document after a batch citation update, that metadata is fixed to the document record and cannot influence the citation clustering, even when citations that matched the old metadata no longer match the new metadata according to the original matching criteria. A goal of this work is to provide a fluid framework for building canonical metadata in which all evidence for the metadata is always considered and easily fetched, and document metadata changes can have immediate impact on citation clustering.

1.3 Relevance and Significance

Since the introduction of the Science Citation Index (Garfield, 1964), researchers, funding agents, promotion and tenure committees, and others have used citation index measures to ascertain the quantity and quality of the impact of articles and authors as well as to explore the topical and social structure of scientific communities (Shiffrin et al.,

2004). However, citations alone can fall short of describing the full network of influence underlying primary scientific communication.  In addition to referencing published material, many researchers choose to document their appreciation of important contributions through acknowledgements.  Acknowledgements may be made for a number of reasons, but often imply significant intellectual debt.  Just as citation indexing proved to be an important tool for evaluating research contributions, acknowledgements can be considered a metric parallel to citations in the academic audit process (Cronin et al., 1993).  Whereas citations are formal expressions of debt, acknowledgements are arguably more personal, singular, or private expressions of appreciation and contribution.

Acknowledgements can be used to develop much more detailed accounts of researchers’ contributions than are possible through authorship and citation counts alone.  The informal dissemination of ideas, methods, and general suggestions for scientific work can cumulate to sizable intellectual contributions and individual acknowledgements of this kind may be considered more direct and valuable contributions than those represented by individual citations (Cronin, 1995).

In addition to analyzing the intellectual contributions of researchers, analysis of acknowledgements indicating financial and instrumental support can give insights into other trends in scientific communities.  For example, acknowledgements of financial support may be used to measure the relative impact of funding agencies and corporate sponsors on scientific research (Cronin and Shaw, 1999; Henderson et al., 2003; Jeschin et al., 1995).  Acknowledgements of instrumental support may be useful for analyzing indirect contributions of research laboratories and universities to research activities.  In short, acknowledgements can help us to better understand the context of scientific research.

1.3.1 Identifying collaborators/facilitators

For social network analysis, the significance of this analysis is simply an extension of the social networks derived from acknowledgements in formal publications.  It is predicted that the data acquired from general research documents will increase the size and complexity of social networks surrounding research activity, providing a richer account of communication and influence.  Comparisons of the social networks derived from general documents with those derived from formal publications will yield information about acknowledgement practices.  If the two networks are quite similar across project domains, increased confidence can be ascribed to acknowledgements in formal publications as yielding a full account of noteworthy contributions.  If the networks differ significantly, potentially important trends in acknowledgement practices may be yielded.  Such information can be used to assess biases in published acknowledgements and possibly to infer causes for phenomena such as network pruning (weeding out certain contributions for publication) and network expansion (inclusion of new contributions during publication).

1.3.2 Providing further measures of impact through dissemination

Scientific work is communicated in a variety of ways in addition to formal communication channels.  The concept of a researcher’s output can be extended beyond authorship on formal published papers and on to general materials publicly available on the web.  If one assumes that the number of informal documents produced under the auspices of a research project correlates to the project’s visibility, the increased communication relating to the project can be important to assessing the impact of the project.  Institutional sponsors of the work will also benefit from the increased visibility and can take this into account in determining the overall return of the project on the provided investment.  Acknowledgements in web-accessible documents can be used to measure the relative web-visibility of projects conducted with the support of specific institutions and provide a fuller view of the penetration of institutions within scientific communities.

1.3.3 Uncovering additional contributions

Perhaps most importantly, analyzing acknowledgements from general webaccessible documents will uncover important contributions to work unpublished in primary scientific communication channels.  Contributions found in formally published work provide only a partial story of influence and communication in science, leaving out contributions to many scientific services and resources.  For example, the DBLP bibliographic service^[3] provides a valuable tool for computer science researchers engaged in scholarly activities, and yet no formal, peer-reviewed publication exists describing the service, leaving no record of the underlying personal and institutional contributions to the project in formal publications.  Likewise, although there are several publications relating to the CiteSeer Digital Library, it is naïve to assume that the contribution of that library and the underlying support network can be fully assessed through formal publications alone.  Institutions and individuals who support the scientific services make valuable contributions to scientific communication through the services provided.  This condition will hold true for any scientific service or educational program, and identifying acknowledgements of support in web-accessible documents will provide more complete assessments of scientific impact.

1.3.4 First Automatic Acknowledgement Indexing Effort

This work represents the first effort of its kind and is thus an exploratory study.  Graphical analysis of acknowledgements and manual review of the acknowledgement extraction results may uncover further trends not previously predicted.  Just as Cronin’s work on acknowledgements uncovered important trends exploited in this and other research, this study will extend the basic research performed on acknowledgements.

1.4 Definition of Terms

The following terms will be used throughout this dissertation to refer to acknowledgements and related information.

1.4.1 Acknowledgement Types

Here, a general definition of ‘acknowledgement’ is provided along with definitions for various types of acknowledgements, loosely adhering to Cronin’s typology described in Chapter 2.

 

Acknowledgement: any published referral to a formal or informal contribution made by some entity to the enclosing body of work, excluding authorship and formal citations to published work.

Peer Interactive Communication: the contribution of intellectual support to a work, e.g., research ideas, suggestion of directions for exploration, or specific suggestions of research and analysis methods to employ.

Financial Support: the provision of monetary resources to a project.

Technical Support: any provision of technical skills or data resources to a project, e.g., providing prepared data, programming, application assistance, or system administration.

Instrumental Support : the contribution of materials to a project including space, computers, or library resources.

Moral Support: any emotional or spiritual support provided to an author of a work, e.g. the patience of a spouse or the guidance of a deity.

Clerical Support: any clerical contribution to a work including, e.g., proofreading or typesetting.

1.4.2 Entity Types

In addition to the various types of acknowledgements, acknowledged entities can be broken down into several categories to facilitate analyses, described below.

 

Acknowledged Entity: any institution, organization, person, or thing acknowledged as having had some impact on the acknowledging work.

Funding Agency: an institution whose primary mission includes the provision of direct monetary support to research initiatives.

Corporation: an institution whose primary mission is to generate revenue through the sales of products or services.

Educational Institution: an institution whose primary mission is to train students, including those institutions that primarily emphasize research but employ students for the assistance of research.

Individual: a person, whether an author, support staff, friend, spouse, or other.

1.5 Overview of the Research Approach

This section summarizes the research that was conducted as well as the research methods and why the approach that was taken is appropriate for the research problem.  Chapter 3 presents an in-depth discussion of the research techniques for acknowledgement indexing; likewise, Chapter 5 discusses techniques for citation matching.

The approach that was chosen for acknowledgement analysis was to build an automatic algorithm for mining the text of research publications in order to extract the names of acknowledged entities.  The construction of a hybrid system based on both machine learning and rule-based language parsing was chosen.  In this way, a declarative rule base is use to exploit regularities in acknowledgement language and labeling while a machine learning tool (in this case, a Support Vector Machine) was trained to find acknowledgements where the rule base proved inadequate.  The system is shown to be highly effective at locating and extracting acknowledgement data in cases where acknowledgements are labeled or where unlabeled acknowledgements exist within text headers or the document body.

Chapter 4 presents an analysis of extracted acknowledgement data from the CiteSeer corpus.  The most acknowledged entities are presented and their impact within the data set is measured.  Acknowledgement trends are explored, including temporal effects and collaborations (whether intended or not) among funding agencies.  Visualizations are presented that characterize acknowledgement network structure and show differences in the areas of the CiteSeer citation and co-authorship networks on which funding agencies make their impact.

The method for citation matching is designed to bridge the clustering and inference properties of traditional batch citation clustering methods with the speed and manageability of one-vs.-one comparisons between citation strings.  New citations and documents can be linked immediately to existing document and citation targets without the need for expensive batch clustering of the entire data set.  At the same time, inference networks inspired by Bayesian methods learn the “correct” metadata for documents and citations over time in order to improve matching performance and dynamically repair previous incorrect matches.  In addition, the system is designed to accept document links from metadata in external databases and user-supplied corrections, but is made robust to corrupt or malicious input by automatically spotting unlikely metadata values.

[1] http://scholar.google.com

[2] This number was discovered by searching for “+the” in the Google Scholar search engine.

[3] http://www.informatik.uni-trier.de/~ley/dblp

CHARACTERIZING SCIENTIFIC CONTRIBUTIONS THROUGH AUTOMATIC ACKNOWLEDGEMENT INDEXING AND CITATION ANALYSIS

Sharing is caring!