INCREMENTAL PLANNING AND BUFFERING IN LANGUAGE PRODUCTION: MODELING LARGE-SCALE CORPUS DATA

Abstract

This dissertation sheds light on some of the representations used in human language production. Language production consists of several distinct stages; in particular, we focus on the stages of lexical retrieval and grammatical encoding. Lexical retrieval involves retrieving the words required to express the intended meaning; grammatical encoding concerns how to combine those words into an utterance. Here, we present studies relating declarative memory to working memory and buffering in fluent language production.

In particular, both this dissertation and previous work provide evidence that some amount of language processing and planning is non-incremental. However, these subtle effects do not outweigh that grammatical encoding and buffering largely proceed incrementally. Indeed, there is no clear concrete strategy that can better explain the data than a strictly incremental one. This is likely because the type of non-incremental buffering we see is not something particularly explainable by a discrete strategy; instead, it is a complicated and nuanced strategy that depends on many external factors, including non-linguistic processing, such as semantic retrieval or reasoning.

We build a series of models to investigate buffering in language production. First, a model of word adoption lays the groundwork for our computation of lexical retrieval from declarative memory. Next, we extend this to discuss a model of grammatical encoding that relies on Combinatory Categorial Grammar (CCG). The third study also builds on the model of declarative memory and extends it with a model of lexical retrieval that occurs across small time-scales. These three initial studies establish a modeling methodology that will inform the next two studies, which attempt to model lexical retrieval strategies from a discrete perspective and a differentiable one. In the end, our studies converge in arguing for mostly incremental planning and buffering of up to five words.

Chapter 1 | Plausible and Accountable Models of Language Production

1.1 Introduction

Models in the field of cognitive science and artificial intelligence have increasingly diverged, as the representations and tasks used to evaluate them have separated. However, while these models have different benefits, both fields are principally concerned with modeling intelligence. The data sets used for evaluation are also different; in cognitive science, close fits to experimental data is seen as validating, while in artificial intelligence, successful models are those that perform some task the best. However, cognitive modeling started with the idea that models should be able to do the tasks that they claim to describe (e.g., Newell, 1990). Meanwhile, interpretability aside, machine learning models often make no claims of plausibility, focusing purely on performance.

This dissertation seeks to make cognitively plausible models of language that are evaluated on big data, similar to other computational models. In particular, I focus on explaining working memory and buffering from the perspective of language production. This dissertation seeks to answer questions about which words can be buffered, how many words are buffered, and which words are usually buffered.

This dissertation is interested in representations and models of language production at various levels. In particular, it is concerned with lexical and syntactic representations. Different levels of representation allow different questions to be asked and different metrics to be used. These representations can include grammar formalisms, models of declarative memory, word embeddings, and different methods to model working memory. The ultimate goal, however, is to provide converging evidence with computational modeling for cognitive phenomena: in this case, buffering during language production.

The goal of this dissertation is then two-fold: to use and introduce methods concerning the evaluation of cognitive models of language on big data, and to use such methods to answer specific scientific questions. In particular, we ask questions about buffering in language production. How far ahead can humans buffer during language production? How incremental is their buffering? How much linguistic material do humans generally buffer? We propose models that can investigate these questions.

Some of the work in this dissertation builds the necessary machinery to investigate these questions. Then, the core of the dissertation investigates these topics from several angles. Finally, the dissertation explores models that show how computational methods can be unified with cognitive models even further.

1.2 Data and Representations for Cognitive Models

As previously stated, computational models are often either model (big) naturalistic data or model (small) experimental data. Models of experimental data often attempt to model a data set that has been presented to participants during an experiment. Then, the model is evaluated by replicating the results of the experiment by creating a model of how the processing may have proceeded (e.g., Lewis and Vasishth, 2005). Alternatively, it could rely on using examples of corner-cases to replicate wellknown effects (e.g., Reitter et al., 2011b). However, in both cases the data and tasks are somewhat artificial, they are created by experimenters to examine some phenomenon. The artificial nature of the task does not invalidate the data; however, intention and attention could play a role in the type of processing used.

Cognitive models attempting to model these experimental tasks frequently use a system like ACT-R (Anderson et al., 2004). ACT-R attempts to model the task at a high-level, providing specific cognitive operations that are designed to map to possible operations in the basal ganglia. ACT-R has both symbolic and subsymbolic components; however, models in canonical ACT-R have only symbolic representations of actual memory contents, while the subsymbolic components only affect the relationship among those components. For some tasks, the ’procedural rules’ of ACT-R may be too high-level to accurately capture the process.

Conversely, machine learning models train on datasets that are more emblematic of daily life, such as simple images. The tasks performed by such models are simpler and more closely resemble ordinary processing, rather than tasks that would evoke individual differences or other types of edge cases. Then, in some sense, they are evaluated on how well they match human performance data; at least, how well they can do tasks that humans can generally do. In the language domain, for instance, a language model attempts to learn which words can go after other words. Recently, neural models generally use distributed, vector-based representations with end to end training. In some sense, they do not intend to be a model of processing, because they make decisions without clear insight into why that decision was made.

Nonetheless, such models achieve state of the art performance on tasks. To focus on language, there are some reasons to believe that some processes of language are governed by statistical mechanisms. For instance, babies response to language varies proportionally with their exposure to that language (e.g., Kuhl, 2010), and readers difficulty in resolving a word’s meaning varies proportionally with that word’s usage (e.g., Rayner and Frazier, 1989). While language processing being governed by statistics does not guarantee it is governed by any particular statistical formalism, one could argue that a closer fit to data is a possible evaluation mechanism for cognitive models as well as models created for the purposes of engineering.

It is not hard to imagine that every methodology has its limits in how precisely it can recreate intelligence. Newer versions of a state of the art model are not always guaranteed to outperform newer methodologies altogether. In some sense, this search for the best type of model is in of itself a non-convex optimization, and cognitive plausibility is one possible tool to avoid local minima.

All of the models in this dissertation use a model of memory specified by ACT-R

(Anderson et al., 2004). All of them are created with language corpora: either Reddit or Switchboard. Traditionally, task performance on corpus data is central to evaluating machine learning models, while cognitive models seek to reproduce experimental effects on smaller data. The models in this dissertation come from corpora but are not evaluated on tasks, instead trying to mirror human performance. Our evaluation metrics include word usage frequencies, edit distance, timing data, and perplexity.

1.3 Models of Memory

Memory is frequently divided into two categories, declarative and procedural. It is also frequently divided into categories like working memory and long-term memory. This dissertation does not take a standpoint on memory in general; however, it does make use of some constructs from ACT-R. In particular, the idea of activation is a well-validated construct that consist of a few different effects, that will be explained in detail later. The important point of activation is that activation is supposed to be a single quantity that determines the entirety of how easy something is to retrieve from declarative memory. Anderson (1983) provides the equations that will be used throughout this dissertation to measure activation. Chapter 1 focuses on formalizing the concept of ACT-R declarative memory for language use and applies it in the context of word adoption.

This dissertation makes most of its contribution in processing models of working memory. In particular, it offers several novel models of various data of language production that come to converging conclusions. The commentary on working memory throughout this dissertation, however, is really focused on the idea of buffering for language production. Working memory in general can be conceptualized a few different ways; for the most part, this dissertation conceptualizes it as a processing buffer. In other words, working memory is where work happens. From the perspective of this dissertation, that work is the combination of words into sentences, and whether or not the buffer for this work overlaps with other buffers is not of interest.

In particular, this dissertation explores how this buffering occurs. Are elements buffered in the same order that they are spoken? How many elements can be buffered? After Chapter 2, these questions become the focus of the dissertation.

Cognitive models can map cleanly to these concepts, with a specified working memory, declarative memory, and procedural memory. For instance, in ACT-R, production rules can be thought of as procedural memory, and the goal buffer can be thought of as working memory, and lastly, declarative memory is called that explicitly (Anderson, 1983; Anderson et al., 2004; Bothell, 2004). Models using ACT-R can then explicitly use these components to comment on language processing (e.g., Lewis and Vasishth, 2005; Reitter et al., 2011b).

Machine learning models rarely specify explicit components.         For instance, Ororbia II et al. (2017) specifies a model of sequential processing over time that could possibly be thought of as to have a declarative memory component, but this is never specified or delineated as part of the model. Instead, neural models specify vectors and matrices that are easy to train with backpropagation. However, some level of symbolic reasoning may be required for high-level tasks. While these symbolic reasoning components could ultimately just be outputs of various neural networks, this type of architecture is somewhat difficult to inspect. Thus, this dissertation will attempt to map its models’ components explicitly to the vocabulary of cognitive science as they relate to memory and processing.

1.4 Syntactic Representations and Memory

Syntactic representation is somewhat controversial; many models do not represent syntax at all. However, there is some reason to believe that syntax exists independently of semantics. For instance, beyond anecdotal evidence (Colorless green ideas sleep furiously (Chomsky, 1957)) odd parts of speech when parsing change reading difficulty (Frazier, 1982; Ferreira, 1986), and syntactic priming suggests that syntax is represented as well (Reitter et al., 2006).

However, the existence of syntax doesn’t inform us what that actual representation is. Historically, syntax from the point of view of language production is often seen within the context of generative grammar, such as CCG (Steedman and Baldridge, 2011). This view of syntax implies that the process of grammatical encoding is primarily solved by resolving syntactic dependencies to form a sentence. It is explored in more detail in Chapter 3. However, there are many other

possibilities.

Chapter 4, beyond exploring lexical memory retrieval processes, also suggests that the process of language production is perhaps driven by memory processes surrounding lexical items rather than generative grammar. Moreover, one of the key problems in Chapter 3 concerns the difficulty in representing context-dependent symbolic syntactic types.

While syntax is generally thought of as the opposite of statistical models,

Gulordava et al. (2018) determined that neural models like Recurrent Neural Networks (RNNs) can in fact learn some of what we consider to be syntax. In some sense, the separation of “pure” language modeling based on probability and syntactic rules could even be arbitrary. Regardless, in Chapter 7, we explore a cognitive model that uses an RNN as its syntactic component.

1.5 Summary

We build a series of models to investigate buffering in language production. First, a model of word adoption lays the groundwork for our computation of lexical retrieval from declarative memory. Next, we extend this to discuss a model of grammatical encoding that relies on Combinatory Categorial Grammar (CCG). The third study also builds on the model of declarative memory and extends it with a model of lexical retrieval that occurs across small time-scales. These three initial studies establish a modeling methodology that will inform the next two studies, which attempt to model lexical retrieval strategies from a discrete perspective and a differentiable one. In the end, our studies converge in arguing for mostly incremental planning and buffering of up to five words.

INCREMENTAL PLANNING AND BUFFERING IN LANGUAGE PRODUCTION: MODELING LARGE-SCALE CORPUS DATA

Sharing is caring!