COMPARISONS OF MEASURES OF CRITICAL THINKING AND ANALYTIC PERFORMANCE:  EXPLORATORY STUDIES

ABSTRACT

There is a substantial body of scholarship on critical thinking in the fields of philosophy, education, and cognitive psychology.  Several mechanisms exist to measure critical thinking skills, abilities, and dispositions, with high measures of reliability and validity.  Exploration of critical thinking specific to intelligence analysis has expanded significantly since 2001 and has become a focused area of inquiry.  However, most researchers in this sub-field equate “critical thinking” with structured analytic methods and largely ignore the complex range of cognitive dimensions that scholars in other domains embrace.  Researchers and other authors in intelligence analysis posit a relationship between critical thinking education and the quality of analytic performance, but findings to date are mixed.  Researchers have not yet explored standard assessment measures of critical thinking as predictive mechanisms to determine which analysts may demonstrate higher or lower levels of job performance.  Furthermore, scholarship in critical thinking for intelligence analysis lacks consistency regarding definitions as well as the understanding and articulation of the task environment, including analytic processes, products, and evaluation.

Within the community of practice, intelligence organizations continue to face strong public and political pressure to reform in the wake of several perceived intelligence failures in recent years, such as a lack of warning regarding the terrorist attacks on New York and Washington, D.C., London, Mumbai, and more recently, Paris, Belgium, Boston, and Germany.  U.S. Intelligence Community reform efforts have been comprehensive, including modifications to organizational structures, increased inter-agency coordination, investment in education and training programs, and emphasis on analytic processes and products.  The need for improved research and practice in critical thinking has emerged as a common theme regarding those reform efforts.  These requirements are even more critical because of the continued expansion of the U.S. national security enterprise and the significant personnel changeover that continues as the “baby boom” generation retires from the workforce.  The demand for skilled entry-level analysts remains high.

To address these challenges, this study explores the relationship between measures of critical thinking abilities and the quality of analytic processes and products.  Specifically, I seek to explore whether standard assessment measures of critical thinking predict the performance of novices in intelligence analysis tasks.  Participatory action research provides an appropriate research approach given the requirement for domain expertise and the lack of relevant research.  Synthesis of prior work and cognitive models yield an integrated framework for critical thinking in intelligence analysis.  I conduct two exploratory studies with multiple methods.  First, I conduct a qualitative research project to understand and summarize how practitioners define and describe the dimensions of analytic quality.  Specific methods include participatory observation, interviews, and expert panels.  This study yields an operationalized model and rubric for analytic quality in the context of intelligence analysis.  The second study explores the relationship between standard assessment mechanisms of critical thinking and the performance of individual novices on intelligence analysis tasks.  Statistical analysis reveals correlations between some elements of critical thinking and analytic performance.  Specifically, certain dispositions for critical thinking may predict analytic performance.  Reflection on the three components of this study yields recommendations for future work, implications for the exploration of critical thinking for intelligence analysis, and implications for broader research efforts.

 

Chapter 1

                                                              INTRODUCTION                                      

 

1.  Introduction and Problem Statement

The public and political pressure to reform the U.S. Intelligence Community (IC) because of perceived intelligence failures (e.g. the 9/11 warning failure, the politically motivated 2002

National Intelligence Estimate on Iraq) is well documented (IRTPA, 2004; WMD Commission, 2005).  Much of the reform effort is targeted towards improving the quality of intelligence processes and products (CIA, 2002; ODNI, 2007 & 2008; DOJ, 2010).   Efforts include incorporating advances from the scientific community into intelligence work (NRC, 2011a & 2011b) and implementation of structured analytic methodologies to augment or replace traditional methods of analysis (Khalsa, 2009; Heuer & Pherson, 2011; Klein, 2011).

Those efforts are clearly worthwhile, but they take for granted a fundamental aspect of intelligence analysis and organizational change that cannot be assumed: a committed, intellectually diverse, and exceptionally capable workforce. I argue that the procedural and structural components of intelligence analysis and the IC overall are secondary to the   intellectual capabilities and expertise of intelligence analysts. Therefore, research to explore and improve the education, skills and tools of intelligence analysts should be the first priority for any reform effort. Theory-based and empirically sound interventions may contribute to the enhanced education of current and aspiring intelligence professionals and more efficient recruitment and hiring practices.

The IC has begun to address the issue of improving analytic skills and processes.  The efforts have resulted in the creation of professional education programs, specific to each of the various U.S. Government agencies, in “critical thinking,” loosely defined  (Moore, 2006; DIA, 2008; CIA, 2009).  The few assessments of those courses show extremely limited efficacy, i.e. the investment of effort, time, and money into the creation of the courses and student participation have not resulted in significant improvement of critical thinking skills of analysts or overall analytical quality of intelligence products (Reiber, 2010; MITRE, 2013).  There is no suggestion from any source (e.g. IRTPA, 2004; WMD Commission, 2005) that the recent intelligence failures were a result of any one individual’s critical thinking deficiencies; attribution is placed on systems failures, lack of interagency cooperation, and failures of distributed cognition.  However, voices from  within the IC (Reiber, 2010; Moore, 2011; MITRE, 2013) suggest that if individual critical thinking is improved, overall analytic quality will increase, therefore mitigating future intelligence failures.

The intelligence analysis community of research is responding as well with dedicated exploration of structured analytic methods (commonly referred to as Structured Analytic

Techniques (SATs)), popularized as a result of Heuer (1999) and Heuer & Pherson (2011).  In the context of intelligence analysis, the terms critical thinking, SATs, and

“tradecraft” (as it is often referred to in the community of practice) are synonymous, i.e.

“critical thinking” refers to the various problem-solving methods described by Heuer & Pherson (2011).  Thus far, the incorporation of SATs in intelligence education and in the workplace has not been shown to improve the quality of analysis processes or products

(Folker, 2000; Cheikes et. al., 2004; Johnston, 2005).

I submit that the limitations of these research and education efforts, since the overall objective is to improve the quality of intelligence processes and products, is largely because “critical thinking” and other relevant terms in the context of intelligence analysis have not been defined (Moore, 2005 & 2006; Derbensteva et. al., 2010; Klein, 2011; Pherson & Pherson, 2013).  Furthermore, the IC and its supportive communities have only recently begun to define the dimensions of “critical analytic thinking,” viewed as core competencies of intelligence analysts (Reiber, 2010; MITRE, 2013).   In other words, it should not be surprising that the efforts in “critical” or “analytical” thinking professional education have not yet succeeded given the lack of clarity, operational definitions, and measurement.

For the purposes of this dissertation, I adopt the definition of critical thinking posited by Facione (1990), which most consider the seminal contribution on the topic; further explanation regarding the various definitions is included in Chapter 2:

“We understand critical thinking to be purposeful, self-regulatory judgment which results in interpretation, analysis, evaluation, and inference, as well as explanation of the evidential, conceptual, methodological, criteriological, or contextual considerations upon which that judgment is based. … The ideal critical thinker is habitually inquisitive, well-informed, trustful of reason, open-minded, flexible, fair-minded in evaluation, honest in facing personal biases, prudent in making judgments, willing to reconsider, clear about issues, orderly in complex matters, diligent in seeking relevant information, reasonable in the selection of criteria, focused in inquiry, and persistent in seeking results which are as precise as the subject and the circumstances of inquiry permit.” 

 

The IC would benefit from an improved ability to observe and assess critical thinking skills when conceptualized as a mechanism to predict potential and performance for analytic tasks.  Exploration of these issues will have significant implications with respect to the recruitment of aspiring intelligence analysts and the relevant higher education programs from which they matriculate. Operationalized metrics of skill dimensions may inform the design of methods to more carefully and objectively assess the potential of individuals in order to better identify those who may perform better as analysts than others. A more complete understanding of the dimensions of critical or analytic thinking may also inform the design of educational programs and supportive information technologies for the purpose of improving individuals’ abilities.

Thus, the overarching objective of this research program is to explore the relationship between measures of critical thinking and the quality of intelligence analysis processes and products.  Elements of achieving this objective include the exploration of 1) a conceptual model for critical thinking in intelligence analysis that is consistent with the broader literature, 2) the observation and evaluation of analytic process and products, 3) the intelligence analysis task environment, including the replication and assessment of analytic processes and products, 4) specific dimensions of critical thinking in the context of intelligence analysis, and finally 5) relationships between standard assessment measures of critical thinking and performance in intelligence analysis tasks.

 

2.  Background

     The genesis of this research agenda was my performance on two education and professional development programs sponsored by the U.S. government: 1) the Intelligence Community Center for Academic Excellence (IC CAE), sponsored by the Office of the Director of National

Intelligence, executed by the Defense Intelligence Agency, and advised by the Central

Intelligence Agency and 2) the CyberCorps; Scholarship for Service Program, authorized by the National Initiative for Cybersecurity Education, executed by the National Science Foundation, and advised by the Department of Homeland Security and the National Security Agency.

Furthermore, in 2017, I served as the interim Director of Diversity Enhancement and Student Programs for a Department of Defense University-Affiliated Research Center.  The objectives of these programs are to recruit, educate, and support students who aspire to U.S. government careers: IC CAE encompasses the entire U.S. national security and intelligence enterprise;

CyberCorps, as the title implies, focuses on cyber security functions; the Diversity and Student

Programs function emphasizes the development of scientists and engineers for the Naval Services.  These programs and those like it are critical at this point in time, with the continued expansion of the national security enterprise and the ongoing and substantial personnel turnover due to the “baby boom” generation retiring from the workforce.   

     At the university performer level, the first challenge in the execution of these programs is how to determine which students have “the right stuff” to 1) be an attractive candidate for U.S. government internships and jobs and 2) succeed once they become entry-level professionals. The question, which students have “the right stuff,” dominates the conversations within the national security communities of practice and the community of educators that support it.  I have interacted and collaborated with faculty representatives from the dozens of IC CAE and CyberCorps partner institutions at program meetings, job fairs, and relevant academic and association conferences and events.  The primary question, phrased in various ways, that we ask our U.S. government colleagues and sponsors is: “how can we distinguish those students that

will be attractive to you [U.S. agencies] as candidates for internships and jobs from those who will not?”

     The responses vary significantly based on the specific organization, functional area, and experience and expertise of the individual queried.  Frequently, responses include those such as “we look for the same qualities all organizations look for: expertise, experience, and briefing and writing abilities” or more specifically e.g. “research experience and writing ability are the most important skills.”  A common response often received is “all of the candidates selected for interviews are exceptional; our selection decisions between the top two or three finalists are subjective and based on the ‘feel’ of the interviewers.”

     Critical thinking, however, is a common theme among all answers to the question.

Respondents inevitably cite such qualities as imagination, creativity, puzzle-solving, outside-thebox thinking, the ability to discern the why instead of restating the what, etc. – all indicators and descriptors of critical thinking.  Agencies’ hiring managers frequently indicate that interview questions of candidates (as well as the curricula for entry-level professional education programs) focus heavily on discerning, developing and improving the critical thinking abilities and potential of aspiring and young analysts.    

     On the surface, I was initially impressed by the emphasis on critical thinking, believing that U.S. intelligence agencies and their exceptional employees possessed esoteric knowledge, or at least special tools and methods, for identifying and measuring various levels of critical thinking or intellect overall.  However, in over a decade of working with intelligence professionals and hiring managers, I came to realize that they do not.  At best, hiring managers rely on existing mechanisms: standardized test scores, personality tests, grade point averages, etc.  At worst, assessments of applicants’ critical thinking ability through interview processes are purely subjective and based on the judgment of one or two interviewers.  There is currently no set of quantifiable, objective predictive tools – mechanisms that may enable hiring managers to more efficiently and effectively identify applicants’ potential for the specialized, knowledge-intensive tasks they aspire to perform.  

 

3.  Research Questions

     I address three primary research issues in this thesis.  The first challenge is to identify extant conceptual models or theoretical frameworks that may be appropriate.  Within the professional research, there is no shortage of opinion (e.g. Moore, 2005, 2006, & 2011; Pherson & Pherson, 2013), but there are a limited number of contributions from the research community.  Models from the broader literature must be adapted within the context presented here.  Thus:

RQ 1:  What theoretical frameworks are most useful for understanding the relationship between critical thinking and analytic quality in the context of intelligence analysis? 

 

The second question addresses the gap in the literature and practice regarding clear definitions and standards for the quality of analytic process and products specific to the intelligence domain (Reiber, 2010).  Qualitative exploration of this question also demands an understanding of the task environment, i.e. the conditions in which intelligence analysis takes place. Thus:

RQ2:  Can/how can researchers define, observe, and assess the quality of intelligence analysis processes and products?

 

     The third research question, representing the quantitative objective of this study, seeks to understand if and how researchers can predict performance in intelligence analysis.  I limit this study to 1) novices, 2) individuals working in a team, collaborative environment, and 3) strategic intelligence tasks (as opposed to intelligence tasks at the tactical or operational level).  Thus:

RQ3:  Do standard assessment measures of critical thinking skills predict individual performance of novices in intelligence analysis tasks?

 

     The activities in this thesis explore these research questions in an attempt to provide a better understanding of the critical thinking and other cognitive tasks that are specific to the challenging domain of strategic intelligence analysis.  My perspective is that of the people who perform these tasks and those that are charged with recruiting, training, mentoring, and supervising them.

 

4.  Thesis Structure

In Chapter 2, I summarize and assess the literature relevant to the intelligence analysis domain.  The integration of multiple perspectives and approaches are necessary for this thesis, including stated requirements for IC Reform, the broad range of critical thinking literature, exploration of critical thinking specific to intelligence analysis, and application of critical thinking assessment in other domains.

In Chapter 3, I provide an overview of and rationale for the research approach.  Participatory Action Research (PAR) serves as an appropriate overall methodology.  PAR enables close coordination with practitioners and allows the researcher to fully participate in the problemsolving team.  Other, more traditional research approaches are less appropriate in this instance because of the infancy of the state of research relevant to the questions posed here.  I also outline and summarize the research approach and methods, detailed in Chapters 5 and 6 respectively, of defining the dimensions of analytic quality and critical thinking, exploring intelligence analysis processes and products, and the testing a relationship between standard assessment measures of critical thinking and performance on intelligence analysis tasks.

The first requirement, articulated in Chapter 4, is to synthesize the broad and domain-specific literature and create an integrated model of critical and analytic thinking, to frame these studies and contribute to future work.  I present a conceptual model that builds on prior work (Klein, 2011) and is also consistent with the broader literature in cognition.

In chapter 5, I explore and define dimensions of analytic performance specific to intelligence analysis.   Building on literature from the community of practice, I employ qualitative research methods and summarize how expert practitioners define and describe analytic thinking, quality, and performance in the intelligence domain.   I test and further refine the evaluation framework in conjunction with practitioners.  As the primary goal of this effort is to explore how researchers can predict performance in intelligence tasks, ensuring a realistic task environment is critical.  Therefore, Chapters 5 and 6 also explore how to replicate the processes and products specific to intelligence analysis.

In Chapter 6, I explore the relationship between standard assessment measures of critical thinking skills and performance on intelligence analysis tasks.  The simplified research question is: Does critical thinking ability predict analytic performance?   Two standard measures of critical thinking assessment, the California Critical Thinking Disposition Inventory (CCTDI) and the basic California Critical Thinking Skills Test (CCTST), are employed and several statistical methods used to identify possible relationships to the rubric developed in Chapter 5.  Again, the study is limited to 1) individuals working in teams, 2) novices, and 3) intelligence tasks at the strategic (as opposed to the tactical) level.

Chapter 7 concludes the thesis with discussions on contributions of the research, lessons learned in terms of both research methods and applications to intelligence analysis education and professional development efforts, recommendations for future work specific to the domain, and implications for broader research in critical thinking.

 

5.  Contributions

     Through this thesis, I contribute the following:  First, in the literature review, I integrate work from both the community of practice and the community of research.  Thus far, the latter has not seriously considered the former while the former includes very few empirical contributions from the latter.  Synthesis of the literature results in an integrated model of critical and analytic thinking, presented in Chapter 4.  Second, I present and validate a consolidated, accessible set of dimensions of critical analytic thinking, processes and products, specific to the tasks and requirements of strategic intelligence analysis, in a manner that enables objective and quantifiable observation and assessment, presented in Chapter 5.  Third, I explore the task environment of intelligence analysis and how to evaluate the quality of analytic processes and products and simulate those tasks for aspiring analysts in a realistic and challenging manner, resulting in a) an integrated rubric to evaluate analytic quality and b) a set of design principles and considerations for the creation of strategic analysis simulation exercises.  Finally, and representing the quantitative goals of the thesis, in Chapter 6, I test and assess standard measures of critical thinking ability to performance on intelligence analysis tasks.  A positive relationship between critical thinking skills and analytic performance may indicate the possibility of mechanisms to better identify individuals with specific cognitive attributes particularly suitable for work in the domain.

 

6.  Summary

Intelligence analysis is a challenging, knowledge-intensive task, conducted in time- and resource-constrained environments, with the highest of stakes: a nation’s strategic interests around the globe and the lives of its citizens.  Furthermore, lives lost because of intelligence failures are often violent and broadcast in real time on global media outlets, events that often influence public opinion and cause reactive policy changes.  Intelligence analysts at the strategic level, those charged with providing assessments and recommendations to elected officials and other decision makers, therefore must be not only well educated but also intellectually agile individuals, i.e. extraordinary critical thinkers.

Educators and researchers in intelligence-analysis relevant academic programs have an obligation to ensure that our students that aspire to and are placed in intelligence analysis careers are as fully prepared and qualified for those tasks as possible.  Universities generally excel in offering traditional coursework, affording students with relevant research opportunities, and professional development via career counseling and guest lectures.  U.S. universities are unparalleled at inspiring and emphasizing critical thinking, broadly defined.  However, U.S. universities and colleges – with a few notable exceptions such as the federal service academies, private and state military colleges, and state merchant marine academies – do not view a student placement in the national security enterprise any greater or less than a placement to graduate school or industry.  We therefore do not look specifically at students’ specific career interests and weigh aspiring government employees any higher than others.  Universities do not distinguish the dimensions of critical thinking specific to strategic analysis – or any other knowledge-intensive domain – from others.  Aside from traditional measures of student success such as grade point average, internships, research experiences, leadership in student organizations, and community service, we cannot predict students’ future success in their chosen careers.  The overarching objective of this thesis – to explore the relationship between measures of critical thinking assessment and the quality of intelligence analysis processes and products – may provide insight to create mechanisms that can, in fact, better predict students’ future success in this context, and better identify students who may hold greater potential for strategic analysis roles than those who may not.

COMPARISONS OF MEASURES OF CRITICAL THINKING AND ANALYTIC PERFORMANCE:  EXPLORATORY STUDIES

Sharing is caring!