A DATA TRIAGE RETRIEVAL SYSTEM FOR CYBER SECURITY OPERATIONS CENTER

ABSTRACT

Triage analysis is a fundamental stage in cyber operations in Security Operations Centers (SOCs). The massive data sources generate great demands on cyber security analysts’ capability of information processing and analytical reasoning. Furthermore, most junior security analysts perform much less efficiently than senior analysts in deciding what data triage operations to perform. To help analysts perform better, retrieval methods need to be proposed to facilitate data triaging through retrieval of the relevant historical data triage operations of senior security analysts. This thesis conducts a research of retrieval methods based on recurrent neural network, including rule-based retrieval and context-based retrieval of data triage operations. It further discusses the new directions in solving the data triage operation retrieval problem.

The present situation is that most novice analysts who are responsible for performing data triage tasks suffer a great deal from the complexity and intensity of their tasks. To fill the gap, we propose to provide novice analysts with on-the-job suggestions by presenting the relevant data triage operations conducted by senior analysts in a previous task. A tracing method has been developed to track an analyst’s data triage operations. This thesis mainly presents a data triage operation retrieval system that (1) models the context of a data triage analytic process, (2) uses recurrent neural network to compare matching contexts, and (3) presents the matched traces to the novice analysts as suggestions. We have implemented and evaluated the performance of the system through both automated testing and human evaluation. The results show that the proposed retrieval system can effectively identify the relevant traces based on an analyst’s current analytic process.

 

TABLE OF CONTENTS

LIST OF FIGURES ………………………………………………………………………………………………….. vi

LIST OF TABLES ……………………………………………………………………………………………………. vii

Acknowledgements …………………………………………………………………………………………………… viii

Introduction …………………………………………………………………………………….. 1

Triage Analysis in SOCs …………………………………………………………………………….. 4

2.1 Data Triage for Cyber SA ……………………………………………………………………………… 5

2.2 Multi-Source Data in SOCs …………………………………………………………………………… 6

2.3 Data Triage Operation …………………………………………………………………………………… 7

Problem Overview ……………………………………………………………………………………… 8

3.1 Difficulties in Data Triage Tasks ……………………………………………………………………. 8

3.2 Experts’ Knowledge of Data Triage ………………………………………………………………… 8

3.3 A Framework for Data Triage Knowledge Retrieval System Designs …………………. 9

3.4 Challenges in Developing Effective Data Triage Knowledge Retrieval Systems ….. 10

Deep Learning based Retrieval of Triage Operations ……………………………………… 12

4.1 Properties in Recurrent Neural Networks ………………………………………………………… 12

4.2 Data Triage Operation Retrieval based on Recurrent Neural Networks ……………….. 13

4.3 Data Triage Model ……………………………………………………………………………………….. 14

4.4 Triage Operations through Time …………………………………………………………………….. 16

4.5 Data Triage Operation and Characteristic Constraint ………………………………………… 17

4.6 Trace and Context ………………………………………………………………………………………… 17

4.7 Insights: Context-Driven and Efficient ……………………………………………………………. 19

4.8 Challenges in Using Machine Learning for Data Triage Operation Retrieval ……….. 20

Evaluation…………………………………………………………………………………………………. 21

5.1 Test Cases Generation …………………………………………………………………………………… 22

5.2 Ground Truth ……………………………………………………………………………………………….. 23

5.3 Random Selection ………………………………………………………………………………………… 24

5.4 Performance Measurement…………………………………………………………………………….. 24

5.5 Human Evaluation ………………………………………………………………………………………… 27

5.5.1 Evaluation Protocol …………………………………………………………………………….. 27

5.5.2 Sequence Length ………………………………………………………………………………… 28

5.5.3 Result ………………………………………………………………………………………………… 29

5.5.4 Case Study …………………………………………………………………………………………. 29

Related Work…………………………………………………………………………………………….. 31

6.1 Rule-Based Data Triage Retrieval System ……………………………………………………….. 31

6.2 Knowledge Representation ……………………………………………………………………………. 32

6.3 An Example of Rule-Based Representation ……………………………………………………… 33

6.4 Knowledge Capturing …………………………………………………………………………………… 34

6.5 Knowledge Matching and Rule Relaxation ……………………………………………………… 35

6.6 Case Study…………………………………………………………………………………………………… 37

6.7 Context-Based Data Triage Knowledge Retrieval System …………………………………. 39

6.8 Knowledge Representation ……………………………………………………………………………. 39

6.9 Knowledge Matching ……………………………………………………………………………………. 40

Discussion and Future Work ……………………………………………………………………….. 41

7.1 Graph-Based Data Triage Knowledge Retrieval System ……………………………………. 41

7.2 Knowledge Representation ……………………………………………………………………………. 42

7.3 Knowledge Matching and Challenges …………………………………………………………….. 44

7.4 Ontology-based Data Triage Operation Retrieval ……………………………………………… 46

Conclusion ………………………………………………………………………………………………… 47

Bibliography ……………………………………………………………………………………………………………. 48

 

LIST OF FIGURES

Figure 1 Experiment measurement ……………………………………………………………………………… 26

Figure 2 The precision-recall curve …………………………………………………………………………….. 26

Figure 3 Data analysis processes in SOCs. …………………………………………………………………… 32

Figure 4 The critical features in an attack graph. …………………………………………………………… 34

Figure 5 Experience relaxation levels. …………………………………………………………………………. 36

Figure 6 Hierarchical experience networks. …………………………………………………………………. 37

Figure 7 The architecture of the rule-based knowledge retrieval system. …………………………. 38

Figure 8 An E-Tree example. ……………………………………………………………………………………… 40

Figure 9 An example of the logical relationships between data triage operations. ……………… 44

 

LIST OF TABLES

Table 1 Examples of data triage operations ………………………………………………………………….. 15

Table 2 Experiment results ………………………………………………………………………………………… 25

Table 3 Satisfaction rate for sequence length ……………………………………………………………….. 28

 

Introduction

There are colossal, complex and undetermined threats in the cyber world. As cyber-attacks are happening on a daily basis and could be launched against an enterprise network at any moment, more and more organizations have established Security Operations Center (SOCs) to coordinate the defenses against cyber-attacks [1] .

When a security incident happens, the top three questions a SOC seeks to answer are: What attack has happened? Why did it happen? What action should be done? While a variety of software tools (e.g., security information management system, host-based security systems) and hardware equipment (e.g., network intrusion detection systems) have been deployed in today’s enterprise networks to detect and correlate security-related events [2] , real-world SOCs still rely on security analysts (and watch officers) to make decisions on “What should I do?”.  Due to several critical limitations (e.g., high false positive rates) of the deployed software tools and hardware equipment, autonomous intrusion response is not yet being adopted by SOCs [3] .

From the perspective of “data to decisions,” the intrusion response decisions made by a SOC can be viewed as the main output of a particular human-in-loop data triage system [4] . Not surprisingly, how soon the right intrusion response decisions can be made heavily depends on the efficiency (i.e., avoid performing useless data triage operations) of the system’s data triage operations [5] . Since there are a large variety of “sensors” monitoring an enterprise network, the enterprise’s SOC will gather a huge amount of heterogeneous data coming from different types of data sources. Accordingly, a critical challenge faced by the SOC is that the massive data sources generate great demands on security analysts’ capability of information processing and analytical reasoning.

To address this critical challenge, SOCs have been putting in a lot of effort to recruit and train security analysts. However, it is widely observed that the amount of time and effort required to train a security analyst is overwhelming. It usually takes a newly hired security analyst several years to complete his or her training and become an experienced analyst.  Moreover, during the long on-job training process, it is observed that most inexperienced (junior) security analysts perform much less efficiently than senior analysts in deciding what data triage operations to perform.

To address these training challenges, several retrieval methods have been proposed to facilitate the data triage of inexperienced security analysts through retrieval of the relevant past data triage operations of experienced (senior) analysts. These research works have shown that data triage operation retrieval could help an inexperienced security analyst a lot in reducing the number of useless triage operations during his or her data triage processes.

In this thesis, we first conduct a review of the existing retrieval methods, including experience-based retrieval and context-driven retrieval of data triage operations. We then discuss the new directions (e.g., apply machine learning techniques) in solving the data triage operation retrieval problem.

The remainder of this paper is organized as follows. In Section 2, we present an overview of data triage in SOCs.  In Section 3, we give an overview of data triage operation retrieval systems. In Section 4, we discuss the main challenges in developing effective triage operation retrieval systems. In Section 5, we conduct an evaluation of the data triage operation retrieval methods, namely, experience-based retrieval and context-driven retrieval of triage operations. In Section 6, some related work will be discussed. In Section 7, we direct some future directions in building better triage operation retrieval systems. We conclude the paper in Section 8.

 

A DATA TRIAGE RETRIEVAL SYSTEM FOR CYBER SECURITY OPERATIONS CENTER

Sharing is caring!