SOCIAL MEDIA COMMUNITY USING OPTIMIZED CLUSTERING ALGORITHM

SOCIAL MEDIA COMMUNITY USING OPTIMIZED CLUSTERING ALGORITHM

Abstract

The aimed of this study was to examine social media community using optimized clustering algorithm. This study focuses on two tasks: clustering and social networks. Both these tasks were considered independent. The clustering task aimed to congregate documents in groups such that documents within a cluster would be as similar as possible. Documents are tweets and webpages. The k-means algorithm was employed for clustering the documents. Two methods were employed for feature extraction, namely, TF-IDF score and the word2vec method. Evaluation of clusters is done by two methods – Within Set Sum of Squares (WSSE) and analyzing the output of the topic analysis team to extract cluster labels and find probability scores for a document. The later strategy is a novel approach for evaluation. This strategy can be used for assessing problems of cluster labeling, likelihood of a document belonging to a cluster, and hierarchical distribution of topics and cluster. This study executed algorithms on different collections and presented the results. The social networking task was extracted information from Twitter data by building graphs. Graph theory concepts were applied for accomplishing this task. Using dimensionality reduction techniques and probabilistic algorithms for clustering, as well as improving the cluster labelling and evaluation, are some of the things that can be improved on the existing work in the future. Also, the clusters that were generated can be used as an input source for Classification, Topic Analysis and Collaborative filtering tasks in order to obtain more accurate results.

CHAPTER ONE

INTRODUCTION

1.1       Background of the study

Social media provides staggering amounts of data. Extracting knowledge from these volumes requires automation. Computing quickly over this data is a challenge for both algorithms and architectures. Recently, microblogging has become a popular trend which is responsible for a large amount of information dissemination. The most prevalent microblogging service is Twitter – a popular tool for short, frequent communication. Twitter users tweet about any topic, status or event and follow others to receive their tweets. Anyone around the world can use Twitter to talk about daily activities and seek information. Observing such a high usage of Twitter has urged many businesses and research groups to analyze these tweets to evaluate possible connections and outcomes (predictions). Several APIs and tools have been developed around this notion to predict future events.

As the importance of Twitter rises, the associated research in the fields of information science, corporate and civil society sectors is booming. We attempt to develop a network graph (based on graph theory) considering users as nodes. Edges between them would attribute to characteristics like following, re-tweet, etc. We also make an effort to group these tweets into clusters in an unsupervised manner via flat clustering. The technology stack will primarily include Java, Scala, Spark and its associated libraries.

1.2       Statement of the problem

A large number of community-detection algorithms have been proposed and applied to several domains. It is realized that analyzing tweets is challenging due to the short text and non-conformance to grammatical rules. These short messages are generated at a prodigious rate, which makes us consider timestamps of the tweets. A tweet can originate from various sources like newspaper, website, TV, etc. Each tweet shows the interest of the publisher. So, it is natural to consider human factors in the Twitter data analysis. Keeping in mind these challenges, this study will first try to build a social network, which shows the interaction between user accounts and the importance of each account. Then, the study can use that information to find the similarity between contents to reveal their importance. For clustering, this study will have to be cautious in identifying the words that form the essence of the tweets and use this further for extracting URLs/webpages. This study will employ clustering for grouping tweets w.r.t. multiple attributes. Clustering tweets effectively will influence the search functionality of the system. The remainder of this report provides further background on clustering in information retrieval and graph representations for social networks.

1.3      Objective of the study

The general objective of this study is to examine social media community using optimized clustering algorithm.

Specific objectives are:

1. To use Scala and SBT as build tool for Eclipse on CDH
2. To evaluate the clustering result with LDA output
3. To run the Scala-Spark program on shell
4. To identify within Set Sum of Squared Errors – WSSE

1.4       Research questions

1. How can we use Scala and SBT as build tool for Eclipse on CDH?
2. What is the clustering result with LDA output?
3. How we run the Scala-Spark program on shell?
4. How can we identify within Set Sum of Squared Errors – WSSE?

1.5 Limitation

The research work cannot contain all it supposes because of the following reasons or limitations.

Time factor:- The aim that is given for the research on this study is very limited making it impossible to go further into the research.

Limited resources: Because of the cost in transportation and other financial demands, it was not possible to get all the information needed for an elaborate work.

SOCIAL MEDIA COMMUNITY USING OPTIMIZED CLUSTERING ALGORITHM

 

Sharing is caring!