• : Ms Word Format
  • : 70 Pages
  • : ₦3,000 | $25 | ₵60 | Ksh 2720
  • : 1-5 Chapters
  • Click to DOWNLOAD Materials



A significant proportion of individuals’ daily activities is experienced through digital devices.  Smartphones, specifically, have become one of the preferred interfaces for content consumption and social interaction. Identifying the content that appears on smartphone screens and the rapid switches in content over time is thus a crucial prerequisite to studying media behavior and the potential impacts of screen content on physical, psychological and social health and well-being.

A need then arises, to effectively extract the content enclosed in digital screenshot and represent it in a machine-readable and efficiently retrievable form. Moreover, screenshot images can depict heterogeneous content and applications, making the a priori definition of adequate taxonomies a cumbersome task, even for humans. Privacy protection of the sensitive data captured on screens means the costs associated with manual annotation are large, as the effort cannot be crowd-sourced. Thus, there is need to examine the utility of unsupervised and semi-supervised methods for classifying digital screenshot. This work introduces the implications of applying clustering on large screenshot sets when only a few labeled data points are available.

We present an end-to-end framework implemented to: (i) extract text from digital screenshots, (ii) index the extracted text through Elasticsearch, (iii) store it a MongoDB collection of JSON documents, with their associated metadata, and (iv) classify the screenshot content through a combination of semi-supervised clustering and Active Learning.

Chapter 1

Introduction and motivation

Daily experiences are increasingly experienced through digital devices and smartphones, in particular, have become a predominant site for consuming, sharing and searching for media content. Thus, a number of studies of human behavior and “just-in-time” intervention planning can stem from intensive and longitudinal collection of smartphone screenshots, providing evidence of individuals’ second-by-second interactions with a wide variety of content.

To successfully represent the fast-paced interplay of activities and contents constituting these life threads – with switches occurring as quickly as every 19 seconds [1] - behavioral researchers need a taxonomy of screen content categories. Paradigmatic use cases include, but are not limited to: early disease detection and prevention planning; HCI models of task switching and its implications for attention and memory; ethnographic assessments on the effects of marketing strategies and profiling on low-income populations; studies of political attitudes and voting expressions across social media and news media, fake news detection, and so forth.

A first pre-requisite is then to represent the information enclosed in each captured frame in a way that is machine-readable, efficient to be stored and later retrieved for further analyses. Given the density of textual information screenshots carry along, effective text extraction (i.e., through Optical Character Recognition, or OCR) is a crucial factor contributing to this aim. Furthermore, screenshots provide a unique combination of graphic and scene text, motivating the evaluation of state-of-the-art OCR methods on this particular data set. Given the unique nature of this data set, a need arises for adopting a general-purpose approach, to handle a variety of fonts and layouts, disambiguate icons and graphical contents from purely textual segments, and recognize textual parts which are embedded in advertisements, logos, or video frames. While organizing these miscellaneous information fragments in a more coherent and interactive repository for further Knowledge Discovery, it is important to consider the organization and accessibility of the extracted data. We particularly focus on the retrieval of screenshot images based on their textual content and metadata, by describing the search engine architecture developed to this aim.

The implemented architecture, and related repository of digital screenshots, are conceived for aiding further analyses on the nature and interplay of contents consumed by media users. As a first effort in that direction, we interrogate on whether the extracted textual information can be leveraged with visual cues present in each digital screenshot, to classify their main content. Classification, however, can be cumbersome especially for screenshots (visual snapshots of screens), that include nested data streams (images, text), presented over diverse templates.  For instance, if Figures 1-1a and 1-1b are compared, one can see how the proportion of text over icons and graphical content is different. Specifically, in Figure 1-1b, more complex frames (i.e., video previews) are alternated with textual contents and titles.  As such, smartphone screenshots form a unique, yet rather unexplored, data typology, providing opportunities to test state-of-the-art methods for data clustering and classification on an unusual collection.

Pattern recognition from a heterogeneous media archive [2] , showcasing different topics and digital affordances competing for users’ attention [3] , with limited prior knowledge and annotation budget available, fosters the search for higher-level and more scalable screenshot representations, through unsupervised and semi-supervised feature learning. Moreover, the confidential nature of screenshots prompts searching for alternatives to crowd-sourcing, when categorizing incoming data streams. Moreover, screenshots enclose visual cues and text in different proportions, based not only on the consumed content (e.g., watching a YouTube video as opposed to reading a news article) but also on the time frame being observed, within the same activity (e.g., as scrolling through a blog post).

Much evidence has been already provided on the benefits of incorporating multi-modal features when disambiguating and categorizing new data sets [4, 5, 6] , however, this hypothesis has not been tested in the specific case of digital screenshots, where a higher degree of varying conditions and entropy is expected, both in visual appearance and textual contents. Consider, for instance, two frames depicting a smartphone home page (Figure 1-1a) and settings menu (Figure 1-1c) respectively. While the two-color palettes are certainly easier to discriminate, even by the human eye, the similar text densities (in terms of number of characters per page) and content overlap (i.e., both frames include the keyword “settings”) could have the two mapped to closer feature spaces. Moreover, (i) OCR errors [7, 8] can be propagated to the analysis of the autonomously-extracted text; and (ii) the presence of spurious characters extracted from icons [7] can introduce additional noise to the pipeline, compromising the usability of the produced library [9] . However, one could also argue that the latter characters, even though not carrying over textual meaningper se, could aid the discrimination of templates embedding different icon sets. Therefore, the influence of multi-modal representations on screenshot classification still needs a careful assessment.

In this scenario, defining adequate taxonomies capturing the full spectrum of relevant activities and applications presents additional challenges. Screen behavior is far-reaching, stretching beyond single, domain-specific and task-oriented taxonomies. For instance, generally annotating all sites for social networking as “Social Media” can potentially create a rather miscellaneous set, especially when different media platforms (and interchanging information threads) are concurrently observed, leading to low inter-field agreement on the chosen labels. While experts in media trend analysis might be more interested in the social connotations of content sharing, researchers studying the expression of voting attitudes through social signals might label that same content as “News” or “Events”, rather than “Social Media” [10] .

A more suitable alternative, is indicating the specific application being used (e.g., Facebook, Instagram, Twitter) or the type of action being performed, when the specific application cannot be inferred (e.g., when watching videos in full screen), explaining the adoption of application-level tags in this work (as shown in Figure 1-1). In fact, this design rationale aids agnostic pattern recognition, i.e., carried out without superimposing field-specific knowledge, ensuring that the learned features are data-driven. Besides, relying on lower-level annotations does not hinder future possibilities to add as many layers of semantic abstraction on top of the chosen taxonomy as required by the specific use case.

In what follows, we present an end-to-end framework to: (i) extract text from digital screenshots, (ii) index the extracted text through Elasticsearch, (iii) store it a MongoDB collection of JSON documents, with their associated metadata, and (iv) classify the screenshot content through a combination of semi-supervised clustering and Active Learning. For the text extraction and image classification components, we also present the specific data preparation and experimental setup used to evaluate the solution.


Sharing is caring!

Leave a Reply