MODELING PHOTOGRAPHIC COMPOSITION VIA TRIANGLES

Abstract

Modeling photographic composition is important to many real-world applications such as digital photography, image retrieval, image understanding, and image aesthetics assessment. The triangle technique is among the indispensable composition methods which professional photographers often rely on. It refers to the organization of visual elements such as lines, edges, and shapes into triangles, resulting in interesting and dynamic compositions. This thesis proposes a system that can detect the presence of the triangle composition in two major categories of photographs: natural scenes and portraits. After the detection, the system further provides on-site composition feedback to the photographers. In the case of natural scene photography, we propose a new image segmentation algorithm for extracting triangles based on agglomerative clustering, utilizing both photographic and geometric cues. We further illustrate how these cues can be directly used to detect the dominant vanishing point in an image without extracting any line segments. For portrait photography, we extract and assemble straight lines to form triangles via a modified RANSAC algorithm. Experimental results have demonstrated that our system can accurately locate preeminent triangles in photographs without any knowledge about the camera parameters or lens choices. Finally, we demonstrate an application of the proposed techniques in providing on-site feedback to photographers.

Chapter 1 Introduction

With the rapid advancement of digital camera and mobile imaging technologies, we have witnessed a phenomenal increase of both professional and amateur photographs in the past decade. Large-scale social media companies, e.g., Flickr, Instagram, and Pinterest, further empowered their users with the capability to share photos with people all around the world. As millions of new photos are added to the Internet on a daily basis, there is an increasing demand for creating automatic systems to manage, assess, and edit these photos.

Currently, most existing online image management or photo sharing services heavily rely on textual information such as tags or descriptions provided by users. A few can also access visual features of photos and infer embedded semantic information from these features. Either textual or semantic information only reveals contents of photos such as present objects, events, themes, and so on. However, the explosive growth of online photos requires those automatic systems to index them with more dimensions other than contents. For example, millions of users upload photos about birthday parties everyday. As a result, searching photos with the keywords “birthday party” generates a huge number of results. Among them, only a small portion has high aesthetic quality and thus has potential appeal to other users. Therefore, an automatic evaluating technique is in demand to filter out these low-quality photos and enable users to easily get access to high-quality photos.

The aesthetics of photography lies in several different aspects: color, lighting, texture, composition, etc. Professional photographers are skillful at adjusting photography parameters such as aperture and shutter speed to improve multiple aspects of photos. For instance, shallow depth-of-field technique is utilized to generate specific textures, i.e., blurred background and sharp foreground, by selecting a large aperture. Compared to the blurred background, the sharp foreground immediately drags viewers’ attention. Such contrast between blurred and sharp textures makes the photo more aesthetically appealing. Once a computer can recognize such contrast, it is able to evaluate aesthetic quality of this photo from the aspect of texture. Hence, some researchers begin to explore the possibility of extracting aesthetics-relevant visual features from a photo in terms of its color, lighting, and texture. Further, the relationship between such visual features and aesthetic quality of photos can be modeled via machine learning techniques. Given a new photo without any texural or semantic information, the evaluating system can predit its aesthetic quality by extracting and tossing those aesthetic-relevant visual features into the learned model.

Accurately predicting aesthetic quality of photos has a wide range of potential applications. Nowadays, the emergence of digital cameras and mobile imaging technology makes people take photos more casually, leaving a lot of low-quality photos in their personal galleries which are unlikely to be accessed in the future. Automatically picking out low-quality photos is beneficial for large-collection personal gallery management [11, 37] . Moreover, such algorithm can also be embedded into online image search engines as a re-ranking tool to deliver high quality images to users [36, 11, 42, 37, 35] . Apart from filtering out low-quality photos, users may want to enhance the quality of these photos. In this case, image processing techniques can be adopted together with the aesthetic quality evaluating technique to enhance the quality of consumer-level photos. Additionally, aesthetic quality assessment can also be used for story illustration [11] and realtime photographic guidance [37, 65] .

Compared to color, lighting, and texture, composition is more difficult to model because it requires an overall semantic comprehension of the photo. Recently, photo composition understanding is becoming a noteworthy area of research in the computer vision community.

Composition is the art of positioning or organization of objects and visual elements (such as color, texture, shape, tone, and depth) within an image or a visual art work. Known principles of organization include balance, contrast, geometry, rhythm, perspective, illumination, and viewing path. Automated understanding of photo composition has been shown to benefit several applications such as summarization of photo collections [41] and assessment of image aesthetics [42] . It can also be used to render feedback to the photographers on the aesthetics of their photos [66, 65] , and to suggest improvements on the image composition through image re-targeting [33, 6] . In the literature, most studies on image composition understanding focus on image-based rules such as the simplicity of the scene, visual balance, the rule of thirds, and the use of diagonal lines. Because of their simplicity, these composition rules have been widely used to guide the photographers at the moment of their creative work.

However, these rules are quite limited in capturing the wide variations in photographic composition. As an expansion, we hereby explore methods to identify an important composition technique, namely, the triangle technique. We focus on two major categories of photographs: natural scenes and portraits. These two categories cover a number of important consumer photography subject types including landscape, architecture, travel, portrait, fashion, family, and baby photography. Other types, such as animal, flower, macro, sport, and event photography, are frequently done by more sophisticated or professional photographers and the triangle technique is not as important for them as some other techniques such as depth-of-field controlling, high speed, motion blurring, and cropping, hence are not covered in this work.

We have developed category-sensitive approaches to detect the presence of the triangle composition and provide on-site feedback to photographers. In a realistic application, users can first inform the system the type of photography being created in order to receive the feedback. This is similar to a command dial selector, often provided on consumer-level digital cameras, where photographers can set the type of photo taking with typical choices as portrait, landscape, macro, sport, and night. In the following section, we discuss the use of triangles in composition.

1.1       Aesthetic Experience in Psychology

To gain deeper understanding of an aesthetic experience, we look into the concept of aesthetics in psychology. Aesthetic experience is defined as a complex interaction among properties of art objects, characteristics of the viewers, and the physical, social, and historical contexts [34, 23] . It intrigues researchers from both psychology and neuroscience, giving rise to two research domains related to perceptual/cognitive aesthetics: experimental aesthetics (psychology of aesthetics) and neuroaesthetics [25] .

Experimental aesthetics is the second oldest branch of experimental psychology [23] . Conceptual models and frameworks proposed in this area mainly focus on three questions: What factors will influence aesthetic experience? What is the underlying neural, perceptual, and cognitive process of aesthetic experience? How does such aesthetic experience evolve/change over time with regard to evolutionary, historical, and cultural changes? The same questions are also approached in the area of neuroaesthetics. The major difference is that researchers from neuroaesthetics investigate the neural underpinnings of aesthetic experience while concerning these questions. With the aid of imaging and neurophysiological techniques such as functional magnetic resonance (fMRI), magnetoencephalography (MEG), and electroencephalography (EEG) [8] , they collect evidence to explain the relationship between biological activation of the brain and psychological process of aesthetic experience [45, 7, 8, 46] .

Aesthetic experience refers to the process that human beings gain aesthetic pleasures while appreciating and evaluating the beauty in a range of entities such as painting, sculpture, music, faces, flowers, food, etc. Although most of previous work focuses on studying aesthetic experience related to visual art, the results and conclusions can be easily extended to other forms of art such as music and dance. Viewer, artwork, and context are the three elements involved in the process of aesthetic experience. To understand the three elements, we can imagine a scene where a person is looking at a painting in a museum. The person is the viewer. His/her individual characteristics such as neural faculty, memory, expertise, and personal preferences will affect the way he/she appreciates the beauty of the painting. Likewise, the content, form, and style of the artwork, which refers to the painting in the above example, also influences the aesthetic experience. Finally, the context can be specifically stated as when and where the aesthetic experience takes place. To bring it up to a more general level, factors like historical, cultural, and economic background as well as the evolutionary process can all be treated as contextual factors in the model of aesthetic experience. These diverse factors were divided into seven perspectives by Jacobsen [23] . Besides, these factors do not prevail over the entire process of aesthetic experience. Instead, it is only a combination of several of these factors that militate in different phases of aesthetic experience.

To understand how these factors affect the way people appreciate artworks, we need to explore more about the underlying perceptual and cognitive process of aesthetic experience. From a psychological perspective, aesthetic experience is regarded as an information processing model involving five stages: perceptual analyses, implicit memory integration, explicit classification, cognitive mastering and evaluation, as well as affective and emotional processing [30] .

First, the artwork is analyzed perceptually. Varieties of aesthetic-specific perceptual visual features are studied. Among which contrast, visual complexity, color, symmetry, grouping and order (where balance is concerned), etc. have positive effect on evoking aesthetic pleasures in human beings. A number of neuroaesthetic researchers explained why people prefer such features from the perspective of biology. For example, the symmetric property of most biologically important objects such as predator, prey, or mate “trained” human beings to be intrigued by symmetric objects [46] .

Implicit memory integration processes features through the faculty of memory. For instance, familiarity and prototypicality (“the amount to which an object is representative of a class of objects” [30] ) are processed in this stage. Similarly, what happens in this stage is also evidenced by work from neuroaesthetics. Ramachandran and Hirstein [46] well interpreted Peak Shift effect^∗ as the action to powerfully activate the same neural mechanisms in viewer’s brain by amplifying the signal.

Until now, aesthetic experience still stays on perceptual level. The third stage, namely explicit classification, takes content and style of the artwork into account, where the viewer generalizes semantic and stylish information from perceptual features abstracted before. Viewers’ expertise as well as contextual information thus come into

play.

Thus, cognitive mastering and evaluation stages together build a feedback loop where the results of cognitive mastering are continuously evaluated. Such mechanism guarantees a gain of aesthetic pleasures after successful understanding of the art object. Conversely, a subjectively perceived failure of understanding the artwork triggers restarting the previous stages where perceptual visual features are extracted and analyzed again. In neuroaesthetic, researchers have stated that the feelings of pleasure are induced because the understanding of artwork activates the rewarding areas in the brain [49, 46] . It has explained that why aesthetic experience is intrinsically positive.

There are two outputs of the model: aesthetic emotion and aesthetic judgments [7,

30].              The state of aesthetic emotion keeps changing during different stages based on

∗

A psychological phenomenon is typically known as its application in animal discrimination learning. In the peak shift effect, animals sometimes respond more strongly to exaggerated versions of the training stimuli. Please refer to wikipedia for more information.

the fluency of information processing [49] . To state it simply, the more fluently one understands the artwork, the more pleasure he/she gains from aesthetic experience.

The last question concerns why human beings are born with the abilities to discern beauty from ugliness. Some researchers answer this question by drawing evidence from evolutionary theories. Such theories usually relate human being’s intrinsic abilities of judging aesthetics to survival demands during evolution. The example about preference for symmetric objects can be one instance. Stebbing [55] also proposed an interesting theory that our aesthetic behavior has evolved from our innate ability to recognize the diversity of organic form for foraging purpose.

According to the psychological model of artistic experience, we have to go through five stages to process all the information contained in an artistic display. During the five stages, a variety of information resources can provide clues for aesthetic judgments such as perceptual and cognitive features of the images, as well as personal preferences and individual experience, etc. However, the state-of-the-art techniques to computationally evaluate image quality only make use of perceptual features such as color [40] , texture [35] , brightness [24] , etc. or a few low-level cognitive information such as familiarity [9] of the images. This is partly due to the lack of mature cognitive information extracting technologies in the domain of image processing. However, it also reveals a large set of potential research questions which can be explored in the future. For instance, considering the diversity of individual cognition abilities and personal preferences, computational models of aesthetic judgments can vary across different users or user groups. Therefore, personalized systems can be studied in the future. Another inspiration of new research question comes from the stage of explicit classification in aesthetic experience where people resort to their expertise and knowledge about artistic styles to infer the aesthetic quality of images. Therefore, we can establish a database of artistic styles or templates for all kinds of visual displays such as painting, photos, and interface design. To build such a database, we can collect training data from artists by constructing an online art community. Artists can contribute their knowledge of artistic styles by correctly labeling images or patches of images with style tags. Then, computational models can be employed to learn the relationship between the style category and the perceptual features extracted from the images or the patches of images. During this process, a computer can learn to “recognize” artistic styles. Similarly, we can also explore other potential research questions concerning different stages of the artistic experience.

For this thesis, we choose to focus on the first stage. Aside from commonlyused perceptual features such as color and lighting, we aim at studying the overall comprehension of photo compositions. In the next section, we will briefly introduce the composition photography technique studied in this work.

1.2       Triangles in Photographic Composition

In pictorial art, good composition is considered as a congruity or agreement among the elements in a design [29] . The design elements appear to belong together as if there are some implicit visual connections between them. Another term to describe this form of unity isharmony. By reflecting this principle in photography, subjects in one scene should not aimlessly scatter around. Instead, they should unify to provide an overall impression to the viewer. To convey such unity in photos, professional photographers

Fig. 1.1 The use of triangles in visual art and architectural works. The suggested triangles are indicated in red. From left to right: (Row 1) Afghan Girl, Eiffel Tower, Girl with a Pearl Earring, Golden Gate Bridge, Mona Lisa, The Arnolfini Portrait, The Scream. (Row 2) Iwo Jima Memorial, Last Supper, Pyramid, Starry Night Over the Rhone, The Birth of Venus.

have designed dozens of executable techniques for composition. One universal and interesting technique is to embed basic geometrical shapes in photographic compositions [61] . Human beings begin to learn about basic geometrical shapes such as circles, rectangles, and triangles since very young age. Even toddlers are capable of recognizing those basic shapes immediately. Valenzuela [61] suggests that we can explicitly or implicitly embed basic geometrical shapes in photos to attract viewers. Moreover, since these shapes are instantly recognized, subjects bounded within such shapes or implicitly constructing such shapes are perceived as a unity.

Among all basic geometric shapes, triangle is arguably the most popular shape utilized by professional photographers to make a composition more interesting. Such compositional technique is called “the triangle technique”. We can also find numerous examples of the use of triangles in visual art and photographic works (Figure 1.1).

In our work, we aim at detecting the usage of the triangle technique in two common types of photographs: natural scenes and portraits. We propose an automated system that can accurately locate a variety of triangles, even those that are carefully designed by professional photographers but difficult to be recognized by amateurs. By detecting the usage of the triangle technique, we can model various composition types and retrieve images based on similarity or dissimilarity in composition. We can also help amateur photographers gain deeper understanding from professional works and inspire them to generate photographs with more interesting composition and higher aesthetic

quality.

1.3      System Overview

We now provide an overview of the proposed composition analysis system. At first, users are asked to indicate the type of photography they are currently engaged in. The system can provide assistance if the selected type is natural scene or portrait. Then, users can take a test shot and upload it to our system as a query image. Given the query image and its category, our system will analyze the composition, particularly the use of triangles, in the image and further provide on-site feedback to users. Figure 1.2 briefly introduces the framework of our composition analysis system.

For natural scenes, the composition of a query image will be represented by a geometric segmentation of the image. The segmentation takes into account both geometric and photometric cues to partition an image into cohesive triangular regions and describes how these regions together construct a composition. In our system, the geometric and photometric cues are obtained by detecting the vanishing point and extracting the contour map of an image, respectively. Utilizing segmentations to represent compositions of images enables us to retrieve images based on compositions. As a result, our system can

Fig. 1.2 Overview of our composition analysis system. recommend high-quality photos with similar compositions as the query image to users. Besides, we can also retrieve images with a variety of compositions and inspire amateur photographers to employ potentially different but more interesting compositions.

For portrait, potential line segments are first extracted from a query image. Then, we filter the line segments using the contour map to remove segments with low confidence scores. Finally, the system identifies all potential triangles from the image by grouping the remaining line segments. Detecting triangles in high-quality photos taken by professional photographers helps amateurs to gain deeper understanding and inspirations about the triangle techniques. Additionally, as on-site feedback to users, our system returns a variety of triangles of different sizes, shapes, and orientations in images which might be easily overlooked by amateurs.

In the following sections, we briefly introduce the major challenges of modeling compositions of natural scenes and portrait photos as well as the algorithms we developed to solve the problem.

1.4         “Pie” Model in Natural Scene Photography

A popular compositional technique in photography is to utilize perspective effects to generate an immersive and stereoscopic scene. Such compositions emphasize a sense of 3D space within 2D display. However, existing composition modeling techniques for natural scenes are mainly concerned with the 2D rendering of objects in an image [41, 42, 66, 65, 33, 6] . In order to model good compositions with high perspective effects, it is necessary to examine the 3D structure of the scene. According to the perspective camera geometry, all parallel lines in 3D space converge to a single vanishing point in a photo, generating a set of triangular regions (Figure 1.3). In order to convey a strong impression of 3D space and depth to viewers, the vanishing point has to lie within or near the image frame and associates with the dominant structures of the scene (e.g., grounds, large walls, bridges).

Fig. 1.3 Applying pie models on natural scene photos.

Figures 1.3 and 1.4 show some examples. We regard such a vanishing point as the dominant vanishing point of a particular photo. As one adjusts shooting angle, both the location of the dominant vanishing point as well as the sizes, shapes, and orientations of triangular regions relating to the vanishing point will change. An experienced photographer often utilizes such technique to produce various image compositions that convey different messages or impressions to viewers. The resulting model looks like a “pie” that has been cut into slices. Each slice is corresponding to one triangle. Hence, we name it the pie compositional model. Such a pie model enables users to extract compositions from photos and further edit the compositions by moving the vanishing point or resizing

the slices.

To extract the pie composition from a landscape photo, we need to first determine where the pie locates with respect to the photo and then “cut” the pie into both photometrically and geometrically consistent slices, e.g., walls, sky, grounds. Locating

(a)                                      (b)                                      (c)

Fig. 1.4 Geometric image segmentation. (a) The original image with the dominant vanishing point detected by our method (shown as a yellow round dot). (b) Region segmentation map produced using a state-of-the-art method. (c) Geometric image segmentation map produced by our method.

the pie is equivalent to detecting the origin of pie, i.e., the dominant vanishing point in the photo. In order to cut the pie, we propose a hierarchical segmentation algorithm based on agglomerative clustering which employs both photometric and geometric cues. As shown in Figure 1.4(c), such a segmentation naturally provides us a holistic comprehension of the 3D spatial structure of the scene and successfully captures a variety of different compositions. Nevertheless, obtaining such a holistic representation is a challenging problem for the following reasons.

First, detecting vanishing point itself is a very challenging problem. Most stateof-the-art methods rely on extracting all straight lines from the image which converge to a same point and identify that point as a vanishing point. Such converging straight lines often come from man-made objects such as edges of buildings. However, some landscape photos depicting natural scenes like mountains or lakes lack cues of straight lines (Figure 3.5). To tackle this problem, we propose a novel vanishing point detecting algorithm utilizing global photometric and geometric cues instead of edges detected from local gradient information. Specifically, we first select several candidate locations of dominant vanishing points from a photo and then apply our segmentation algorithm which aggregates global photometric and geometric cues within potential slices. The segmentation with maximum photometric and geometric distances between adjacent slices identifies the most confident location of the dominant vanishing point. We have shown that our algorithm succeeds in detecting vanishing points in outdoor natural scene photos where straight lines are often absent (Figure 3.5).

Second, once the dominant vanishing point is detected, segmenting this pie into geometrically consistent planes is also not trivial. There are a number of examples where adjacent planes do not differ significantly in color or texture (e.g., the walls and the ceiling in the second picture of Figure 1.4). However, existing image segmentation algorithms primarily employ photometric cues to calculate distance between adjacent regions (see Figure 1.4(b) for examples). To overcome this problem, we propose a geometric distance to describe the locational relationship of two regions with respect to the dominant vanishing point. Specifically, consider a polar coordinate system whose origin locates at the dominant vanishing point in the image, a region can be represented as a set of points that cover a specific range of angles. The geometric distance measures how two regions are overlapping with each other in terms of their ranges of angles. Two regions that belong to distinct geometrically consistent planes are less likely to have overlapped angles than those locating within the same plane. With such geometric cues in complement with photometric cues, our method is able to preserve essential geometric regions in the image (Figure 1.4(c)).

Finally, our segmentation method succeeds to extract a variety of compositions from landscape photos, and thus sets stage for many potential composition-based applications ranging from composition-based image retrieval to enhancement of composition design. In this thesis, we employ this technique to build an image retrieval system which provides on-site recommendation of high-quality images with similar compositions to amateur users. Given a query image, which can be a test shot taken by users, we first detect its dominant vanishing point and segment it into slices. By comparing both the location of vanishing point and the segmentation result of the query image with those of all exemplar high-quality images stored in our database, we can select exemplar images with similar compositions to the query image and show users more possible ways of enhancing the aesthetic quality of their photos.

1.5         Modeling Composition in Portrait Photography

Two fundamental questions are often raised when analyzing the composition of a portrait photograph: where are the human subjects in the photo and how do they pose? Traditional composition rules provide us with guidelines to answer the first question. For example, the rule of thirds suggests that putting the human subjects near the 1/3 point of an image is more appealing than at the center. Based on these rules, several methods have been developed to model and assess the positioning of human subjects in a photo.

Nevertheless, the second question remains a challenge. To address this problem, we leverage an important observation in portrait photograph: experienced portrait photographers often use triangle techniques to create interesting and good-looking poses for human subject. For example, a widely-used rule for posing is that one should try to avoid 90-degree body angles, because it typically looks unnatural and forced. In addition, triangle techniques are also frequently used to unify multiple human subjects and the surrounding environment, such as chairs and lamps, in a portrait photo.

Despite the popularity of triangle techniques in portrait photography, it is often difficult for amateurs to recognize such triangles, because most triangles do not have explicit edges and sometimes are even constructed by different objects. Moreover, triangles in portraits can be of various sizes, shapes, orientations, and appearances. Hence, our goal is to automatically detect potential triangles from professional photographers’ work in order to help amateurs recognize and learn from the usage of triangle techniques.

Our algorithm can be divided into two steps: First, a line segment detection module is used to extract candidate line segments from an image, which are subsequently filtered using the global contour information in the image. Second, the filtered line segments are fed into a triangle detection module as the candidate sides of triangles. Specifically, a RANSAC algorithm is developed to randomly pick two sides from all the candidates and fit the triangle. Two metrics, Continuity Ratio and Total Ratio, are defined to evaluate the fitness of these triangles. Only those triangles with high fitness scores will be shown to the users.

1.6      Contributions

Overall, this thesis makes the following major contributions:

• Composition modeling: We model the composition of a natural scene image by examining the perspective effects and partitioning the image into photometrically and geometrically consistent regions using our novel hierarchical image segmentation algorithm. We model the composition of a portrait by detecting the lines that can form dominant triangles.
• Dominant vanishing point detection: By aggregating the photometric and geometric cues using our segmentation algorithm, we develop an effective method to detect the dominant vanishing point in an arbitrary image.
• Triangle Detection in Portraits: We propose a RANSAC algorithm to detect triangles in portraits with a variety of sizes, shapes, orientations, and appearances, helping amateurs to understand and learn from the usage of triangle techniques in professional photographers’ work.
• Composition-sensitive retrieval for on-site feedback: We develop triangle detection techniques for photographic composition understanding so that photographs with similar or dissimilar composition as the query photo can be retrieved from a collection of photos to assist the photographer.

Admittedly, our technique for natural scenes cannot yet model all potential compositions in such photos, especially when there is a lack of triangles. While in this thesis we focus on the use of triangle techniques in photography, we point out that there are studies that explore other important aspects of composition, including the semantic features (e.g., buildings, trees, roads) [21, 22, 16, 17] . It would be ideal to integrate all these features in order to gain a deeper understanding of the image composition, but a thorough discussion on this topic is beyond the scope of this thesis.

1.7       Outline of the Thesis

The remainder of the thesis is organized as follows: The related work is discussed in Chapter 2. The methods for modeling composition in natural scenes and portraits are presented in Chapters 3 and 4, respectively. In Chapter 5, experiments and results are described. We conclude and suggest future research in Chapter 6.

MODELING PHOTOGRAPHIC COMPOSITION VIA TRIANGLES

Sharing is caring!