- : Ms Word, Ms Word Format
- : 70 Pages
- : ₦5,000
- : 1-5 Chapters
- Click to DOWNLOAD Materials
A COMPREHENSIVE STUDY OF THE REGULATION AND BEHAVIOR OF WEB CRAWLERS
Abstract
Search engines and many web applications such as online marketing agents, intelligent shopping agents, and web data mining agents rely on web crawlers to collect information from the web, which has led to an enormous amount of web traffic generated by crawlers alone. Due to the unregulated open-access nature of the web, crawler activities are extremely diverse. Such crawling activities can be regulated from the server side by deploying the Robots Exclusion Protocol in a file called robots.txt. Ethical crawlers (and many commercial) will follow the rules specified in robots.txt files. Since the Robots Exclusion Protocol has become a de facto standard for crawler regulation, a thorough study of the regulation and behavior of crawlers with respect to the Robots Exclusion Protocol allows us to understand the impact of search engines and the current situation of privacy and security issues related to web crawlers.
The Robots Exclusion Protocol allows websites to explicitly specify an access preference for each crawler by name. Such biases may lead to a “rich get richer” situation, in which a few popular search engines ultimately dominate the web because they have preferred access to resources that are inaccessible to others. We propose a metric to evaluate the degree of bias to which specific crawlers are subjected. We have investigated 7,593 websites covering education, government, news, and business domains, and collected 2,925 distinct robots.txt files. Results of content and statistical analysis of the data confirm that the crawlers of popular search engines and information portals, such as Google, Yahoo, and MSN, are generally favored by most of the websites we have sampled. The biases toward popular search engines are verified by applying the bias metric to 4.6 million robots.txt files from the web. These results also show a strong correlation between the search engine market share and the bias toward particular search engine crawlers.
Since the Robots Exclusion Protocol is only an advisory standard, actual crawler behavior may differ from the regulation rules. In other words, crawlers may ignore robots.txt files or violate part of the rules in robots.txt files. A thorough analysis of web access logs reveals many potential ethical and privacy issues in web crawler generated visits. We present the log analysis results of three large scale websites and the applications of the data extracted from the log analysis including estimating the crawler population and user stability measures.
To minimize negative aspects of crawler generated visits on websites, the ethical issues of crawler behavior with respect to the crawling rules specified in websites is studied in this thesis. As many web site administrators and policy makers have come to rely on the informal contract set forth by the Robots Exclusion Protocol, the degree to which web crawlers respect robots.txt policies has become an important issue of computer ethics. We analyze the behaviors of web crawlers in a crawler honeypot, a set of websites where each site is configured with a distinct regulation specification using the Robots Exclusion Protocol in order to capture specific behaviors of web crawlers. A set of ethicality models is proposed to measure the ethicality of web crawlers computationally based on their conformance to the regulation rules. The results show that ethicality scores vary significantly among crawlers. Most commercial web crawlers receive good ethicality scores; however, many commercial crawlers still consistently violate certain robots.txt rules.
The bias and ethicality measurement results calculated based on our proposed metrics are important resources for webmasters and policy makers to design websites and policies. We design and develop BotSeer, a web-based robots.txt and crawler search engine that makes these resources available for users. BotSeer currently indexes and analyzes 4.6 million robots.txt files obtained from 17 million websites as well as three large web server logs and provides search services and statistics of web crawlers for researching web crawlers and trends in Robot Exclusion Protocol deployment and adherence. BotSeer serves as a resource for studying the regulation and behavior of web crawlers as well as a tool to inform the creation of effective robots.txt files and crawler implementations.
Table of Contents
List of Figures viii
List of Tables x
Chapter 1 Introduction 1
1.1 Methodology 7
Chapter 2 Related Work 9
2.1 Web Crawlers 9
2.2 Robots Exclusion Protocol 9
2.3 The robots.txt Files 12
2.4 Log Analysis 14
2.5 Crawler Behavior 16
2.6 Crawler Ethics 17
2.7 Population Estimation 18
Chapter 3 Biases toward Crawlers 20
3.1 Data Collection 20
3.1.1 Data Sources 20
3.1.2 Crawling for Robots.txt 20
3.2 Usage of the Robots Exclusion Protocol 21
3.3 Robot Bias 23
3.3.1 The GetBias Algorithm 23
3.3.2 Measuring Overall Bias 25
3.3.3 Examining Favorability 25
3.4 Bias Results 26
3.4.1 History of Bias 27
3.4.2 Search Engine Market vs. Robot Bias 29
3.4.3 Results on Larger Data Set 29
Chapter 4 Log Analysis 34
4.1 Crawler Identification 35
4.2 Crawler Traffic Statistics 36
Chapter 5 BotSeer System 39
5.1 DATA 41
5.1.1 Robots.txt Files 41
5.1.2 Web Server Logs 44
5.1.3 Open Source Crawlers 45
5.2 WEB APPLICATION 45
5.2.1 Robots.txt File Search 45
5.2.2 Crawler Search 47
5.2.3 Data Analysis 49
5.2.3.1 Bias Analysis 50
5.2.3.2 Dynamic Bias Analysis 51
5.2.3.3 Robot Generated Log Analysis 52
5.3 DISCUSSION 53
Chapter 6 Crawler Behavior Analysis 56
6.1 Models 56
6.1.1 Vector Model of Crawler Behavior 56
6.1.2 Ethicality Metrics 57
6.1.2.1 Binary Model 57
6.1.2.2 Probabilistic Model 57
6.1.2.3 Relative Model 58
6.1.2.4 Cost Model 58
6.2 Experiments 60
6.2.1 Crawler Behavior Test: Honeypot 60
6.2.2 Results 64
6.2.2.1 Binary Ethicality 65
6.2.2.2 Probabilistic Ethicality 65
6.2.2.3 Relative Ethicality 68
6.2.2.4 Cost Ethicality 69
6.2.3 Temporal Ethicality 70
6.2.4 Compare to Favorability 71
6.3 Applications 71
6.4 Effectiveness of Search Engines 72
Chapter 7 Estimating Crawler Population 75
7.0.1 Capture-Recapture Models 75
7.0.1.1 Lincoln-Peterson Model 76
7.0.1.2 Dependency of Capture Sources 76
7.0.1.3 Model M0 78
7.0.1.4 Model Mh 79
7.0.1.5 Model Mt 80
7.0.1.6 Model Mth 81
Chapter 8 Conclusions and Future Work 83
8.1 Future Work 84
8.1.1 Bias Analysis 84
8.1.2 Crawler Ethics 86
8.1.3 BotSeer Service 87
Bibliography 89
List of Figures
| 1.1 | The high-level architecture of a general web crawler system. . . . . . . . . . . . . | 2 |
| 1.2 | An example of fetching process in crawlers. . . . . . . . . . . . . . . . . . . . . . | 3 |
| 1.3 | The flow of an expected crawler activity. . . . . . . . . . . . . . . . . . . . . . . . | 4 |
| 3.1 | Probability of a website that has robots.txt in each domain. . . . . . . . . . . . . | 21 |
| 3.2 | Distribution of robots.txt by domain suffixes. Because of long-tailed distribution, | |
| only the top 10 suffixes are shown . . . . . . . . . . . . . . . . . . . . . . . . . . | 22 | |
| 3.3 | Most frequently used robot names in robots.txt files. The height of the bar repre- | |
| sents the number of times a robot appeared in our dataset. . . . . . . . . . . . . | 26 | |
| 3.4 | The distribution of a robot being used. . . . . . . . . . . . . . . . . . . . . . . . . | 27 |
| 3.5 | Top 10 and Bottom 10 robots ranked by ∆P(r), the proportion of the difference | |
| between favored and disfavored robots. . . . . . . . . . . . . . . . . . . . . . . . . | 28 | |
| 3.6 | The search engine market share for 4 popular search engines between 12/05 and | |
| 09/06, and ∆P rating of favorability of these engines. . . . . . . . . . . . . . . . | 33 | |
| 3.7 | Search engine market share vs. robot bias. . . . . . . . . . . . . . . . . . . . . . . | 33 |
| 4.1 | Distribution of visits per day from each unique IP address. . . . . . . . . . . . . | 34 |
| 4.2 | Distribution of visits per day from each unique IP address. . . . . . . . . . . . . | 35 |
| 4.3 | The comparison of crawler visits and user visits as a function of date. . . . . . . | 37 |
| 4.4 | The geographical distribution of web crawlers. . . . . . . . . . . . . . . . . . . . . | 37 |
| 4.5 | The geographical distribution of web crawlers named as Googlebot. The blue and | |
| red circles point out the well behaved and badly behaved Googlebots respectively. | 38 | |
| 5.1 | The architecture of BotSeer system. . . . . . . . . . . . . . . . . . . . . . . . . . | 41 |
| 5.2 | The Homepage of BotSeer system. . . . . . . . . . . . . . . . . . . . . . . . . . . | 42 |
| 5.3 | The architecture of BotSeer crawler. . . . . . . . . . . . . . . . . . . . . . . . . . | 42 |
| 5.4 | Data selection module between physical storage and applications. . . . . . . . . . | 44 |
| 5.5 | Distribution of visits per day from each unique IP address. . . . . . . . . . . . . | 45 |
| 5.6 | BotSeer robots.txt search component response to query “botname:msnbot”. . . | 46 |
| 5.7 | The crawler search result page for query “googlebot”. . . . . . . . . . . . . . . . | 48 |
| 5.8 | The crawler search result page for query “googlebot”. . . . . . . . . . . . . . . . | 49 |
| 5.9 | The geographical distribution of web crawlers that visit CiteSeer. Gray points are the location of crawlers that visit CiteSeer. The blue and red circles point out the | |
| well behaved and bad behaved “Googlebot” respectively. . . . . . . . . . . . . . . | 49 | |
| 5.10 Detailed bias analysis of a website. . . . . . . . . . . . . . . . . . . . . . . . . . . | 50 | |
| 5.11 Bias analysis result page of 1,858 named crawlers. . . . . . . . . . . . . . . . . . | 51 | |
| 5.12 Dynamic Bias analysis on the query “Robots Exclusion Protocol”. . . . . . . . . | 52 | |
| 5.13 The statistics of crawler generated traffic in CiteSeer. . . . . . . . . . . . . . . . . | 53 |
| 5.14 The crawler generated traffic analysis and monitor. . . . . . . . . . . . . . . . . .
5.15 Comparison of user visits and crawler visits. The press release shows two dates |
54 |
| that BotSeer is reported in news media. . . . . . . . . . . . . . . . . . . . . . . . | 55 |
| 6.1 Site structure and robots.txt file for Honeypot 1. . . . . . . . . . . . . . . . . . . | 61 |
| 6.2 Site structure and robots.txt file for Honeypot 2. . . . . . . . . . . . . . . . . . . | 62 |
| 6.3 Site structure and robots.txt file for Honeypot 3. . . . . . . . . . . . . . . . . . . | 62 |
| 6.4 Site structure and robots.txt file for Honeypot 4. . . . . . . . . . . . . . . . . . . | 62 |
| 6.5 Site structure and robots.txt file for Honeypot 5. . . . . . . . . . . . . . . . . . . | 63 |
| 6.6 Site structure and robots.txt file for Honeypot 6. . . . . . . . . . . . . . . . . . . | 63 |
| 6.7 Site structure and robots.txt file for Honeypot 7. . . . . . . . . . . . . . . . . . . | 63 |
| 6.8 Site structure and robots.txt file for Honeypot 8. . . . . . . . . . . . . . . . . . . | 64 |
| 6.9 Site structure and robots.txt file for Honeypot 9. . . . . . . . . . . . . . . . . . . | 64 |
| 6.10 Temporal changes in ethicality of web crawlers. . . . . . . . . . . . . . . . . . . . | 71 |
| 6.11 A rule based system to monitor the ethicality of web crawlers. . . . . . . . . . . . | 72 |
| 6.12 Comparison of the effectiveness of Google, Yahoo, MSN and Baidu. . . . . . . . . | 74 |
| 7.1 Set drawing of three capture sources C1, C2 and C3. . . . . . . . . . . . . . . . .
7.2 The distribution of population size N with model M0. The maximum likelihood |
77 |
| is obtained at N = 11,844. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 79 |
| 7.3 The simulation of population size N in model Mh. . . . . . . . . . . . . . . . . . | 80 |
| 7.4 The simulation of population size N in model Mt. . . . . . . . . . . . . . . . . . | 81 |
| 7.5 The simulation of population size N in model Mth. . . . . . . . . . . . . . . . . . | 82 |
List of Tables
| 1.1 | HTTP request. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 3 |
| 2.1 | An example of robots.txt file from http://botseer.ist.psu.edu. . . . . . . . . . . . | 11 |
| 2.2 | Examples of matching between request URL and Record Path. . . . . . . . . . . | 12 |
| 3.1 | The average size (in bytes) and average length (in number of lines) of the collected | |
| robots.txt files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 22 | |
| 3.2 | The frequency of Crawl-Delay rules for different robots. . . . . . . . . . . . . . . | 23 |
| 3.3 | Top 10 favored and disfavored robots. σ is the standard deviation of ∆P(r). . . . | 29 |
| 3.4 | Top 5 favored and disfavored robots on the USA university websites. σ is the | |
| standard deviation of ∆P(r). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 30 | |
| 3.5 | Top 5 favored and disfavored robots in the Government websites. σ is the standard | |
| deviation of ∆P(r). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 30 | |
| 3.6 | Top 5 favored and disfavored robots in the newspaper websites. σ is the standard | |
| deviation of ∆P(r). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 31 | |
| 3.7 | Top 5 favored and disfavored robots in the company websites. σ is the standard | |
| deviation of ∆P(r). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 31 | |
| 3.8 | Top 5 favored and disfavored robots in the European university websites. σ is the | |
| standard deviation of ∆P(r). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 32 | |
| 3.9 | Top 5 favored and disfavored robots in the Asian university websites. σ is the | |
| standard deviation of ∆P(r). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 32 | |
| 4.1 | Statistics for IPs addresses visiting a digital library in one week. . . . . . . . . . | 36 |
| 6.1 | Statistics of the rules in robots.txt files of 2.2 million websites. . . . . . . . . . . | 61 |
| 6.2 | Probability of a rule being violated or misinterpreted. . . . . . . . . . . . . . . . | 65 |
| 6.3 | The binary violation table of web crawlers derived from honeypot access logs. . . | 66 |
| 6.4 | The full violation table of web crawlers derived from honeypot access logs. . . . . | 67 |
| 6.5 | Probability Ethicality of web crawlers. . . . . . . . . . . . . . . . . . . . . . . . . | 68 |
| 6.6 | Relative ethicality of web crawlers. . . . . . . . . . . . . . . . . . . . . . . . . . . | 68 |
| 6.7 | Content ethicality scores for crawlers that visited the honeypot. . . . . . . . . . . | 69 |
| 6.8 | Access ethicality scores for crawlers that visited the honeypot. . . . . . . . . . . | 69 |
| 6.9 | Additional ethical behavior of crawlers. . . . . . . . . . . . . . . . . . . . . . . . | 70 |
| 6.10 Comparison of crawler probabilistic ethicality and favorability scores. . . . . . . . | 71 | |
| 6.11 Effectiveness of search engines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . | 73 | |
| 6.12 Comparison of the effectiveness of Google, Yahoo, MSN and Baidu. . . . . . . . . | 74 | |
Chapter 1
Introduction
Web crawlers (a.k.a. “spiders,” “robots,” “bots” or “harvesters”) are self-acting agents that navigate around the clock through the hyperlinks of the web, harvesting topical resources with zero cost in human management [12, 11, 51] . Web crawlers are essential to search engines; without crawlers, there would probably be no search engines. Web search engines, digital libraries, and many other web applications such as offline browsers, internet marketing software and intelligent searching agents heavily depend on crawlers to acquire documents [13, 28, 73] . For example, Google, Yahoo and MSN crawlers traverse billions of web pages periodically to support the variety of services provided by these search engines. Shopping bots and price bots bring users discounted products and price comparisons everyday. There is even a dating bot[1] that provides possible dating information to users. The crawler functions and activities are extremely diverse because of the variety of tasks. These functions and activities include not only regular crawls of web pages for general-purpose indexing and public services, but also different types of unethical functions and activities such as automatic extraction of email and personal identification information as well as service attacks. Even general-purpose web page crawls can lead to unexpected problems for Web servers such as a denial of service attack in which crawlers may overload a website such that normal user access is impeded. Crawler-generated visits can also affect log statistics significantly so that real user traffic is overestimated.
A general high-level crawler architecture is shown in Figure 5.1. Multithreaded fetchers download web pages from the World Wide Web and pass the web pages to a link parser that extracts hyperlinks (URLs) from these pages. The extracted URLs will then be filtered by a URL filter that encodes the regulation rules provided by corresponding websites. Valid URLs will be queued and scheduled for fetchers to download from the World Wide Web. A crawling cycle includes downloading web pages, extracting and filtering URLs, and generating new tasks.
Figure 1.1. The high-level architecture of a general web crawler system.
Therefore, a crawler can traverse the web by following hyperlinks.
The traversal strategy of a crawler can be designed based on specific tasks. The goal of web crawlers is to visit all or part of the nodes (web pages) on the web graph. Thus, the traversal strategy is essentially a graph search algorithm. A general web crawler typically implements a breadth first search (BFS), a depth first search (DFS) or a focused search to traverse the web graph. BFS is a search algorithm that starts with the root node and explores all the nodes connecting to it. For each of these nodes, BFS explores their unexplored connecting nodes until it reaches the goal. DFS is a search algorithm that starts with the root node and explores as far as possible along each branch before backtracking. Both BFS and DFS can make sure web crawlers traverse connected web subgraphs. Each has advantages and disadvantages that make the choice problem dependent, however. BFS crawlers need more space to store all visited pages in each node level than DFS crawlers, which only need to store visited pages in one branch of the web graph. DFS crawlers, however, can be trapped by infinite link loops. BFS crawlers also yield higher quality pages [47] .
Focused crawling was brought to attention by both researchers and practitioners in the late 90’s [45] . A focused crawler (a.k.a. topical crawler) attempts to download only web pages that are relevant to predefined topics. In practice, BFS, DFS and focused crawling are all used in web crawler designs for different tasks.
A typical fetching process is illustrated in Figure 1.2. A URL is parsed into three parts: protocol, host and request. The protocol and host are translated to port number and IP address respectively which are used to establish socket connection to the remote web server. When the socket connection is successfully established, the request will be sent to the server to download corresponding document. A typical HTTP request is listed in Table 1.1. In the HTTP request fields, the HTTP method (first line), Host, and connection are required fields in HTTP specification. The rest of the fields are recommended. The User-Agent and Referrer fields, however, are very important to study crawler as well as user behavior and click patterns. After the request
Figure 1.2. An example of fetching process in crawlers.
| GET /about.html HTTP/1.1
Host: botseer.ist.psu.edu connection: close Accept-CharSet: UTF-8 Accept: text/html User-Agent: Mozilla/4.0 (compatible; SomeBotName 7.0; Windows NT 5.1) Referrer: /index.jsp |
Table 1.1. HTTP request.
is sent to web server, the server will parse the request and follow five steps to respond to the request: 1. identify connection type; 2. lookup Host configuration; 3. locate requested files in local directories; 4. send response to client; 5. log request.
On account of dramatically increasing web related services, web crawlers have become much more complicated, and their crawling functions and activities vary significantly. As a result, the regulation of web crawlers has become a difficult and important problem. Because of the highly automated nature of the crawlers, rules must be made to regulate their crawling activities in order to prevent an undesired impact to the server workload and to prevent access to information that is not to be offered to the public. In addition, these rules should assist welcome crawlers index the website more efficiently.
The Robots Exclusion Protocol has been proposed [37] to provide advisory regulations for crawlers to follow. A file called robots.txt, which contains crawler access policies, is deployed in the root directory of a website and is accessible to all crawlers. The Robots Exclusion Protocol allows website administrators to indicate to visiting robots which parts of their site should not be visited as well as a minimum time interval between visits. If there is no robots.txt file on a website, robots are free to crawl all content. Ethical crawlers read this file and obey the rules during their visit to the website. An expected crawler activity is illustrated in Figure 1.3. The
Figure 1.3. The flow of an expected crawler activity.
expected crawler activity should be parsing the domain from a URL and examine whether the request is restricted by the robots.txt files before fetching web pages.
The robots.txt convention has been adopted by the community since the late 1990s and has continued to serve as one of the predominant means of crawler regulation. Our study shows that more than 30% of active websites deploy this standard to regulate crawler activities [70] . Although the robots.txt convention has become a de facto standard for crawler regulation, little work has been done to investigate its usage in detail, especially on the scale of the Web.
More importantly, as websites may favor or disfavor certain crawlers by assigning to them different access policies, this bias can lead to a “rich get richer” situation whereby some popular search engines are granted exclusive access to certain resources, which in turn could make them even more popular. Considering the fact that users often prefer a search engine with broad (if not exhaustive) information coverage, this “rich get richer” phenomenon may introduce a strong influence on users’ choice of search engine, which will eventually be reflected in the search engine market share. On the other hand, it is often believed (although this is an exaggeration) that “what is not searchable does not exist,” and this phenomenon may also introduce a biased view of the information on the Web.
The robots.txt files also play an important role in indexing digital repositories over OAI-PMH protocol [42] . The OAI-PMH describes a set ofverbs and specifications for harvesters to request interoperable metadata of digital objects from online digital repositories [16] . A digital repository (data provider) typically provides the metadata of its digital objects to service providers (e.g., search engines) with a set of base URLs through which OAI-PMH requests are submitted. The metadata of a digital object also contains the URL where the actual object can be located.
Major search engines including Google and Yahoo already built harvesters to collect resources from large repositories [42] . Other efforts have been made to convert the metadata of digital objects to general web pages for general web crawlers to index [77] .
Both the OAI-PMH base URLs and the URLs of the digital objects are regulated by robots.txt files since major search engine crawlers follow the robots.txt rules whenever they are fetching a URL. In such situation, the robots.txt files may prevent the harvesters from requesting resources from digital repositories. Thus, improper design of robots.txt rules can lead to unexpected results. It is reported that robots.txt files protecting both the OAI-PMH base URLs and the object URLs which results in that the major search engines fail to index the repositories even though the intent of implementing OAI-PMH for the repositories is to share their metadata [42] . Therefore, the robots.txt file is very important to the search engine coverage of digital repositories. The research of search engine coverage of the OAI-PMH corpus suggests that robots.txt files in digital repositories should be carefully designed so that it does not prevent service providers from requesting the repositories over OAI-PMH and also provides necessary protections at the same time.
Alternative techniques are used to regulate the behavior of Web crawlers. The Robots META tag can be written in an HTML page to prevent ethical crawlers from crawling or indexing the page. Similar to the Robots Exclusion Protocol, the Robots META tag is an advising rule that will only be obeyed by ethical crawlers. Enforcing techniques such as robot traps[2] or IP restrictions are also used in Web servers. These techniques can usually keep unwelcome crawlers away from the site. Although the enforcing techniques seem more powerful in regulating crawlers, they are also vulnerable if the new crawling techniques can avoid the traps [27] . In addition to attempts to restrict web crawlers, efforts have been made to assist crawlers to better index websites. A new tool, Sitemaps [61] , helps search engines by specifying how often pages are changed. However, the Sitemap protocol does not yet include specifications of load balancing for websites.
Although the Robots Exclusion Protocol has been widely adopted by websites, it is an advisory standard that cannot prevent misbehaved crawlers. How web crawlers really behave on websites is the key to designing regulation rules and standards to guide and regulate web crawlers. According to our research, a significant portion of web crawlers fails to identify themselves. A significant amount of crawler visits also violate the rules specified in robots.txt. This diverse crawler behavior brings up ethical issues related to automated computer software as well.
Previous research on computer ethics (a.k.a. machine ethics) has primarily focused on how humans use technology. With the growing role of autonomous agents on the internet, however, the ethics of machine behavior has come to the attention of the research community [2, 4, 24, 32, 53, 73] . The field of computer ethics is especially important in the context of web crawling since collecting and redistributing information from the web often leads to considerations of information privacy and security.
Because the Robots Exclusion Protocol serves only as an unenforced advisory to crawlers, web crawlers may ignore the rules and access part of the forbidden information on a website. It is reported that many web crawlers, including some commercial crawlers, disregard part of the rules or disregard robots.txt completely.[3] One company even states that it does not follow regulation standards in its crawler description documents.[4] Crawlers can attempt to acquire hidden documents by guessing URLs from restricted directory names listed in the robots.txt files.[5] The crawler visit interval is also important to websites, especially when the access bandwidth is limited. Webmasters complain about crawlers occupying too much bandwidth because users experience delay in regular access of websites.[6] Clearly, the usage of the Robots Exclusion Protocol and the behavior of web crawlers with respect to the robots.txt rules provide a foundation for a quantitative measure of web crawler ethics.
It is difficult to interpret ethicality in different websites. The unethical actions in one website may not be considered unethical in others. For example, crawling more than 1 page per second may significantly affect the normal usage of a small website. The same crawling speed, however, may be expected for a large website with fast updating content. Little research has been done in the area of developing computational models to formulate ethicality and to provide measures of web crawler ethics.
This thesis studies the regulation and behavior of web crawlers based on the Robots Exclusion Protocol and web access log analysis. The contributions of the thesis are as follows:
The thesis proposes a quantitative metric to measure the bias in the regulations of web crawlers. By applying the metric to a large sample of websites, the thesis presents the findings about the most favored and disfavored crawlers and suggests complements to the Robots Exclusion Protocol [70, 69] .
The thesis analyzes the statistics of crawler traffic for different websites and estimates the population of active web crawlers with capture-recapture models.
The thesis formally defines crawler ethicality based on common concepts of ethical crawlers in the research community and proposes a vector space model of web crawler ethics based on the Robots Exclusion Protocol. A honeypot is designed to capture crawler behaviors. The ethicality results of major search engine crawlers that visited our honeypot are presented in this thesis.
A robots.txt and web crawler search engine, BotSeer,[7] is designed and developed to provide resources for analyzing web crawler behavior and trends in the Robots Exclusion Protocol deployment [68] .
The goal of the thesis is to conduct a comprehensive survey of web crawler regulations and behavior based on the Robots Exclusion Protocol. The widely adopted Robots Exclusion Protocol provides a foundation for the study. As introduced in previous chapters, web crawlers may be regulated by different rules in the robots.txt files. Such bias may have a significant impact on the indexable content of each crawler. Since the Robots Exclusion Protocol specifies a set of advisory rules, the crawler may behave differently than the regulation rules specify. Ethical issues arise with crawler behavior violations.
This thesis attempts to explore the following questions:
How is the Robots Exclusion Protocol used?
Does a robot bias exist?
How should such a bias be measured quantitatively?
What are the implications for such a bias?
How many crawlers are there on the web?
How do crawlers actually behave on the web?
How can we measure a crawler’s ethicality towards a website?
A large-scale data collection is the basis for answering these questions and for showing the significance of the quantitative analysis. For this thesis, 4.6 million robots.txt files were collected and 200GB of web access logs were processed to support the quantitative analysis.
The study of crawler regulation is focused on the analysis of the bias in robots.txt files. In the first step, we collect real-world robots.txt files from 17 million unique websites with different functionalities, covering the domains of education, government, news and business. Thus, regulation bias can be compared across domains. A quantity metric is proposed based on the Robots Exclusion Protocol to measure its biases in terms of web crawlers. A formal definition of bias in robots.txt files is proposed, and an overall favorability metric is proposed to measure the bias toward each web crawler over a set of websites. With the favorability metric, the most favored and disfavored crawlers on the Web are identified.
To summarize the analysis of the robots.txt files and crawler behavior and to provide information to the public, we design and develop BotSeer,[8] the first robots.txt and crawler search engine to deliver data and measures to the public.
To study the behavior of web crawlers, we process 200GB of the access logs from three independent websites. Statistics from the crawler-generated logs provide answers to the fifth questions above. We also set up a honeypot, a set of websites designed based on the specifications in the Robots Exclusion Protocol and common rules derived from 4.6 million sample websites.
The honeypot includes all cases where the robots.txt rules can be violated by crawlers. Once a crawler visits the honeypot, its behaviors are recorded and analyzed. With the honeypot design and data, we proposed a set of models to measure the ethicality of web crawlers. Thus, ethical issues can be explored by examining the ethicality measure of web crawlers. The most ethical and unethical crawlers are identified based this measure.
The Robots Exclusion Protocol will be discussed in detail in Chapter 2. Chapter 3 discusses the methodology of the thesis in the study of web crawlers. Chapter 4 presents the study of usage and the bias analyses of robots.txt files. Chapter 5 presents the statistics of web crawler traffic and log analysis results. Chapter 6 presents BotSeer, the first robots.txt and crawler search engine to assist crawler related research and regulation policy design. The crawler behavior study and ethicality measures are presented in Chapter 7 with the honeypot, a set of websites to test the behavior of web crawlers, and presents the measurement of crawler ethicality. Chapter 8 presents the capture-recapture models used to estimate web crawler population.
[1] http://en.wikipedia.org/wiki/Windows Live Agents
[2] http://www.fleiner.com/bots/
[3] http://labnol.blogspot.com/2007/01/google-spiders-sometimes-ignore-meta.html
[4] http://www.munax.com/crawlingfaq.htm
[5] http://forums.searchenginewatch.com/archive/index.php/t-2786.html
[6] http://www.webmasterworld.com/forum30/33074.htm
[7] http://botseer.ist.psu.edu
[8] http://botseer.ist.psu.edu/
A COMPREHENSIVE STUDY OF THE REGULATION AND BEHAVIOR OF WEB CRAWLERS
