Web Robot Detection in Academic Publishing
Athanasios Lagopoulos

Grigorios Tsoumakas

Georgios Papadopoulos

Aristotle University of Thessaloniki
Thessaloniki, Greece
lathanag@csd.auth.gr

Aristotle University of Thessaloniki
Thessaloniki, Greece
greg@csd.auth.gr

Atypon Systems Inc.
Santa Clara, California, USA
georgios@atypon.com

arXiv:1711.05098v1 [] 14 Nov 2017

ABSTRACT
Recent industry reports assure the rise of web robots which comprise more than half of the total web traffic. They not only threaten
the security, privacy and efficiency of the web but they also distort
analytics and metrics, doubting the veracity of the information
being promoted. In the academic publishing domain, this can cause
articles to be faulty presented as prominent and influential. In
this paper, we present our approach on detecting web robots in
academic publishing websites. We use different supervised learning algorithms with a variety of characteristics deriving from both
the log files of the server and the content served by the website.
Our approach relies on the assumption that human users will be
interested in specific domains or articles, while web robots crawl a
web library incoherently. We experiment with features adopted in
previous studies with the addition of novel semantic characteristics
which derive after performing a semantic analysis using the Latent
Dirichlet Allocation (LDA) algorithm. Our real-world case study
shows promising results, pinpointing the significance of semantic
features in the web robot detection problem.

KEYWORDS
web robot detection, crawler, semantics, content analysis

1

the filtering out of their activities are the actions need to be taken
so as to maintain the higher resolution picture of unbiased and
unaltered information shared in the web [8].
This paper introduces a novel web robot detection approach
for content rich websites. The key assumption of the proposed
approach is that humans are typically interested in specific topics,
subjects or domains, while robots typically crawl all the available
resources irrespectively of their content, with the exception of a
special class of web robots, called focused crawlers [4]. Based on
this assumption, our main contribution is a novel class of features,
for representing user sessions, that capture the semantics of the
content of requested resources. Correspondingly, our main research
question is whether such features can improve the results of supervised learning approaches to web robot detection in content
rich websites. Empirical results on real-world data from the digital
library of an academic publisher provide evidence in favor of a
positive answer to our research question.
This paper is structured as follows: after providing the background and related work in Section 2 , we introduce our approach
on extracting semantics from sessions in Section 3. In Section 4, we
describe our real world case study by presenting the data and the
steps taken before creating a detection model, while in Section 5 we
discuss the results of our study proving our assumption. Finally, in
Section 6, we review our approach and draw some future directions.

INTRODUCTION

Web (ro)bots, also known as web crawlers, are computer programs
that request resources from web servers across the Internet without
human intervention. The constant growth of Web 3.0 technologies
and social media generate a huge amount of valuable information
ready to be accessed by both traditional web crawlers and emerging
advanced robots representing Internet of Things devices, such as
smart watches, cars and digital assistants [15]. As of 2016, the web
traffic originated by web bots constitutes more than a half (51.8%)
of the total web traffic, being in an uptrend after three years of
decline [23].
Malicious bots threaten the security, privacy and performance
of a web application. Non-malicious bots are involved in analytics
skewing, affecting the reliability of metrics and, by extension, the
decision making process [5]. A recent industry report [13] points
out that large websites with unique content, such as blogs, online newspapers and digital libraries of academic publications, are
the most attractive to bots. The most common threat that such
websites need to deflect is skewing: their metrics and ratings are
altered, intentionally by malicious robots and unintentionally by
non-malicious robots, rendering their validity questionable and
giving the false impression that some piece of information is highly
popular and recommended by many [10]. In addition to this, social
bots contribute further to the spread of unverified information or
rumors. Therefore, the detection of web robots and by extension

2 BACKGROUND AND RELATED WORK
2.1 Web Robot Taxonomy
Several categories of malicious web robots have been defined based
on their behavioral characteristics and range of capabilities. Scrappers, for example, collect application content and data for use elsewhere and obtain limited-availability goods and services by unfair
methods [12]. Hacker tools are involved in credit card, credential
and token cracking and target resources of the application and
database servers to achieve denial of service (DDOS), while impersonators’ functionality spans from account creation and spamming
to ad fraud via false clicks and sniping by performing last minute
bids for goods [21]. In addition, a taxonomy listing the automated
threats on web applications [9] has been created by the community
of the Open Web Application Security Project1 (OWASP).
Similarly, a variety of different categories of non-malicious bots
have been defined. Search engine crawlers collect information to
improve their ranking algorithms, feed fetchers carry the information of a website to a web or mobile application, while monitoring
bots help developers keep track of the health and function of their
website [7]. Interestingly enough, feed fetchers comprise more than
12% of the total traffic, with Facebook’s mobile application feed
1 http://www.owasp.org/

fetcher being the most active web robot accounting for 4.4% of all
website traffic [23].

2.2

are considered those that visit the same page and with the
same HTTP method.
• HTTP requests. This feature consist of 4 different values.
Each value represents the percentage of requests of the
different HTTP response codes. We separate the HTTP
response codes in 4 categories depending on the first digit
of their code: Successful requests - 2xx, Redirection - 3xx,
Client Errors - 4xx and Server Errors - 5xx.
• Specific Type Requests. The percentage of type X requests
over the number of all requests. This feature can be modified according to the application.

Web Robot Detection Approaches

There are four main categories of web robot detection approaches:
syntactic log analysis, where string processing techniques are used;
analytical learning, which make use of machine learning algorithms
and features that contain different properties deriving from the
user sessions in the server; traffic pattern analysis, which search
for statistical diversity between the features of human and robots;
Turing test systems, which identify a robot in real-time by means of a
Turing test. Recent work on web robot detection mainly focuses on
analytical learning approaches, which achieve considerably better
results than the other categories of approaches, since the latter rely
on procedures or algorithms that a properly engineered bot can
evade [5].

2.3

Many websites are based on content management systems or
specialized web applications that log additional information about
the identity of the visitors of a website, beyond that found in web
server access logs. Such information may be the country that the
request is coming from, by checking the user’s IP using a geolocation service; the username of a logged in user or an indicator if the
request originates from a web service. This kind of information can
be a great source of valuable features for a web robot detector, but
unfortunately it normally is application dependent and not always
available.

Analytical Learning

An important first step in analytical learning approaches is session
identification, which is concerned with breaking the click stream
into sessions. Various timeout thresholds have been investigated
in the past for session identification, such as 10 minutes [1], 30
minutes [20] and using dynamically adaptive thresholds ranging
from 30 minutes to 60 minutes [16].
An important second step concerns feature extraction from the
identified sessions, based on the variety of information found in
the entries of web server access log files, such as: the IP address
of the host that made the request to the server, the date and time
that the request was received, the resource requested, the HTTP
method used (e.g GET, HEAD, POST), the HTTP response code sent
back to the client (200, 404 etc.), the size of the returned object,
the Referer HTTP request header, which is the page that links to
the resource requested and the User-Agent String that identifies the
client’s browser. Figure 1 shows an example of an entry in a server
access log file.
Some of the typical features used by analytical learning approaches are:

2.4

Related Work

Several supervised and unsupervised analytical learning approaches
have been developed in the past based on a variety of learning algorithms and features. Tan et al. [20] used decision trees (C4.5
algorithm) to train a model using 25 different features that were
extracted from each user session. The feature vector included percentages of the different content (images, multimedia, HTML etc.),
time characteristics (average time, total time etc.), request types
(GET, POST, HEAD etc.) and other (IP, user-agent etc.). Bomhardt
et al. [3] used neural networks and included features like total number of bytes and percentage of response codes (200, 2xx, 404 etc.).
Stassopoulou et al. [17] used a heuristic semi-automatic method to
label the training data and introduced a Bayesian approach to classify the sessions. Stevanovic et al [18] experimented with a variety
of classifiers (C4.5, RIPPER, k-nearest, Naive Bayesian, Bayesian
Network, SVM and Neural Networks) and introduced two novel
features considering the page depth of a session’s requests and
the sequentiality of HTTP requests. Finally, the recent research
of Doran et al. [6] presents a novel approach, that in contrast to
the approaches mentioned earlier, can also be used for real-time
detection of web robots. Their approach is based on a first-order
discrete time Markov chain model and the request patterns of the
visitors. Also, in order to embed the detection algorithm in a realtime system they present a new way to identify sessions from log
file entries. In contrast with the above supervised approaches, the
study of Stevanonic et al. [19] used unsupervised neural networks
to detect humans and robots and to further analyze the behavior of
malicious and non-malicious web robots, while Zabihi et al. [22]
used the DBSCAN clustering algorithm with just four different
features.
Few studies have addressed web robot detection in the particular domain of academic publishing, where our empirical study
is focused. Huntington et al. [11] was the first to examine this

• Total Requests. The total number of requests in a session.
• Session Duration. Total time, in seconds, elapsed between
the first and the last request of a session.
• Average Time. Average time, in seconds, between two
consecutive requests.
• Standard Deviation Time. The standard deviation of time between two consecutive requests. This feature corresponds
to the visitors browsing behavior. Users usually don’t
spend the same time in each page. A page of an interesting
article will keep them, in the page, for a much longer time
than just a reference page. On the other hand, bots usually keep a stable browsing pattern where they visit each
page for almost the same amount of time. Therefore, the
standard deviation of humans will be greater than that of
robots.
• Repeated Requests. Percentage of repeated requests of a
page in a session. As repeated request is counted when
two identical requests occur in a session. Identical requests
2

Figure 1: Example of an entry in a web server access log file.
• Page Similarity (PS). The percentage of unique topics with
non-zero probability over all the topics with non-zero probability of a session.

domain, comparing the activity of robots in open access and restricted full text articles of a biology journal. Robots were identified
using different heuristic methods and behavioral pattern techniques
without using any machine learning. A second recent study benchmarked existing web robot detection approaches in Open Access
institutional repositories [10]. By performing a close review of the
literature, system documentation and open source code, the study
concludes that web robot detection is most successful when a variety of data and techniques are combined and pinpoints that non of
the examined methods leads to usage statistics that are completely
free of robot activity.

3

UT
TT
This feature models the dissimilarity of the different pages
visited during a session. The lower its value, the more
semantically similar the requested resources.
• Page Variance (PV). The semantic variance of the pages of
a session.
Í qÍk
2
i
j=1 (pi j − p j )
PV =
,
n
Í
where p = n1 ni=1 pi is the mean of the vectors pi associated with each request of the session. This feature
computes the mean Euclidean distance of the topic distribution of the resource of each request with that of the
mean topic distribution. The lower this distance, the higher
the semantic similarity of the requested resources in the
session.
• Boolean Page Variance. It is a boolean version of PV, where
prior to its calculation we set all non-zero pi j values to be
equal to 1.
PS =

EXTRACTING SEMANTICS FROM SESSIONS

As already mentioned in the introduction, this work is based on
the assumption that humans typically look for specific information
on a particular topic, while on the other hand, most of the web
robots go through the content of a website in a uniform fashion,
without favoring specific pages or content. Building a web robot
detection approach on top of this assumption, requires measuring
the semantic (in)coherence of the content visited during a session.
To achieve this, we start with topic modeling of the content of a
website using latent Dirichlet allocation (LDA) [2]. LDA describes
each document or, in this case, each web resource, as a probability
distribution over a user-defined number, k, of topics, where each
topic is a probability distribution over words.
Consider a session, S, comprising n requests for web pages (or
other textual resources, such as PDF files). Let pi j, be the probability
of topic j, 1 ≤ j ≤ k, for the web page associated with request i,
1 ≤ i ≤ n. Let also pi be a vector containing the distribution over
the k topics for the web page associated with request i. We extract
the following semantic features from S:

To the best of our knowledge, this is the first work to consider
modeling the semantics of the content of web sites for detecting
web robots.

4

REAL WORLD CASE STUDY

In this section, we present the real world case study where we
applied the proposed approach. First, we present the dataset and its
pre-processing procedure. Then, we discuss our session identification method and we define the features extracted for our detection
model. Finally, we unfold our three-step labeling process.

• Total Topics (TT). The number of topics with non-zero probability.
TT = |{(i, j) : 1 ≤ i ≤ n, 1 ≤ j ≤ k, pi j , 0}|

4.1

The higher the total number of topics with non-zero probability in all requests of a session, the lower the semantic
coherence of the session.
• Unique Topics (UT). The number of unique topics with nonzero probability in a session.

Dataset Preparation

Our data come from the web portal of a large commercial academic publisher. The data concern log files of the server requests
for an entire month (January 2014). The log files contain in total
25.318.451 requests without including those entries that don’t contain sufficient length of text or any semantic value. These entries
may include the home page, search results, log in page, about page,
contact page, etc. The remaining entries include only the following
requests regarding the available articles in the library:

UT = |{j : 1 ≤ j ≤ k, (∃i)pi j , 0}|
This feature also measures the semantic inconsistency of a
session, but without counting the same topic twice.
3

• Web Service. Indicates whether the session comes from a
web service or an application programming interface (API)
of the web application or not.
Note also that we customize the feature Specific Type Requests to
PDF requests, measuring the percentage of PDF requests over the
number of all requests, so as to match our case study.
To extract the semantic features we first applied the LDA algorithm on the full corpus of the 2.253.533 available articles, since
every page is associated with an article as specified in Section 4.1.
The number of topics was set to k = 5000, but for each article only
the top-10 topics with the highest probabilities were considered.
Most of the articles (2.177.524 - ≈ 96.6%) have non-zero probabilities
in 10 topics and 6.446 (≈ 0.02%) articles have non-zero probability
in just 1 topic. Finally, for each session, we constructed the five
features proposed in Section 3.

• Abstract (HTML): The web page containing the abstract of
an article.
• Full-text (HTML): The web page containing the full-text of
an article.
• Full-text (PDF): The PDF file containing the full-text of an
article.
• References (HTML): The web page containing the references of an article.
• Supplementary Material (HTML): The web page containing
supplementary material (Tables, Data etc.) of an article.
The aim of keeping pages that have these particular text content
is to reduce the size of the log entries while keeping all the useful
information that will enhance the semantically rich knowledge of
which the semantic features proposed take advantage of.
Besides the log files we were also provided with the full corpus
of the available articles in the digital library. In total, there are
2.253.533 different articles.

4.2

4.4

Session Identification

In this step of our pipeline we identify distinctive user sessions. Log
entries or requests are grouped together forming sessions. Generally, a session contains requests coming from a specific user, who
is identified by both the IP address and the user-agent string. The
user’s requests break into different groups by applying a timeout
threshold. In our case, the timeout threshold was set to 30 minutes
following what appears to be the standard. Furthermore, it is reasonable to assume that any session with less than 3 requests is too
short to produce significant features; thus, we ignore sessions with
less than 3 requests in total.
The session identification step further reduces the amount of
data by recognizing 10.039.241 sessions and the sessions that have
more than two requests are just 1.727.568. The sessions seems to
vary a lot for both the number of requests and the duration. They
have an average (median) of 7.8 (4) requests while there are sessions
with more than 10.000 requests. The average (median) duration
of session is 629.5 (208.5) seconds and the average time-threshold
between two consecutive requests is 131.6 seconds. The users of
the sessions have in total visited 1.653.999 unique articles.

4.3

Session Labeling

We identify and label a session as human or robot. The robot labeling procedure consists of three stages. During the first stage we
label each session using the API from useragentstring.com2 . The
API indicates if the user-agent string of a session belongs to one of
the following categories: Cloud Client, Console, Offline Browser,
Link Checker, Crawler, Feed Fetcher, Library, Mobile Browser, Validator, Browser, Unknown and Other. We consider as web robots
only the user-agents labeled as Crawler.
In the second stage, we use two lists containing regular expressions that match with the user-agent string of known bots. The
first one3 is the official list of user agents that are regarded as
robots/spiders by project COUNTER4 , which provides a code of
practice that helps librarians and publishers record and report online resource usage stats in a consistent and credible way. The
second list5 is a regularly updated list that the open source web analytics software Piwik6 uses. The sessions whose user-agent string
matches one of the regular expressions are labeled as robots. After
sampling and manually checking some of the labeled sessions, we
decided to remove the following questionable regular expressions
of the first list: ˆMozilla$, ˆMozilla.4\.0$ and ˆMozilla.5\.0$. Beside
the sessions categorized as Crawlers by the previous step, the two
lists label as robots all the sessions categorized as Cloud Client,
Offline Browser, Link Checker, Feed Fetcher, Library, Validator and
Other, and some of the sessions identified as Unknown.
In the third stage, and in order to label more sessions, we manually label the unique user-agents marked as Unknown from the API.
From the total of 2.562 user-agent strings 1.946 are identified as
robots. These user-agents are mostly WordPress plugins, reference
and citation management tools and custom applications.
In order to identify humans, we use information from the logs of
the publisher’s web application. In particular, we label as human all
the sessions that come from a logged in user. Sessions previously
categorized as Browser by the API, can not be considered humans
since robots can mask their user-agent string with one of a known
browser [19].

Feature Extraction

We extracted two sets of features from the identified sessions: simple
features, based on past approaches of the literature, and semantic
features, based the approach that we presented in this paper.
Simple features, include the 10 features discussed in Section
2.3 (HTTP requests is broken down into 4 features), plus the following features that come from the particular web application of the
publisher:
• Unique Content. Text content usually exists in different
formats in a web page (pdf, eps, html etc.). This feature represents the number of the unique requests to this content
regardless its format.
• Multiple Countries. Indicating if the requests are coming
from different countries (> 1) during the sessions. The
web application uses a geolocation service to accurately
determine the location of the visitors using their IP address.

2 http://www.useragentstring.com/
3 https://github.com/atmire/COUNTER-Robots
4 https://www.projectcounter.org/
5 https://goo.gl/5WQ6ds
6 https://piwik.org/

4

Following the above labeling strategy we managed to label 67.484
sessions, of which 37.922 (≈ 56%) are labeled as robot and 29.562
(≈ 44%) as human. Interestingly, the distribution of the target
variable in our dataset is similar to the reported distribution of
human and bot traffic on the Web (≈ 50% each). In addition, we
note that our original data remain largely unlabeled, exposing the
inability of heuristic algorithms and syntactical log analysis to
effectively detect robots [5].

5

RESULTS

Here, we contribute empirical results concerning the utility of the
proposed semantic features for web robot detection in our realworld case study. We first present and discuss measures of the
dependency between each feature (both simple and semantic) and
the class variable (human vs bot). Then we compare and contrast
simple, semantic and both sets of features in conjunction with
a variety of machine learning algorithms for building web robot
detection models.

5.1

Figure 2: A histogram comparing the distribution of human
and robots sessions’ unique topics in the log scale.

Feature Evaluation

We discuss the dependency between each feature and the class, as
measured by two univariate statistical tests: the F-test in ANOVA
and the χ 2 test. Table 1 presents the scores of all features according
to these two tests, in descending order.
We first notice that all semantic features are highly ranked by at
least one of the two tests. In particular, four out of the five semantic
features (Boolean Page Variance, Page Similarity, Page Variance and
Unique Topics) are among the top-5 feature according to the F-test,
while two out the five semantic features (Total Topics and Unique
Topics) are among the top-3 features according to the χ 2 test. These
findings are in line with our initial hypothesis that semantic features
make a useful representation of sessions for web robot detection in
content rich websites.
We also notice that simple features like Repeated Requests, Session
Duration and Total Requests are also ranked high by both tests. This
is expected, since long sessions with many and repeated requests
is typical of the behavior of web robots.
The Unique Topics semantic feature is the only feature to be found
among the top-5 features according to both tests. Figure 2 contrasts
the distribution of Unique Topics in human sessions with that of
robot sessions. For this particular graph we randomly under-sample
the robot sessions in order to match the number of human sessions
(≈ 29k). It is evident that robot sessions exhibit a much higher
number of Unique Topics compared to human sessions.

5.2

Table 2 presents the F-measure, Balanced Accuracy and G-mean
of each model using only the simple features, only the semantic
features and, finally, both the simple and the semantic features.
We first see that the best results in all three evaluation measures are achieved by RBF when the semantic features are used by
themselves, and by GB when the simple features are used either
by themselves or in tandem with the semantic features (in bold
typeface). Considering these best results per feature space used,
we notice that a decent level of web robot detection accuracy can
be achieved using semantic features alone. Simple features lead to
better results compared to semantic features when these two types
of features are used by themselves. However, the best results in
all three evaluation measures are achieved when using both the
simple and the semantic features (in underlined typeface). In particular, the F-measure is increased by by 1.06%, Balanced Accuracy by
1.26% and G-mean by 1.34% compared to using the simple features
alone. These findings are evidence that semantic features can lead
to improved web robot detection accuracy in content rich websites.
We conclude the discussion of the results with a learning curve
plotting the Balanced Accuracy of the best learning algorithm, GB,
using the best representation of sessions, both simple and semantic
features, for a varying number of training examples (Figure 3).
We see that the training and testing accuracy curves converge
when almost half of the available data is used. We can therefore
conclude that our model has low variance and there is no need
for additional training data to improve the current results. Instead,
the complexity of our model should be increased, either by getting
additional features or by using polynomial features.

Predictive Modeling

We split the original training data in two parts: a training set
containing 70% and a test set containing the rest 30%. The split
is done in a time-ordered way, so that the training set contains
only sessions that occurred before the test set, in accordance with
a real-world deployment.
We experiment with four different models: a support vector
machine with an RBF kernel (RBF), a gradient boosting (GB) model
and a multi-layer perceptron (MLP) using scikit-learn [14], as well
as an eXtreme Gradient Boosting (XGB) model7 .

6

CONCLUSION & FUTURE WORK

In this work we discuss the use of semantic features on detecting
web robots. Based on the assumption that humans typically look
for specific information on a particular subject, while on the other
hand, most of the web robots go through the content of a website
in a uniform fashion, we propose a novel set of semantic features

7 https://github.com/dmlc/xgboost

5

Table 1: The scores of the features in descending order.
χ 2 test

F-test
Feature

Score

Feature

Score

Boolean Variance

55346.73

Session Duration

Page Similarity

21268.92

Total Topics

43153243.50
6265437.49

Repeated Requests

16867.69

Unique Topics

2188149.91

Page Variance

13679.18

Total Requests

628569.58

Unique Topics

9695.67

Unique Content

310769.30

Unique Content

5626.69

Average Time

259439.45

Session Duration

3736.19

Standard Deviation Time

153834.77

HTTP Client Error

2221.33

Boolean Page Variance

72611.97

Total Requests

1593.08

Page Similarity

2803.16

Total Topics

1584.91

Page Variance

2208.23

Average Time

909.14

Repeated Requests

678.14
267.90

Standard Deviation Time

574.04

HTTP Client Error

PDF Requests

458.06

PDF Requests

HTTP Successful

100.65

83.89

HTTP Successful

25.02

HTTP Server Error

35.54

Webservice

22.45

HTTP Redirection

25.27

HTTP Server Error

18.08

Webservice

22.49

HTTP Redirection

9.63

3.19

Multiple Countires

3.18

Multiple Countires

Table 2: Result of the experiments conducted.
F-measure

Features

Balanced Accuracy

G-mean

RBF

MLP

GB

XGB

RBF

MLP

GB

XGB

RBF

MLP

GB

XGB

Simple

0.6552

0.7844

0.9075

0.905

0.6551

0.7685

0.9007

0.898

0.5835

0.7432

0.8989

0.8961

Semantic

0.8489

0.7497

0.8482

0.846

0.8484

0.7712

0.845

0.8418

0.8475

0.7673

0.8432

0.8395

Simple & Semantic

0.6484

0.8166

0.9181

0.9177

0.6518

0.801

0.9133

0.9127

0.5656

0.7815

0.9123

0.9116

that can potentially increase the discrimination capability of a supervised learning model. We performed an empirical study on real
world data, from an academic digital library, by comparing the use
of features previous research used and the semantic characteristics
we engineered. We focused on the significance of features and we
experimented with various learning algorithms. The results of our
study seems promising, reaching a balanced accuracy over 90%
when using both semantic and common features while in most of
the classifiers tested the use of semantic features increases considerably the classification accuracy. Finally, the scores of the features,
deriving by statistical tests, clearly indicate the importance of the
semantic features and the consideration need to be taken when
selecting the different features.
In the future, we plan to make the most out of the semantic
analysis by creating more and different features that can better

characterize the (in)coherence of a session. Our study showed that
the increase of semantic features may augment the discrimination
capabilities of a model. Toward this, we are gonna fine tune the
parameters of LDA or even use other algorithms like Word2Vec to
extract meaningful information from the web pages a user visits.
Our future plans also include the modification of the classes
we use for our detection. Following the path of previous research
we are going to transform the problem to a multi-class problem,
since web robots come with different characteristics and functions.
An analysis of the significance of semantic features on the different types of web robots may expose their true intentions while
unsupervised techniques may reveal new uncategorized robots.

6

[19] Dusan Stevanovic, Natalija Vlajic, and Aijun An. 2013. Detection of malicious
and non-malicious website visitors using unsupervised neural network learning.
Applied Soft Computing 13, 1 (2013), 698–708.
[20] Pang-Ning Tan and Vipin Kumar. 2004. Discovery of web robot sessions based
on their navigational patterns. In Intelligent Technologies for Information Analysis.
Springer, 193–222.
[21] Bing Wang, Yao Zheng, Wenjing Lou, and Y Thomas Hou. 2015. DDoS attack
protection in the era of cloud computing and software-defined networking.
Computer Networks 81 (2015), 308–319.
[22] Mahdieh Zabihi, Majid Vafaei Jahan, and Javad Hamidzadeh. 2014. A density
based clustering approach to distinguish between web robot and human requests
to a web server. The ISC International Journal of Information Security 6, 1 (2014).
[23] Igal Zeifman. 2016. Bot Traffic Report. (2016). Retrieved August 1, 2017 from
https://www.incapsula.com/blog/bot-traffic-report-2016.html

Figure 3: Learning curve of the Gradient Boosting algorithm
using both the simple and the semantic features.

REFERENCES
[1] Yasmin A AlNoamany, Michele C Weigle, and Michael L Nelson. 2013. Access
patterns for robots and humans in web archives. In Proceedings of the 13th
ACM/IEEE-CS joint conference on Digital libraries. ACM, 339–348.
[2] David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research 3, Jan (2003), 993–1022.
[3] Christian Bomhardt, Wolfgang Gaul, and Lars Schmidt-Thieme. 2005. Web robot
detection-preprocessing web logfiles for robot detection. In New developments
in classification and data analysis. Springer, 113–124.
[4] Michelangelo Diligenti, Frans Coetzee, Steve Lawrence, C Lee Giles, Marco Gori,
et al. 2000. Focused Crawling Using Context Graphs.. In VLDB. 527–534.
[5] Derek Doran and Swapna S Gokhale. 2011. Web robot detection techniques:
overview and limitations. Data Mining and Knowledge Discovery 22, 1 (2011),
183–210.
[6] Derek Doran and Swapna S Gokhale. 2016. An integrated method for real time
and offline web robot detection. Expert Systems 33, 6 (2016), 592–606.
[7] Derek Doran, Kevin Morillo, and Swapna S Gokhale. 2013. A comparison of web
robot and human requests. In Proceedings of the 2013 IEEE/ACM International
Conference on Advances in Social Networks Analysis and Mining. ACM, 1374–1380.
[8] Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro
Flammini. 2016. The rise of social bots. Commun. ACM 59, 7 (2016), 96–104.
[9] OWASP Foundation. 2016. OWASP Automated Threat Handbook Web Application Version 1.1. (2016). Retrieved October 1, 2017 from https://www.owasp.org/
index.php/File:Automated-threat-handbook.pdf
[10] Joseph W Greene. 2016. Web robot detection in scholarly Open Access institutional repositories. Library Hi Tech 34, 3 (2016), 500–520.
[11] Paul Huntington, David Nicholas, and Hamid R Jamali. 2008. Web robot detection
in the scholarly information environment. Journal of Information Science 34, 5
(2008), 726–741.
[12] Junsup Lee, Sungdeok Cha, Dongkun Lee, and Hyungkyu Lee. 2009. Classification
of web robots: An empirical study based on over one billion requests. computers
& security 28, 8 (2009), 795–802.
[13] Distil Networks. 2017. Bad Bot Report. (2017). Retrieved October 10, 2017 from
https://resources.distilnetworks.com/whitepapers/2017-bad-bot-report
[14] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M.
Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine
Learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830.
[15] Nathan Rude and Derek Doran. 2015. Request type prediction for web robot and
internet of things traffic. In Machine Learning and Applications (ICMLA), 2015
IEEE 14th International Conference on. IEEE, 995–1000.
[16] Athena Stassopoulou and Marios D Dikaiakos. 2007. A probabilistic reasoning
approach for discovering Web crawler sessions. In Advances in Data and Web
Management. Springer, 265–272.
[17] Athena Stassopoulou and Marios D Dikaiakos. 2009. Web robot detection: A
probabilistic reasoning approach. Computer Networks 53, 3 (2009), 265–278.
[18] Dusan Stevanovic, Aijun An, and Natalija Vlajic. 2012. Feature evaluation for web
crawler detection with data mining techniques. Expert Systems with Applications
39, 10 (2012), 8707–8717.
7

