The Clickbait Challenge 2017: Towards a Regression Model for Clickbait Strength
Martin Potthast Tim Gollub Matthias Hagen Benno Stein


The Clickbait Challenge 2017: Towards a Regression Model for Clickbait StrengthThe Clickbait Challenge 2017: Towards a Regression Model for Clickbait Strength





Leipzig University
martin.potthast@uni-leipzig.de


Bauhaus-Universität Weimar
tim.gollub@uni-weimar.de


Martin-Luther-Universität Halle-Wittenberg
matthias.hagen@informatik.uni-halle.de


Bauhaus-Universität Weimar
benno.stein@uni-weimar.de



Clickbait has grown to become a nuisance to social media users and social media operators alike. Malicious content publishers misuse social media to manipulate as many users as possible to visit their websites using clickbait messages. Machine learning technology may help to handle this problem, giving rise to automatic clickbait detection. To accelerate progress in this direction, we organized the Clickbait Challenge 2017, a shared task inviting the submission of clickbait detectors for a comparative evaluation. A total of 13 detectors have been submitted, achieving significant improvements over the previous state of the art in terms of detection performance. Also, many of the submitted approaches have been published open source, rendering them reproducible, and a good starting point for newcomers. While the 2017 challenge has passed, we maintain the evaluation system and answer to new registrations in support of the ongoing research on better clickbait detectors.




    Benno Stein
    December 30, 2023
=====================





§ INTRODUCTION


This paper reports on the results of the Clickbait Challenge 2017.
[<https://clickbait-challenge.org>]
The main goal of the challenge was to kickstart research and development on the novel task of clickbait detection in social media <cit.>. The term “clickbait” refers to social media messages that are foremost designed to entice their readers into clicking an accompanying link to the posters' website, at the expense of informativeness and objectiveness. Typical examples include exaggerated statements, such as “You won't believe …”, “… will change your life”, or “… will blow your mind”, as well as unnecessary omissions of informative details as in “This actor fell …”, “This city started …”, “This fact about …”.

Spreading content through clickbait has meanwhile become an established practice on social media, even among reputed news publishers <cit.>. Clickbait works: it has a measurable effect on page impressions, which explains its widespread usage. But clickbait also works to the detriment of all stakeholders:
news publishers succumbing to economic pressure undermine their established reputations and journalistic codes of ethics;
social media platform operators increase user engagement on their networks at the expense of user experience;
users of social media platforms unconsciously have their curiosity “tickled”, often not realizing they are being manipulated. When they finally do realize, clickbait is rather perceived as a nuisance, much like spam.

To step out of this vicious circle, an automatic detection of clickbait, e.g., as part of ad-blockers within browsers, would enable users to opt out. The more users choose to opt out, the less effective clickbait becomes, thus forcing publishers and social networks to find other means of reaching their respective audience. Deployed at the social network itself, clickbait detection technology may unfold its impact much more rapidly. A necessary prerequisite, however, is machine learning technology capable to reliably detect clickbait.

To raise awareness and to build a community of researchers interested in this new task, we decided to organize the Clickbait Challenge 2017. In what follows, after a review of related work in Section <ref>, we outline the challenge design and the evaluation results obtained from its first edition. Regarding its design, in Section <ref> we argue for our decision to cast clickbait detection as a regression task to measure clickbait strength; in Section <ref>, the Webis Clickbait Corpus 2017 <cit.> is reviewed, which was used as evaluation dataset for the challenge; and in Section <ref>, organizational details of the challenge are discussed. Section <ref> presents the submitted approaches and reports on their achieved performance. We conclude with a description of how, even after the 2017 challenge has passed, researchers can still use our evaluation-as-a-service platform TIRA to evaluate clickbait detection algorithms against the ones submitted before.



§ RELATED WORK






To the best of our knowledge, Table <ref> lists all clickbait-related datasets that have been published to date. In the last row, our Webis Clickbait Corpus 2017 <cit.> is listed, which has been compiled and used for the Clickbait Challenge 2017. In the columns of Table <ref>, apart from the respective publication, the datasets are classified with respect to the annotation scale at which clickbait is assessed, the type of teaser message used in the dataset, whether the articles referred to in the teasers are included in the dataset, as well as the datasets' size. Comparing our clickbait challenge dataset with the others along these attributes, it becomes apparent that our dataset is the first one that measures clickbait on a graded scale, a decision further motivated in Section <ref>. Other than this, the clickbait challenge dataset adopts the construction principles of our previous Webis Clickbait Corpus 2016 <cit.>, while providing for an order of magnitude more training examples. Its construction is summarized in Section <ref>.

-3
All except the last publication listed in Table <ref> also propose an approach for the automated detection of clickbait alongside their dataset. An additional clickbait detection approach is presented by <cit.>, who use the dataset of <cit.> for evaluation. In what follows, we highlight the differences of the published approaches concerning the features used for clickbait detection, as well as the (best-performing) classification technology applied. The same analysis is repeated in Section <ref> for the approaches submitted to the clickbait challenge.

Since rather different datasets have been used in the literature for evaluation, the respective approaches can hardly be compared performance-wise—a strong incentive for the organization of the clickbait challenge. In terms of the F1-measure, clickbait detection performance scores around 0.75 (<cit.>) and 0.95 (<cit.>) have been reported. Given the deficiencies of the used datasets <cit.>, these scores must be taken with a grain of salt, though.

Features for clickbait detection can be derived from three sources: the teaser message, the linked article, and metadata for both. While all reviewed approaches derive features from the teaser message, the linked article and the metadata are considered only by <cit.> and <cit.>, who use variants of decision trees (random forest and gradient boosted decision tree, respectively) as their classifiers. Examples of features derived from the linked article are text readability scores and the semantic similarity of the teaser with (parts of) the article. Examples of metadata-based features are the publisher name, whether the teaser contains multimedia content, and the frequency of specific characters in the article URL. As for features derived from the teaser message itself, the above two approaches, as well as that of <cit.> (their best performance comes from an SVM classifier), use a multitude of structural, word-level, n-gram, and linguistically motivated features. The remaining approaches are based on embeddings as feature representation of the teaser message. <cit.> takes pre-trained word embeddings as input for a CNN, which are further updated during the training of the classifier. <cit.> use both character and word embeddings to train an RNN. Lastly, <cit.> average the sub-word embeddings of a teaser message to obtain a feature representation which is then used to train a linear classifier (not further specified).



§ CLICKBAIT STRENGTH REGRESSION



Previous work considered clickbait as a binary phenomenon: either a teaser message is clickbait or it is not. Though there are messages that are obviously clickbait (“You won't believe what happened!”), we noticed that making a decision is often not as straightforward. The reason for this is that, since the very purpose of teaser messages is to attract the attention of readers, every message containing a link baits user clicks to some extent. The question is whether this baiting is perceived as immoderate or deceptive by the reader—a subjective notion. To work around this subjectivity we asked how heavily a teaser message makes use of clickbaiting techniques, leaving open the question whether it is clickbait.

The teaser messages depicted in Figure <ref> serve as an illustration of the varying degrees of clickbait strength. The messages are arranged by increasing clickbait strength  from left to right as judged by many assessors. The first teaser message on the left makes a clear statement about the information conveyed in the linked article (links have been omitted), and likely only very few readers would consider this message clickbait. In the second teaser message, the text of the message is comparably more clickbaiting, but the image below provides valuable additional clues (the image “spoils” the information absent from the text message). The same is true for the third teaser message; however, the image only conveys a very vague idea about the information missing from the teaser text. Whether these two tweets should be classified as being clickbait is difficult to decide, but they are obviously more clickbaiting than the first one. The fourth teaser message on the right finally makes heavy use of clickbaiting techniques, and the vast majority of readers would classify this teaser message as a paragon of clickbait.

What this sequence of teaser messages exemplifies, and what our large-scale crowdsourcing study corroborates, is that there is a whole continuum of teaser messages between the extremes of clickbait and non-clickbait. Although it is apparent that teaser messages that fall in-between could have been formulated in a more informative way to render them less clickbaiting, it is questionable whether they are clickbaiting enough to be perceived as clickbait in general. Because of this, to account for different degrees of clickbait strength, we opted for casting clickbait detection as a regression problem in the clickbait challenge, and used a graded scale to annotate the teaser messages in our dataset. The graded scale has four values (Likert scale with forced choice) and ranges from “not” via “slightly” and “considerably” to “heavily” clickbaiting.





§ EVALUATION DATASET



To provide a high-quality, representative evaluation dataset for the clickbait challenge, we compiled the Webis Clickbait Corpus 2017. The corpus is authentic, representative, rich in terms of potential features, unbiased, and large-scale. Since the dataset has been described at length by <cit.>, we only summarize the most important points here.

Table <ref> gives an overview of the main characteristics of our acquisition and annotation process. We build on and extend the approach taken to construct the smaller Webis Clickbait Corpus 2016 <cit.>, rendering both corpora comparable.




As teaser type for our corpus, we chose Twitter tweets since
the platform has a large user base, and
virtually all major US news publishers disseminate their articles through Twitter.
Our sample of news publishers is governed by publisher importance in terms of retweets. Restricting ourselves to English-language publishers, we obtain a ranking of the top-most retweeted news publishers from the NewsWhip social media analytics service.
[https://www.newswhip.com]

Taking the top 27 publishers, we used Twitter's API to record every tweet they published in the period from December 1, 2016, through April 30, 2017. To enable the research community to develop and experiment with a rich set of features, we included the tweet text, media attachments, and the metadata provided by Twitter.Furthermore, we crawled the news article advertised using the Webis Web Archiver <cit.>, which records the whole communication that takes place between a client (browser) requesting the web page of a news article and the publisher's web server hosting it, storing it in web archive (WARC) files (including, e.g., HTML, CSS, Javascript, and images). This way, every article page that forms part of our corpus can be reviewed as it was on the day we crawled it, allowing for corpus reviews even after years, hence maximizing its reproducibility. Nevertheless, users of our corpus will not have to handle the raw WARC files. For convenience, we applied publisher-specific wrappers extracting a set of content-related fields (cf. fields prefixed with “target” in Table <ref>).

To obtain a sample of tweets that has a high topic diversity, we crawled news as described above for five months in a row, yielding almost half a million tweets that fit our criteria and that were successfully archived. From this population, we drew a random sample for annotation where, for budgetary reasons, the goal was to draw at least 30,000 tweets and at most 40,000. Since the distribution of tweets per publisher is highly skewed, we apply stratified sampling to avoid a corresponding publisher bias. Similarly, we ensure that tweets are sampled from each day of the five months worth of tweets to cover the whole time period. Selecting a maximum of ten tweets per day and publisher yielded a set of 38,517 tweets and archived articles, which were then subjected to manual annotation.

The annotation of the tweets regarding clickbait strength was implemented with the crowdsourcing platform Amazon Mechanical Turk (AMT). For each tweet, we requested annotations from five different workers. To guarantee a high-quality dataset, all crowdsourced assessments were reviewed and, if necessary, discarded, resubmitting the respective assignment to AMT.

The histogram in Table <ref> shows the distribution of tweets across the four classes of our graded scale as stacked bars. To classify a tweet into one of the four classes, the mode of its annotations is used, where, in case of multiple modes, the fifth annotation is used as a tie breaker. The different colors in the bars encode different levels of agreement. With a value of 0.21 in terms of Fleiss' κ, the annotator agreement is between slight and fair. However, when binarizing the classes by joining the first and last two classes into one, κ becomes 0.36, which corresponds to the respective value of 0.35 reported for our previous clickbait corpus <cit.>. Furthermore, also the distribution of tweets across the binarized classes matches that of our previous corpus. Recalling that our previous corpus has been assessed by trained experts, we conclude that our crowdsourcing strategy lives up to the state of the art and that it can be considered as successful: the two independently designed and operationalized annotation studies still achieve the same result, hence our annotation experiment can be understood as a reproduction of our previous efforts, only at a larger scale. 



§ CHALLENGE ORGANIZATION



To organize and manage the clickbait challenge, we use the eval­uation-as-a-service platform TIRA <cit.>,
[<https://tira.io>]
which provides the means to host competitions that invite software submissions. In contrast to run submissions, challenge participants get access to a virtual machine on which they deploy (=submit) their clickbait detection approach. The deployed approach is evaluated on a test dataset hosted at TIRA, so that participants cannot gain direct access to it, giving rise to blind evaluation. For task organizers, this procedure has the advantage that it allows, given the consent of the authors, to reevaluate the submitted clickbait detection approaches also on new datasets (since the approaches are maintained in executable form in the virtual machines), and to evaluate future clickbait detection approaches in a meaningful way (since the test data are kept private).

To participate in the clickbait challenge, teams had to develop regression technology that rates the “clickbaitiness” of a social media post on a [0,1] range (a value of 1 denoting heavy clickbaiting). For each post, the content of the post itself as well as the main content of the linked target web page are provided as JSON-Objects in our datasets. As primary evaluation metric, mean squared error (MSE) with respect to the mean judgment of the annotators is used. For informational purposes, we also compute the F1 score with respect to a binarized clickbait class (from the mode of the judgments), and we measure the runtime of the classification software.

We published 19,538 tweets from our dataset as a training dataset for the challenge participants, and kept the remaining 18,979 tweets for the private test dataset. As a strong baseline for the challenge, we used a modified version of our own seminal classifier <cit.>. To account for the recast of clickbait detection as a regression task, we used the same feature set but replaced the random forest classifier with a ridge regression algorithm. A rather weaker baseline just predicts the average true clickbait score of the test data for every tweet.



§ SUBMITTED APPROACHES AND RESULTS



-2
From the 100 teams that registered for the Clickbait Challenge 2017, 33 finally requested a virtual machine when asked. From these 33 teams, 13 teams followed through, made a successful submission, and evaluated their approach on the test dataset. Their performance results are shown in Table <ref>.
[To render talking about the respective approaches more consistent (and more fun), we adopted a naming scheme for this task: each team chose a “code name” for their approach from a list of fish names, loosely alluding to the “bait” part of clickbait.]
As can be observed in the second column of Table <ref>, 6 of the 13 submitted approaches outperformed our strong baseline in terms of minimizing the mean squared error (MSE), the best performance being achieved by zingel <cit.> with a mean squared error of 0.033. The runner-ups are emperor (no notebook submitted, approach uses a CNN on the teaser text) and carpetshark <cit.>, both achieving a mean squared error of 0.036. Furthermore, eight teams, as well as our strong baseline, outperform the weak baseline. This can be seen from the normalized mean squared error (NMSE) scores in the third column of Table <ref>, where the mean squared error achieved by a clickbait detection approach is divided by the weak baseline's performance (MSE=0.0735). Hence, an NMSE score less than 1 means that the approach outperforms the weak baseline.



In terms of the F1 measure (fourth column), the six top approaches and three others outperform the strong baseline; a result at least partly rooted in the fact that some of the approaches have been optimized with respect to F1 instead of MSE. In terms of F1, the top-performing approach is again zingel. Together with the observation that this approach is also the fastest one (eighth column), this underlines its high quality. Regarding precision and recall individually (columns six and seven), one can observe that the approaches outperforming the baseline in terms of F1 succeed by trading small losses in precision with significant gains in recall.

Of the 13 teams that made a successful submission, 9 also submitted a notebook paper, referenced in the last column of Table <ref>. In what follows, we briefly summarize each approach.

Zingel by <cit.> is the best-performing approach of the Clickbait Challenge 2017. It employs a neural network architecture with bidirectional gated recurrent units (biGRU) and a self-attention mechanism to assess clickbait strength (namely, the mean of the clickbait annotations for a tweet). As input to the network, only the teaser text is considered, which is represented as a sequence of word embeddings that have been pre-trained on Wikipedia using Glove (but then updated during training).

Carpetshark by <cit.> employs an ensemble of SVM regression models (so-called extremely randomized forests), one each trained separately for the text fields provided in the training data. Besides the teaser text, these fields are the keywords, descriptions, image captions, paragraphs, and the title of the linked article. In addition to the objective of predicting clickbait strength, predicting the standard deviation of the annotations is considered, improving the performance of the ensemble. An attempt at augmenting our challenge dataset with own data failed to improve the approach's performance and was hence omitted from the final submission.

Pineapplefish by <cit.> relies on a “linguistically infused” neural network with two sub-networks. An LSTM sub-network that takes as input a sequence of 100 word embeddings (pre-trained on a Twitter dataset and then updated during training). The sequence consists of the first 100 content words of the teaser text, and, in case the teaser text is shorter than 100 content words, text from the linked article's fields. The second sub-network consists of two dense layers which take as input a vector of linguistic features extracted from the teaser message and the linked article text. The authors also experimented with object detection technology to obtain features from teaser images, however, these features were not included in their final model due to a lack of performance gain.

Whitebait by <cit.> analyzes all text fields available for each training example (=tweet), as well as the publication time of the tweet. For each of the text fields, an LSTM is trained that takes as input a sequence of word embeddings (initialized randomly). For the publication date, a neural network with one hidden layer is trained. As input, the publication time is binned into one-hour ranges and then converted into one-hot encodings. In a second step, the individually trained networks are fused by concatenating the last dense layer of the individual networks. The authors state that this two-step procedure performs better than training a complete model from scratch. Attempts at exploiting the teaser images have not made it into the final version of this model.

Pike by <cit.> computes a set of 175 linguistic features from the teaser message, two features related to the linked article, as well as three features that capture semantic relations between the teaser message and the linked article. This feature set is then fed into a random forest regression algorithm, which achieved the best performance compared to other alternatives tested in a 10-fold cross validation on the training dataset.

Tuna by <cit.> consists of a deep neural network with three sub-networks. The first sub-network is a bidirectional LSTM with attention which gets as input a sequence of word embeddings (pre-trained on a Google News dataset and updated during training) representing the teaser text. The other two sub-networks are Siamese neural networks, the first of them producing a similarity score between the teaser text and the description field of the linked article, the second one producing a similarity score between the teaser image and the target description. As representations for the teaser text and the target description, doc2vec embeddings of the fields are employed. To represent the teaser image, a pre-trained object detection network was applied to the image, and the activation on a convolutional layer was taken as image representation.

Torpedo by <cit.> uses pre-trained Glove word embeddings (on Wikipedia and a further dataset) to represent the teaser message of a tweet. For this, the word embeddings for the different words in the teaser text are averaged. In addition, seven handcrafted linguistic features are added to the representation. With this feature set, a linear regression model is trained to predict the clickbait strength of tweets.

Salmon by <cit.> applies gradient boosting (XGBoost) to a tweet representation that consists of three feature types:
teaser image-related features encoding whether there is a teaser image, and, using OCR on the image, whether there is text in the image,
linguistic features extracted from the teaser text and the linked article fields, and
features dedicated to detect so-called abusers that are supposed to capture user behavior patterns.

-2
Snapper by <cit.> trains separate logistic regression classifiers on different feature sets extracted from the teaser text, the linked article title, and the teaser images (features extracted first using the Caffe library).
[<http://caffe.berkeleyvision.org>]
In a second step, the predictions of the individual classifiers are taken as input to train a final logistic regression classifier.



§ CONCLUSION


The Clickbait Challenge 2017 stimulated research and development towards clickbait detection: 13 approaches have been submitted to the challenge. Many of these approaches have been released open source by their authors.
[We collected them here: https://github.com/clickbait-challenge]
Together with the working prototype deployed within virtual machines at TIRA, this renders the proceedings of the clickbait challenge reproducible, and newcomers have an easier time following up on previous work.

Several more approaches have been proposed and submitted to TIRA after the challenge had ended. Together with zingel, these four additional approaches are the top five best-performing clickbait detectors on the leaderboard at the time of publishing the current challenge overview.
[https://www.tira.io/task/clickbait-detection/]
The leading approach, albacore by <cit.>, like zingel, employs a biGRU network, initialized by Glove word embeddings. The runner-up anchovy is also an adaptation of zingel, whereas icarfish by <cit.> demonstrates that our baseline <cit.> is still competitive: when optimizing the selection of features using a newly proposed feature selection approach, the baseline approach improves substantially. For the two approaches anchovy and ray, at the time of writing, no written reports have surfaced. More teams have registered after the first challenge has passed, now working on new approaches to solve the task. We will keep the evaluation system running for as long as possible to allow for a continued and fair evaluation of these new approaches.







21


















#1     #1#1#1     #1     #1     #1     #1#1          #1#1           






[AgrawalAgrawal2016]
        agrawal:2016
authorpersonA. Agrawal.
  year2016.
Clickbait detection using deep learning. In
  booktitle2016 2nd International Conference on Next
  Generation Computing Technologies (NGCT). pages268–272.

<https://doi.org/10.1109/NGCT.2016.7877426>



[Anand, Chakraborty, and ParkAnand
  et al2017]
        anand:2017
authorpersonAnkesh Anand, personTanmoy
  Chakraborty, and personNoseong Park.
  year2017.
We Used Neural Networks to Detect Clickbaits: You
  Won't Believe What Happened Next!. In booktitleAdvances in
  Information Retrieval - 39th European Conference on IR Research, ECIR
  2017, Aberdeen, UK, April 8-13, 2017, Proc..
  pages541–547.

<https://doi.org/10.1007/978-3-319-56608-5_46>



[Biyani, Tsioutsiouliklis, and BlackmerBiyani
  et al2016]
        biyani:2016
authorpersonPrakhar Biyani, personKostas
  Tsioutsiouliklis, and personJohn Blackmer.
  year2016.
"8 Amazing Secrets for Getting More Clicks":
  Detecting Clickbaits in News Streams Using Article Informality. In
  booktitleProc. of the Thirtieth AAAI Conference on
  Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA.
  pages94–100.

<http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/11807>




[Cao, Le, and ZhangCao et al2017]
        pike
authorpersonXinyue Cao, personThai Le,
  and personJason Zhang. year2017.
Machine Learning Based Detection of Clickbait Posts
  in Social Media.
journalCoRR  volumeabs/1710.01977
  (year2017).

<http://arxiv.org/abs/1710.01977>




[Chakraborty, Paranjape, Kakarla, and
  GangulyChakraborty et al2016]
        chakraborty:2016
authorpersonAbhijnan Chakraborty,
  personBhargavi Paranjape, personSourya Kakarla, and
  personNiloy Ganguly. year2016.
Stop Clickbait: Detecting and preventing clickbaits
  in online news media. In booktitle2016 IEEE/ACM
  International Conference on Advances in Social Networks Analysis and Mining,
  ASONAM 2016, San Francisco, CA, USA, August 18-21, 2016.
  pages9–16.

<https://doi.org/10.1109/ASONAM.2016.7752207>



[Elyashar, Bendahan, and PuzisElyashar
  et al2017]
        salmon
authorpersonAviad Elyashar, personJorge
  Bendahan, and personRami Puzis.
  year2017.
Detecting Clickbait in Online Social Media: You
  Won't Believe How We Did It.
journalCoRR  volumeabs/1710.06699
  (year2017).

<http://arxiv.org/abs/1710.06699>




[Gairola, Lal, Kumar, and KhattarGairola
  et al2017]
        tuna
authorpersonSiddhartha Gairola,
  personYash Kumar Lal, personVaibhav Kumar, and
  personDhruv Khattar. year2017.
A Neural Clickbait Detection Engine.
journalCoRR  volumeabs/1710.01507
  (year2017).

<http://arxiv.org/abs/1710.01507>




[Glenski, Ayton, Arendt, and VolkovaGlenski
  et al2017]
        pineapplefish
authorpersonMaria Glenski, personEllyn
  Ayton, personDustin Arendt, and personSvitlana
  Volkova. year2017.
Fishing for Clickbaits in Social Images and Texts
  with Linguistically-Infused Neural Network Models.
journalCoRR  volumeabs/1710.06390
  (year2017).

<http://arxiv.org/abs/1710.06390>




[Gollub, Stein, and BurrowsGollub
  et al2012]
        stein:2012k
authorpersonTim Gollub, personBenno Stein,
  and personSteven Burrows. year2012.
Ousting Ivory Tower Research: Towards a Web
  Framework for Providing Experiments as a Service. In
  booktitle35th International ACM Conference on Research and
  Development in Information Retrieval (SIGIR 2012).
  publisherACM, pages1125–1126.
978-1-4503-1472-5

<https://doi.org/10.1145/2348283.2348501>



[GrigorevGrigorev2017]
        carpetshark
authorpersonAlexey Grigorev.
  year2017.
Identifying Clickbait Posts on Social Media with an
  Ensemble of Linear Models.
journalCoRR  volumeabs/1710.00399
  (year2017).

<http://arxiv.org/abs/1710.00399>




[Indurthi and OotaIndurthi and Oota2017]
        torpedo
authorpersonVijayasaradhi Indurthi and
  personSubba Reddy Oota. year2017.
Clickbait detection using word embeddings.
journalCoRR  volumeabs/1710.02861
  (year2017).

<http://arxiv.org/abs/1710.02861>




[Kiesel, Kneist, Alshomary, Stein, Hagen, and
  PotthastKiesel et al2018]
        stein:2018v
authorpersonJohannes Kiesel, personFlorian
  Kneist, personMilad Alshomary, personBenno Stein,
  personMatthias Hagen, and personMartin Potthast.
  year2018.
Reproducible Web Corpora: Interactive Archiving
  with Automatic Quality Assessment.
journalJournal of Data and Information Quality
  (JDIQ) volume10, number4 (dateOct.
  year2018), pages17:1–17:25.

<https://doi.org/10.1145/3239574>



[Omidvar, Jiang, and AnOmidvar
  et al2018]
        albacore
authorpersonAmin Omidvar, personHui Jiang,
  and personAijun An. year2018.
Using Neural Network for Identifying Clickbaits in
  Online News Media.
journalCoRR  volumeabs/1806.07713
  (year2018).

<http://arxiv.org/abs/1806.07713>




[Papadopoulou, Zampoglou, Papadopoulos, and
  KompatsiarisPapadopoulou et al2017]
        snapper
authorpersonOlga Papadopoulou, personMarkos
  Zampoglou, personSymeon Papadopoulos, and
  personIoannis Kompatsiaris. year2017.
A Two-Level Classification Approach for Detecting
  Clickbait Posts using Text-Based Features.
journalCoRR  volumeabs/1710.08528
  (year2017).

<http://arxiv.org/abs/1710.08528>




[Potthast, Gollub, Komlossy, Schuster,
  Wiegmann, Garces Fernandez, Hagen, and SteinPotthast
  et al2018]
        stein:2018l
authorpersonMartin Potthast, personTim
  Gollub, personKristof Komlossy, personSebastian
  Schuster, personMatti Wiegmann, personErika
  Patricia Garces Fernandez, personMatthias Hagen, and
  personBenno Stein. year2018.
Crowdsourcing a Large Corpus of Clickbait on
  Twitter. In booktitleProc. of the 27th International
  Conference on Computational Linguistics (COLING 2018),
  pages1498–1507.

<https://aclanthology.info/papers/C18-1127/c18-1127>




[Potthast, Gollub, Rangel, Rosso, Stamatatos,
  and SteinPotthast et al2014]
        stein:2014j
authorpersonMartin Potthast, personTim
  Gollub, personFrancisco Rangel, personPaolo Rosso,
  personEfstathios Stamatatos, and personBenno
  Stein. year2014.
Improving the Reproducibility of PAN's Shared
  Tasks: Plagiarism Detection, Author Identification, and Author Profiling.
  In booktitleInformation Access Evaluation meets
  Multilinguality, Multimodality, and Visualization. 5th International
  Conference of the CLEF Initiative (CLEF 2014).
  publisherSpringer, addressBerlin Heidelberg New York,
  pages268–299.
978-3-319-11381-4

<https://doi.org/10.1007/978-3-319-11382-1_22>



[Potthast, Köpsel, Stein, and
  HagenPotthast et al2016]
        stein:2016b
authorpersonMartin Potthast, personSebastian
  Köpsel, personBenno Stein, and personMatthias
  Hagen. year2016.
Clickbait Detection. In
  booktitleAdvances in Information Retrieval. 38th European
  Conference on IR Research (ECIR 2016) (seriesLecture Notes
  in Computer Science),
  Vol. volume9626. publisherSpringer,
  addressBerlin Heidelberg New York, pages810–817.

<https://doi.org/10.1007/978-3-319-30671-1_72>



[Rony, Hassan, and YousufRony
  et al2017]
        rony:2017
authorpersonMd Main Uddin Rony,
  personNaeemul Hassan, and personMohammad Yousuf.
  year2017.
Diving Deep into Clickbaits: Who Use Them to What
  Extents in Which Topics with What Effects?
journalCoRR  volumeabs/1703.09400
  (year2017).

<http://arxiv.org/abs/1703.09400>




[ThomasThomas2017]
        whitebait
authorpersonPhilippe Thomas.
  year2017.
Clickbait Identification using Neural Networks.
journalCoRR  volumeabs/1710.08721
  (year2017).

<http://arxiv.org/abs/1710.08721>




[Wiegmann, Völske, Stein, Hagen, and
  PotthastWiegmann et al2018]
        icarfish
authorpersonMatti Wiegmann, personMichael
  Völske, personBenno Stein, personMatthias
  Hagen, and personMartin Potthast.
  year2018.
Heuristic Feature Selection for Clickbait
  Detection.
journalCoRR  volumeabs/1802.01191
  (year2018).

<http://arxiv.org/abs/1802.01191>




[ZhouZhou2017]
        zingel
authorpersonYiwei Zhou.
  year2017.
Clickbait Detection in Tweets Using Self-attentive
  Network.
journalCoRR  volumeabs/1710.05364
  (year2017).

<http://arxiv.org/abs/1710.05364>






