CoAID: COVID-19 Healthcare Misinformation Dataset

As the COVID-19 virus quickly spreads around the world, unfortunately, misinformation related to COVID-19 also gets created and spreads like wild fire. Such misinformation has caused confusion among people, disruptions in society, and even deadly consequences in health problems. To be able to understand, detect, and mitigate such COVID-19 misinformation, therefore, has not only deep intellectual values but also huge societal impacts. To help researchers combat COVID-19 health misinformation, therefore, we present CoAID (Covid-19 heAlthcare mIsinformation Dataset), with diverse COVID-19 healthcare misinformation, including fake news on websites and social platforms, along with users' social engagement about such news. CoAID includes 1,896 news, 183,564 related user engagements, 516 social platform posts about COVID-19, and ground truth labels. The dataset is available at: https://github.com/cuilimeng/CoAID.


Introduction
COVID-19 is believed to be caused by a novel coronavirus called SARS-CoV-2, which was initially discovered in Wuhan, China in December 2019, and has quickly spread throughout the world. WHO declared the outbreak a Public Health Emergency of International Concern on January 30, 2020 [16] and characterized COVID-19 as a pandemic on March 11, 2020 [18]. As of May 6, 2020, more than 3.5 million cases of COVID-19 and almost 250,000 deaths worldwide have been reported to WHO [17]. Common symptoms of COVID-19 includes cough, shortness of breath, fever, sore throat, and loss of taste or smell [7]. The incubation period is estimated to extend to 14 days, with a median time of 5.1 days [11]. Mortality for COVID-19 (3-4%) appears higher than that for seasonal influenza (<0.1%) [15]. Until now, there has been no known vaccine or specific cure for COVID-19 [14].
While COVID-19 situation has gotten severely worse, misinformation related to COVID-19 has rapidly risen and caused serious social disruptions as well. One the one hand, fake cures for COVID-19 have seriously threatened people's lives. For example, an Arizona man was dead and his wife was hospitalized after the couple ingested a form of Chloroquine to prevent COVID-19 1 . On the other hand, the prevalent misinformation is disrupting social order. For example, 77 cell phone towers have been set on fire due to the conspiracy that 5G mobile networks can spread COVID-19 2 .
In recent years, in mitigating these problems, many computational methods have been developed to auto-detect diverse types of misinformation in different genres [20,26,23,3]. However, debunking COVID-19-related misinformation exhibits its own set of unique challenges. First, aided by the fear of the unknown, misinformation about emerging diseases can spread more quickly than ordinary misinformation does before being debunked 3 . Second, when one keeps seeing a piece of fake news in the news feed, she tends to think that it is true, even if she had doubts before 4 . Third, after people were convinced by misinformation, "myths correction message" from authority may be ineffective or even reduce people's beliefs in other facts about an epidemic [2]. Therefore, it is not clear how easy or difficult for researchers to build satisfactory computational models to auto-detect COVID-19-related misinformation.
To aid these computational efforts, we have constructed a benchmark dataset, named as CoAID (Covid-19 heAlthcare mIsinformation Dataset). CoAID includes confirmed fake and true news articles from the fact-checked or reliable websites and posts at social platforms. We also conducted quick data analysis to demonstrate the utility of CoAID. We contrasted the distinctive features between misinformation and facts about COVID-19, and ran several state-of-the-art misinformation detection methods on CoAID to show where solutions currently lie. Our aim is to call out for public attention to the problems of COVID-19-related misinformation and work together to help develop accurate detection and deterrence of such misinformation.

Preliminaries
According to the Oxford English Dictionary, both misinformation 5 and disinformation 6 are either wrong or misleading information, but disinformation is spread "deliberately" while misinformation is not necessarily. Literature provides more fine-grained definitions of various examples of misinformation on social media, including rumor, fake news, hoax, satire, propaganda, and even conspiracy [22,13], such that: (1) fake news is the presentation of fake or misleading claims as news, where the claims are misleading deliberately [8], (2) hoax is deliberately fabricated falsehood, with the intention to deceive a certain group of the population [8], (3) rumor is unverified but relevant information that arise in contexts such as danger or potential threat, that helps people make sense and manage risk [6].
In this work, we do not differentiate between misinformation and disinformation as it is virtually impossible to computationally determine one's intention. We also interchangeably use fake news and misinformation in this work.
CoAID comprehensively collects various misinformation examples across different platforms.

Related Work
The construction of fake news dataset aim to extract useful features that can better distinguish fake news from true ones.
Researchers have proposed a series of methods to extract news and verify the truthfulness of the news. There exists several benchmark datasets for fake news detection, that contain the linguistic features of news.
LIAR [25] is collected from political fact-checking website PolitiFact. It has 12.8K human labeled short statements from PolitiFact API, and each statement includes statement content, speaker and context. The truthfulness of each statement is evaluated by an editor from PolitiFact, by using detailed judgments and fine-grained labels: pants-fire, false, barely true, half-true, mostly-true, and true.
FA-KES [21] includes the news articles around the Syrian war. The authors first identified the major events in the Syrian war from VDC (Violations Documentation Center), and then retrieved 804 news articles related to the events from the following several news outlets. The truthfulness of each article is justified by comparing the articles against the VDC dataset.
Dhoju et al. [5] proposed a method to collect a healthcare news dataset. They first identified reliable (WHO, CDC/NIH/NCBI, mainstream media outlets) and unreliable media outlets (Dr. Melissa Zimdars's list, fact-checking websit snopes.com), and then used news article scraping tool Newspaper3k 7 to crawl the news articles. Finally they used Google Cloud Natural Language API to filter out non-health news articles.
Besides the news content, other researchers also collected user engagement features of the news, such as user engagement on online social media.
Ghenai et al. [9] collected 126 unproven cancer treatments that are scrutinized by an oncologist from medical sources DC Science, Wikipedia and PubMed. Topic keywords were generated from the treatments and used to find related tweets.
FakeNewsNet [24] is collected from fact-checking websites PolitiFact and GossipCop. News contents and the related ground truth labels are crawled from the two websites. And the author collected social engagement through Twitter's Advanced Search API. The dataset includes news ID, URL, article title, tweet IDs of tweets sharing the news and scripts to crawl news contents and social engagement. It contains 1,056 articles from PolitiFact and 22,864 articles from GossipCop. Each article is labeled as fake or true.
FakeHealth [4] is collected from healthcare information review website Health News Review, which reviews whether a news is satisfactory from 10 criteria. Daily publication of new content on this website was ceased at the end of 2018. The authors crawled users' replies, retweets and profiles by using article URL and article title by using Twitter API.
In addition, existing datasets often focus only on the news coverage or articles, and little attention has been paid to social media posts. However, fake social media posts can easily go viral on multiple social platforms, and cause serious social disruptions. For example, a fake video on YouTube claiming that a US patent on the coronavirus was applied in 2006, quickly went viral on multiple social media platforms 8 , causing much confusion and discomfort.
As shown in Table 1, CoAID dataset not only includes both true and fake news, but also has short claims and social media posts. Compared with news articles, a claim is much shorter-only one or two sentences-such as "Eating garlic prevents infection with the new coronavirus." In this work, we term "myth" and "rumor" as "fake claim", in order to contrast those with "true claim." Our dataset collects all possible features of interests, including multi-modal information and user engagement data. In addition, CoAID can be updated automatically to fetch the latest news. The statistics of COVID-19 news dataset is shown in Table 2.

Dataset Construction
In this section, we introduce how we have collected healthcare misinformation related to COVID-19, their reliable ground truth labels, and associated user engagement features, and how we manage to update the dataset automatically.

Facts and Misinformation on Websites
We have collected both facts and misinformation related to COVID-19, including news articles and claims. The publishing dates of the collected information range from December 1, 2019 to May 1, 2020. The topics include: COVID-19, coronavirus, pneumonia, flu 9 , lock down, stay home, quarantine and ventilator.
• News Article: An example of news article is shown in Figure 1(a). We identified reliable media outlets and fact-checking websites and obtained URLs of misinformation (i.e., fake news) and facts (i.e., true news). To collect true news articles, we have crawled from 9 reliable media outlets that have been cross-checked as reliable, including: Healthline 10 , ScienceDaily 11 , NIH 12 (National Institutes of Health), MNT 13    • Claim: To collect claims (of one or two sentences), we referred to the WHO official website 26 , WHO official Twitter account 27 and MNT 28 . We specifically separated true and fake claims. For example, "Only older adults and young people are at risk" is a fake claim, while "5G mobile networks DO NOT spread COVID-19" is a true claim.
After we obtained all URLs to true and fake news related to COVID-19, we used the Newspaper3k 29 to fetch their corresponding title, content, abstract, and keywords. In total, we have obtained 135 fake news articles, 1,568 true news articles, 27 fake claims and 166 true claims.

User Engagement
We used Twitter API to fetch user engagement data of both facts and misinformation. Figure 1(b) shows user tweets related to the news in Figure 1(a), and Figure 1(c) shows user replies under a tweet.
• Tweets: We used the titles of news articles as the search queries, and specified start and end dates to get the tweets discussing about the news in question in a certain period. The retrieved user engagement features include: user ID, tweets, replies, favorites, retweets, and location. In total, we obtained 9,218 tweets about fake articles, 87,324 tweets about true news articles, 457 tweets about fake claims and 6,342 tweets about true claims, which is summarized in Table 2.
• Replies: We further obtained users' replies of each tweet, by using tweet IDs. In total, we obtained 5,721 replies under fake articles, 64,115 replies under true news articles, 623 tweets about fake claims and 9,764 tweets under true claims, which is summarized in Table 2.

Social Platform Posts
In addition to traditional mass media, on social media, individual users act as the producers of user generated contents, forming the so-called self media. An example of such self-media is shown in Figure 2. As both true and fake news spread through self-media as well, we have also collected both fake and true posts originated from five well-known social media platforms-Facebook, Twitter, Instagram, Youtube, and TikTok-also fact-checked by: Lead Stories, PolitiFact, FactCheck.org, CheckYourFact, AFP Fact Check, and Health Feedback. We only count once for each distinctive fake post. For example, if a post was originally posted by a Facebook user and then posted again by another user on Twitter, it is counted as 1 (i.e., no duplicate fake posts). The numbers in parentheses in Table 2 represent the total number of posts, before duplicates were removed.
Per each post, we only crawled its title without news content, and listed the posts under "Claim" in Table 2.

Automatic Updates
CoAID can be updated automatically with the latest news and tweets. We record the timestamp of the most recently added data each time, and use it as the start date for the next search.
In summary, we list the description of the extracted features in

Data Analysis
CoAID includes multi-modal news, related ground truth labels, and user engagement. The detailed statistics of the dataset are shown in Table 2. In this section, we perform preliminary data analysis in order to illustrate the features of CoAID. Then, we perform several fake news detection methods to evaluate the challenges of COVID-19 healthcare misinformation detection.

Prescriptive Analysis
We use VADER [10] to analyze users' sentiments in tweets related to fake and true news articles. We exclude the tweets that are completely neutral, and plot the sentiment scores of tweets related to fake articles in Figure 3(a) and true news articles in Figure 3(b), respectively. The bars show the numbers of tweets with different negative and positive degrees, while the scatter plot shows the negative and positive scores. The intensity of the hive represents the density of data points. We can see from Figure 3 that tweets related to fake news are more negative, and have stronger sentiment polarities than those related to true news articles.
Next, we analyze the top 30 frequent hashtags in tweets related to fake and true news articles, respectively. For a more intuitive view, we delete the most frequent hashtags "#coronavirus", "#covid_19", "#covid19" and "#covid" as they appear almost in every tweet. We show the frequency of hashtags in tweets related to fake and true news articles in Figure 4(a) and Figure 4(b), respectively. We find that the hashtag distributions of tweets about fake and true news articles are quite different. While the hashtags in tweets about true news articles are mainly related to healthcare, those in tweets about fake news cover more diverse topics, including conspiracy (#bioweapon) and fake cure (#vitaminc). In Figure 5, next, we list ten most common claims of COVID-19 on Twitter, and their corresponding frequencies.
We merge similar claims into one, such as "Eating garlic prevents infection with the new coronavirus" and "Garlic protects against coronaviruses." We can see that the most frequently discussed claim is "COVID-19 is just like the flu", followed by "COVID-19 is airborne" and "The new coronavirus can be transmitted through mosquito bites". We then plot the daily counts of tweets related to two claims, 'COVID-19 is just like the flu" and "5G mobile networks spread COVID-19" in Figure 6(a) and 6(b), respectively. We can see that there were most tweets about "COVID-19 is just like the flu" at the beginning of the global outbreak, which is around March 12, 2020. Note that this topic is mentioned almost everyday. The tweets about "5G mobile networks spread COVID-19" were mostly on the April 10, 2020, when 5G towers were being set on fire, but afterward the number declined as the claim was debunked.

COVID-19 Misinformation Detection
To demonstrate the main utility of m, here, we conduct comparative experiments on the misinformation detection task. We include both simple methods and state-of-the-art methods as baselines. We consider the following baseline methods for COVID-19 misinformation detection: • CSI [20]: CSI is a hybrid deep learning-based misinformation detection model that utilizes information from article content and user response. The article representation is modeled via an LSTM model with the article embedding via Doc2Vec [12] and user response. • SAME\v [3]: SAME uses news image, content, metadata and users' sentiments for fake news detection. As the majority of news does not have a cover image, the visual part of the model is ignored, and the baseline is termed as SAME\v. • HAN [27]: HAN has two levels of attention mechanisms applied at the word and sentence-level, enabling it to attend differently to more and less important content for document classification. • dEFEND [23]: dEFEND utilizes the hierarchical attention network as HAN on article content, and a co-attention mechanism between article content and user comment for misinformation detection.
To evaluate the performance of misinformation detection algorithms, we use the following metrics, which are commonly used to evaluate classifiers in related areas: PR-AUC, Precision, Recall, and F1 score.
We implement all models with Keras. We randomly use the labels of 75% news articles for training and predict the remaining 25%. We set the hidden dimension of all models to 100. The word embeddings are initialized by GloVe [19] and the dimension of pre-trained word embeddings is set to 100. For deep learning methods, we use Adam with a minibatch of 50 articles on the dataset, and the training epoch is set as 10. For a fair comparison, we use the cross-entropy loss for all methods. Finally, we run each method five times and report the average score in Table 4.
From Table 4, we can see that state-of-the-art methods perform better than simple methods, as they incorporate signals from user engagement, better capturing contextual information. However, as the dataset is quite imbalanced, the models tend to generate many fake positive cases. Thus, the recall and F1 values are far from being satisfactory. As the proportion of true and fake information is likely to be even more skewer, practical detection solutions need to handle this type of imbalance setting more.
In addition, with regard to healthcare news, it is often more scarce and rare for lay persons to give discriminating comments due to lack of professional knowledge. For example, WHO tweeted "There is no evidence that regularly rinsing the nose with saline solution has protected people from infection with the new coronavirus." The majority of comments for the tweet are unrelated, useless, and even includes hate speech and other misinformation about

Conclusion
In this paper, we present a comprehensive COVID-19 misinformation dataset CoAID, which contains news articles, user engagement, and social platform posts. We describe how we collected the dataset. In addition, we conduct brief data analysis to show the distinctive features between misinformation and facts, and demonstrate the future research directions through a fake news detection task over several state-of-the-art methods. We hope researchers to find COVID-19 useful for their research and together contribute to flatten the curve of COVID-19.