Abstract
Abstract-Although citizen science projects can engage a very large number of volunteers to collect volumes of data, they are susceptible to issues with data quality. Our experience with eBird, which is a broad-scale citizen science project to collect bird observations, has shown that a massive effort by volunteer experts is needed to screen data, identify outliers and flag them in the database. The increasing volume of data being collected by eBird places a huge burden on these volunteer experts and other automated approaches to improve data quality are needed. In this work, we describe a case study in which we evaluate an automated data quality filter that improves data quality by identifying outliers and categorizing these outliers as either unusual valid observations or mis-identified (invalid) observations. This automated data filter involves a two-step process: first, a data-driven method detects outliers (ie. observations that are unusual for a given region and date). Next, we use a data quality model based on an observer's predicted expertise to decide if an outlier should be flagged for review. We applied this automated data filter retrospectively to eBird data from Tompkins Co., NY and found that that this automated process significantly reduced the workload of reviewers by as much as 43% and identifies 52% more potentially invalid observations.
INTRODUCTION
Citizen science enlists the help of volunteers from the general public (citizen scientists) in scientific research [1] . Data collected by citizen scientists can be achieved at little cost, enabling scientific research to gather data over longer periods of time and across broader spatial extents [5] . Engaging citizen scientists in meaningful projects also broadens the public understanding of the scientific process, which in turn can lead to better-informed decision making at all levels of society [23] .
Recruiting volunteers in broad-scale citizen science projects to gather biodiversity information can generate enormous quantities of data across broad spatial and temporal domains [22] . However, maximizing the information gathered from citizen-science data depends on finding the proper balance between data quantity and quality [7] . Data quantity is essential because obtaining sufficient volumes of data of low per-datum information can contain as much information as data with high information content but gathered in smaller amounts [16] . Due to the importance of data quality, citizen science projects must take into consideration the ease of the data gathering process, the ability to limit data entry errors and identify questionable observations, and the offering of incentives to contributors to submit high-quality observations [26] .
The most significant data quality issue in broad-scale citizen-science is individual variability in detection and classification of organisms to species. While large citizenscience projects can engage a broad network of tens of thousands of individuals contributing their observations, each participant has different identification skills. Data collected by inexperienced citizen scientists is often of lower quality due to their lack of expertise in accurately detecting and identifying organisms. On the other hand, data collected by experts is much more accurate though not completely free of mistakes.
Our strategy for addressing data quality in broad-scale citizen science is to examine contributions from many citizen scientists in a given geographic region. We can then identify general observational patterns for a specific geographic region, which allows for quality filters to emerge from the data. Furthermore, the expertise level of a citizen scientist can be used to screen that individual's contributions to a broad-scale citizen-science project. For instance, if an individual is a novice and he is frequently reporting rare, difficult-to-detect birds, then his records may be flagged more frequently for review than an expert's records. However, the expertise level of a participant needs to be estimated, which can be accomplished by using data mining techniques based on their historical observations. Thus, the challenge is to identify outliers (i.e. observations of a species that is unusual for a location or date) and categorize these outliers as either unusual valid observations, or mis-identified (invalid) observations.
In this work, we describe a case study in which we evaluate the effectiveness of an automated data filter using eBird (http://www.ebird.org) as our exemplar broad-scale citizen science project [22] . Our automated data filter combines two parts --an emergent filter, derived from observed frequency patterns in the data, and a data quality model from our prior work [27] that predicts the observer's expertise. This observer expertise model was previously only evaluated on its ability to predict observer expertise and species observations. Our current work applies the observer expertise model from [27] for a different purpose -namely that of data quality control. The main contribution of our current work is the evaluation of the effectiveness of the overall automated data filter in a retrospective study using data from Tompkins Co., NY. This evaluation of data quality control, which was not performed in [27] , is particularly challenging for eBird because there is no real ground truth (i.e. it is never known whether all species at a given location were detected and identified by the observer).
Nevertheless, we will show that the observe combined with the emergent filter prod results for improving the data quality for a b science project.
A. The eBird Project
The eBird project [22, 26] is a citizen-sc engages a global network of bird watchers t species and report their observations to a cen Anyone can submit their observations of bird web, and more than 50,000 individuals have v million hours to collect more than 70 million for eBird from more than 200 countries; largest biodiversity data collection project in 1 depicts an overview of the eBird system. eBird data contain information on the o visit, and species observed. Observer info name, ID and contact information allow ever to be attributed to a specific person. Locatio site name, the coordinates where the observ and the geographic information, are stored w that location. Information about a specific v date and time of the visit, the amount of ef whether all the species observed were observations consist of a checklist of birds o many individuals of each species were counte eBird data reveal patterns of bird occurr and through time, providing a data-rich understanding the broad-scale dynamic populations [7] . Recently, the United States Interior used eBird data as the basis for the Birds Report, which estimated the occu populations on public lands [17] . Figure 2 sho a species distribution estimate of Western M the western US in June 2009 based on eBird o Data quality is a major issue for th particularly regarding an observer's abili correctly identify birds to the taxonomic lev network of bird distribution experts volun create expert-defined filters to provide the b regional-specific checklists of birds for data experts have a thorough knowledge of the se bird occurrence for a specific region. Based o a regional checklist filter delineates, when each species are expected in that region. on the checklist they must take eport a species that would not s record is flagged for review. r an area as large as a country or Presently eBird project employs filters. 550 volunteers review flagged red by an editor range from an uth America), to a single county da. The reviewers contact those gged records to obtain additional tes or photographs, in order to 010, 4% (720k observations) of bmitted to eBird were flagged for vations) were marked as invalid ds, their flags and their review d database [9] . and time of year, an editor will ek (average of 200). About 80% ed fairly quickly in 5-10 seconds. ing up with the observer and the process is semi-automated, it eview each record. About 1% of n more time, as the editor will n a series of emails, explaining y and the review process, and y have. These communications rticipants are the most frequent have with people from eBird.
Thus, there are three major data quality ch the eBird project. First, given the larg participants' expertise, invalid observations m identified to improve data quality. Second, t defined filters have generated an enorm observations for review, which overwhelm volunteer editors. This problem becomes e with the tremendous growth of observations as illustrated in Figure 3 . Finally, due to eBird, data filters need to be created for new existing regional experts.
II. RELATED WORK
The eBird project is an example of a crow [8] that engages a large number of people to automated sensors and computers cannot re Crowdsourcing systems have been succes computer vision [4] and natural language pro Galaxy Zoo project [12] uses volunteers t from the Sloan Digital Sky Survey.
One way to improve data quality in crow is to have multiple observers annotate the sa and Skeene [2] first addressed this problem and developed a general model by treating th a latent variable. Smyth et al. [20] used a si labeling Venus images. Unlike previous wor estimating the label of the objects, Raykar et an EM algorithm to estimate the label and to jointly. Whitehill et al. [25] took one step fur difficulty of labeling an image into accoun [24] proposed a Bayesian generative model annotator's attributes, including expertise, bias. In a different line of work, Sheng et al reduce uncertainty of the labels by repeated-l
The eBird project has characteristics tha from other citizen science projects and th approaches for improving data quality. First in the eBird project collect the actual annotating data that has already been collec [12] ). Since observers vary greatly in expended, their observations cannot be sim ground truth. Second, eBird is a high-volu datum information system, meaning that a area may have multiple observations from wdsourcing system perform tasks that eadily accomplish. ssfully applied in ocessing [21] . The to classify images dsourcing systems ame object. Dawid in diagnostic tests he ground truth as imilar approach in rk that focused on t al. [18] proposed o learn a classifier rther by taking the nt. Welinder et al. that can model an competence and l. [19] proposed to labeling. at make it distinct hus require novel t, citizen scientists data, rather than cted (eg. unlike in skill and effort mply accepted as ume and low perwider geographic m many observers.
Although these observations a location and time (making the improving data quality not viab and temporal scale of eBird, a observational patterns which outliers and assessing the reliab
The approach in this work other citizen science projects with a wide range of expertise of low per-datum information.
III. M
The automated data filter filter and the data quality m describe in the following subse
A. Emergent data filter
We can use the large vol from eBird as the basis for au checklists. We replace the expe driven filters that emerge from regional checklists and identify
The emergent data filter in frequency of reporting a specie the number of checklists that r the total number of checklists These frequencies are easily u and thus the emergent filters ar is a measure of the likelihood within that region. Since each where and when a bird was frequencies of bird occurrence date of year. Figure 4 sho Sparrow's weekly occurrence f in Tompkins Co., NY.
For any specific region (e.g data filter automatically iden follows: the frequency of occu species that have been reported that day of year. The frequen online checklist by including frequency is past a threshold. T observations falling below th observations with the data quali r month since 2003. Figure 4 . The weekly occurrence Tompkin are rarely from the exact same multiple observers strategy for ble for eBird), the broad spatial allows the emergence of general can be effective for detecting bility of the data collected. k is more broadly applicable to that rely on citizen scientists, levels, to collect a high volume
In all cases examined, the expert-defined checklist filters for Tompkins Co. accepted observations over a broader temporal window than the emergent data filters. Three general categories for the expert-defined filter were apparent. First, an expert may have had a particular interest or knowledge of certain species and these filters could be very accurate (e.g. Figure 8B American Tree Sparrow Jan. -May). Second, the expert-generated filter may accurately describe the bird's biology, which may be quite different from what eBird contributors report. For example, Chipping Sparrows (see Figure 8A ) are a common breeding bird in Tompkins Co., which are often found in close proximity to lawns and gardens, and have a very distinctive plumage and song. However, immediately after the breeding season (end of July) they stop singing, disperse, and begin to molt into a less distinctive plumage. They become more cryptic and harder to detect, which would lower the probability that they are reported to eBird. The final category included expert-defined filters that accepted observations, even when it was very unlikely that the bird would be encountered. For example, although expert filters allowed either Swamp or Savannah Sparrow to be reported for any month of the year in Tompkins Co., observations falling outside the typical pattern of occurrence, especially during winter, should be reviewed. 
For the emergent data filter, the tempo the 5% threshold in frequency created a m window of occurrence than that develope defined filter. Since the emergent data filte observer submissions, they matched the patte eBird volunteers reported a particular spec Co. However, the emergent data filters signi the number of flagged records. The eme flagged more than 35425 observations for rev 4006 observations that were flagged by t filters. We conclude that the emergent data cut-off accurately represented the patterns eBird for the majority of observations, and identification of any outliers. However, conservative filter, which resulted in a sign the number of flagged records that a reg review. If the automated frequency fi employed, it would lead to a greatly increa the regional editors. One alternative for red of flagged records would be to make conservative (e.g. set the cutoff to be 3% o this would increase the possibility misidentifications to become part of eBird da Figure 7 represents a schematic of the ex and emergent data filter for a single spec falling outside of the bars were flagged fo mentioned previously, in our data, the em always a shorter window of acceptance defined filter and was thus more conservati and expert-defined filters in Figure 7 crea regions labeled A, B, and C that we will use of evaluation metrics.
Records falling in region A were not filters and added to the eBird database with these records were not reviewed, we did not about the actual misidentifications in region number of actual misidentifications in re identical for both the emergent and expert-de Second, records falling in region B we emergent filter but not by the expert-defined B corresponded to a time period in which were common, such as after a particular bird for migration and before their return. Since Region B were also not reviewed in our retro we did not have ground truth about actual mi ral resolution and more conservative ed by the experters were based on erns of when most cies for Tompkins ificantly increased ergent data filters view, compared to the expert-defined filters set at a 5% s of reporting to d allowed the easy it was a very nificant increase in gional editor must ilter alone were ased workload for ducing the number e the filter less of detections), but y of allowing atabase. xpert-defined filter cies. Observations or review. As was mergent filter was than the expertive. The emergent ated three distinct e in our discussion flagged by both hout review. Since t have information n A. However, the egion A will be efined filters. ere flagged by the d filter. The region misidentifications d species departed e these records in ospective analysis, isidentifications.
Finally, records falling in filters. Unlike for regions A an reviewed by experts and then d added to eBird or designated i the validity of these records as filter in region C.
METHODS
consists of the emergent data model, both of which we will ections. lume of historical observations utomatically generating regional ert-defined data filters with datam the historical data to generate unusual observations. n eBird project is based on the s. The frequency is calculated as reported the species divided by submitted for a specific region. updated as new data is reported, re constantly updated. The result of observing a specific species observation contains details of detected, we can calculate the at any spatial level and for any ows an example of Chipping frequency across a year window g. county) and date, the emergent ntifies unusual observations as urrence estimates is made for all d to eBird for that region and for ncy is then used to generate an all species whose occurrence The emergent data filter flags the he threshold and processes the ity model. 
B. Data quality model
The eBird data are provided by tens of thousands of observers with a wide range of expertise in identifying birds and with variable effort made in contributing to eBird. For example, at one extreme, several thousand observers with high identification skill levels contribute "professional grade" observations to eBird, whereas at the other extreme tens of thousands of participants contribute data of more variable quality. While there is much variability in the number of checklists that eBird volunteers submit, the top third of eBird contributors submit more than 90% of all data. Although the identification skills of this subset of contributors are unknown, it is probably skewed to the more skilled because individuals who regularly contribute tend to become better observers [6] .
This inter-observer variation must be taken into account during analysis because valid outlier observations (i.e. those observations that are unusual but valid) could provide potentially important information on unique or changing patterns of occurrence. Since eBird engages a significant number of skilled observers who are motivated to detect rare species or are skilled in detecting elusive and cryptic species, being able to accurately distinguish their observations from those of less-skilled observers is crucial. The challenge is to obtain an objective measure of observer expertise that can be used to classify unusual observations.
In this case study, we investigate using a data quality model based on an observer's predicted expertise. In order to predict expertise, we use the Occupancy-Detection-Expertise (ODE) model from our prior work [27] because it was successful at accurately predicting an observer's expertise. The ODE model is a probabilistic graphical model [11] that extends a wellknown model in Ecology called the Occupancy-Detection (OD) model [13] . We will provide an overview of the ODE model by first describing the OD model and then showing how to extend it with observer expertise in the ODE model. More details about the ODE model can be found in [27] .
1) Occupancy-Detection model
Ecologists often build species distribution models (SDM), which predict the occupancy of a site by a species. Occupancy determines if a geographic site is viable habitat for a species and it depends on environmental covariates such as temperature, precipitation and land use. A general form for SDMs is shown as the occupancy function in Equation 1, in which the Boolean random variable represents the occupancy of site i, are the set of environmental covariates (i.e. features), and are the parameters of the model. Many different machine learning approaches have been used to model ; . In our work, we use logistic regression.
Data for SDMs are often collected by observers who record occurrences of species in the field. Detecting the species in the field can pose a major problem, particularly if the members of the species are camouflaged, only appear at night, or are evasive. If an observer wrongly reports a species to be absent at a site when it was in fact present at that site, species distribution models built from such data will underestimate the true occupancy of that species for that site.
To address this issue, Mackenzie et al. [13] proposed the OD model, which models the relationship between occupancy and detection. In the OD model, the true occupancy of a
, examples of which include the effort expended, the time of day, and the current weather conditions.
In Equation 2, the Boolean random variable is influenced by both the true occupancy of a site and by the detection function , which we model using a logistic function with parameters .
The OD model makes two key assumptions. First, the population closure assumption [14] assumes that the species occupancy status at a site stays constant over the course of the visits. Second, the OD model does not allow the observers to report false detections. False detections occur when observers incorrectly report a species to be present at a site when in fact the species does not occupy that site.
2) Occupancy-Detection-Expertise model
The detection process can also be affected by the expertise level of an observer. The ODE model extends the OD model by adding an expertise component. In this expertise component, as shown in Equation 3, the function predicts , which is a binary random variable representing the expertise of birder j (i.e. taking values expert or novice), from a set of expertise features . Examples of expertise features include features derived from the birder's personal information and history of observations, such as number of years in bird watching, number of observations submitted to eBird and number of observations invalidated. We model ; using a logistic function with parameters .
The detection function in the ODE model is now influenced by the detection features and by the expertise of the birder that supplied observation . Equation 4 illustrates the detection process in the ODE model, where the 
By adding the expertise variable to , , ,
we distinguish the detection process of expert and novice birders. In effect, we use a mixture model in which one mixture component models the experts' detection process and one models the novices' detection process. Furthermore, in order to improve the model fit on the eBird data, we also modified the assumptions of the OD model to allow false detections by both experts and novices Given the observations from a set of labeled expert and novice birders, we estimate the ODE model parameters using Expectation Maximization [3] . With the learned ODE model parameters, we can perform inference to predict a birder's expertise based on their checklists. In deployment, we update the ODE model annually.
IV. EVALUATION

A. Data description
For this case study we analyzed eBird data from Tompkins Co., which is an average sized county (1,270 km 2 ) in the ecologically rich Finger Lakes Region of west-central NY. Participation in eBird is high in this county, with more than 48,000 checklists representing almost 700,000 observations. A regional expert developed a checklist filter for this county, which was the basis for all following comparisons. To evaluate the expert-defined filter and automated data filter, we applied both filters to eBird data collected from January 1, 2003 to June 23, 2011.
B. Emergent data filter
To generate the emergent data filter, we calculated the frequency of occurrence based on all data reported for that species at the county level and date range, and compared with eBird submissions. This frequency was calculated as follows. First, a day-of-year value was assigned to each checklist ranging from 1 to 365, and then a raw daily frequency was associated with this day. However, there were large variations in the raw daily frequency, which ranged from 3 to 125 checklists. To account for this variation, we replaced each raw daily frequency with a value computed by taking the highest raw daily frequency of a day within a sliding 7-day window (3 days before to 3 days after the current day). In this study, we calculated the day-of-year frequencies for every species observed in Tompkins Co., NY based on eBird data gathered from 2003 to 2011. We used a frequency threshold of 5%.
C. Estimating an observer's expertise level
For our analysis, we used observations from the original eBird Reference Data [15] from New York State during MayJuly in years 2009 and 2010. To train the ODE model, we used the observations from a list of birders with their expertise levels labeled. The eBird project leaders manually labeled the expertise of these birders using a variety of criteria including personal knowledge of the birder, the number of misidentified observations, the frequency of poor spatial accuracy in checklist submissions and manual inspection of their eBird submissions. There were a total of 134 expert birders and 229 novice birders used to train the ODE model.
We divided the checklists into training and test sets according to the observers submitting them. Birders that submitted checklists from Tompkins Co. in 2009 and 2010 were placed into an independent test set while labeled birders were placed into a training set. The test set consisted of 176 birders. We trained the ODE model and then used the trained model to predict the probability of a birder from the test set being an expert. To get a more reliable estimation of observer expertise, we applied the ODE model to 18 species (8 common species and 10 uncommon species) and the final expertise prediction was based on the average score over all 18 species.
V. RESULTS AND DISCUSSION

B. The automated data filter
In Figures 8A and 8B , predictions of expertise in rela two filters. What is most strik low level of eBird expertise t Tree Sparrow and Chipping windows of occurrence more are very similar looking spar feeders and easily observed. confuse these species, and m particularly at their first season region C were flagged by both nd B, these records were in fact designated to be either valid and invalid and discarded. We used s a measure of the accuracy of a we illustrate the ODE model ation to records flagged by the king is how individuals with a ended to report both American Sparrow outside their typical frequently. These two species rrows that are attracted to bird Many inexperienced observers misidentification is a problem nal arrival. The observers of low
A) Chipping Sparrow and (B) American
The time periods of the emergent filter ilter (light grey) are shown as horizontal s falling within the time period indicated atically accepted into eBird and are not alling outside of the emergent filter were triangles (from novices) or circles (from ded black while invalid observations are predicted expertise reported more American Tree Sparrows earlier in fall than observers of high predicted expertise, and their observations fell outside the general patterns of the frequency graphs. This example shows the significant contribution that the automated filter process could have for identifying outlier reports for birds that are relatively common, and which would normally pass as valid records under the expert-defined filter model. Table 1 provides examples of a variety of bird species with difference occurrence patterns in Tompkins Co. and the percent of expert/novice observations that were flagged by the emergent filter. These results indicate that expert observers tend to identify more unusual birds than novice observers. Use of the automated data filter would significantly reduce the number of flagged records that must be reviewed since it accepts records from expert observers. Species that occur year round at frequencies below the emergent filter. 2 Species that occur periodically in the county. 3 Species that are locally common breeders in the county. 4 Species that are locally uncommon breeders in the county. 5 Migrant species that are locally common when they pass through the county. 6 Species that are locally common breeders and uncommon throughout the year. Table 2 shows the number of flagged records from all three filters. The emergent data filter significantly increased the total number of observations for review to 35425, but when the emergent filter was combined with the ODE model in the automated data filter, the number of flagged records decreased by 93% to 2303. When compared to the expert-defined filter, the automated data filter decreased the number of flagged records by as much as 43%, showing the potential of the automated data filter for substantially reducing the workload of reviewers. Under current expert-defined filters, each reviewer spends approximately 5 hours per week reviewing flagged records; this cost reduces to 2.85 hours (i.e. 2.15 hours saving per week) with the automated data filter. These savings become even larger due to the fast growth of eBird project. Although the automated data filter can substantially reduce the number of records to be reviewed, it must also not carelessly discard any truly erroneous records that should indeed be reviewed. In order to measure the accuracy of the automated data filter in our retrospective analysis of eBird data from Tompkins Co., we compared how many of the records in region C were designated as valid or invalid after being reviewed. Figure 9 (left) illustrates the fraction of valid and invalid records among all the records in region C, and then the amounts broken down by experts and novices (middle and right pie charts). Only 137 flagged records (5%) from experts were invalid, while 848 records (65%) from novices were invalid. The automated data filter would have allowed the 137 flagged expert records to pass through, but all 848 novice records would have been flagged.
The analysis above only covers region C and gives a partial picture as to the accuracy of the automated data filter as the 2303 records flagged by the automated data filter are in both regions B and C. Records in region B were not reviewed, and as a result, we did not have any ground truth as to their validity. However, we can estimate the number of invalid records by making the assumption that, as in region C, novices submitted invalid records 65% of the time. This assumption is conservative because misidentifications tend to increase in region B as compared to region C. Under this assumption, 65% of the 2303 records flagged by the automated data filter were invalid (i.e. 1497 records). This amount is higher than the 985 invalid observations flagged by the expert-defined filter by as much as 52%, thus showing how effective the automated data filter is at identifying truly erroneous outliers. VI. CONCLUSION Data quality is a major challenge in any sensor network, especially when the sensor network consists of a massive number of volunteer observers that have differing abilities to accurately identify birds. This paper assessed the performance of a more automated process for addressing a major data quality need in broad-scale citizen-science projects: filtering misidentified organism occurrences. Our automated data filter was based on both the patterns of submissions within a predefined spatial and temporal extent, as well as the contributor's skill level.
We presented the results of applying the automated data filter retrospectively to historical records from Tompkins Co., NY. Our automated data filter allowed us to reduce the workload of reviewers by about 43% as compared to the existing expert-defined filter, which results in about 2.15 hours savings per week for a reviewer. The automated data filter also identified as many as 52% more invalid outliers than the expert-defined filter. These results demonstrate that our automated process has the potential to play a critical role in improving data quality in broad-scale citizen-science projects.
For future work, we will test these results more broadly across the US. In addition, we will model an individual's expertise regionally, as a birder can be an expert observer in their home region, but less so outside of that region.

{'paper_id': '9222013', 'title': 'Automated data verification in a large-scale citizen science project: A case study', 's2_url': 'https://api.semanticscholar.org/CorpusID:9222013', 'source': 'S2ORC'}