How to take multiple factors into account when evaluating a Game with a Purpose? How is player behaviour or participation influenced by different incentives? How does player engagement impact their accuracy in solving tasks? In this paper, we present a detailed investigation of multiple factors affecting the evaluation of a GWAP and we show how they impact on the achieved results. We inform our study with the experimental assessment of a GWAP designed to solve a multinomial classification task.
@cite_10: Citizen participation in online communities of scientific investigations has recently become more popular. Enhancing the engagement of citizens within these communities is a focus of attention for researchers and practitioners who want to amplify the impact on learning, science and society. This study investigates the relationship between engagement factors and behaviour patterns in an online community that requires high levels of citizen participation. While other studies explore engagement in communities where citizens contribute data, the current research investigates a community to support citizens in facilitating their own scientific investigations. Data were collected from log files and questionnaires, and multiple measures of engagement were examined: engagement metrics, roles, motivation, attitude, satisfaction and belonging to the community. The results allowed comparison of the engagement levels among different types of citizen participation communities and categorised members in engagement profiles, according to their behaviour patterns. Findings indicate a need for differing design approaches based on the type of citizen participation community and individual engagement profiles.
@cite_3: Human computation is a computing approach that draws upon human cognitive abilities to solve computational tasks for which there are so far no satisfactory fully automated solutions even when using the most advanced computing technologies available. Human computation for citizen science projects consists in designing systems that allow large crowds of volunteers to contribute to scientific research by executing human computation tasks. Examples of successful projects are Galaxy Zoo and FoldIt. A key feature of this kind of project is its capacity to engage volunteers. An important requirement for the proposal and evaluation of new engagement strategies is having a clear understanding of the typical engagement of the volunteers; however, even though several projects of this kind have already been completed, little is known about this issue. In this paper, we investigate the engagement pattern of the volunteers in their interactions in human computation for citizen science projects, how they differ among themselves in terms of engagement, and how those volunteer engagement features should be taken into account for establishing the engagement encouragement strategies that should be brought into play in a given project. To this end, we define four quantitative engagement metrics to measure different aspects of volunteer engagement, and use data mining algorithms to identify the different volunteer profiles in terms of the engagement metrics. Our study is based on data collected from two projects: Galaxy Zoo and The Milky Way Project. The results show that the volunteers in such projects can be grouped into five distinct engagement profiles that we label as follows: hardworking, spasmodic, persistent, lasting, and moderate. The analysis of these profiles provides a deeper understanding of the nature of volunteers' engagement in human computation for citizen science projects.
@cite_23: This report documents the program and the outcomes of Dagstuhl Seminar 17272 "Citizen Science: Design and Engagement". In this report, we detail the briefly summarise the content of three invited keynote talks and two invited tutorials. We further outline the findings of five parallel working groups, which met on the first and third day of the workshop, in the areas of: sustainability, measuring success, community engagement, linking and quality.
@cite_20: Online citizen science projects have been increasingly used in a variety of disciplines and contexts to enable large-scale scientific research. The successes of such projects have encouraged the development of customisable platforms to enable anyone to run their own citizen science project. However, the process of designing and building a citizen science project remains complex, with projects requiring both human computation and social aspects to sustain user motivation and achieve project goals. In this paper, we conduct a systematic survey of 48 citizen science projects to identify common features and functionality. Supported by online community literature, we use structured walkthroughs to identify different mechanisms used to encourage volunteer contributions across four dimensions: task visibility, goals, feedback, and rewards. Our findings contribute to the ongoing discussion on citizen science design and the relationship between community and microtask design for achieving successful outcomes.
Other metrics to evaluate GWAPs can be borrowed from studies of social community @cite_20 and citizen science evolution @cite_23 ; in those cases, however, user participation's success'' is measured through simple indicators like number of participants and contributions, while a deeper investigation is needed to assess the effectiveness of participation. Behavioural studies in HCI research have investigated volunteer characterization in citizen science, defining engagement metrics and profiles @cite_3 @cite_10 , which may or may not apply to GWAP players.