In this paper, we present CrowdTone, a system designed to help people set the appropriate tone in their email communication. CrowdTone utilizes the context and content of an email message to identify and set the appropriate tone through a consensus-building process executed by crowd workers. We evaluated CrowdTone with 22 participants, who provided a total of 29 emails that they had received in the past, and ran them through CrowdTone. Participants and professional writers assessed the quality of improvements finding a substantial increase in the percentage of emails deemed "appropriate" or "very appropriate" - from 25 to more than 90 by recipients, and from 45 to 90 by professional writers. Additionally, the recipients' feedback indicated that more than 90 of the CrowdTone processed emails showed improvement.
@cite_30: Despite decades of research attempting to establish conversational interaction between humans and computers, the capabilities of automated conversational systems are still limited. In this paper, we introduce Chorus, a crowd-powered conversational assistant. When using Chorus, end users converse continuously with what appears to be a single conversational partner. Behind the scenes, Chorus leverages multiple crowd workers to propose and vote on responses. A shared memory space helps the dynamic crowd workforce maintain consistency, and a game-theoretic incentive mechanism helps to balance their efforts between proposing and voting. Studies with 12 end users and 100 crowd workers demonstrate that Chorus can provide accurate, topical responses, answering nearly 93 of user queries appropriately, and staying on-topic in over 95 of responses. We also observed that Chorus has advantages over pairing an end user with a single crowd worker and end users completing their own tasks in terms of speed, quality, and breadth of assistance. Chorus demonstrates a new future in which conversational assistants are made usable in the real world by combining human and machine intelligence, and may enable a useful new way of interacting with the crowds powering other systems.
@cite_35: In story writing, the diverse perspectives of the crowd could support an author's search for the perfect character, setting, or plot. However, structuring crowd collaboration is challenging. Too little structure leads to unfocused, sprawling narratives, and too much structure stifles creativity. Motivated by the idea that individual creative leaders and the crowd have complementary creative strengths, we present an approach where a leader directs the high-level vision for a story and articulates creative constraints for the crowd. This approach is embodied in Ensemble, a novel collaborative story-writing platform. In a month-long short story competition, over one hundred volunteer users on the web started over fifty short stories using Ensemble. Leaders used the platform to direct collaborator work by establishing creative goals, and collaborators contributed meaningful, high-level ideas to stories through specific suggestions. This work suggests that asymmetric creative contributions may support a broad new class of creative collaborations.
@cite_18: Micro-task markets such as Amazon's Mechanical Turk represent a new paradigm for accomplishing work, in which employers can tap into a large population of workers around the globe to accomplish tasks in a fraction of the time and money of more traditional methods. However, such markets typically support only simple, independent tasks, such as labeling an image or judging the relevance of a search result. Here we present a general purpose framework for micro-task markets that provides a scaffolding for more complex human computation tasks which require coordination among many individuals, such as writing an article.
@cite_7: The physical constraints of smartwatches limit the range and complexity of tasks that can be completed. Despite interface improvements on smartwatches, the promise of enabling productive work remains largely unrealized. This paper presents WearWrite, a system that enables users to write documents from their smartwatches by leveraging a crowd to help translate their ideas into text. WearWrite users dictate tasks, respond to questions, and receive notifications of major edits on their watch. Using a dynamic task queue, the crowd receives tasks issued by the watch user and generic tasks from the system. In a week-long study with seven smartwatch users supported by approximately 29 crowd workers each, we validate that it is possible to manage the crowd writing process from a watch. Watch users captured new ideas as they came to mind and managed a crowd during spare moments while going about their daily routine. WearWrite represents a new approach to getting work done from wearables using the crowd.
@cite_9: Without the benefit of paralinguistic cues such as gesture, emphasis, and intonation, it can be difficult to convey emotion and tone over electronic mail (e-mail). Five experiments suggest that this limitation is often underappreciated, such that people tend to believe that they can communicate over e-mail more effectively than they actually can. Studies 4 and 5 further suggest that this overconfidence is born of egocentrism, the inherent difficulty of detaching oneself from one’s own perspective when evaluating the perspective of someone else. Because e-mail communicators “hear” a statement differently depending on whether they intend to be, say, sarcastic or funny, it can be difficult to appreciate that their electronic audience may not.
@cite_3: Crowdsourcing has been shown to be an effective approach for solving difficult problems, but current crowdsourcing systems suffer two main limitations: (i) tasks must be repackaged for proper display to crowd workers, which generally requires substantial one-off programming effort and support infrastructure, and (ii) crowd workers generally lack a tight feedback loop with their task. In this paper, we introduce Legion, a system that allows end users to easily capture existing GUIs and outsource them for collaborative, real-time control by the crowd. We present mediation strategies for integrating the input of multiple crowd workers in real-time, evaluate these mediation strategies across several applications, and further validate Legion by exploring the space of novel applications that it enables.
@cite_24: This paper presents the MicroWriter, a system that decomposes the task of writing into three types of microtasks to produce a single report: 1) generating ideas, 2) labeling ideas to organize them, and 3) writing paragraphs given a few related ideas. Because each microtask can be completed individually with limited awareness of what has been already done and what others are doing, this decomposition can change the experience of collaborative writing. Prior work has used microtasking to support collaborative writing with unaffiliated crowd workers. To instead study its impact on collaboration among writers with context and investment in the writing project, we asked six groups of co-workers (or 19 people in total) to use the MicroWriter in a synchronous, collocated setting to write a report about a shared work goal. Our observations suggest ways that recent advances in microtasking and crowd work can be used to support collaborative writing within preexisting groups.
@cite_25: This paper introduces architectural and interaction patterns for integrating crowdsourced human contributions directly into user interfaces. We focus on writing and editing, complex endeavors that span many levels of conceptual and pragmatic activity. Authoring tools offer help with pragmatics, but for higher-level help, writers commonly turn to other people. We thus present Soylent, a word processing interface that enables writers to call on Mechanical Turk workers to shorten, proofread, and otherwise edit parts of their documents on demand. To improve worker quality, we introduce the Find-Fix-Verify crowd programming pattern, which splits tasks into a series of generation and review stages. Evaluation studies demonstrate the feasibility of crowdsourced editing and investigate questions of reliability, cost, wait time, and work time for edits.
Crowdsourcing is another domain being explored for writing assistance. Crowd-powered projects like Soylent @cite_25 , Legion @cite_3 and Chorus @cite_30 have encouraged researchers to harness the crowd for complex tasks --- and have even been implemented on different interfaces, including smart-watches @cite_7 . Projects like Turkomatic @cite_9 , Ensemble @cite_35 , Crowdforge @cite_18 , and others @cite_24 have succeeded in converting macro tasks into micro chunks that makes it easier for crowd workers to accomplish expert-level tasks such as writing articles and stories.