{'fraction_non_alphanumeric': 0.0341119650071383, 'fraction_numerical': 0.013000820145196075, 'mean_word_length': 4.794086589229145, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Wisdom and Persuadability of Threads

Online discussion threads are important means for individual decision-making and for aggregating collective judgments, e.g. the `wisdom of crowds'. Empirical investigations of the wisdom of crowds are currently ambivalent about the role played by social information. While some findings suggest that social information undermines crowd accuracy due to correlated judgment errors, others show that accuracy improves. We investigate experimentally the accuracy of threads in which participants make magnitude estimates of varying difficulty while seeing a varying number of previous estimates. We demonstrate that, for difficult tasks, seeing preceding estimates aids the wisdom of crowds. If, however, participants only see extreme estimates, wisdom quickly turns into folly. Using a Gaussian Mixture Model, we assign a persuadability score to each participant and show that persuadability increases with task difficulty and with the amount of social information provided. In filtered threads, we see an increasing gap between highly persuadable participants and skeptics.


Introduction
Social information in the form of opinions and judgments by other people is sampled sequentially. We read the news, hear rumors, listen to debates on TV, and scroll through comments on social media platforms and blogs. These activities inform us and influence our decisions, but researchers still debate under what conditions these types of social information improve decisions-making [1][2][3][4], lead us astray [5][6][7][8][9], or simply add to our confusion [10,11].
Collective estimates of a diverse group of people can outperform the majority of its members because any random confusion at the individual level is likely to average out and let the most accurate estimate prevail [12][13][14][15]. Then again, confusion is not always randomly scattered around the truth. Systematic biases in individual perceptions may create measurable disruptions in the wisdom of crowds [16][17][18]. Social information may add to those biases and create cascades, echo chambers, bandwagoning and herding behavior [19][20][21][22]. Partially sampled social information may lead to rich-get-richer dynamics [23] and to belief misattributions, which uphold harmful social practices despite being rejected by a majority of people [24][25][26][27][28]. Social information may also have been intentionally filtered or manipulated in various ways, for instance through group pressure [29], algorithmic filtering [30], false cues [10,31,32], or simply by plain misinformation [33], often with highly detrimental consequences for our economy and our health.
Observational data of decision-making processes is acutely sensitive to the social context in which people find themselves. Thus, researchers find it difficult to separate observational data into its social and individual components. How may we know how much weight an individual puts on her own 'independent' estimate relative to the weight put on the estimates by others? Randomized experimental studies have attempted to solve this problem by first letting participants make a magnitude estimate of an object without social information (ex ante), and subsequently ask them to revise their estimate after hav-ing received information about other people's estimates of that very object (ex post) [3,4,6,34,35]. This setup presumes that people change their mind because of the social information they receive. Other studies, however, have shown that people routinely change their mind all by themselves, and that it may be more correct to assume an 'inner crowd' in the sense that people sample randomly from a probability distribution in their own mind [36][37][38]. Such a psychological mechanism -and perhaps others such as hedging strategies due to anticipated regrets [39], and/or disappointments [40] -make it difficult to differentiate between 'inner' samples and 'outer' influences. It is therefore desirable to develop an alternative framework that is able to infer the extend of individual bias and social influence from a single estimation task.
We propose to use a probabilistic Gaussian Mixture Model (GMM) as it has properties that are highly valuable in crowd aggregation research. First, a GMM associates a measure to each data point, describing the influence of social information on that particular participant without any prior knowledge about that participant. Second, a GMM is comprised of several Gaussians which fit well to the right-skewed and heavy-tailed distributions emerging from free response elicitations. Finally, as a statistical model, a GMM provides confidence bounds to the estimates, which further adds a measure of model uncertainty.
We also propose the experimental mechanism of dotguessing games [41], where participants guess the number of dots in an image. While dot estimations have been used previously in numerosity experiments [42][43][44] they have only recently been proposed as useful 'model organisms' for crowd aggregation research [41,45] due to their advantages in terms of cultural neutrality, resistance to expertise and/or prior knowledge, and their qualities as captchas [46]. In addition, dot estimation tasks are easy to implement and easy to understand. Most importantly, they have an objective solution and are tunable, allowing for nearly-continuous difficulty levels and performance measures. There is a clear relation between d and v showing that social information plays a dual role in difficult tasks: When v is large, pristine and unmanipulated social information (left) counteracts underestimation bias and improves thread accuracy. When the social information is manipulated (right), however, a strong overestimation bias emerges for large v. The red lines, corresponding to the control groups with v = 0, are identical in both plots.
We collected a total of 11,748 estimates from 6,196 unique participants on Amazon Mechanical Turk. 5,990 estimates were collected from participants placed in 12 different threads where they successively estimated the number of dots, d ∈ {55, 148, 403, 1097}, in an image, while seeing the v ∈ {1, 3, 9} preceding estimates (historical threads). Another 3,934 estimates were collected from participants who were placed in 12 other threads and shown the same images while seeing the v ∈ {1, 3, 9} highest estimates made so far (manipulated threads). Finally, 1.824 estimates were collected from participants who were shown the same images, but with v = 0, corresponding to control conditions for each d containing no social information.
We interpret the number of dots, d, as the task difficulty, while the number of visible previous estimates, v, is interpreted as the degree of social information. Participants were placed randomly in one of the 28 threads (2 × 3 × 4 + 4 controls) and made their estimate one after another. In order to keep the estimates in a somewhat realistic range, participants could not submit numbers below 10 and above 1.000.000. No participant who had seen a certain image would be able to participate in another thread containing the same image again. In addition to a participation fee and a variable waiting fee, all participants in all threads received a bonus of $1 if their estimate was within 10% of the true value. See the Methods section and the Supplementary Information for additional information about the experimental design.
Free response elecitation of absolute values is known to create right-skewed distributions with long tails, which inflate the means. While it is still debated which mea-sure is best suited to aggregate such data [18], we follow the lead of Galton [12] and focus on the median as it is easy to interpret, robust against outliers, and best expresses the opinion of the crowd in the sense that the majority deems every other estimate as too high or too low. For the statistical analysis of thread aggregates, we therefore compare the log-ratio of thread medians, y dv = log(M (d,v) /d), using a linear normal model to quantify differences between threads in terms of log(d) and v, the latter as a categorical variable. The effects of social information on individual descision-making are analysed using a GMM on the log-ratio of individual estimates Y i with the weighted geometric mean of the social information as the explanatory variable. The model assigns a persuadability score, β w , to each participant, which is high when people are highly influenced by the social information they can see, and small when people are not influenced, or when they are skeptical about the information (see Methods for details).

Results
In accordance with previous findings, participants do well in tasks without social information, especially when estimating small numbers. For higher difficulties estimates vary widely and biases become substantial [16,18,43,44,47]. The median tends to underestimate the true value and the mean tends to overestimate the true value.

Analysis of thread performance
The relationship between the observed median log-ratio y dv and the number of dots d are found suitable, according to quantile-quantile plots (see Supplementary Infor-0  100  200  300  400  500  −2 −1 0   1   2   3   4   5 History: history Dots:  mation), when modeling y dv against log d and v. For more details, see the Methods section. Historical and manipulated threads are modelled separately to allow for differing error variances. Figure 1 (left) shows that the collective performance of historical threads declines significantly with increasing task difficulty d, but improves with the amount of social information. In contrast to [6][7][8] and in concert with [2][3][4]48] these findings support the claim that crowds indeed may become wise under (pristine) social influence. It should be noted, however, that the overlapping confidence intervals reveal where thread performances are comparable. Thus, the negative effects of task difficulty and the positive effects of social information are only discernible in situations where people have hard problems to solve, and at the same time have social information in abundance. In fact, the median estimate of historical threads with v = 9 is 'wise' in the sense of being statistically indistinguishable from the true value for all d.
In manipulated threads, Fig. 1 (right), the manipulation results in a large positive bias for v = 3, 9 which increases with d, implying that when a task becomes more demanding, the amount of (filtered) social information has a highly detrimental impact on thread performance. For v = 1 there is still a small negative trend, implying that the manipulation is not very effective. This resonates well with the findings in [4] which shows that providing a moderate amount of incorrect information may counterbalance underestimation bias and improve collective performance.

Analysis of Persuadability
In Fig. 2 two exemplary threads show how the GMM model framework reveals interesting features of the data. Each thread is represented by three figures. The top figures show the log-ratio of estimates as the thread evolves over time with the weighted geometric mean of the social information shown by a dotted line. Each estimate has an associated color, given by the persuadability score β w , which measures the influence of social information on each participant. A high value (β w > 0.6) in dark green and blue colors implies a large social influence effect, a low value (β w ≈ 0) in red implies a small or no effect, and a medium value (β w ≈ 0.4) in brown and light green colors suggests a compromise between the two extremes. The RGB color scale (right) is the same for all threads and plots. The bottom left plots show the log-ratio of the estimates as a function of the log-ratio of the social information, indicating how differently people use their social information: The more they align with the identity line, the more they tend to follow the social information.
The bottom right plots show the cumulative distribution of the persuadability scores with 95% confidence intervals derived from the fitted model. They clearly show how people may be categorized into those 'sleeping dogs' or 'lost causes' [49] in red that do not take other peo- ples estimates into account, the 'persuadables' that tend to follow others, and a large group of green 'compromisers', who try to strike a balance between what they see others have guessed, and what they believe themselves.
Of course, such labels are only proximate. Given a par-ticipant has a personal estimate much in line with the social information seen, this participant may be labelled as a persuadable or as a compromiser, when in fact this is only partially true. In general, however, the distributions are remarkable stable across threads, which suggests that people indeed may be partitioned into such overlapping response types.

Thread dynamics
What becomes abundantly clear when examining the results from the GMMs, is a sharp contrast between how people act in pristine historical threads, and how people act in manipulated, filtered threads. This is shown in Figure 3, where we compare typical historical threads (left) with their corresponding manipulated treatments (right). When the task difficulty is low and v = 1, participants are not significantly influenced, no matter whether they see the preceding or highest estimate in their thread (top row in Figure 3). But as soon as the amount of social information increases (second row), more people tend to follow -or at least to compromise. This indicates that there is a bandwagon effect at work [20,50], making it more likely for people to follow others, the more people have done so already. If we increase the difficulty of the task from d = 55 to d = 148 and keep v constant at v = 3, as shown in the third row of Figure 3, the dynamics between the historical thread on the left diverges strongly from the dynamics of the manipulated thread on the right. While the majority of participants in the historical thread are easier to persuade (green colors), people in the manipulated thread become more diverse and slowly start to split bewteen those, who are highly persuadable (blue) and those who are not (red). Increasing difficulty even further to d = 403, and also increasing the amout of social information to v = 9, as shown in the fourth row of Figure 3, makes this split in behaviour even more visible. For d = 1097 and v = 9, as shown in the bottom row, the difference is very clear: In the historical thread, a large majority of participants are medium to strongly influenced, having a persuadability score around 0.5, probably because most of the social information they see is regarded as more or less reasonable. However, as can be seen in the β w -distribution, there is a high uncertainty in the scores, because we only model the weighted geometric mean of the nine preceding estimates, while people presumably are much more diverse in the way they process these estimates.

Discussion
Returning to the initial question of whether social information helps crowd wisdom, or whether it does more harm than good, our answer is both. Social information in online estimation threads does help when people have difficult questions to answer and when the information available to them is pristine and therefore representative. However, if the social information is filtered and not representative of the thread population, it becomes manipulative and may fool a substantial proportion of people, which, of course, interferes with any aspiration to harness the powers of collective intelligence.
Moving from the aggregate to the individual level, we find that people can be assigned a persuadability score by using a Gaussian Mixture Model. Persuadability generally increases with task difficulty and with the amount of social information provided. In the case of strong manipulation, we may see a split between a minority of persuadables and a majority of compromisers and skeptics. We do not know how much these influencing effects are transferable to other domains. However, due to the general nature of dot estimations, we suspect the effects to be substantial in other settings as well, and also for other questions -especially for those that are more emotional, subjective and political in nature. In computational social science and decision-making research, there is an acute need to further investigate the types of 'disturbances' in online threads and crowds, such as filters, rankings, likes, and recommendations, as well as the types of reaction to those disturbances. To the end of designing future collective intelligence systems, one needs to be vigilant about the way social information is gathered and framed. Crowd knowledge and thread wisdom is fragile, and can only be maintained with strong controls upon the way it is cultivated and recovered.

Methods
All experiments were coded in otree 2.1 [51]. The code itself is designed along the same lines as the classical information cascade experiments by Anderson and Holt [19]. A participant makes an estimate, and the next one receives the information about the estimate of the previous participant(s).
We obtained a total of 11.748 estimates from 6.196 unique participants. Any dot-image was only seen once by a participant, i.e. we had a total of 3.157 participants seeing only one image, 1.259 participants seeing two images, 1.047 participants seeing three images, and 733 participants seeing all four images (either in the unmanipulated or manipulated conditions and with v ∈ {0, 1, 3, 9}). After providing informed consent, participants waited in a 'waiting room' until the 'choice room' became available. When entering the choice room participants could see an image d together with v ∈ {0, 1, 3, 9} previous estimates. After making an estimate, participants were thanked and paid a participation fee of $0.10 and bonus of $1 if their estimate was within 10% of the true number. The average time used was less than two minutes, see Supplementary Information for screenshot and detailed design descriptions.

Analysis of thread medians
For the observed medians M dv , d ∈ {55, 148, 403, 1097}, v ∈ {0, 1, 3, 9} the log ratios y dv = log(M dv /d) were modeled using a linear normal model. For a given thread, log d was used as a quantitative variable whereas v was used as a categorical factor. Hence, the model structure for the medians was µ dv = αv + βv log d. Models were fitted separately to historical and manipulated threads to allow for different variance estimatesσ 2 between these groups. Goodness of fit was assessed by quantile-quantile (QQ) plots of the residuals (see Supplementary Information, Figure 6). Although some deviation is present, with the available number of observations we do not discard the models based on these plots. The low number of observations also implies wide confidence bands, hence conclusions from the models can be viewed as conservative.

GMM model for social influence
A Gaussian Mixture Model (GMM) was used to cope with the heavy tails and skewness present in the data. The GMM employs a fixed number of states to fit a weighted sum of Gaussian distributions to each observation, hence the mixture labeling.
Let Xi = {1, . . . , k} denote the state variable and Yi the observation for participant i = 1, . . . , n. Then, conditional on Xi = j, j = 1, . . . , k, Yi follows a normal distribution hence the unconditional distribution of Yi is obtained by integrating (i.e. summing since Xi is discrete) over Xi Including social information as an observed variable in the model, Yi depends on both Xi and Si, if v > 0. Furthermore, if v > 0, then Yi influences Si+1 as the next participant Yi+1 will see the estimate of Yi (assuming the history provided is the previous views). A graphical presentation of model dependencies (with preceding views available) are shown in Figure 4. Thus, given k independent states and using δij = P (Xi = j), then Yi is normally distributed with parameters
Note that µ w i and σ w i refer to the weighted estimates, unique to each observation yi, i = 1, . . . , n. This implies, that the effect of social information becomes a weighted sum of βj estimates Similarly for the α w i parameter, modeling the accompanying intercept.
It should be emphasized that the weighted α w i , β w i are unique for each observation due to the weights δij, contrary to a standard regression model where all observations are assumed to adhere to the same effects. For the GMM this can be interpreted as each participant is modeled as a weighted average of k strategies. Hence, the personal strategy is unique, due to δij, but weighted among k general dimensions.
For each model, the number of states were set to k = 2, 3, 4, 5 and the model with the lowest Bayesian Information Criteria (BIC) was chosen as the preferable one. The BIC was used rather than the more usual Akaikes Information Criteria (AIC), since the BIC penalized the number of parameters more heavily. The aim was to settle with a decent model fit, with fewer parameters to avoid overfitting. In our case AIC: 6k − 2 ln(L) and BIC: 3k ln(n) − 2 ln(L), where L refers to the likelihood value. Note that both criteria are scaled with 3k parameters for k states. The BIC penalizes the number of parameters weighted by the (log) number of observations n, as 3k, corresponding to (δi, µi, σ 2 i ) for each state, whereas AIC only penalizes by a factor 2. Using objective criteria to choose a number of states/clusters is often prone to simply increase k continuously [52] and as such the AIC yielded much larger k values (k 5), practically implying that k = 5 for all models, due to us capping this parameter at this value. This effect was not seen for the BIC measure for 2 ≤ k ≤ 5, further supporting this choice of objective measure.
The goodness of fit for the GMM was assessed by calculating standardized residualŝ and assessing the empirical distribution ofεi, i = 1, . . . , n against a standard N (0, 1) distribution. After removing the 5% most extreme observations, i.e. estimates below the 2.5% and above the 97.5% thresholds, Fig. 7 in the Supplementary Information shows the QQ-plots of the fitted models. Fig. 8 in the SI-text shows the estimatedβ w distributions, with 95% confidence bounds provided by the models. Fig. 9 in the Supplementary Information shows the observed threads, colored by the individualβ w values, and the available social information. Fig. 10 shows the observed estimates against the available social information, again colored by their individualβ w values.

Modeled values
The observations in the model were transformed to achieve a better fit. The estimates were standardized with the relevant number of dots and then log-transformed. Hence, for an observed estimate ei on d dots (1) , . . . , e * (v) |e * j = ej, j < i}, where e * (1) ≥ e * (2) ≥ · · · ≥ e * (v) ≥ e * (l) denote the (decreasingly) ordered estimates e * j among e1, . . . , ej, for v < l ≤ j < i. The superscripts h, m refer to the type of information available, either historical (h) or manipulated (m). Hence, the historical information are the preceding v estimates, whereas the manipulated information are the v largest estimates among all preceding estimates. Following [4] the aggregation of these estimates was a weighted geometric mean. However, contrary to [4], the actual estimates were available to the participants, hence in this context si becomes a proxy for the available social information. Letting zij denote the j'th visible estimate for the i'th participant in either type of threads, then where v j=1 wj = 1. Thus, the log-transform implies that the modeled values (log-normalized estimates) are aggregated as a simple weighted mean. The pair (yi, si), i = 1, . . . , n were then used in the GMM framework. The weights wj, j = 1, . . . , v were determined from density estimates based on the control threads of v = 0. Hence, for d dots, a density was estimated from the v = 0 thread and then used as a proxy for how individuals in a thread with v > 0 would estimate without any social information. Using this, an individual in a thread with v > 0 is then assumed to assess the available previous estimates based on this density. This way extreme estimates are naturally filtered and a participants information is therefore almost exclusively weighted among sensible estimates. Practically this implied that estimates within boundaries considered extreme, were almost equally weighted, however extreme inputs, either by deliberate efforts to mislead the thread or otherwise just to be considered nuisances were weighted almost at 0, yielding a more reasonable aggregated social information input si. For manipulated threads, or threads with v = 1, the social information available show these extreme guesses. This is simply what a participant have seen and will thus be discarded by the parameter estimation, fitting a low β w value for this participant (assuming that the current participant submits a realistic estimate and not taking the extreme info into account). In the case of manipulated threads, due to the removal of most extreme estimates to filter out deliberate extremities, the social information available can be higher than the cutoff limit of the 97.5% quantile. However, as in the case of v = 1, this will simply be the available information that a participant can follow or disregard. The model will thus fit a suitable β w value based on the current participants use of the info available. The R package depmixS4 [53] was used to fit the model, using an EM algorithm to maximize the log-likelihood function of the model.

Acknowledgments
The experiments were implemented by Robin Engelhardt and Mikkel Birkegaard Andersen. Server infrastructure and devops was handled by Mikkel Birkegaard Andersen. The authors wish to thank Philipp Chapkovski for help with the cascade design, Rasmus Rendsvig for modeling discussions and Ulrik Nash and Peter Norman Sørensen for their comments on an earlier draft of this paper. This research was approved by the Institutional Review Board at the University of Copenhagen and included informed consent by all participants in the study. The authors gratefully acknowledge the support provided by The Carlsberg Foundation under grant number CF 15-0212.

Author contributions
Robin Engelhardt designed the experiment, analysed the data, and contributed as a lead author; Vincent F. Hendricks contributed as the project manager and co-author; Jacob Staerk-Østergaard designed the statistical methods, analysed the data and contributed as a lead-author.

Experimental setup
Controlled lab experiments of thread dynamics are rare in the research literature due to the difficulties in keeping a large number of participants in a queue. We designed our experiment along the same lines as the classical information cascade laboratory experiments by Anderson and Holt [19]. While this design is not feasible in normal laboratory conditions, it is well suited for online labor markets and crowdsourcing platforms such as Amazon Mechanical Turk (Mturk), an online labor markets and crowdsourcing platform that has become a highly valued tool for social scientists who wish to conduct experimental research on the real time dynamics of large groups. Mturk has repeatedly been shown to meet or exceed the standards set by data collection methods using other means [1,2]. The platform provides an integrated participant compensation system, has a large participant pool, and has been shown to be reliable, replicable, and significantly more diverse than typical American college samples [3][4][5][6][7]. Our experimental setup on Mturk is simple. After accepting our 'HIT' (Mturk acronym for a 'human intelligence task') and providing informed consent, participants are asked to wait in a 'waiting room' until the 'choice room' becomes available. After entering the choice room, participants are asked to take a look at the image and make an estimate (see screenshot in Figure 5). Depending on the view condition, participants can see v ∈ {0, 1, 3, 9} preceding estimates. We chose to present the 'oldest' previous estimate on the top of the list and the last estimate made on the bottom. When dealing with news, or financial data, users typically want to see the most recent activity first (think tweets, online banking transactions, news updates). With conversations it is different because there is a context to consider of whatever message came before and after the one you are looking at (think blogs or Facebook comments). We have chosen to use the conversation thread design (oldest on top) because there is no particular news criteria when guessing the weight of an ox or estimating the number of dots on a screen. Participants have one minute to think about the image and make their estimate. This might sound as a severe time constraint, but exploratory trials had shown that participants in general use less than a minute when performing this task. After submission, participants are thanked for their participation and the experiment ends. Waiting times are compensated with $0.20 per minute (maximally 5 minutes), participation fee is $0.10, and a bonus of $1 is paid if an estimate is within 10% of the true number.

Mturk settings and data quality
When working with Mturk it is important to consider the right settings in order to obtain the best data quality possible [8]. Fair wage, attrition rates, removal of duplicate participants and informative feedback are some of the most important issues to address.
Average wage for participants in our experiments was ∼ $12 per hour, which is considered good according to Mturk guidelines and certainly above the estimated average of $6 per hour when excluding un-submitted and rejected work [9].
Quitting a study before completing it is prevalent on Mturk, and varies systemically across experimental conditions [10]. On average 20 participants accepted our HITs within the first minute; after 10 minutes the average acceptance rate had dropped to 2-3 participants per minute and after an hour to less than one participant per minute. Our scripts were coded in such a way that participants were automatically assigned to a 'waiting room' in which they were asked to wait for maximally five minutes before entering the choice room. This meant that a lot of participants waited in vain. Due to such a high attrition rate in the first couple of experiments we changed the script slightly later on: Now the waiting room could contain a maximum of five participants, and when the waiting room was full, participants were told to come back and finish the HIT at a later point in time. This reduced the attrition rate to 6.5% on average.
All participants automatically received an image-specific qualification when accepting a HIT. This qualification ensured that participants could not accept any other HITs that use the same image. Further data inspection showed that 32 participants somehow managed to accept two HITs with the same image anyway. The reason may be that the time interval between accepting two HITs with the same image was too short for the qualification to register in the Mturk interface. All 32 duplicate participants were removed before data analysis. In addition, we set the qualification that participants should have completed at least 100 HITs and have an accepted HIT rate of 98% or above. This ensured that we would get only experienced and qualified participants.
Mturk participant attention was expected to be equal to or better than undergraduate participant's attention [11], while various forms of dishonesty (practical joking or telling others about the true value offline or on an Mturk participants web page) was expected to be rare. Our screening of data files before data analyses revealed that a small fraction of participants submitted ridiculously high estimates across images seen, thereby skewing thread averages substantially. These estimates were included in the calculations of the geometric mean of the social information. However, they were given a low weight determined from density estimates based on the control threads with v = 0 (see section Modeled values above).
During our experiments, participants had easy access to our email for questions and possible bug reports. Apart from some minor difficulties when typing from a mobile device (less than 1%) participants had few comments or complaints.

Dataset
Anonymized data set of all dots-experiments can be found the the file dots.xlsx. Parameters: task = type of experiment; d = number of dots in image; v = number of visible preceding estimates; session = thread name; hashed turker = anonymized participant id; decision order = order in which participants enter the queue; hist = list of guesses seen by participant; guess = estimate by participant