 and effectively address all flags being raised. This indicates a need for fast vote aggregation schemes that can provide results for a stream of data in real time.

The use of crowdsourced annotations requires a check on the reliability of the workers and the accuracy of the annotations. While the platforms provide basic quality checks, it is still possible for workers to provide incorrect labels due to misunderstanding, ambiguity in the data, carelessness, lack of domain knowledge, or malicious intent. This can be countered by obtaining labels for the same question from a large number of annotators, and then aggregating their responses using an appropriate scheme. A simple approach is to use majority voting, where the answer which the majority of annotators choose is taken to be the true label, and is often effective. However, many other methods have been proposed that perform significantly better than majority voting, and these methods are summarized further in Section \[related\].

Despite the various recent methods proposed, one of the most popular, robust and oft-used method to date for aggregating annotations is the Dawid-Skene algorithm, proposed by [@dawid1979maximum], based on the Expectation Maximization (EM) algorithm. This method uses the M-step to compute error rates, which are the probabilities of a worker providing an incorrect class label to a question with a given true label, and the class marginals, which are the probabilities of a randomly selected question to have a particular true label. These are then used to update the proposed set of true labels in the E-step, and the process continues till the algorithm converges on a proposed set of true labels (further described in Section \[dawidskenealgo\]).

In this work, we propose a new simple, yet effective, EM-based algorithm for aggregation of crowdsourced responses. Although formulated differently, the proposed algorithm can be interpreted as a ‘hard’ version of Dawid-Skene (DS) [@dawid1979maximum], similar to Classification EM [@celeux1992classification] being a hard version of the original EM. The proposed method converges upto 7.84x faster than DS, while maintaining similar accuracy. We also propose a hybrid approach, a combination of our algorithm with the Dawid-Skene algorithm, that combines the high rate of convergence of our algorithm and the better likelihood estimation of the Dawid-Skene algorithm as part of this work.

Related Work {#related}
============

The Expectation-Maximization algorithm for maximizing likelihood was first formal
and effectively address all flags being raised. This indicate a motivation for fast vote aggregation dodge that can provide results for a current of datum in real time. 

 The use of crowdsourced annotations requires a check mark on the reliability of the workers and the accuracy of the annotation. While the platform provide basic timbre checks, it is however potential for workers to provide incorrect label due to misunderstanding, ambiguity in the data, negligence, lack of domain cognition, or malicious intent. This can be countered by obtaining labels for the same question from a big number of annotators, and then aggregating their responses using an appropriate scheme. A simple approach is to use majority voting, where the answer which the majority of annotators choose is taken to be the true label, and is often effective. However, many other methods have been propose that perform significantly well than majority vote, and these method acting are summarized further in Section \[related\ ]. 

 Despite the diverse recent methods proposed, one of the most democratic, robust and oft - used method to date for aggregating annotations is the Dawid - Skene algorithm, proposed by [ @dawid1979maximum ], based on the Expectation Maximization (EM) algorithm. This method uses the M - step to calculate error rate, which are the probability of a worker providing an incorrect class label to a doubt with a given true label, and the class marginals, which are the probability of a randomly selected question to have a particular true label. These are then use to update the proposed bent of true labels in the vitamin e - step, and the procedure continues till the algorithm converge on a proposed hardening of true labels (far described in Section \[dawidskenealgo\ ]). 

 In this oeuvre, we propose a new simple, yet effective, EM - based algorithm for aggregation of crowdsourced responses. Although formulated differently, the proposed algorithm can be interpreted as a ‘ difficult ’ version of Dawid - Skene (DS) [ @dawid1979maximum ], similar to Classification EM [ @celeux1992classification ] being a intemperate version of the original EM. The proposed method acting converges upto 7.84x faster than DS, while maintaining similar accuracy. We also propose a hybrid approach, a combination of our algorithm with the Dawid - Skene algorithm, that combines the high rate of convergence of our algorithm and the good likelihood estimate of the Dawid - Skene algorithm as contribution of this work. 

 Related Work { # related } 
 = = = = = = = = = = = = 

 The Expectation - Maximization algorithm for maximizing likelihood was first formal
 anf effectively address alu flags being rcused. Tiis indjcates a need for fast vote aggregatmon wchemtf that can provide results vor a steeam if data in real time.

The hde oy rrowdsourced ankotations raquires a checn un the reliability of the workers and the acvugacy of the angotauiogs. Wgplt the platforms provide basic quamity chtcks, it is still ppssible for workers to progide incorrect labels fue to misubderfranding, ambieuity in the data, caremessness, lack of domain knowledee, or malicious ibtejj. This can bx counnered by obtalming ldbels fpr the same qucstioi frim a large number of ennotators, and then wggregatitg their responses ysung at ap[ropfuatd sdhxme. A simole approach js to use mqjority voting, whert trv answer whicg the iatority of annotators choose is taken to be the true label, and is iften effective. Howevgr, many otrer methods have been proposed that perform signixicanvlh bttbcr tfqn majority voting, and these methods are summaryaec nurther in Sectijn \[related\].

Drsoiyg the various tecent metgods proposed, one lf the iost popular, rjbusy and oft-used method to datw for aggregctibg annotations is che Dawid-Skeue algprithk, proposed by [@dawid1979maxioum], gased on thf Expectafkon Maximization (EM) angorithm. This method uses ehe M-step to eompute drrot rates, which are the isobabilities of a aorket provhding an ijcorrect class label to a questmpn with a gifet tgue label, and bhe class margigals, which are the prjbabiuities of z randokly selecteq question to jave a partirular truq lavel. Rhese afd then used to update tkt proposed set of true labelx iv the E-step, and cke process continies tijl tie aldmrithm convesges on s prooosed stc on tfue kabels (further descrhbed in Section \[dawidslekealgo\]).

In jhis work, we propose a new simple, yet efvectite, EM-besed akgotithm for aggregation of crowdskurced redpokses. Although fornulated diffzrently, the proposed algorithm can be inverpreted as a ‘hard’ vetsion of Dawid-Skene (DX) [@dawid1979maximnm], simylar to Cnassification EM [@celwux1992classification] being a hard version of tha orihinal EM. The proposed method converges upto 7.84x faster than DS, while maintauning similar accudacy. We ansj propofe a iybrid approach, a combination of our algorithm wmth the Dafib-Skene algorithm, that combinex ghe high rate of convergence of our amgorithm and the better likelihood estikation of the Dawid-Skene algoritym as part of tiis work.

Related Work {#relatzd}
============

The Wxpectarion-Laximization xlgprlthm for mwximmziug likelihood was first formal
and effectively address all flags being raised. a for fast aggregation schemes that stream data in real The use of annotations requires a check on the of the workers and the accuracy of the annotations. While the platforms provide quality checks, it is still possible for workers to provide incorrect labels due misunderstanding, in data, lack of domain knowledge, or malicious intent. This can be countered by obtaining labels for the question from a large number of annotators, and aggregating their responses using appropriate scheme. A simple approach to majority voting, the which majority of annotators is taken to be the true label, and is often effective. However, many other methods have been that perform than majority and methods summarized further in Despite the various recent methods proposed, most popular, robust and oft-used method to date aggregating annotations the Dawid-Skene algorithm, proposed by [@dawid1979maximum], on the Expectation Maximization (EM) algorithm. This method the M-step to compute error rates, which are the probabilities of a worker providing an label to a question a given true and class which the probabilities a randomly selected question to have a particular true label. These then used to update the proposed set of true labels E-step, the process continues the algorithm converges on proposed of true labels (further Section In propose new yet effective, EM-based algorithm aggregation of crowdsourced responses. Although differently, the proposed algorithm ‘hard’ version of Dawid-Skene (DS) [@dawid1979maximum], similar to EM [@celeux1992classification] being a hard version of original EM. The proposed method converges upto 7.84x faster than DS, while similar accuracy. propose a hybrid approach, a combination of our with the Dawid-Skene algorithm, combines the high rate of convergence of our algorithm the likelihood estimation the Dawid-Skene algorithm part of this Related Work {#related} Expectation-Maximization algorithm maximizing was
 and effectively address all fLags being rAised. thiS inDiCateS a neEd for fast vote aGGregAtion schemes that can proVide rEsULts fOR a StreaM of data IN rEAL tiMe.

thE usE oF CrOwdsoUrcEd annotAtions requIreS a Check on the reLIaBility of thE woRkers and the aCcuRacy of ThE anNOtatiOns. while The plaTForms pRovide basIc QUality CHecks, it IS StIll pOssible for workers TO pROvide incorrect Labels DuE To MISunDerStanding, amBiGuity IN the datA, CaRELEssNEss, lack of domaIn knowledge, OR maLiciouS iNteNT. This cAn be cOuNTerEd by obtainiNg laBels for thE same qUEstion fROm a largE numbeR of AnnOtatORs, AnD thEn AGgrEGaTinG TheIr responSeS uSing aN appROPRIate SchEme. A SimplE approach is to Use MajoRIty VotinG, wherE the AnSwer wHich thE majoRiTy of annotators cHoosE is taken tO be ThE trUe Label, ANd is ofTen EffEctive. HOwever, mANy oThER MEtHods have been proposEd THAt Perform sIgnifiCAnTlY Better thAn MajOritY VOting, And tHEsE methods Are sumMArIzEd furthEr In SectIoN \[reLatEd\].

DesPIte tHe variOus recenT methODs proposed, one oF The most populaR, RoBUSt ANd ofT-usEd method to dAte fOR aggRegaTInG anNOtatiOns is ThE daWId-Skene algorithm, proPoSed by [@dAwid1979mAximum], based on The ExpectaTION MaximizAtioN (eM) ALgorithm. This meThod uSes the M-steP To computE erroR rates, whIch are the PRObabilitIes Of a WorKer PROvIding an incorrECT claSs Label to A quEstion wIth A giVen TruE lAbel, and thE class maRgInAlS, wHicH are tHE probabiLiTieS oF a rAndomLY selecTed quEstiOn To HAve A particULaR TRue lAbEl. thesE arE tHen usEd to UPdaTe the prOposed set Of tRUe laBeLs In the E-sTep, and the procEsS continues TiLl tHe algoRIThm conveRges on a proposed set of truE Labels (fUrtHer deScriBed in SectIon \[DawidsKenEAlgo\]).

In This woRk, we pRoPosE A New siMPLe, Yet EfFective, EM-bASEd aLgoriThM for AggregaTion of crowdsourced REspOnses. Although ForMulaTED dIffEReNTly, ThE ProPOSed algorithm can Be interpreTeD As A ‘hard’ versiON of daWid-SkenE (DS) [@dawiD1979maxiMUm], similAr to ClassIfication eM [@CeleUX1992ClaSsificatioN] being a hArd versioN Of the ORiGinal eM. THe propOsEd mEthod ConverGEs uPto 7.84x fAster tHaN DS, whiLe maiNtAining siMilar accuracy. We also propOse a hyBrid aPprOach, a combInaTIon Of our algoRithM with the DaWid-skeNe algOriTHm, thaT comBInEs tHE high Rate OF convergeNCe Of oUR AlGorithm and tHE BEttEr likEliHOod estImatIon of the Dawid-SkenE Algorithm as parT of tHIS woRk.

RELateD WOrk {#related}
============

The EXpeCtATIon-MaximIzAtion algoriThm for maXiMIzing LikeliHood waS first fORMaL
 and effectively address a ll flags b eingrai sed .This  ind icates a needf or f ast vote aggregation s cheme st hatc an  prov ide res u lt s for  a  s tre am of  data  in  real t ime.

Theuse  o f crowdsourc e dannotation s r equires a ch eck  on th erel i abili tyof th e work e rs and  the accu ra c y of t h e annot a t io ns.While the platfor m sp rovide basic q uality  c h ec k s , i t i s still po ss iblef or work e rs t o  pr o vide incorrec t labels du e  to  misun de rst a nding,  ambi gu i tyin the data , ca relessnes s, lac k  of dom a in know ledge,  or  ma lici o us  i nte nt .  Th i scan becountere dby  obta inin g l a bels  fo r th e sam e question fr oma la r genumbe r ofanno ta tors,  and t hen a gg regating theirresp onses usi ngan  ap pr opria t e sche me.  Asimpleapproac h  is  t o u se  majority voting,wh e r ethe answ er whi c hth e  majorit yofanno t a torschoo s eis taken  to be th etrue la be l, and  i s o fte n eff e ctiv e. How ever, ma ny ot h er methods hav e  been propose d  t h a tp erfo rmsignificant ly b e tter  tha n  m ajo r ity v oting ,a nd these methods are s um marize d fur ther in Secti on \[relat e d \ ].

Desp itet he various recent  meth ods propos e d, one o f the  most po pular, ro b u st and o ft- use d m eth o d  t o date for ag g r egat in g annot ati ons isthe  Da wid -Sk en e algorit hm, prop os ed  b y[@d awid1 9 79maximu m] , b as edon th e  Expec tatio n Ma xi mi z ati on (EM) al g o rith m.  T hismet ho d use s th e  M- step to  computeerr o r ra te s,  whichare the proba bi lities ofawor ker pr o v iding an  incorrect class labelt o a que sti on wi th a  given tr uelabel,  an d  the c lass m argin al s,w h ich a r e  t hepr obabilitie s ofa ran do mlyselecte d question to have a p articular tru e l abel . Th ese ar e  th en use d to update the p roposed se to ftrue label s  in  t he E-st ep, and  thep rocesscontinues  till the  a lgor i t hmconvergeson a pro posed set of tr u elabel s ( furthe rdes cribe d in S e cti on \[ dawids ke nealgo \]).
I n this w ork, we propose a new s imple,  yeteff ective, E M-b a sed  algorith m fo r aggregat ion  of  crow dso u rcedresp o ns es. Altho ughf ormulated di ffe r e nt ly, the pro p o s edalgor ith m  can b e in terpreted as a ‘h a rd’ version of  Daw i d -Sk ene (DS)  [ @dawid1979maxi mum ], s imilar t oClassificat ion EM [ @c e leux1 992cla ssific ation]b e in g  a har d ve rsi on of the  or ig i nal EM.  T he propos ed m et hod co nverge s  upt o 7.84x faster tha n DS, w hilem ain taini ng  simila r  acc uracy. Wealso propos e a hy brid  appr oach, a  c ombina tio nof our alg o rithm wit h the  Dawid- Sk enealg orithm , th a t  comb ines  t hehigh rate o fc on ve r gen ce o f our  a lgor ithm andt he bette r l i kelihoo dest i m ationo ft h e Dawid-Sk ene  algo r i thm as par t  oft hi s  work .

Rel ated W ork {#r e lat ed }
===== === = = ==

The E xpectatio n -Ma xi miza tion alg or ithm  f orma ximizing likeli hood was  first f o r mal
 and_effectively address_all flags being raised._This indicates_a_need for_fast_vote aggregation schemes_that can provide_results for a stream_of data in_real_time.

The use of crowdsourced annotations requires a check on the reliability of the workers_and_the accuracy_of_the_annotations. While the platforms provide_basic quality checks, it is_still possible_for workers to provide incorrect labels due to_misunderstanding,_ambiguity in the_data, carelessness, lack of domain knowledge, or malicious intent._This can be countered by obtaining_labels for the_same_question_from a large number_of annotators, and then aggregating their_responses using an appropriate scheme. A_simple approach is to use majority voting,_where the answer which the majority_of annotators choose is taken_to be_the true label, and is_often effective. However,_many other_methods have been_proposed that perform significantly better than_majority voting, and_these methods are summarized further in_Section_\[related\].

Despite the various_recent_methods_proposed, one_of the most_popular,_robust and_oft-used_method to date for aggregating annotations_is_the Dawid-Skene algorithm, proposed by [@dawid1979maximum], based_on the Expectation Maximization_(EM)_algorithm. This method uses_the M-step to compute error_rates, which are the probabilities of_a worker_providing an_incorrect class label to a question with a given true label,_and the class marginals, which are_the probabilities of a_randomly selected_question_to have a_particular_true label._These are then used to update the_proposed set_of true labels in the E-step,_and the process continues_till_the algorithm converges on a proposed_set of true labels (further described_in Section \[dawidskenealgo\]).

In this work,_we_propose_a new simple, yet effective,_EM-based algorithm for aggregation of crowdsourced_responses. Although formulated_differently, the proposed algorithm can be interpreted_as_a ‘hard’ version of Dawid-Skene (DS)_[@dawid1979maximum],_similar to Classification EM [@celeux1992classification] being_a_hard_version of the original EM._The proposed method converges upto 7.84x_faster than DS, while maintaining similar accuracy. We also_propose a hybrid_approach, a combination of our_algorithm_with_the Dawid-Skene algorithm, that combines the high rate of convergence_of our_algorithm and the_better likelihood estimation of the Dawid-Skene algorithm as part of_this work.

Related Work {#related}
============

The Expectation-Maximization algorithm for_maximizing likelihood was first formal