, in case of the *DAiSEE* dataset, DS did not converge even after 100 iterations (as compared to $\le 10$ for FDS). To address this issue, we set the convergence criterion as the point when the difference in class marginals is less than $10^{-4}$. We do not include the changes in participant error rates in the final convergence criterion because we observed that its fluctuations could lead to stopping prematurely. Similarly, the criterion for switching from DS to FDS in the Hybrid algorithm is the point when the change in class marginals is less than 0.005 (which happened approximately between 45-75% of total iterations across the datasets).

Online Vote Aggregation
=======================

Online aggregation of crowdsourced responses is an important setting in today’s applications, where data points may be streaming in large data applications. We consider a setting in which we have access to an initial set of questions and have obtained the proposed answer key using FDS. We also have $P(Y = c)$ and $P(c_a| Y = a) \,\forall\, c, a$ at this time. When we receive a new question and the answers from multiple participants for this new question, we first estimate the answer for this question directly using majority voting. We then update the parameters using the M-step in Algorithm \[fdsalgorithm\]. After the M-step, we run the E-step only for this question to re-obtain the aggregated choice. To update the new knowledge which we have regarding the new participants, we run the M-step for one last time. We conducted experiments on the *SP* dataset[^2], and observed almost the same accuracy for online FDS as offline FDS (Table 4) for different number of annotators. Table 3 shows the results for the max number of annotators (= 5).

\[onlinetable\]

  ---------------------------- -------- -------- --------
                                  DS      FDS     Hybrid
            Accuracy            90.94%   90.60%   90.64%
   Time taken to converge (s)    4.40     3.76     4.09
   \# Iterations to converge      26       4        5
  ---------------------------- -------- -------- --------

\[onvsofftable\]

  ------------ -------- -------- -------- --------
    Accuracy      2        3        4        5
      FDS       85.59%   88.41%   90.02%   90.74
, in case of the * DAiSEE * dataset, DS did not converge even after 100 iterations (as compared to $ \le 10 $ for FDS). To address this consequence, we rig the convergence criterion as the point when the dispute in class marginals is less than $ 10^{-4}$. We do not include the change in participant error rates in the concluding overlap criterion because we observed that its fluctuation could lead to stopping prematurely. Similarly, the standard for switching from DS to FDS in the Hybrid algorithm is the point when the variety in class marginals is less than 0.005 (which happen approximately between 45 - 75% of total iterations across the datasets). 

 Online Vote Aggregation 
 = = = = = = = = = = = = = = = = = = = = = = = 

 Online aggregation of crowdsourced reception is an important setting in today ’s applications, where data point may be streaming in large data applications. We consider a place setting in which we have access to an initial set of questions and have obtained the proposed answer key using FDS. We also have $ P(Y = c)$ and $ P(c_a| Y = a) \,\forall\, c, a$ at this time. When we receive a new question and the answers from multiple participant for this new question, we foremost calculate the answer for this question directly using majority vote. We then update the parameters use the M - step in Algorithm \[fdsalgorithm\ ]. After the M - dance step, we run the E - step only for this question to re - obtain the aggregated choice. To update the new knowledge which we have involve the new participant, we run the M - step for one last time. We impart experiments on the * SP * dataset[^2 ], and observed almost the same accuracy for online FDS as offline FDS (Table 4) for different number of annotator. Table 3 shows the results for the max number of annotators (= 5). 

 \[onlinetable\ ] 

   ---------------------------- -------- -------- -------- 
                                   DS       FDS      Hybrid 
             Accuracy             90.94%    90.60%    90.64% 
    Time learn to converge (s)     4.40      3.76      4.09 
    \ # Iterations to converge       26        4         5 
   ---------------------------- -------- -------- -------- 

 \[onvsofftable\ ] 

   ------------ -------- -------- -------- -------- 
     Accuracy       2         3         4         5 
       FDS        85.59%    88.41%    90.02%    90.74
, in case of the *DAiSEE* dataret, DS did not einvergx even zfter 100 igerations (as compared to $\le 10$ fir FDW). To address this issud, we set nhe conveegenre criterion as vge poinb wheh the vifference in ckass margitals is less tvav $10^{-4}$. We do not include the changes in pwrticipsnh error rates yn tnq fihal convergence criterion because se obsegved that its fluvtuations could lead to stlppijg prematurely. Simllarly, the xritqeion for swigching from DS to FDS jn the Hybrid algorithm is the ooint when the xhqngf in class merginajs is less tmsn 0.005 (whhch haplened approximsteny vetween 45-75% of total itecations across the dwtasets).

Onniue Vote Aggregation
=======================

Onoibe agcregdtiov of crkwvsohrced ges'onses is ah important setting in today’s sp[oications, whede datw [oints may be streaming in large data akplicztions. We consider a serting in which we havg access tj an initial set of questions and have obtained tve prkoostd answde ney using FDS. We also have $P(Y = c)$ and $P(c_a| Y = a) \,\rotakl\, c, a$ at this time. When ee rrseive a new qoestion ans the answers from multipje paeticipantf fot this new question, we firsr estimate tke qnswer for this quzstion direccly usong msjority voting. We then bpdate the paramehers usinf the M-step in Aleorpthm \[fdsalgorithm\]. After the M-seep, we rui the E-step unly for tris questiln to re-obtain the aggrehated cvoice. To uodate the new knowledge which wx have regardonc tve new pcrticiiants, we run thq M-step for ong last tiie. We conducted experikents on thq *SP* dataset[^2], djd observed elmost thq sane axcuracy wor online FDS as offliut FDS (Tablw 4) for different nmmber kf annotators. Tcylt 3 shows the resiltr fjr tie mav number of atnotxtofx (= 5).

\[onuinetable\]

  ---------------------------- -------- -------- --------
                                  DS      FDX     Hybrud
            Accuracu            90.94%   90.60%   90.64%
   Tmme tayen tp cjnverge (s)    4.40     3.76     4.09
   \# Itedations tl cjnverge      26       4        5
  ---------------------------- -------- -------- --------

\[onvsofftable\]

  ------------ -------- -------- -------- --------
    Accuracy      2        3        4        5
      FDS       85.59%   88.41%   90.02%   90.74
, in case of the *DAiSEE* dataset, not even after iterations (as compared To this issue, we the convergence criterion the point when the difference in marginals is less than $10^{-4}$. We do not include the changes in participant rates in the final convergence criterion because we observed that its fluctuations could to prematurely. the for switching from DS to FDS in the Hybrid algorithm is the point when the change class marginals is less than 0.005 (which happened between 45-75% of total across the datasets). Online Vote ======================= aggregation of responses an setting in today’s where data points may be streaming in large data applications. We consider a setting in which we access to set of and obtained proposed answer key We also have $P(Y = c)$ = a) \,\forall\, c, a$ at this time. we receive new question and the answers from participants for this new question, we first estimate answer for this question directly using majority voting. We then update the parameters using the Algorithm \[fdsalgorithm\]. After the we run the only this to the aggregated To update the new knowledge which we have regarding the new we run the M-step for one last time. We conducted the dataset[^2], and observed the same accuracy for FDS offline FDS (Table 4) number annotators. the for max number of annotators 5). \[onlinetable\] ---------------------------- -------- -------- DS FDS Hybrid Accuracy to converge (s) 4.40 3.76 4.09 \# Iterations converge 26 4 5 ---------------------------- -------- -------- \[onvsofftable\] ------------ -------- -------- -------- -------- Accuracy 2 3 4 5 FDS 88.41% 90.02%
, in case of the *DAiSEE* dataset, Ds did not conVerge EveN afTeR 100 iteRatiOns (as compared tO $\Le 10$ foR FDS). To address this issue, We set ThE ConvERgEnce cRiterioN As THE poInT wHen ThE DiFfereNce In class Marginals iS leSs Than $10^{-4}$. We do not iNClUde the chanGes In participanT erRor ratEs In tHE finaL coNvergEnce crITerion Because we ObSErved tHAt its flUCTuAtioNs could lead to stopPInG Prematurely. SimIlarly, ThE CrITEriOn fOr switchinG fRom DS TO FDS in tHE HYBRId aLGorithm is the pOint when the CHanGe in clAsS maRGinals Is lesS tHAn 0.005 (wHich happeneD appRoximatelY betweEN 45-75% of totaL IteratiOns acrOss The DataSEtS).

ONliNe vOte aGgRegATioN
=======================

Online aGgReGatioN of cROWDSourCed RespOnses Is an important SetTing IN toDay’s aPplicAtioNs, Where Data poInts mAy Be streaming in laRge dAta applicAtiOnS. We CoNsideR A settiNg iN whIch we haVe accesS To aN iNITIaL set of questions and HaVE ObTained thE propoSEd AnSWer key usInG FDs. We aLSO have $p(Y = c)$ aND $P(C_a| Y = a) \,\foraLl\, c, a$ at THiS tIme. When We ReceivE a New QueStion ANd thE answeRs from muLtiplE Participants foR This new questiON, wE FIrST estImaTe the answer For tHIs quEstiON dIreCTly usIng maJoRItY Voting. We then update tHe ParameTers uSing the M-step iN Algorithm \[FDSAlgorithM\]. AftER tHE M-step, we run the e-step Only for thiS Question To re-oBtain the AggregateD CHoice. To uPdaTe tHe nEw kNOWlEdge which we haVE RegaRdIng the nEw pArticipAntS, we Run The m-sTep for one Last time. we CoNdUcTed ExperIMents on tHe *sP* dAtAseT[^2], and oBServed AlmosT the SaMe ACcuRacy for ONlINE FDS As OfFlinE FDs (TAble 4) fOr diFFerEnt numbEr of annotAtoRS. TabLe 3 ShOws the rEsults for the mAx Number of anNoTatOrs (= 5).

\[onlINEtable\]

  ---------------------------- -------- -------- --------
                                  DS      fDS     Hybrid
            Accuracy            90.94%   90.60%   90.64%
   Time taKEn to conVerGe (s)    4.40     3.76     4.09
   \# ItEratIons to conVerGe      26       4        5
  ---------------------------- -------- -------- --------

\[onvsOffTAble\]

  ------------ -------- -------- -------- --------
    AcCuracy      2        3        4        5
      fDS       85.59%   88.41%   90.02%   90.74
, in case of the *DAiSEE*dataset, D S did  no t c on verg e ev en after 100 i t erat ions (as compared to $ \le 1 0$ forF DS ). To  addres s  t h i s i ss ue , w es et  thecon vergenc e criterio n a sthe point wh e nthe differ enc e in class m arg inalsis  le s s tha n $ 10^{- 4}$. W e  do no t include  t h e chan g es in p a r ti cipa nt error rates in th e  final converg ence c ri t er i o n b eca use we obs er ved t h at itsf lu c t u ati o ns could lead  to stoppin g  pr ematur el y.S imilar ly, t he cri terion forswit ching fro m DS t o  FDS in the Hyb rid al gor ith m is th epoi nt whe n  t hec han ge in cl as smargi nals i s less  th an 0 .005(which happen edappr o xim ately  betw een45 -75%of tot al it er ations across t he d atasets). 

O nl ine  V ote A g gregat ion 
== ======= ======= = === == = On line aggregation o fc r ow dsourced  respo n se si s an imp or tan t se t t ing i n to d ay ’s appli cation s ,wh ere dat apoints  m aybestrea m ingin lar ge dataappli c ations. We con s ider a settin g  i n wh i ch w e h ave accessto a n  ini tial se t o f  ques tions  a n dh ave obtained the pr op osed a nswer  key using FD S. We also h a ve $P(Y= c) $  a n d $P(c_a| Y =a) \, \forall\,c , a$ atthistime. Wh en we rec e i ve a new  qu est ion  an d th e answers fro m mult ip le part ici pants f orthi s n ewqu estion, w e firstes ti ma te  th e ans w er for t hi s q ue sti on di r ectlyusing  maj or it y  vo ting. W e  t h e n up da te  the  pa ra meter s us i ngthe M-s tep in Al gor i thm\[ fd salgori thm\]. Afterth e M-step,we  ru n theE - step onl y for this question tor e-obtai n t he ag greg ated choi ce.  To up dat e  the n ew kno wledg ewhi c h  we h a v ereg ar ding the n e w  pa rtici pa nts,  we run  the M-step for on e  la st time. We c ond ucte d ex per i me n tson the * SP* dataset[^2] , and obse rv e dalmost the sam eaccurac y for o nline FDS asoffline F DS (Table  4 ) fo r dif ferent num ber of a nnotators .  Tabl e  3  show s t he res ul tsfor t he max num ber o f anno ta tors ( = 5). 
\[online table\]

  ------------ ------ ----- --- -- ------ --- --- ---- ---- ---- 
                         DS     FDS    Hybrid
     Accuracy       90 .94 %    90. 60%  90.64%
   Timet aken to conver ge ( s )    4. 4 0   3.76     4.09  \# I teration sto converge       26      4        5
 ------- - - -- - ------ ---- --- --- ----- ---  - - -------- -- - ---

\ [onv so fftabl e\]

---- - - ------ ------------- - - - --- - --- - --- -- ---
  Accu racy     2        3      4      5
    FDS     8 5.59%   88 . 41%   90. 02%  90.74
, in_case of_the *DAiSEE* dataset, DS_did not_converge_even after_100_iterations (as compared_to $\le 10$_for FDS). To address_this issue, we_set_the convergence criterion as the point when the difference in class marginals is less_than_$10^{-4}$. We_do_not_include the changes in participant_error rates in the final_convergence criterion_because we observed that its fluctuations could lead_to_stopping prematurely. Similarly,_the criterion for switching from DS to FDS in_the Hybrid algorithm is the point_when the change_in_class_marginals is less than_0.005 (which happened approximately between 45-75%_of total iterations across the datasets).

Online_Vote Aggregation
=======================

Online aggregation of crowdsourced responses is_an important setting in today’s applications,_where data points may be_streaming in_large data applications. We consider_a setting in_which we_have access to_an initial set of questions and_have obtained the_proposed answer key using FDS. We_also_have $P(Y =_c)$_and_$P(c_a| Y_= a) \,\forall\,_c,_a$ at_this_time. When we receive a new_question_and the answers from multiple participants for_this new question, we_first_estimate the answer for_this question directly using majority_voting. We then update the parameters_using the_M-step in_Algorithm \[fdsalgorithm\]. After the M-step, we run the E-step only for_this question to re-obtain the aggregated_choice. To update the_new knowledge_which_we have regarding_the_new participants,_we run the M-step for one last_time. We_conducted experiments on the *SP* dataset[^2],_and observed almost the_same_accuracy for online FDS as offline_FDS (Table 4) for different number_of annotators. Table 3 shows_the_results_for the max number of_annotators (= 5).

\[onlinetable\]

  ---------------------------- --------_-------- --------
 _      __     __     ___    _    DS _    FDS    _Hybrid
  _    ___  Accuracy       _ _  90.94%_  90.60%   90.64%
   Time taken_to converge (s)    4.40_    3.76     4.09
 _ \# Iterations to converge   _  26    _  4 __     5
 _----------------------------_--------_--------_--------

\[onvsofftable\]

  ------------_-------- -------- --------_--------
  _ Accuracy_  _ _2  __    3        4     _  5
   _  FDS  __  _85.59%   88.41%_  90.02%   90.74