 no regularization. However if this attempt fails, then EM is run again with regularization. As in @scikit-learn, regularization involves adding a regularization factor to the diagonal of the covariance matrices to ensure that they are positive. This method does not modify the eigenvectors of the covariance matrix, making it rotationally invariant [@ledoit].

`AutoGMM` has a `scikit-learn` compliant API and is freely available at <https://github.com/tathey1/graspy>.

Reference Clustering Algorithms
-------------------------------

We compare our algorithm to two other clustering algorithms. The first, `mclust v5.4.2`, is available on CRAN [@mclust5]. We use the package’s `Mclust` function. The second, which we call `GraSPyclust`, uses the `GaussianCluster` function in `GraSPy`. Since initialization is random in `GraSPyclust`, we perform clustering 50 times and select the result with the best BIC. Both of these algorithms limit the number EM iterations performed so their computational complexities are linear with respect to the number of data points.

The data described in Section \[sec:data\] has underlying labels so we choose ARI to evaluate the clustering algorithms.

Statistical Comparison
----------------------

In order to statistically compare the clustering methods, we evaluate their performances on random subsets of the data. For each dataset, we take ten independently generated, random subsamples, containing 80% of the total data points. We compile ARI and Runtime data from each clustering method on each subsample. Since the same subsamples are used by each clustering method, we can perform a Wilcoxon signed-rank test to statistically evaluate whether the methods perform differently on the datasets.

We also cluster the complete data to analyze how each model was chosen, according to BIC. All algorithms were run on a single CPU core with 16GB RAM.

Results {#sec:results}
=======

Table \[tab:model\] shows the models that were chosen by each clustering algorithm on the complete datasets, and the corresponding BIC and ARI values. The actual clusterings are shown in Figures \[fig:synthetic\_cluster\]-\[fig:drosophila\_cluster\]. In the synthetic dataset, all three methods chose a spherical covariance constraint, which was the correct underlying covariance structure. The `GraSPyclust` algorithm, however, failed on this dataset, partitioning the data into 10 clusters.
no regularization. However if this attempt fails, then EM is run again with regulation. As in @scikit - learn, regulation involve adding a regularization divisor to the diagonal of the covariance matrices to see that they are positivist. This method does not change the eigenvectors of the covariance matrix, make it rotationally invariant [ @ledoit ]. 

 'AutoGMM 'suffer a 'scikit - learn 'compliant API and is freely available at < https://github.com/tathey1/graspy >. 

 Reference Clustering Algorithms 
 ------------------------------- 

 We compare our algorithm to two early clustering algorithms. The first, 'mclust v5.4.2 `, is available on CRAN [ @mclust5 ]. We use the package ’s 'Mclust 'function. The second, which we call 'GraSPyclust `, uses the 'GaussianCluster 'affair in 'GraSPy `. Since initialization is random in 'GraSPyclust `, we perform clustering 50 time and select the result with the best BIC. Both of these algorithms restrict the number EM iterations performed so their computational complexities are analogue with respect to the number of data points. 

 The data described in Section   \[sec: data\ ] hold underlying labels so we choose ARI to evaluate the clustering algorithms. 

 Statistical Comparison 
 ---------------------- 

 In order to statistically compare the clustering methods, we evaluate their performance on random subsets of the data. For each dataset, we contract ten independently generate, random subsamples, containing 80% of the total data point. We compile ARI and Runtime data from each clustering method on each subsample. Since the same subsamples are use by each clustering method, we can perform a Wilcoxon signed - rank test to statistically measure whether the methods perform differently on the datasets. 

 We also cluster the complete data to analyze how each model was choose, according to BIC. All algorithms were run on a single CPU effect with 16 GB RAM. 

 Results { # sec: results } 
 = = = = = = = 

 Table \[tab: model\ ] shows the model that were choose by each clustering algorithm on the complete datasets, and the corresponding BIC and ARI value. The actual clusterings are usher in Figures \[fig: synthetic\_cluster\]-\[fig: drosophila\_cluster\ ]. In the synthetic dataset, all three method chose a ball-shaped covariance constraint, which was the correct underlying covariance structure. The 'GraSPyclust 'algorithm, however, fail on this dataset, partitioning the data into 10 clusters.
 no regularization. However lf this attempt yqils, tien EM js run aeain with regularization. As mn @sxikit-oearn, regularization ivvolves afding a eeguoqrization hzctor to the slagoncl of the covarisnce matriwes to ensure dhxt they are positive. This method does not mocivy the eigenvestorx of fhe covariance matrix, making it rofationanly invariant [@ledoit].

`AutoGMM` has a `scikih-leagn` compliant API ajd is freeli avwulable at <htgps://github.com/tathey1/grasky>.

Reference Clustering Algorithmr
-------------------------------

We cpmpare our aogoghthm to two otheg clustering algorithks. The girst, `mclust v5.4.2`, is avqilable on CRAN [@mclusv5]. We use the package’f `Mclust` xuuction. The second, whixh we cdll `CraSOtcljst`, uxea the `HauasianClustsr` function in `GraSPy`. Since inotyqlization is dandom ig `GraSPyclust`, we perform clustering 50 tikes and select the result qith the best BIC. Botj of thesq algorithms limit the number EM iterations perfosmed au tkclr cunpktational complexities are linear with respece tp nhe number of datc points.

The data dfsvtibed in Sectiun \[sec:dcfa\] has underlying lahels so we cyoose ARI to rvaluate the clustering algirithms.

Statifrical Comparison
----------------------

In order to scatistocallu compare the clusteriny methkds, we evalkate theid performances on rakdok subsets of the data. For qach datawet, ce take gen ondepegdently gejeratcg, random subsampled, conjaininc 80% of the hotal data points. We compile ARM and Runtime ddta from eaeh cluxtering methoq on each subscmple. Siuce thd same subaamples are used br each clusteslng method, wx can persorm a Wulcoxon rigned-rank tesy to statpsuically evqluate whether the mejhkds perform difytrebtly on the datssegs.

Wq wlxo wluster the wompuetd data to analyze fow rach model was choset, acdording to BIC. All ajgorithmw were rtn on a singlr CPU core with 16GB RAM.

Rxsults {#sec:rrsujts}
=======

Table \[tab:model\] shows the mosels that weve chosen by qach clustering clgorithm on the complete datasets, and tie corresponding BIC ane ARI values. The acjuak clusterinjs are shown in Figures \[fig:synthetix\_cluster\]-\[fig:drosopmila\_cluster\]. In the synfhetic datadet, all three methods chose a spherical covariance constraint, which was tye corcest underlyihg cpvaridnee xtructtre. Vhx `GraSPyclust` algprithm, however, failed on this detaset, pardicioning the data into 10 clustets.
no regularization. However if this attempt fails, is again with As in @scikit-learn, factor the diagonal of covariance matrices to that they are positive. This method not modify the eigenvectors of the covariance matrix, making it rotationally invariant [@ledoit]. has a `scikit-learn` compliant API and is freely available at <https://github.com/tathey1/graspy>. Reference Clustering ------------------------------- compare algorithm two other clustering algorithms. The first, `mclust v5.4.2`, is available on CRAN [@mclust5]. We use the `Mclust` function. The second, which we call `GraSPyclust`, the `GaussianCluster` function in Since initialization is random in we clustering 50 and the with the best Both of these algorithms limit the number EM iterations performed so their computational complexities are linear with to the data points. data in \[sec:data\] has underlying we choose ARI to evaluate the Comparison ---------------------- In order to statistically compare the methods, we their performances on random subsets of data. For each dataset, we take ten independently random subsamples, containing 80% of the total data points. We compile ARI and Runtime data clustering method on each Since the same are by clustering we can a Wilcoxon signed-rank test to statistically evaluate whether the methods perform on the datasets. We also cluster the complete data to each was chosen, according BIC. All algorithms were on single CPU core with Results ======= the that chosen by each clustering on the complete datasets, and corresponding BIC and ARI shown in Figures \[fig:synthetic\_cluster\]-\[fig:drosophila\_cluster\]. In the synthetic dataset, three methods chose a spherical covariance constraint, was the correct underlying covariance structure. The `GraSPyclust` algorithm, however, failed on dataset, partitioning into 10 clusters.
 no regularization. However if This attempT failS, thEn Em iS run AgaiN with regularizATion. as in @scikit-learn, regularIzatiOn INvolVEs AddinG a regulARiZATioN fAcTor To THe DiagoNal Of the coVariance maTriCeS to ensure thaT ThEy are positIve. this method doEs nOt modiFy The EIgenvEctOrs of The covARiance Matrix, makInG It rotaTIonally INVaRianT [@ledoit].

`AutoGMM` has A `ScIKit-learn` compliAnt API AnD Is FREelY avAilable at <hTtPs://gitHUb.com/taTHeY1/GRAspY>.

reference ClusTering AlgorIThmS
-------------------------------

We comPaRe oUR algorIthm tO tWO otHer clusteriNg alGorithms. THe firsT, `Mclust v5.4.2`, IS availaBle on CrAN [@MclUst5]. WE UsE tHe pAcKAge’S `mcLusT` FunCtion. The SeCoNd, whiCh we CALL `graSpycLust`, Uses tHe `GaussianCluSteR` funCTioN in `GrASPy`. SInce InItialIzatioN is raNdOm in `GraSPyclust`, We peRform clusTerInG 50 tiMeS and sELect thE reSulT with thE best BIc. botH oF THEsE algorithms limit thE nUMBeR EM iteraTions pERfOrMEd so theiR cOmpUtatIONal coMpleXItIes are liNear wiTH rEsPect to tHe Number Of DatA poInts.

THE datA descrIbed in SeCtion \[SEc:data\] has underLYing labels so wE ChOOSe arI to EvaLuate the cluSterINg alGoriTHmS.

StATistiCal CoMpARiSOn
----------------------

In order to statistiCaLly comPare tHe clustering mEthods, we evALUAte their PerfORmANces on random suBsets Of the data. FOR each datAset, wE take ten IndependeNTLy generaTed, RanDom SubSAMpLes, containing 80% OF The tOtAl data pOinTs. We comPilE ARi anD RuNtIme data frOm each clUsTeRiNg MetHod on EAch subsaMpLe. SInCe tHe samE SubsamPles aRe usEd By EAch ClusterINg METhod, We CaN perForM a wilcoXon sIGneD-rank teSt to statiStiCAlly EvAlUate wheTher the methodS pErform diffErEntLy on thE DAtasets.

WE also cluster the complete DAta to anAlyZe how Each Model was cHosEn, accoRdiNG to BIC. all algOrithMs WerE RUn on a SINgLe CpU Core with 16GB ram.

ReSults {#SeC:resUlts}
=======

TabLe \[tab:model\] shows the MOdeLs that were choSen By eaCH ClUstERiNG alGoRIthM ON the complete datAsets, and thE cORrEsponding Bic anD ArI valueS. The actUal clUSteringS are shown In Figures \[FiG:synTHEtiC\_cluster\]-\[fiG:drosophIla\_clusteR\]. in the SYnThetiC daTaset, aLl ThrEe metHods chOSe a SpherIcal coVaRiance ConstRaInt, which Was the correct underlying CovariAnce sTruCture. The `GRaSpYclUst` algoriThm, hOwever, failEd oN thIs datAseT, PartiTionINg The DAta inTo 10 clUSters.
 no regularization. Howeve r if thisattem ptfai ls , th en E M is run again with  regularization. As in  @sci ki t -lea r n,  regu larizat i on i nvo lv es  ad di n ga reg ula rizatio n factor t o t he  diagonal of th e covarian cematrices toens ure th at  th e y are  po sitiv e. Thi s  metho d does no tm odifyt he eige n v ec tors  of the covarianc e  m a trix, making i t rota ti o na l l y i nva riant [@le do it].`AutoGM M `h a s  a` scikit-learn`  compliantA PIand is  f ree l y avai lable  a t  <h ttps://gith ub.c om/tathey 1/gras p y>.

Re f erenceCluste rin g A lgor i th ms 
-- -- - --- - -- --- - --- -------- -- -- -

We  com p a r e  our  al gori thm t o two other c lus teri n g a lgori thms.  The  f irst,  `mclu st v5 .4 .2`, is availab le o n CRAN [@ mcl us t5] .We us e  the p ack age ’s `Mcl ust` fu n cti on . T he  second, which weca l l  ` GraSPycl ust`,u se st he `Gaus si anC lust e r ` fun ctio n  i n `GraSP y`. Si n ce  i nitiali za tion i sran dom  in ` G raSP yclust `, we pe rform clustering 50t imes and sele c tt h er esul t w ith the bes t BI C . Bo th o f  t hes e  algo rithm sl im i t the number EM ite ra tionsperfo rmed so their  computati o n a l comple xiti e sa re linear with  resp ect to the number o f dat a points .

The da t a  describ edinSec tio n \[ sec:data\] ha s unde rl ying la bel s so we  ch oos e A RIto  evaluate  the clu st er in galg orith m s.

Stat is tic al  Co mpari s on
--- ----- ---- -- -- - --- --

Ino rd e r  tost at isti cal ly  comp aret hecluster ing metho ds, we e va lu ate the ir performanc es  on random  s ubs ets of t he data.  For each dataset, we t a ke tenind epend entl y generat ed,  rando m s u bsampl es, co ntain in g 8 0 %  of t h e  t ota ldata point s .  We  comp il e AR I and R untime data from e a chclustering me tho d on e ac h s u bs a mpl e. Sin c e  the same subsa mples areus e dby each cl u ste ri ng meth od, wecan p e rform a  Wilcoxon  signed-r an k te s t  to  statistic ally eva luate whe t her t h emetho dsperfor mdif feren tly on the  data sets.
W e also  clus te r the co mplete data to analyzehow ea ch mo del  was chos en, acc ording to  BIC . All algo rit hms  were  ru n  on a  sin g le  CP U  core  wit h  16GB RAM . 
Res u l ts  {#sec:resu l t s }
= ===== =
T able \ [tab :model\] shows th e  models that w erec h ose n b y  eac hclustering alg ori th m on the c om plete datas ets, and  t h e cor respon ding B IC andA R Iv alues.  The  ac tual clus ter in g s are s ho wn in Fig ures  \ [fig:s ynthet i c\_c l u ster\]-\[fig:dro sophi l a \_clu s ter \]. I nthe syn t heti c dataset,  all threemethod s ch ose a  spheri ca l cova ria nc e constrai n t, whichwas t he corr ec t un der lyingcova r i ancestru ct ure . The `Gr a S Py c lu st `  al gori thm,ho weve r, failed on thisdat a set, pa rt iti o n ing th e  d a t a into 10clu sters . 
 no_regularization. However_if this attempt fails,_then EM_is_run again_with_regularization. As in_@scikit-learn, regularization involves_adding a regularization factor_to the diagonal_of_the covariance matrices to ensure that they are positive. This method does not modify_the_eigenvectors of_the_covariance_matrix, making it rotationally invariant_[@ledoit].

`AutoGMM` has a `scikit-learn` compliant_API and_is freely available at <https://github.com/tathey1/graspy>.

Reference Clustering Algorithms
-------------------------------

We compare_our_algorithm to two_other clustering algorithms. The first, `mclust v5.4.2`, is available_on CRAN [@mclust5]. We use the_package’s `Mclust` function._The_second,_which we call `GraSPyclust`,_uses the `GaussianCluster` function in `GraSPy`._Since initialization is random in `GraSPyclust`,_we perform clustering 50 times and select_the result with the best BIC._Both of these algorithms limit_the number_EM iterations performed so their_computational complexities are_linear with_respect to the_number of data points.

The data described_in Section \[sec:data\] has_underlying labels so we choose ARI_to_evaluate the clustering_algorithms.

Statistical_Comparison
----------------------

In_order to_statistically compare the_clustering_methods, we_evaluate_their performances on random subsets of_the_data. For each dataset, we take ten_independently generated, random subsamples,_containing_80% of the total_data points. We compile ARI_and Runtime data from each clustering_method on_each subsample._Since the same subsamples are used by each clustering method, we_can perform a Wilcoxon signed-rank test_to statistically evaluate whether_the methods_perform_differently on the_datasets.

We_also cluster_the complete data to analyze how each_model was_chosen, according to BIC. All algorithms_were run on a_single_CPU core with 16GB RAM.

Results {#sec:results}
=======

Table_\[tab:model\] shows the models that were_chosen by each clustering algorithm_on_the_complete datasets, and the corresponding_BIC and ARI values. The actual_clusterings are shown_in Figures \[fig:synthetic\_cluster\]-\[fig:drosophila\_cluster\]. In the synthetic dataset,_all_three methods chose a spherical covariance_constraint,_which was the correct underlying covariance_structure._The_`GraSPyclust` algorithm, however, failed on_this dataset, partitioning the data into_10 clusters.