 higher proportions of polarizing material. Three of the authors (A. M. J., S. Q., and L. A.) coded relevant tweets as “provaccine,” “antivaccine,” or “neutral” using a codebook developed by 1 of the authors (A. M. J.). When coders disagreed, we employed a second round of annotation. We resolved any remaining disagreements by a fourth annotator (D. A. B.). We compared all users’ proportions of polarized (nonneutral) tweets to users with bot scores below 20% (likely humans). We also tested the hypothesis that nonneutral content posted by bots and trolls was more likely to be antivaccine by comparing the relative proportions of polarized tweets that were antivaccine across all user types. We assessed all hypothesis tests for statistical significance using the distribution-free χ2 goodness of fit test. Thematic analysis of tweets by Russian trolls. During annotation, an unfamiliar hashtag, #VaccinateUS, appeared 25 times in tweets posted by known Russian troll accounts identified by NBC News and documenting Russian interference in the US political system.20 Searching Twitter on February 20, 2018, we found only 5 messages including this hashtag, suggesting that #VaccinateUS had been primarily used by suspended accounts and that most instances had been purged. Turning to data stored in the vaccine stream, we identified 253 messages with #VaccinateUS. We conducted an exploratory thematic analysis of these messages to identify and describe major themes. Our goal was to explore unifying patterns in the #VaccinateUS data30 and to illustrate some of the behaviors that known Russian trolls exhibit on Twitter. Consistent with this aim, we annotated messages as pro- or antivaccine. Next, 1 author (A. M. J.) categorized messages into pro- and antivaccine themes using a combination of inductive and deductive codes.31 We determined these categories loosely from existing research,12 and we incorporated emergent themes in the data into them. We compared these tweets with the randomly selected vaccine-relevant tweets we used in the second analysis, which we considered representative of the general vaccine discourse.

RESULTS Section: Choose Top of page Abstract METHODS RESULTS << DISCUSSION REFERENCES CITING ARTICLES Raw counts of tweets by source are shown in Table C (available as a supplement to the online version of this article at http://www.ajph.org). Figure 1 shows that accounts identified by NBC News
higher proportions of polarizing material. Three of the authors (A. M. J., S. Q., and L. A.) gull relevant tweet as “ provaccine, ” “ antivaccine, ” or “ neutral ” using a codebook develop by 1 of the generator (A. M. J.). When coders disagreed, we hire a second turn of annotation. We decide any remaining disagreements by a fourth annotator (D. A. B.). We compare all users ’ proportions of polarize (nonneutral) tweet to users with bot scores below 20% (probable humans). We besides test the hypothesis that nonneutral content posted by bot and trolls was more likely to be antivaccine by comparing the relative proportions of polarize tweets that were antivaccine across all user types. We assessed all hypothesis test for statistical significance using the distribution - free χ2 goodness of fit test. Thematic psychoanalysis of tweet by Russian troll. During annotation, an unfamiliar hashtag, # VaccinateUS, appeared 25 times in tweets posted by known Russian troll accounts identify by NBC News and documenting Russian interference in the US political system.20 Searching Twitter on February 20, 2018, we discover entirely 5 messages including this hashtag, hint that # VaccinateUS had been chiefly used by suspended accounts and that most case had been purged. Turning to data stored in the vaccine stream, we identified 253 messages with # VaccinateUS. We conducted an exploratory thematic analysis of these messages to identify and identify major themes. Our finish was to research consolidative patterns in the # VaccinateUS data30 and to illustrate some of the behaviors that know Russian trolls exhibit on Twitter. Consistent with this purpose, we annotated messages as pro- or antivaccine. Next, 1 author (A. M. J.) categorized message into pro- and antivaccine themes use a combination of inductive and deductive codes.31 We determined these categories broadly from existing research,12 and we incorporated emergent themes in the data into them. We compare these tweets with the randomly choose vaccine - relevant tweets we used in the second analysis, which we view representative of the general vaccine discourse. 

 RESULTS Section: Choose Top of page Abstract METHODS RESULTS < < DISCUSSION REFERENCES CITING ARTICLES Raw counts of tweets by source are shown in Table C (available as a addendum to the online version of this article at http://www.ajph.org). human body 1 shows that accounts identified by NBC News
 hihher proportions of polavizing material. Jheee of the ahthors (A. M. J., S. Q., and L. A.) coded relevent rweetw as “provaccine,” “antivazcine,” or “jeutral” ysinj a codebook devxmoped bn 1 of bhe abtiors (A. M. J.). When coders divagreed, we empnoheb a second round of annotation. We refolved snj remaining difagrtemqnts by a fourth annotator (D. A. B.). We cojpared ell users’ propottions of polarized (nonneuhral) tweets to users wlth bot scotss fwlow 20% (likely humans). We also tested the hypothesis that nonneutral contznt posted vy bohv and trollw was more likely to be attivaccone by comparikg thx reoative proportions of polarized tweets jhat were dncivaccine across all yswr tykes. Wa asrwssdd zlk gypothfsia tests fod statisticql significance usimg nne distributjon-freq χ2 goodness of fit test. Thematic analysis of tweets by Russian trolos. During annotation, wn unfamijiar hashtag, #VaccinateUS, appeared 25 times in tweetv posvea bv knowv Rkssian troll accounts identified by NBC News whd dpcumenting Rusfian interfrrfnvg in the US pouitical sgstem.20 Searching Twltter og Febeuary 20, 2018, wt foumd only 5 messages including this hashtay, syggesting that #VaceinateUS had beem prikarily used by suspendeb accohnts and thwt most ihrtances had been pugged. Turning to data stored in the vaccmne scream, we idemtifieq 253 messaged witm #VaccinateUS. We cojductgd an axploratorj thematic analysis of these messages to idenjiff atd descrnbe mamor themes. Our doal was to exklore uniyying oatterns ih the #VeccinateUS dwta30 and to ilnostrate some mf the bqhavuors that kvuwn Russian trplls exhiynt on Twirter. Consistent wibh thka aim, we annotactd nessages as pro- or aneinacrine. Gaxt, 1 author (D. M. G.) cxyegorkzed messagcs knto pro- and antivaccine thejes using a combinstlon of ineuctive wnd deductive codes.31 We determintd theve rategotief loosely from existing researdh,12 and we ingorporated emqrgekt tremes in tke data into them. We compared these tweevs with the randomly sglected vaccine-relevaut tweets we nsed ig the secmnd analysis, which ww considered reprtsentative of the generzl vacwine fiscourse.

RESULTS Section: Choose Top of page Abstract METHODS RESULTS << DISXUSSIOI WEFERENCES DITIMG ARNICJZS Raw sounvs of tweets by solrce are shown in Table C (availaule as a sgp'lement to the online version ow this articlg at http://www.ajph.org). Figure 1 shows yhat accounts identified by NBC News
higher proportions of polarizing material. Three of (A. J., S. and L. A.) “antivaccine,” “neutral” using a developed by 1 the authors (A. M. J.). When disagreed, we employed a second round of annotation. We resolved any remaining disagreements a fourth annotator (D. A. B.). We compared all users’ proportions of polarized tweets users bot below 20% (likely humans). We also tested the hypothesis that nonneutral content posted by bots and was more likely to be antivaccine by comparing relative proportions of polarized that were antivaccine across all types. assessed all tests statistical using the distribution-free goodness of fit test. Thematic analysis of tweets by Russian trolls. During annotation, an unfamiliar hashtag, #VaccinateUS, 25 times posted by Russian accounts by NBC News Russian interference in the US political on February 20, 2018, we found only 5 including this suggesting that #VaccinateUS had been primarily by suspended accounts and that most instances had purged. Turning to data stored in the vaccine stream, we identified 253 messages with #VaccinateUS. an exploratory thematic analysis these messages to and major Our was to unifying patterns in the #VaccinateUS data30 and to illustrate some of behaviors that known Russian trolls exhibit on Twitter. Consistent with we messages as pro- antivaccine. Next, 1 author M. categorized messages into pro- themes a and codes.31 determined these categories loosely existing research,12 and we incorporated themes in the data tweets with the randomly selected vaccine-relevant tweets we in the second analysis, which we considered of the general vaccine discourse. RESULTS Section: Choose Top of page Abstract RESULTS << CITING ARTICLES Raw counts of tweets by source shown in Table C as a supplement to the online version of this at Figure 1 that accounts identified NBC News
 higher proportions of polariZing materiAl. ThrEe oF thE aUthoRs (A. M. j., S. Q., and L. A.) coded rELevaNt tweets as “provaccine,” “anTivacCiNE,” or “nEUtRal” usIng a codEBoOK DevElOpEd bY 1 oF ThE authOrs (a. M. J.). When Coders disaGreEd, We employed a sECoNd round of aNnoTation. We resoLveD any reMaIniNG disaGreEmentS by a foURth annOtator (D. A. B.). we COmpareD All userS’ PRoPortIons of polarized (noNNeUTral) tweets to usErs witH bOT sCORes BelOw 20% (likely huMaNs). We aLSo testeD ThE HYPotHEsis that nonneUtral contenT PosTed by bOtS anD Trolls Was moRe LIkeLy to be antivAcciNe by compaRing thE RelativE ProportIons of PolAriZed tWEeTs ThaT wERe aNTiVacCIne Across alL uSeR typeS. We aSSESSed aLl hYpotHesis Tests for statiStiCal sIGniFicanCe usiNg thE dIstriBution-Free χ2 GoOdness of fit test. themAtic analySis Of TweEtS by RuSSian trOllS. DuRing annOtation, AN unFaMILIaR hashtag, #VaccinateUs, aPPEaRed 25 times In tweeTS pOsTEd by knowN RUssIan tROLl accOuntS IdEntified By NBC NEWs AnD documeNtIng RusSiAn iNteRfereNCe in The US pOlitical SysteM.20 searching TwittER on February 20, 2018, we FOuND OnLY 5 mesSagEs including This HAshtAg, suGGeStiNG that #vacciNaTEUs Had been primarily useD bY suspeNded aCcounts and thaT most instaNCES had been PurgED. TURning to data stoRed in The vaccine STream, we iDentiFied 253 messAges with #VACCinateUS. we cOndUctEd aN EXpLoratory themaTIC anaLySis of thEse MessageS to IdeNtiFy aNd Describe mAjor themEs. ouR gOaL waS to exPLore unifYiNg pAtTerNs in tHE #VacciNateUs datA30 aNd TO ilLustratE SoME Of thE bEhAvioRs tHaT knowN RusSIan Trolls eXhibit on TWitTEr. CoNsIsTent witH this aim, we annOtAted messagEs As pRo- or anTIVaccine. NExt, 1 author (A. M. J.) categorized MEssages IntO pro- aNd anTivaccine TheMes usiNg a COmbinaTion of InducTiVe aND DeducTIVe CodEs.31 we determinED TheSe catEgOrieS looselY from existing reseaRCh,12 aNd we incorporaTed EmerGENt TheMEs IN thE dATa iNTO them. We compared These tweetS wITh The randomlY SelEcTed vaccIne-releVant tWEets we uSed in the sEcond analYsIs, whICH we Considered RepresenTative of tHE geneRAl VacciNe dIscourSe.

rESuLTS SEction: cHooSe Top Of page abStract mETHOdS rESULTS << DiSCUSSION REFERENCES CITInG ARTIcLES RAw cOunts of twEetS By sOurce are sHown In Table C (avAilAblE as a sUppLEment To thE OnLinE VersiOn of THis articlE At HttP://WWw.Ajph.org). FiguRE 1 SHowS that AccOUnts idEntiFied by NBC News
 higher proportions of pol arizing ma teria l.Thr ee  oftheauthors (A. M. J.,S. Q., and L. A.) code d rel ev a nt t w ee ts as  “prova c ci n e ,”“a nt iva cc i ne ,” or  “n eutral”  using a c ode bo ok developed by  1 of theaut hors (A. M.J.) . When  c ode r s dis agr eed,we emp l oyed a  second r ou n d of a n notatio n .  W e re solved any remain i ng disagreementsby a f ou r th a nno tat or (D. A.B. ). We compare d  a l l use r s’ proportion s of polari z ed(nonne ut ral )  tweet s tous e rswith bot sc ores  below 20 % (lik e ly huma n s). Wealso t est edtheh yp ot hes is tha t  n onn e utr al conte nt  p osted  byb o t s  and  tr olls  wasmore likely t o b e an t iva ccine  by c ompa ri ng th e rela tivepr oportions of po lari zed tweet s t ha t w er e ant i vaccin e a cro ss alluser ty p es.  W e a ss essed all hypothes is t es ts for s tatist i ca ls ignifica nc e u sing t he di stri b ut ion-free  χ2 go o dn es s of fi ttest.Th ema tic  anal y sisof twe ets by R ussia n  trolls. Durin g  annotation,a nu n fa m ilia r h ashtag, #Va ccin a teUS , ap p ea red 25 ti mes i nt we e ts posted by knownRu ssiantroll  accounts ide ntified by N B C News a nd d o cu m enting Russian  inte rference i n  the USpolit ical sys tem.20 Se a r ching Tw itt eronFeb r u ar y 20, 2018, w e foun donly 5mes sages i ncl udi ngthi shashtag,suggesti ng  t ha t#Va ccina t eUS hadbe enpr ima rilyu sed by  susp ende dac c oun ts andt ha t most  i ns tanc esha d bee n pu r ged . Turni ng to dat a s t ored  i nthe vac cine stream,we  identifie d253  messa g e s with # VaccinateUS. We conduct e d an ex plo rator y th ematic an aly sis of  th e se mes sagesto id en tif y and d e s cr ibe  m ajor theme s .  Ou r goa lwasto expl ore unifying patte r nsin the #Vacci nat eUSd a ta 30a nd toil l ust r a te some of thebehaviorsth a tknown Russ i antr olls ex hibit o n Twi t ter. Co nsistentwith this  a im,w e  an notated me ssages a s pro- or antiv a cc ine.Nex t, 1 a ut hor  (A.M. J.) cat egori zed me ss ages i nto p ro - and an tivaccine themes usinga comb inati onof induct ive and  deductiv e co des.31 Wedet erm inedthe s e cat egor i es  lo o selyfrom existingr es ear c h ,1 2 and we in c o r por atedeme r gent t heme s in the data int o  them. We comp ared t hes e t w eets  w ith the random lyse l e cted vac ci ne-relevant  tweetswe usedin the  secon d analy s i s, whichwe c ons idered re pre se n tativeof  t h e gene ralva ccinediscou r se.R ESULTS Section:Choos e Top o f  pa ge Ab st ract ME T HODS  RESULTS < < DISCUSSIO N REFE RENC ES CI TING AR TI CLES R awco unts of tw e ets by so urceare sho wn  inTab le C ( avai l a ble a s asu ppl ement tot h eo nl in e  ve rsio n ofth is a rticle at http://w ww. a jph.org ).  Fi g u re 1 s h ow s that accou nts  iden t i fied by NB C  New s 
 higher_proportions of_polarizing material. Three of_the authors_(A._M. J.,_S._Q., and L._A.) coded relevant_tweets as “provaccine,” “antivaccine,”_or “neutral” using_a_codebook developed by 1 of the authors (A. M. J.). When coders disagreed, we_employed_a second_round_of_annotation. We resolved any remaining_disagreements by a fourth annotator_(D. A._B.). We compared all users’ proportions of polarized_(nonneutral)_tweets to users_with bot scores below 20% (likely humans). We also_tested the hypothesis that nonneutral content_posted by bots_and_trolls_was more likely to_be antivaccine by comparing the relative_proportions of polarized tweets that were_antivaccine across all user types. We assessed_all hypothesis tests for statistical significance_using the distribution-free χ2 goodness_of fit_test. Thematic analysis of tweets_by Russian trolls._During annotation,_an unfamiliar hashtag,_#VaccinateUS, appeared 25 times in tweets_posted by known_Russian troll accounts identified by NBC_News_and documenting Russian_interference_in_the US_political system.20 Searching_Twitter_on February_20,_2018, we found only 5 messages_including_this hashtag, suggesting that #VaccinateUS had been_primarily used by suspended_accounts_and that most instances_had been purged. Turning to_data stored in the vaccine stream,_we identified_253 messages_with #VaccinateUS. We conducted an exploratory thematic analysis of these messages_to identify and describe major themes._Our goal was to_explore unifying_patterns_in the #VaccinateUS_data30_and to_illustrate some of the behaviors that known_Russian trolls_exhibit on Twitter. Consistent with this_aim, we annotated messages_as_pro- or antivaccine. Next, 1 author_(A. M. J.) categorized messages into_pro- and antivaccine themes using_a_combination_of inductive and deductive codes.31_We determined these categories loosely from_existing research,12 and_we incorporated emergent themes in the data_into_them. We compared these tweets with_the_randomly selected vaccine-relevant tweets we used_in_the_second analysis, which we considered_representative of the general vaccine discourse.

RESULTS_Section: Choose Top of page Abstract METHODS RESULTS <<_DISCUSSION REFERENCES CITING_ARTICLES Raw counts of tweets_by_source_are shown in Table C (available as a supplement to_the online_version of this_article at http://www.ajph.org). Figure 1 shows that accounts identified by_NBC News