 10987) of the relations are across-sentence relations. The inter-annotator agreement (IAA) for the final annotation schema is 0.89. It was calculated for the 61 reports annotated by both annotators on a token level using the Cohen’s Kappa coefficient ($\kappa$) [@Cohen:1960].

Discussion and Conclusions {#sec:discussion}
==========================

We presented a manually annotated corpus of entities and relations in radiology reports written in Spanish. The goal was twofold: to have an annotated dataset available for evaluating NER and RE algorithms results and for training of supervised models and to present an annotation guideline that can be used by other researchers with similar needs. The creation of the corpus was not an easy task. Many annotation-revision iterations had to be performed in order to arrive to a stabilized annotation schema. According to what we expected, $\kappa$ improved in each annotation iteration step. Data had to be anonymized. Furthermore, the shortness of the texts, the abundance of abbreviations and acronyms (about 6% of the AE and FI are written as such and there are 105 different abbreviations or acronyms in 513 reports), the specificity of the medical language, the existence of multi-segment terms and the existence of relations between sentences, makes not only the NER and RE tasks, but also the annotation task a difficult one.

The relation of findings with temporal terms, negation terms and uncertainty terms should be taken into account to determine their factuality. The abundance of negated findings (56%) might lead to the implementation of methods to detect negated findings in reports (see, and for Spanish). The difference of criteria among the annotators helps us determine that the evaluation of NER systems is not an easy task.

Finally, the pre-annotation helped us speed the annotation process, although it might have biased the annotation results.

[^1]: Ultrasound reports are texts describing what has been observed in a type of imaging study called ultrasound examination.

[^2]: <http://radlex.org/>

[^3]: <https://www.nlm.nih.gov/research/umls/>

[^4]: In “right lobe of the liver has the usual size”, “right lobe of the liver” should be annotated, although it is not associated to any finding.

[^5]: <http://brat.nlplab.org/>

[^6]: All the annotationsrevision
10987) of the relations are across - sentence relations. The inter - annotator agreement (IAA) for the final note schema is 0.89. It was account for the 61 reports annotated by both annotator on a token horizontal surface using the Cohen ’s Kappa coefficient ($ \kappa$) [ @Cohen:1960 ]. 

 Discussion and Conclusions { # sec: discussion } 
 = = = = = = = = = = = = = = = = = = = = = = = = = = 

 We present a manually gloss corpus of entity and relative in radiology report written in Spanish. The goal was double: to accept an annotated dataset available for evaluating NER and RE algorithms results and for training of supervised models and to salute an annotation guideline that can be used by early researchers with similar need. The creation of the principal was not an easy job. Many annotation - revision iterations give birth to be performed in order to arrive to a stabilized annotation schema. According to what we expect, $ \kappa$ better in each annotation iteration step. Data had to be anonymized. Furthermore, the shortness of the texts, the abundance of abbreviations and acronyms (approximately 6% of the AE and FI are spell as such and there are 105 unlike abbreviation or acronym in 513 reports), the specificity of the medical language, the existence of multi - segment terms and the existence of relations between sentences, lay down not only the NER and RE tasks, but also the annotation task a difficult one. 

 The relation of findings with temporal terms, negation term and doubt terms should be taken into account to determine their factuality. The abundance of negated findings (56 %) might lead to the implementation of method to detect negated findings in reports (see, and for Spanish). The dispute of criteria among the annotators help us determine that the evaluation of NER organization is not an easy undertaking. 

 Finally, the pre - note helped us speed the annotation summons, although it might have biased the note results. 

 [ ^1 ]: Ultrasound reports are text describing what has been observed in a type of imaging study call ultrasound examination. 

 [ ^2 ]: < http://radlex.org/ > 

 [ ^3 ]: < https://www.nlm.nih.gov/research/umls/ > 

 [ ^4 ]: In “ correct lobe of the liver has the usual size ”, “ right lobe of the liver ” should be annotated, although it is not associated to any finding. 

 [ ^5 ]: < http://brat.nlplab.org/ > 

 [ ^6 ]: All the annotationsrevision
 10987) ov the relations are acrors-sentence relajiins. Thx inter-znnotatof agreement (IAA) for the finap qnnotqtion schema is 0.89. It war calculaned for tye 61 ceports annotatev by botm annkbatorv on a token leyel using tve Cohen’s Kappd zozfficient ($\kappa$) [@Cohen:1960].

Discussion and Sonclusoojs {#sec:discussijn}
==========================

We [ressnted a manually annotated corpus kf entiuies and relations in radiology reports writhen ln Spanish. The goap was twofood: tj have an annutated dataset availabme for evaluating NER and RE aleoritkms results abd vmr training of slpervised modcks and to prexent an annotabion juidwline that can be usev by other researchets with sikimar needs. The creqtuon ox tha cofpus waa iot an eady vask. Many ahnotation-recision iterations hsd np be performsd in jrqer to arrive to a stabilized annotatiot sdhema. According to what we expected, $\kappa$ imkroved in qach annotation iteration step. Data had to be anotymizxd. Fuxbmermuee, the shortness of the texts, the abundance of wgbteniations and acrokyms (about 6% of the AF sgd FI are wrijten as sudh and there are 105 fifferegt abvreviatiogs ot acronyms in 513 reports), the wpecificity jd the medical langbage, the exirtenve of multi-segment terms and the existence lf relatikvs between sentevcex, kakes nou only the NER and RE tasks, but also tfe amnotatyon task a difflwult one.

The relatiln of fhndings wihh temporal terms, negation terms and uncertaimtf tvrms shoujd be taken into ascount to detetmine thenr facguality. Thv abundanre of negateq findings (56%) mhhht lead to vhe impleientqtiob of megfods to detect negated ynndings ib reports (see, and nor Skahish). The differzuct if criteria ampng thq wniotatjss helps us getefmivr thag the evaluctkon pf NER systems is nod an easy task.

Finally, yhc pre-annojation hejped us speed the annotation prlcess, alvhough it might have biased the annotatjon resulhs.

[^1]: Mltrasound re[ortw are texts bescribing what has been observed in a tbpe of imaging study caoled ultrasound exaklnation.

[^2]: <http://cadlex.jrg/>

[^3]: <https://fww.nlm.nih.gov/research/ymls/>

[^4]: In “right lobt of the liver has the hsual vize”, “gight lobe of the liver” should be annotated, although it is not associated to anb sinding.

[^5]: <httl://bray.nlpldb.jxg/>

[^6]: All ehe eniotationsrevision
10987) of the relations are across-sentence relations. agreement for the annotation schema is the reports annotated by annotators on a level using the Cohen’s Kappa coefficient [@Cohen:1960]. Discussion and Conclusions {#sec:discussion} ========================== We presented a manually annotated corpus of and relations in radiology reports written in Spanish. The goal was twofold: to an dataset for NER and RE algorithms results and for training of supervised models and to present an annotation that can be used by other researchers with needs. The creation of corpus was not an easy Many iterations had be in to arrive to stabilized annotation schema. According to what we expected, $\kappa$ improved in each annotation iteration step. Data had be anonymized. shortness of texts, abundance abbreviations and acronyms of the AE and FI are and there are 105 different abbreviations or acronyms 513 reports), specificity of the medical language, the of multi-segment terms and the existence of relations sentences, makes not only the NER and RE tasks, but also the annotation task a The relation of findings temporal terms, negation and terms be into account determine their factuality. The abundance of negated findings (56%) might lead the implementation of methods to detect negated findings in reports for The difference of among the annotators helps determine the evaluation of NER not easy pre-annotation us the annotation process, although might have biased the annotation [^1]: Ultrasound reports are observed in a type of imaging study called examination. [^2]: <http://radlex.org/> [^3]: <https://www.nlm.nih.gov/research/umls/> [^4]: In lobe of the liver has the usual size”, “right lobe of the should be it is not associated to any finding. [^5]: [^6]: All the annotationsrevision
 10987) of the relations are across-seNtence relaTions. the IntEr-AnnoTatoR agreement (IAA) fOR the Final annotation schema iS 0.89. It waS cALculATeD for tHe 61 reporTS aNNOtaTeD bY boTh ANnOtatoRs oN a token Level using The coHen’s Kappa coeFFiCient ($\kappa$) [@cohEn:1960].

Discussion And concluSiOns {#SEc:disCusSion}
==========================

WE preseNTed a maNually annOtATed corPUs of entITIeS and Relations in radiolOGy REports written iN SpaniSh. tHe GOAl wAs tWofold: to haVe An annOTated daTAsET AVaiLAble for evaluaTing NER and Re AlgOrithmS rEsuLTs and fOr traInINg oF supervised ModeLs and to prEsent aN AnnotatIOn guideLine thAt cAn bE useD By OtHer ReSEarCHeRs wITh sImilar neEdS. THe creAtioN OF THe coRpuS was Not an Easy task. Many aNnoTatiON-reVisioN iterAtioNs Had to Be perfOrmed In Order to arrive to A staBilized anNotAtIon ScHema. ACCordinG to WhaT we expeCted, $\kapPA$ imPrOVED iN each annotation iteRaTIOn Step. Data Had to bE AnOnYMized. FurThErmOre, tHE ShortNess OF tHe texts, tHe abunDAnCe Of abbreViAtions AnD acRonYms (abOUt 6% of The AE aNd FI are wRitteN As such and there ARe 105 different abBReVIAtIOns oR acRonyms in 513 repOrts), THe spEcifICiTy oF The meDical LaNGuAGe, the existence of mulTi-SegmenT termS and the existeNce of relatIONS between SentENcES, makes not only tHe NER And RE tasks, BUt also thE annoTation taSk a difficULT one.

The rElaTioN of FinDINgS with temporal TERms, nEgAtion teRms And unceRtaIntY teRms ShOuld be takEn into acCoUnT tO dEteRmine THeir factUaLitY. THe aBundaNCe of neGated FindInGs (56%) MIghT lead to THe IMPlemEnTaTion Of mEtHods tO detECt nEgated fIndings in RepORts (sEe, AnD for SpaNish). The differEnCe of criterIa AmoNg the aNNOtators hElps us determine that the eVAluatioN of nER syStemS is not an eAsy Task.

FiNalLY, the prE-annotAtion HeLpeD US speeD THe AnnOtAtion proceSS, AltHough It MighT have biAsed the annotation rESulTs.

[^1]: Ultrasound rEpoRts aRE TeXts DEsCRibInG WhaT HAs been observed iN a type of imAgINg Study calleD UltRaSound exAminatiOn.

[^2]: <httP://Radlex.oRg/>

[^3]: <https://wwW.nlm.nih.goV/rEseaRCH/umLs/>

[^4]: In “right lObe of the Liver has tHE usuaL SiZe”, “rigHt lObe of tHe LivEr” shoUld be aNNotAted, aLthougH iT is not AssocIaTed to any Finding.

[^5]: <http://brat.nlplab.orG/>

[^6]: All thE annoTatIonsrevisIon
 10987) of the relations a re across- sente nce  re la tion s. T he inter-annot a toragreement (IAA) for th e fin al anno t at ion s chema i s  0 . 8 9.It  w asca l cu lated  fo r the 6 1 reportsann ot ated by both an notators o n a  token level  us ing th eCoh e n’s K app a coe fficie n t ($\k appa$) [@ Co h en:196 0 ].

Dis c u ss ionand Conclusions { # se c :discussion}
= ====== == = == = = === === ======

We  p resen t ed a ma n ua l l y  an n otated corpus  of entitie s  an d rela ti ons in rad iolog yr epo rts written  inSpanish.The go a l was t w ofold:to hav e a n a nnot a te ddat as e t a v ai lab l e f or evalu at in g NER  and R E algo rit hmsresul ts and for tr ain ingo f s uperv isedmode ls  andto pre sentan  annotation gui deli ne that c anbe  us ed  by o t her re sea rch ers wit h simil a r n ee d s .  T he creation of the  c o r pu s was no t an e a sy  t a sk. Many  a nno tati o n -revi sion it erations  had t o  b eperform ed  in or de r t o a rrive to a  stabi lized an notat i on schema. Acc o rding to what we e xp e cted , $ \kappa$ imp rove d  ineach an not a tionitera ti o ns tep. Data had to be  a nonymi zed.Furthermore,the shortn e s s  of thetext s ,t he abundance o f abb reviations and acro nyms(about 6 % of theA E  and FIare  wr itt ena s  s uch and there a re 1 05  differ ent  abbrev iat ion s o r a cr onyms in513 repo rt s) ,th e s pecif i city ofth e m ed ica l lan g uage,the e xist en ce ofmulti-s e gm e n t te rm sandthe  e xiste nceo f r elation s between  se n tenc es ,makes n ot only the N ER  and RE ta sk s,but al s o  the ann otation task a difficul t  one.
The  rela tion  of findi ngs  withtem p oral t erms,negat io n t e r ms an d un cer ta inty terms s hou ld be  t aken  into a ccount to determin e  th eir factualit y.Thea b un dan c eo f n eg a ted f indings (56%) m ight leadto th e implemen t ati on  of met hods to  dete c t negat ed findin gs in rep or ts ( s e e,and for Sp anish).The diffe r enceo fcrite ria  among  t heannot atorsh elp s usdeterm in e that  theev aluation  of NER systems is notan eas y tas k.
Finally,  th e  pr e-annotat ionhelped usspe edthe a nno t ation  pro c es s,a lthou gh i t  might ha v ebia s e dthe annotat i o n  re sults .
[ ^1]: U ltra sound reports are texts describi ng w h a t h asb eenob served in a ty peof i maging s tu dy called u ltrasoun de xamin ation. 

[^2] : <http : / /r a dlex.o rg/> 

[ ^3]: <htt ps: // w ww.nlm. ni h. g ov/res earc h/ umls/> 

[^4] :  In“ r ight lobe of the  live r has t h e u sualsi ze”, “r i ghtlobe of th e liver” sh ould b e an notat ed, alt ho ugh it  is  n ot associa t ed to any  find ing.

[ ^5 ]: < htt p://br at.n l p lab.o rg/> 
[^6 ]: All th e an n ot at i ons revi sion
 10987)_of the_relations are across-sentence relations._The inter-annotator_agreement_(IAA) for_the_final annotation schema_is 0.89. It_was calculated for the_61 reports annotated_by_both annotators on a token level using the Cohen’s Kappa coefficient ($\kappa$) [@Cohen:1960].

Discussion and_Conclusions_{#sec:discussion}
==========================

We presented_a_manually_annotated corpus of entities and_relations in radiology reports written_in Spanish._The goal was twofold: to have an annotated_dataset_available for evaluating_NER and RE algorithms results and for training of_supervised models and to present an_annotation guideline that_can_be_used by other researchers_with similar needs. The creation of_the corpus was not an easy_task. Many annotation-revision iterations had to be_performed in order to arrive to_a stabilized annotation schema. According_to what_we expected, $\kappa$ improved in_each annotation iteration_step. Data_had to be_anonymized. Furthermore, the shortness of the_texts, the abundance_of abbreviations and acronyms (about 6%_of_the AE and_FI_are_written as_such and there_are_105 different_abbreviations_or acronyms in 513 reports), the_specificity_of the medical language, the existence of_multi-segment terms and the_existence_of relations between sentences,_makes not only the NER_and RE tasks, but also the_annotation task_a difficult_one.

The relation of findings with temporal terms, negation terms and uncertainty_terms should be taken into account_to determine their factuality._The abundance_of_negated findings (56%)_might_lead to_the implementation of methods to detect negated_findings in_reports (see, and for Spanish). The_difference of criteria among_the_annotators helps us determine that the_evaluation of NER systems is not_an easy task.

Finally, the pre-annotation_helped_us_speed the annotation process, although_it might have biased the annotation_results.

[^1]: Ultrasound reports_are texts describing what has been observed_in_a type of imaging study called_ultrasound_examination.

[^2]: <http://radlex.org/>

[^3]: <https://www.nlm.nih.gov/research/umls/>

[^4]: In “right lobe_of_the_liver has the usual size”,_“right lobe of the liver” should_be annotated, although it is not associated to any_finding.

[^5]: <http://brat.nlplab.org/>

[^6]: All_the annotationsrevision