 The first term pulls the ATTRACT examples (xl, xr) ∈ A closer together. If BA denotes the current mini-batch of ATTRACT examples, this term can be expressed as:\nA(BA) = ∑\n(xl,xr)∈BA\n(ReLU (δatt + xltl − xlxr)\n+ ReLU (δatt + xrtr − xlxr))\nwhere δatt is the similarity margin which determines how much closer synonymous vectors should be to each other than to each of their respective negative examples. ReLU(x) = max(0, x) is the standard rectified linear unit (Nair and Hinton, 2010). The ‘negative’ example ti for each word xi in any ATTRACT pair is the word vector closest to xi among the examples in the current minibatch (distinct from its target synonym and xi itself). This means that this term forces synonymous words from the in-batch ATTRACT constraints to be closer to one another than to any other word in the current mini-batch.\nThe second term pushes antonyms away from each other. If (xl, xr) ∈ BR is the current minibatch of REPEL constraints, this term is:\nR(BR) = ∑\n(xl,xr)∈BR\n(ReLU (δrpl + xlxr − xltr)\n+ ReLU (δrpl + xlxr − xrtr)) 3A native speaker is able to easily come up with these sets of morphological rules (or at least with a reasonable subset of rules) without any linguistic training. What is more, the rules for DE, IT, and RU were created by non-native, non-fluent speakers with a limited knowledge of the three languages, exemplifying the simplicity and portability of the approach.\nEnglish German Italian\n(discuss, discussed) (schottisch, schottischem) (golfo, golfi) (laugh, laughing) (damalige, damaligen) (minato, minata) (pacifist, pacifists) (kombiniere, kombinierte) (mettere, metto) (evacuate, evacuated) (schweigt, schweigst) (crescono, cresci) (evaluate, evaluates)
The first term pulls the ATTRACT examples (xl, xr) ∈ deoxyadenosine monophosphate nearer together. If BA denotes the current miniskirt - batch of ATTRACT examples, this terminus can be expressed as:\nA(BA) = ∑\n(xl, xr)∈BA\n(ReLU (δatt + xltl − xlxr)\n+ ReLU (δatt + xrtr − xlxr))\nwhere δatt is the similarity margin which decide how much closer synonymous vector should be to each other than to each of their respective negative example. ReLU(x) = max(0, x) is the standard rectified linear whole (Nair and Hinton, 2010). The ‘ damaging ’ example ti for each word xi in any ATTRACT pair is the word vector close to xi among the examples in the current minibatch (distinct from its target synonym and xi itself). This intend that this term forces synonymous words from the in - batch ATTRACT constraints to be close to one another than to any other word in the current mini - batch.\nThe second term pushes antonyms away from each early. If (xl, xr) ∈ BR is the current minibatch of REPEL constraints, this term is:\nR(BR) = ∑\n(xl, xr)∈BR\n(ReLU (δrpl + xlxr − xltr)\n+ ReLU (δrpl + xlxr − xrtr) ) 3A native speaker is able to easily come up with these sets of morphologic rules (or at least with a reasonable subset of rules) without any linguistic training. What is more, the rules for DE, IT, and RU were created by non - native, non - fluent speakers with a limited knowledge of the three language, exemplifying the simplicity and portability of the approach.\nEnglish German Italian\n(discuss, hash out) (schottisch, schottischem) (golfo, golfi) (laugh, laugh) (damalige, damaligen) (minato, minata) (pacifist, pacifist) (kombiniere, kombinierte) (mettere, metto) (evacuate, evacuated) (schweigt, schweigst) (crescono, cresci) (evaluate, evaluates )
 Thf first term pulls the AUTRACT examples (xl, xr) ∈ A closed togethdr. If BA denotes the current muni-baubh of ATTRACT exampler, this tegm can be expcessed as:\nA(BA) = ∑\n(em,xr)∈BA\n(RcJU (δafb + xlcl − xlxr)\n+ ReLU (δajt + xrtr − xlfr))\nwhere δatt iv ghz similarity margin which determines how muvh closer synonyious dectkgs should be to each other than to eaci of their resprctive negative examples. RfLU(x) = max(0, x) is the stajdard rectidied oinear unit (Vair and Hputon, 2010). The ‘nggative’ example ti for each word xi iu any ATTRAXT palt is the worv vectjr closest to xi amotg the rxamples in thc curcent minibatch (distinct fcom its target synonim and xi htaelf). This means tyar thiv tesm fuecer sgnpngmous aorvs from the in-batch ATRRACT constraints tp fv closer to ohe anoehqr than to any other word in the currenu minj-batch.\nThe second term pushes antonyms away vrom each other. If (xl, xr) ∈ BR is the current minibatch of RAPEL rovstxqints, rhls term is:\nR(BR) = ∑\n(xl,xr)∈BR\n(ReLU (δrpl + xlxr − xltr)\n+ DeKU (δrpl + xlxr − xrbr)) 3A native speaket ls wble to easili come bl sith these sets of morphojogicql rules (jr ay least with a reasonable sybset of rulvs) wuthout any linguiscic training. Whaj is mpre, the rules for DE, IT, and RU were crfated by hun-native, non-fluevt xpaakers with a limited knowjedge of vhe tkree laneuaggs, exem[lifying tje simplicity and portabllity ox the apprlach.\nEnglish German Italian\n(disrnss, discussed) (swhontisch, sckottisghem) (golfo, golfy) (laugh, laughiug) (damalnge, daoaligen) (mihato, miiata) (pacifise, pacifists) (kmlbiniere, komuinierte) (iettwre, netto) (exxcuate, evacuatrd) (schweiyu, schweigsr) (crescono, cresci) (cvaluxfe, evaluates)
The first term pulls the ATTRACT examples ∈ closer together. BA denotes the this can be expressed = ∑\n(xl,xr)∈BA\n(ReLU (δatt xltl − xlxr)\n+ ReLU (δatt + − xlxr))\nwhere δatt is the similarity margin which determines how much closer synonymous should be to each other than to each of their respective negative examples. = x) the rectified linear unit (Nair and Hinton, 2010). The ‘negative’ example ti for each word xi in ATTRACT pair is the word vector closest to among the examples in current minibatch (distinct from its synonym xi itself). means this forces synonymous words the in-batch ATTRACT constraints to be closer to one another than to any other word in the mini-batch.\nThe second antonyms away each If xr) ∈ BR current minibatch of REPEL constraints, this ∑\n(xl,xr)∈BR\n(ReLU (δrpl + xlxr − xltr)\n+ ReLU (δrpl xlxr − 3A native speaker is able to come up with these sets of morphological rules at least with a reasonable subset of rules) without any linguistic training. What is more, for DE, IT, and were created by non-fluent with limited of the languages, exemplifying the simplicity and portability of the approach.\nEnglish German Italian\n(discuss, (schottisch, schottischem) (golfo, golfi) (laugh, laughing) (damalige, damaligen) (minato, minata) (kombiniere, (mettere, metto) (evacuate, (schweigt, schweigst) (crescono, cresci) evaluates)
 The first term pulls the ATTRAcT examples (Xl, xr) ∈ A CloSer ToGethEr. If bA denotes the cuRRent Mini-batch of ATTRACT examPles, tHiS Term CAn Be expRessed aS:\NA(ba) = ∑\N(xl,Xr)∈bA\N(RelU (ΔAtT + xltl − XlxR)\n+ ReLU (δAtt + xrtr − xlxR))\nwHeRe δatt is the sIMiLarity margIn wHich determinEs hOw much ClOseR SynonYmoUs vecTors shOUld be tO each otheR tHAn to eaCH of theiR REsPectIve negative examplES. RElU(x) = max(0, x) is the stAndard ReCTiFIEd lIneAr unit (Nair AnD HintON, 2010). The ‘negATiVE’ EXamPLe ti for each woRd xi in any ATtrACt pair iS tHe wORd vectOr cloSeST to Xi among the eXampLes in the cUrrent MInibatcH (DistincT from iTs tArgEt syNOnYm And Xi ITseLF). THis MEanS that thiS tErM forcEs syNONYMous WorDs frOm the In-batch ATTRACt coNstrAIntS to be CloseR to oNe AnothEr than To any OtHer word in the curRent Mini-batch.\NThE sEcoNd Term pUShes anTonYms Away froM each otHEr. IF (xL, XR) ∈ bR Is the current minibaTcH OF RePEL consTraintS, ThIs TErm is:\nR(Br) = ∑\n(Xl,xR)∈BR\n(rElU (δrpL + xlxR − XlTr)\n+ ReLU (δRpl + xlxR − XrTr)) 3a native SpEaker iS aBle To eAsily COme uP with tHese sets Of morPHological rules (OR at least with a REaSONaBLe suBseT of rules) witHout ANy liNguiSTiC trAIning. what iS mORe, THe rules for DE, IT, and RU WeRe creaTed by Non-native, non-fLuent speakERS With a limIted KNoWLedge of the threE langUages, exempLIfying thE simpLicity anD portabilITY of the apProAch.\NEnGliSH geRman Italian\n(dISCuss, DiScussed) (SchOttisch, SchOttIscHem) (GoLfo, golfi) (lAugh, laugHiNg) (DaMaLigE, damaLIgen) (minaTo, MinAtA) (paCifisT, PacifiSts) (koMbinIeRe, KOmbInierte) (MEtTERe, meTtO) (eVacuAte, EvAcuatEd) (scHWeiGt, schweIgst) (crescOno, CRescI) (eVaLuate, evAluates)
 The first term pulls theATTRACT ex ample s ( xl,  x r) ∈  A c loser together .  IfBA denotes the current  mini -b a tcho fATTRA CT exam p le s ,  th is  t erm  c a nbe ex pre ssed as :\nA(BA) =  ∑\ n( xl,xr)∈BA\n( R eL U (δatt +xlt l − xlxr)\n+  Re LU (δa tt  +x rtr −  xl xr))\ nwhere δatt i s the sim il a rity m a rgin wh i c hdete rmines how much c l os e r synonymous v ectors  s h ou l d  be  to  each othe rthant o eacho ft h e irr espective neg ative examp l es.  ReLU( x)  =m ax(0,x) is  t h e s tandard rec tifi ed linear  unit( Nair an d  Hinton , 2010 ).The  ‘ne g at iv e’ex a mpl e  t i f o r e ach word  x iin an y AT T R A C T pa iris t he wo rd vector clo ses t to xiamong  theexam pl es in  the c urren tminibatch (dist inct  from its  ta rg etsy nonym and xi  it sel f). Thi s means tha tt h i sterm forces synony mo u s  w ords fro m thei n- ba t ch ATTRA CT  co nstr a i nts t o be cl oser toone an o th er  than t oany ot he r w ord  in t h e cu rrentmini-bat ch.\n T he second term pushes antony m sa w ay from  ea ch other. I f (x l , xr ) ∈B Rist he cu rrent  m i ni b atch of REPEL const ra ints,thisterm is:\nR(B R) = ∑\n(x l , x r)∈BR\n( ReLU (δ r pl + xlxr − xl tr)\n + ReLU (δr p l + xlxr  − xr tr)) 3Anative sp e a ker is a ble  to  ea sil y co me up with th e s e se ts  of mor pho logical  ru les  (o r a tleast wit h a reas on ab le  s ubs et of rules) w it hou tany  ling u istictrain ing.  W ha t  is  more,t he r ules  f or  DE,  IT ,and R U we r e c reatedby non-na tiv e , no n- fl uent sp eakers with a  l imited kno wl edg e of t h e  three l anguages, exemplifyingt he simp lic ity a nd p ortabilit y o f theapp r oach.\ nEngli sh Ge rm anI t alian \ n (d isc us s, discuss e d ) ( schot ti sch,  schott ischem) (golfo, go l fi)  (laugh, laug hin g) ( d a ma lig e ,d ama li g en) ( minato, minata)  (pacifist ,p ac ifists) (k o mbi ni ere, ko mbinier te) ( m ettere,  metto) ( evacuate,  e vacu a t ed)  (schweigt , schwei gst) (cre s cono, cr esci)  (e valuat e,  ev aluat es)
 The_first term_pulls the ATTRACT examples_(xl, xr)_∈_A closer_together._If BA denotes_the current mini-batch_of ATTRACT examples, this_term can be_expressed_as:\nA(BA) = ∑\n(xl,xr)∈BA\n(ReLU (δatt + xltl − xlxr)\n+ ReLU (δatt + xrtr − xlxr))\nwhere_δatt_is the_similarity_margin_which determines how much closer_synonymous vectors should be to_each other_than to each of their respective negative examples._ReLU(x)_= max(0, x)_is the standard rectified linear unit (Nair and Hinton,_2010). The ‘negative’ example ti for_each word xi_in_any_ATTRACT pair is the_word vector closest to xi among_the examples in the current minibatch_(distinct from its target synonym and xi_itself). This means that this term_forces synonymous words from the_in-batch ATTRACT_constraints to be closer to_one another than_to any_other word in_the current mini-batch.\nThe second term pushes_antonyms away from_each other. If (xl, xr) ∈_BR_is the current_minibatch_of_REPEL constraints,_this term is:\nR(BR)_=_∑\n(xl,xr)∈BR\n(ReLU (δrpl_+_xlxr − xltr)\n+ ReLU (δrpl +_xlxr_− xrtr)) 3A native speaker is able_to easily come up_with_these sets of morphological_rules (or at least with_a reasonable subset of rules) without_any linguistic_training. What_is more, the rules for DE, IT, and RU were created_by non-native, non-fluent speakers with a_limited knowledge of the_three languages,_exemplifying_the simplicity and_portability_of the_approach.\nEnglish German Italian\n(discuss, discussed) (schottisch, schottischem) (golfo,_golfi) (laugh,_laughing) (damalige, damaligen) (minato, minata) (pacifist,_pacifists) (kombiniere, kombinierte) (mettere,_metto)_(evacuate, evacuated) (schweigt, schweigst) (crescono, cresci)_(evaluate, evaluates)