sum_{k=1}^Y \exp(\mathbf{W}_k \cdot \mathbf{x}+ \mathbf{b}_k) }
\end{split}$$ Where $\mathbf{x}$ denotes a feature vector, $\mathbf{W}_k$ and $\mathbf{b}_k$ are the weight vectors and bias for class $k$. In our case, we treat words as classes to be predicted. Calculating the denominator thus requires summing over all of the words in the (large) vocabulary, an expensive operation. To avoid this computational bottleneck, we approximate the term $P(w_i|\mathbf{e}_k)$ with  Hierarchical Softmax.[^3]

To learn meaningful user embeddings, we seek representations that are predictive of individual word-usage patterns. In light of this motivation, we approximate $P(w_i|\mathbf{u}_j)$ via the following hinge-loss objective which we aim to minimize:

(w\_i,\_j) &=\
& \_[w\_l V, w\_l S]{}(0,1 - \_i + \_l )

Here, each $w_l$ (and corresponding embedding, $\mathbf{e}_l$) is a *negative example*, i.e., a word not in the sentence under consideration, which was authored by user $j$. The intuition is that in the aggregate, such words are less likely to be employed by user $j$ than words observed in sentences she has authored. Thus minimizing this objective attempts to induce a representation that is discriminative with respect to word usage.

In practice, $V$ will be very large and hence we approximate the objective via *negative sampling*, a variant of Noise Contrastive Estimation. The idea is to approximate the objective function in a binary classification task by learning to discriminate between observed positive examples (sampled from the true distribution) and *pseudo*-negative examples (sampled from a large space of predominantly negative instances). Intuitively, this shifts probability mass to plausible observations. See for notes on Negative Sampling and Noise Contrastive Estimation.

Previous work by showed that this approach works well in representation learning tasks when a sufficient amount of training data is available. However, we have access to only a limited amount of text for each user (see Section \[sec:experiments\]). We hypothesize that this problem can be alleviated by carefully selecting the negative samples that mostly contribute to ‘push’ the
sum_{k=1}^Y \exp(\mathbf{W}_k \cdot \mathbf{x}+ \mathbf{b}_k) } 
 \end{split}$$ Where $ \mathbf{x}$ denotes a feature vector, $ \mathbf{W}_k$ and $ \mathbf{b}_k$ are the weight vectors and diagonal for course $ k$. In our casing, we treat actor's line as class to be predicted. Calculating the denominator therefore requires summing over all of the words in the (bombastic) vocabulary, an expensive operation. To avoid this computational constriction, we approximate the term $ P(w_i|\mathbf{e}_k)$ with    Hierarchical Softmax.[^3 ] 

 To learn meaningful user embeddings, we search representations that are predictive of individual word - usage patterns. In light of this motivation, we estimate $ P(w_i|\mathbf{u}_j)$ via the following hinge - loss objective which we aim to understate: 

 (w\_i,\_j) & = \ 
 & \_[w\_l V, w\_l S]{}(0,1 - \_i + \_l) 

 Here, each $ w_l$ (and corresponding embedding, $ \mathbf{e}_l$) is a * negative example *, i.e., a word not in the conviction under consideration, which was authored by user $ j$. The intuition is that in the aggregate, such words are less probable to be employed by user $ j$ than words observed in sentences she has authored. therefore minimize this objective attempts to induce a representation that is discriminative with respect to word usage. 

 In practice, $ V$ will be very large and hence we approximate the aim via * damaging sampling *, a version of Noise Contrastive Estimation. The theme is to approximate the objective function in a binary classification undertaking by learning to discriminate between observed positive examples (sample from the true distribution) and * pseudo*-negative examples (sampled from a large space of predominantly negative instances). Intuitively, this shifts probability mass to plausible observations. See for notes on Negative Sampling and Noise Contrastive Estimation. 

 former work by show that this overture works well in representation learning tasks when a sufficient sum of training data is available. However, we have entree to only a circumscribed amount of text for each user (see Section \[sec: experiments\ ]). We hypothesize that this trouble can be alleviated by carefully selecting the negative samples that largely contribute to ‘ advertise ’ the
sum_{n=1}^Y \exp(\mathbf{W}_k \cdot \mathnf{x}+ \mathbf{b}_k) }
\end{split}$$ Wiere $\mafhbf{x}$ devotes a feature vector, $\mathbh{W}_k$ qnd $\mqthbf{b}_k$ are the weight vectors wnd bias for xlass $k$. In our casc, we fveat cocds as classes jo be predicded. Calculatinc ghz denominator thus requires summing jver alk lf the words ig tht (lwrge) nogabulary, an expensive operation. Fo avoiv this computatoonal bottleneck, we approxlmatf the term $P(w_i|\mathhf{e}_k)$ with  Huerawxhical Softmxx.[^3]

To learn meaningful oser embeddings, we seek represengatious that are peedlwtive of invividuwl word-usage patternv. In libht of this mobivatmon, qe approximate $P(w_i|\matibf{u}_j)$ via the followyng hinge-noas objective whicy qe aik to minknizd:

(w\_i,\_n) &=\
& \_[w\_m V, w\_l S]{}(0,1 - \_i + \_l )

Here, each $w_l$ (ane corresponding embtddybg, $\mathbf{e}_l$) ia a *nedaeive example*, i.e., a word not in the sentetce under consideration, whuch was authored by uder $j$. The intuition is that in the aggregate, such words ara lesa lijejh tl be employed by user $j$ than words observed ig stntvnces she has autmored. Thus minimizonh yris objective attem'fs to induce a repredentatijn thqt is dissrimonative with respect to wore usage.

In prcctuce, $V$ will be very large and kence ee aplroximate the objective via *negative swmpling*, a xariant of Noise Coktrdstive Estimation. The idea is to ap'roxikate thd obkectivq function in a binary classificatlon tcsk bf learning to discriminate between observxv positive exsm[lev (sampleb from the true diseribution) and *kseudo*-negctive dxamples (szmpled hrom a large space of preglminantly nejative inftanxes). Untuitixdly, this shiftx probability mass ro plausible obseryatiova. See for notes on Negative Samplong anq Jomse Cjttrastive Esdimagiov.

Lreviuus work by rhowrd that this approacv wodks well in represrnbation leqrning twsks when a sifficient amount ov tramning vata ix adailable. However, we have accesa to only a jimited amougt on tevt for eack user (see Section \[sec:experiments\]). We hypithesize that this problem can be allgvisted by carxfully selectinc the negative samplws that mostly coktribute to ‘push’ the
sum_{k=1}^Y \exp(\mathbf{W}_k \cdot \mathbf{x}+ \mathbf{b}_k) } \end{split}$$ denotes feature vector, and $\mathbf{b}_k$ are for $k$. In our we treat words classes to be predicted. Calculating the thus requires summing over all of the words in the (large) vocabulary, an operation. To avoid this computational bottleneck, we approximate the term $P(w_i|\mathbf{e}_k)$ with Hierarchical To meaningful embeddings, seek representations that are predictive of individual word-usage patterns. In light of this motivation, we approximate via the following hinge-loss objective which we aim minimize: (w\_i,\_j) &=\ & V, w\_l S]{}(0,1 - \_i \_l Here, each (and embedding, is a *negative i.e., a word not in the sentence under consideration, which was authored by user $j$. The intuition that in such words less to employed by user words observed in sentences she has this objective attempts to induce a representation that discriminative with to word usage. In practice, $V$ be very large and hence we approximate the via *negative sampling*, a variant of Noise Contrastive Estimation. The idea is to approximate the in a binary classification by learning to between positive (sampled the true and *pseudo*-negative examples (sampled from a large space of predominantly negative Intuitively, this shifts probability mass to plausible observations. See for Negative and Noise Contrastive Previous work by showed this works well in representation when sufficient data available. we have access to a limited amount of text each user (see Section problem can be alleviated by carefully selecting the samples that mostly contribute to ‘push’ the
sum_{k=1}^Y \exp(\mathbf{W}_k \cdot \mathbF{x}+ \mathbf{b}_k) }
\End{spLit}$$ wheRe $\MathBf{x}$ dEnotes a feature VEctoR, $\mathbf{W}_k$ and $\mathbf{b}_k$ arE the wEiGHt veCToRs and Bias for CLaSS $K$. In OuR cAse, We TReAt worDs aS classeS to be prediCteD. CAlculating thE DeNominator tHus Requires summIng Over alL oF thE Words In tHe (larGe) vocaBUlary, aN expensivE oPEratioN. to avoid THIs CompUtational bottleneCK, wE Approximate the Term $P(w_I|\mAThBF{E}_k)$ wIth  hierarchicAl softmAX.[^3]

To learN MeANINgfUL user embeddinGs, we seek repREseNtatioNs ThaT Are preDictiVe OF inDividual worD-usaGe patternS. In ligHT of this MOtivatiOn, we apProXimAte $P(W_I|\mAtHbf{U}_j)$ VIa tHE fOllOWinG hinge-loSs ObJectiVe whICH WE aim To mInimIze:

(w\_i,\_J) &=\
& \_[w\_l V, w\_l S]{}(0,1 - \_i + \_l )

Here, EacH $w_l$ (aND coRrespOndinG embEdDing, $\mAthbf{e}_L$) is a *nEgAtive example*, i.e., a Word Not in the sEntEnCe uNdEr conSIderatIon, WhiCh was auThored bY UseR $j$. tHE InTuition is that in the AgGREgAte, such wOrds arE LeSs LIkely to bE eMplOyed BY User $j$ Than WOrDs observEd in seNTeNcEs she haS aUthoreD. THus MinImiziNG thiS objecTive atteMpts tO Induce a represeNTation that is dIScRIMiNAtivE wiTh respect to Word USage.

in prACtIce, $v$ Will bE very LaRGe ANd hence we approximatE tHe objeCtive Via *negative saMpling*, a varIANT of Noise contRAsTIve Estimation. THe ideA is to approXImate the ObjecTive funcTion in a biNARy classiFicAtiOn tAsk BY LeArning to discrIMInatE bEtween oBseRved posItiVe eXamPleS (sAmpled froM the true DiStRiBuTioN) and *pSEudo*-negaTiVe eXaMplEs (samPLed froM a larGe spAcE oF PreDominanTLy NEGatiVe InStanCes). inTuitiVely, THis Shifts pRobabilitY maSS to pLaUsIble obsErvations. See fOr Notes on NegAtIve sampliNG And Noise contrastive Estimation.

PrEVious woRk bY showEd thAt this appRoaCh workS weLL in repResentAtion LeArnING taskS WHeN a sUfFicient amoUNT of TrainInG datA is avaiLable. However, we have ACceSs to only a limiTed AmouNT Of TexT FoR EacH uSEr (sEE section \[sec:experIments\]). We hyPoTHeSize that thIS prObLem can bE alleviAted bY CarefulLy selectiNg the negaTiVe saMPLes That mostly ContribuTe to ‘push’ tHE
sum_{k=1}^Y \exp(\mathbf{W }_k \cdot\math bf{ x}+  \ math bf{b }_k) }
\end{sp l it}$ $ Where $\mathbf{x}$ d enote sa  fea t ur e vec tor, $\ m at h b f{W }_ k$  an d$ \m athbf {b} _k$ are  the weigh t v ec tors and bia s  f or class $ k$.  In our case , w e trea twor d s ascla ssesto bep redict ed. Calcu la t ing th e  denomi n a to r th us requires summi n go ver all of the  words  i n  t h e  (l arg e) vocabul ar y, an expensi v eo p e rat i on. To avoidthis comput a tio nal bo tt len e ck, we  appr ox i mat e the term$P(w _i|\mathb f{e}_k ) $ withHierarc hicalSof tma x.[^ 3 ]
T o l ea r n m e an ing f uluser emb ed di ngs,we s e e k repr ese ntat ionsthat are pred ict iveo f i ndivi dualword -u sagepatter ns. I nlight of this m otiv ation, we  ap pr oxi ma te $P ( w_i|\m ath bf{ u}_j)$via the fol lo w i n ghinge-loss objecti ve w hi ch we ai m to m i ni mi z e:

(w\_ i, \_j ) &= \ & \_[ w\_l V,  w\_l S] {}(0,1 -\_ i + \_l  ) 

Here ,eac h $ w_l$( andcorres pondingembed d ing, $\mathbf{ e }_l$) is a *n e ga t i ve exam ple *, i.e., aword notin t h esen t enceunder  c o ns i deration, which was  a uthore d byuser $j$. The  intuition i s  that in  the ag g regate, such w ordsare less l i kely tobe em ployed b y user $j $ than wor dsobs erv edi n  s entences sheh a s au th ored. T hus  minimi zin g t his  ob je ctive att empts to  i nd uc ea r epres e ntationth atis  di scrim i native  with  res pe ct toword us a ge . 
Inpr ac tice , $ V$  will  bev ery  largeand hence  we appr ox im ate the  objective vi a*negativesa mpl ing*,a variantof Noise Contrastive Es t imation . T he id ea i s to appr oxi mate t heo bjecti ve fun ction  i n a b inary c la ssi fi cation tas k bylearn in g to  discri minate between obs e rve d positive ex amp les( s am ple d  f r omth e  tr u e  distribution)and *pseud o* - ne gative exa m ple s(sample d froma lar g e space  of predo minantlyne gati v e  in stances).Intuitiv ely, this shift s  p robab ili ty mas stoplaus ible o b ser vatio ns. Se efor no tes o nNegative  Sampling and Noise Con trasti ve Es tim ation.

P rev i ous  work byshow ed that th isapp roach  wo r ks we ll i n  r epr e senta tion learningt as ksw h en  a sufficie n t amo unt o f t r aining  dat a is available. H o wever, we have  acc e s s t o o n ly a  l imited amountofte x t  for eac huser (see S ection \ [s e c:exp erimen ts\]).  We hyp o t he s ize th at t his  problemcan  b e  allevi at ed by car eful ly  selec ting t h e ne g a tive samples tha t mos t l y con t rib ute t o‘push’t he
sum_{k=1}^Y \exp(\mathbf{W}_k_\cdot \mathbf{x}+_\mathbf{b}_k) }
\end{split}$$ Where $\mathbf{x}$_denotes a_feature_vector, $\mathbf{W}_k$_and_$\mathbf{b}_k$ are the_weight vectors and_bias for class $k$._In our case,_we_treat words as classes to be predicted. Calculating the denominator thus requires summing over_all_of the_words_in_the (large) vocabulary, an expensive_operation. To avoid this computational_bottleneck, we_approximate the term $P(w_i|\mathbf{e}_k)$ with  Hierarchical Softmax.[^3]

To learn_meaningful_user embeddings, we_seek representations that are predictive of individual word-usage patterns._In light of this motivation, we_approximate $P(w_i|\mathbf{u}_j)$ via_the_following_hinge-loss objective which we_aim to minimize:

(w\_i,\_j) &=\
& \_[w\_l V,_w\_l S]{}(0,1 - \_i + \_l_)

Here, each $w_l$ (and corresponding embedding, $\mathbf{e}_l$)_is a *negative example*, i.e., a_word not in the sentence_under consideration,_which was authored by user_$j$. The intuition_is that_in the aggregate,_such words are less likely to_be employed by_user $j$ than words observed in_sentences_she has authored._Thus_minimizing_this objective_attempts to induce_a_representation that_is_discriminative with respect to word usage.

In_practice,_$V$ will be very large and hence_we approximate the objective_via_*negative sampling*, a variant_of Noise Contrastive Estimation. The_idea is to approximate the objective_function in_a binary_classification task by learning to discriminate between observed positive examples (sampled_from the true distribution) and *pseudo*-negative_examples (sampled from a_large space_of_predominantly negative instances)._Intuitively,_this shifts_probability mass to plausible observations. See for_notes on_Negative Sampling and Noise Contrastive Estimation.

Previous_work by showed that_this_approach works well in representation learning_tasks when a sufficient amount of_training data is available. However,_we_have_access to only a limited_amount of text for each user_(see Section \[sec:experiments\])._We hypothesize that this problem can be_alleviated_by carefully selecting the negative samples_that_mostly contribute to ‘push’ the