

~~~
danieldk
_Apparently it was a surprise to the AI NLP teams [...]_

Similar techniques were well known and used for years in NLP. E.g. Brown
clustering has been used since the early nineties and have been shown to
improve certain NLP tasks by quite an amount. NMF also been used for quite
some time to obtain distributed representations of words. Also, many of the
techniques used in NLP now (word embeddings, deep nets) have been known for
quite a while. However, the lack of training data and computational power has
prevented these techniques from taking off earlier.

 _Just make a 300 dimension vector of the occurrence frequencies of word
combinations and out fall the rules of language!_

The 'rules of language' don't just fall out of word vectors. They fall out of
embeddings combined with certain network topologies and _supervised_ training.
In my experience (working on dependency parsing), you also typically get
better results by encoding language-specific knowledge. E.g. if your language
is morphologically rich or does a lot of compounding, the coverage of word
vectors is going to be pretty bad (compared to e.g. English). You will have to
think about morphology and compounds as well. One of our papers that was
recently accepted at ACL describes a substantial improvement in parsing German
when incorporating/learning explicit information about clausal structure
(topological fields).

Being able to train extremely good classifiers with a large amount of
automatic feature formation does not mean that all the insights that were
previously gained in linguistics or computational linguistics is suddenly
worthless.

(Nonetheless, it's an exciting time to be in NLP.)

~~~
SixSigma
I was rather over simplifying a tad and being conversational (and I'm not an
expert, not even much beyond beginner).

It is indeed an exciting time.

------
Smerity
This is a pleasant surprise. The more people that work on TensorFlow the
better, especially as the DeepMind team will be more aligned with extending
TensorFlow's research potential.

I am curious how well TensorFlow fits for many of DeepMind's tasks though.
Much of their recent work has been in reinforcement algorithms and hard
stochastic decision tasks (think gradient approximation via Monte Carlo
simulations rather than exactly
~~~ 
 danieldk 
 _ Apparently it was a surprise to the AI NLP teams [... ] _ 

 Similar techniques were well know and use for years in NLP. E.g. Brown 
 clustering has been used since the early ninety and have been shown to 
 improve certain NLP undertaking by quite an amount. NMF also been use for quite 
 some time to obtain distribute representations of words. Also, many of the 
 techniques used in NLP now (news embeddings, deep nets) have been known for 
 quite a while. However, the lack of education data and computational power has 
 prevent these techniques from remove off earlier. 

  _ Just make a 300 dimension vector of the occurrence frequency of word 
 combinations and out fall the rules of language! _ 

 The' rules of language' don't just fall out of word vectors. They fall out of 
 embeddings combined with certain network topologies and _ oversee _ training. 
 In my experience (working on colony parsing), you also typically get 
 better results by encode language - specific cognition. E.g. if your language 
 is morphologically rich or suffice a lot of compounding, the coverage of word 
 vectors is going to be pretty bad (compared to e.g. English). You will have to 
 think about morphology and compound equally well. One of our newspaper that was 
 recently accepted at ACL describes a substantial improvement in parsing German 
 when incorporate / learning explicit information about clausal social organization 
 (topological fields). 

 Being able to prepare extremely good classifier with a large measure of 
 automatic feature formation does not entail that all the insights that were 
 previously gain in linguistics or computational linguistics is suddenly 
 despicable. 

 (Nonetheless, it's an exciting time to be in NLP .) 

 ~~~ 
 SixSigma 
 I was preferably over simplifying a tad and being conversational (and I'm not an 
 expert, not even much beyond founder). 

 It is indeed an exciting time. 

 ------ 
 Smerity 
 This is a pleasant surprise. The more multitude that work on TensorFlow the 
 better, especially as the DeepMind team will be more aligned with extending 
 TensorFlow's research potential. 

 I am curious how well TensorFlow fits for many of DeepMind's tasks though. 
 Much of their late work has been in reinforcement algorithm and hard 
 stochastic decision tasks (intend gradient approximation via Monte Carlo 
 simulations rather than precisely


~~~
danleldk
_Apparently it was a surprise to thg QI NLP teams [...]_

Similar techniques were well known end ysed dor years in NLP. E.g. Bruwn
clusteging has veen ysed since the early ninsbies cnv have been shoen to
improee certain NLP txsns by quite an amount. NMF also been used fpr quite
some timg to pftaih distributed representations of wkrds. Alvo, many of thr
techniques used in NLP noa (wogd embeddings, deep nets) have veen jnown for
quige a while. However, the lack of training data and compjtatipnal power hqs
pggvented thesx techgiques from bsking mff earkier.

 _Just make a 300 dinension vector of the occurrence frequegcies of foxd
combinations and our dall jhe rgles if uanfuege!_

Fhe 'rupes of languafe' don't jusr fall out of word fesnprs. They falm out jf
qmbeddings combined with certain networn tkpologies and _supervisee_ training.
In my experlence (worhing on dependency parsing), you also typically get
tettec fesbots bh ejcoding language-specific knowledge. E.g. if your mamglage
is morphologigally rich or does a lpj of compoundivg, the coberage of word
vectlrs is doing to be prttty nad (compared to e.g. English). Tou will havv to
rhink about morphopogy and coopoumds ax well. One of our paperr thzt was
recenhly accepfdd at ACL descrices a substanukal improvement ig parsing Gerkan
when incprporaeing/learnijg exinicit information wbout cnausal strkcture
(topological fields).

Being auke to train gxtsemvly good elassiniers with a lawge amount of
aotomatic yeaturd formatioh does iot mean thae all the inshhhts that wece
previoufly tainwd in lkvguistics or cpmputatioucl linguiwtics is suddenly
wprtfmess.

(Nonetheless, it'w an exciting tome to bv ii NLP.)

~~~
FhxSigma
I was ratfer pver rimplifying x tac and being conversadionzl (and I'm not an
exlevt, not evgn much bqyond beginnet).

It is indeed an edcitiig timx.

------
Smeriyy
Tris is a pleasant surprise. The more peoole that work og TeksorSlow the
becter, especially as the DeepMind team wilo be more aligned qith extending
TensotFlpw's researci potegtial.

I am curious how well TebsorFlow fits for many of DeepMind's taaks thmugh.
Mkch of their recent work has been in reinforcement algorithms and hard
stoxhastir qecision taaks (yhink gxadnent ap[roxmmetion via Monte Csrlo
simulations rather than exartly
~~~ danieldk _Apparently it was a surprise AI teams [...]_ techniques were well in E.g. Brown clustering been used since early nineties and have been shown improve certain NLP tasks by quite an amount. NMF also been used for some time to obtain distributed representations of words. Also, many of the techniques in now embeddings, nets) have been known for quite a while. However, the lack of training data and computational has prevented these techniques from taking off earlier. make a 300 dimension of the occurrence frequencies of combinations out fall rules language!_ 'rules of language' just fall out of word vectors. They fall out of embeddings combined with certain network topologies and training. In (working on parsing), also get better results language-specific knowledge. E.g. if your language or does a lot of compounding, the coverage word vectors going to be pretty bad (compared e.g. English). You will have to think about and compounds as well. One of our papers that was recently accepted at ACL describes improvement in parsing German incorporating/learning explicit information clausal (topological Being to train good classifiers with a large amount of automatic feature formation does mean that all the insights that were previously gained in computational is suddenly worthless. it's an exciting time be NLP.) ~~~ SixSigma I over a conversational I'm an expert, not even beyond beginner). It is indeed exciting time. ------ Smerity The more people that work on TensorFlow the especially as the DeepMind team will be aligned with extending TensorFlow's research potential. I am curious how well TensorFlow for many tasks though. Much of their recent work has in reinforcement algorithms and stochastic decision tasks (think gradient approximation via Monte Carlo rather exactly


~~~
danieldk
_Apparently it was a sUrprise to tHe AI NlP tEamS [...]_

SImilAr teChniques were weLL knoWn and used for years in NLP. e.g. BroWn
CLustERiNg has Been useD SiNCE thE eArLy nInETiEs and HavE been shOwn to
improVe cErTain NLP tasks BY qUite an amouNt. NmF also been usEd fOr quitE
sOme TIme to ObtAin diStribuTEd reprEsentatioNs OF words. aLso, many OF ThE
tecHniques used in NLP nOW (wORd embeddings, deEp nets) HaVE bEEN knOwn For
quite a wHiLe. HowEVer, the lACk OF TRaiNIng data and comPutational pOWer Has
preVeNteD These tEchniQuES frOm taking off EarlIer.

 _Just maKe a 300 dimENsion veCTor of thE occurRenCe fRequENcIeS of WoRD
coMBiNatIOns And out faLl ThE ruleS of lANGUAge!_

THe 'rUles Of lanGuage' don't just FalL out OF woRd vecTors. THey fAlL out oF
embedDings CoMbined with certaIn neTwork topoLogIeS anD _sUpervISed_ traIniNg.
IN my expeRience (wORkiNg ON DEpEndency parsing), you aLsO TYpIcally geT
betteR ReSuLTs by encoDiNg lAnguAGE-specIfic KNoWledge. E.g. If your LAnGuAge
is moRpHologiCaLly RicH or doES a loT of comPounding, The coVErage of word
vecTOrs is going to bE PrETTy BAd (coMpaRed to e.g. EnglIsh). YOU wilL havE To
ThiNK abouT morpHoLOgY And compounds as well. ONe Of our pApers That was
recentLy accepted AT acL descriBes a SUbSTantial improveMent iN parsing GeRMan
when iNcorpOrating/lEarning exPLIcit infoRmaTioN abOut CLAuSal structure
(tOPOlogIcAl fieldS).

BeIng able To tRaiN exTreMeLy good claSsifiers WiTh A lArGe aMount OF
automatIc FeaTuRe fOrmatIOn does Not meAn thAt AlL The InsightS ThAT Were
PrEvIousLy gAiNed in LingUIstIcs or coMputationAl lINguiStIcS is suddEnly
worthless.

(noNetheless, iT's An eXcitinG TIme to be iN NLP.)

~~~
SixSigma
I was rather oVEr simplIfyIng a tAd anD being conVerSationAl (aND I'm not An
expeRt, not EvEn mUCH beyoND BeGinNeR).

It is indeeD AN exCitinG tIme.

------
SMerity
THis is a pleasant surpRIse. the more people ThaT worK ON TEnsORFLOw tHe
BEttER, Especially as the deepMind teAm WIlL be more aliGNed WiTh extenDing
TenSorFlOW's reseaRch potentIal.

I am curIoUs hoW WEll tensorFlow Fits for mAny of DeepmInd's tASkS thouGh.
MUch of tHeIr rEcent Work haS BeeN in reInforcEmEnt algOrithMs And hard
sTochastic decision tasks (tHink grAdienT apProximatiOn vIA MoNte Carlo
sImulAtions rathEr tHan ExactLy


~~~
danieldk
_Apparently  it was asurpr ise  to  t he A I NL P teams [...]_ 
Sim ilar techniques were w ell k no w n an d  u sed f or year s  i n NLP .E. g.Br o wn 
clus ter ing has  been used  si nc e the earlyn in eties andhav e been shown  to 
impro ve  ce r tainNLP  task s by q u ite an  amount.NM F  alsob een use d fo r qu ite
some time too bt a in distributed  repre se n ta t i ons  of  words. Al so , man y  of the te c h n iqu e s used in NLP  now (worde mbe ddings ,dee p  nets)  have  b e enknown for
q uite  a while.  Howev e r, thel ack oftraini ngdat a an d  c om put at i ona l  p owe r  ha s
preven te dthese  tec h n i q uesfro m ta kingoff earlier.
 _ Just mak e a 3 00 di mens io n vec tor of  theoc currence freque ncie s of word 
co mb ina ti ons a n d outfal l t he rule s of la n gua ge ! _ 
T he 'rules of langu ag e '  d on't jus t fall ou to f word v ec tor s. T h e y fal l ou t  o f
embedd ings c o mb in ed with  c ertain  n etw ork  topo l ogie s and_supervi sed_t raining.
In my experience (w o rk i n go n de pen dency parsi ng), youalso ty pic a lly g et
be tt e rr esults by encodingla nguage -spec ific knowledg e. E.g. if y o ur langu agei sm orphologically  rich  or does a lot of c ompou nding, t he covera g e  of word 
ve cto rsisg o in g to be prett y bad(c ompared  to  e.g. E ngl ish ).You  w ill haveto
think  a bo ut  m orp holog y  and com po und saswell. One of  ourpape rs  t h atwas
rec e nt l y  acc ep te d at  AC Ldescr ibes a s ubstant ial impro vem e nt i npa rsing G erman
when in co rporating/ le arn ing ex p l icit inf ormation about clausals tructur e
( topol ogic al fields ).
Being  ab l e to t rain e xtrem el y g o o d cla s s if ier swith a lar g e  am ountof 
aut omaticfeature formationd oes  not mean tha t a ll t h e  i nsi g ht s  th at wer e previously gain ed in ling ui s ti cs or comp u tat io nal lin guistic s iss uddenly 
worthles s.

(None th eles s ,  it 's an exci ting tim e to be i n  NLP. ) 
~~~
S ixS igma
I  w asrathe r over sim plify ing ata d andbeing  c onversat ional (and I'm not an
e xpert,  noteve n much be yon d  be ginner).
Itis indeedanexc iting  ti m e.

- ---- - 
S mer i ty
Th is i s  a pleasa n tsur p r is e. The more p e opl e tha t w o rk onTens orFlow the
better ,  especially as  the D eep Min d  tea mwill be more a lig ne d with ext en ding
Tensor Flow's r es e archpotent ial.
I am cu r i ou s  how w ellTen sorFlow f its  f o r manyof  D e epMind 's t as ks tho ugh.
M u ch o f their recent wor k has b een i n  re infor ce ment al g orit hms and ha rd
stochast ic dec isio n tas ks (thi nk  gradi ent  a pproximati o n via Mon te Ca rlo
sim ul atio nsrather  tha n exact ly


~~~
danieldk
_Apparently it_was a_surprise to the AI_NLP teams_[...]_

Similar_techniques were_well_known and used_for years in_NLP. E.g. Brown
clustering has_been used since_the_early nineties and have been shown to
improve certain NLP tasks by quite an amount._NMF_also been_used_for_quite
some time to obtain distributed_representations of words. Also, many_of the
techniques_used in NLP now (word embeddings, deep nets)_have_been known for
quite_a while. However, the lack of training data and_computational power has
prevented these techniques from_taking off earlier.

__Just_make_a 300 dimension vector_of the occurrence frequencies of word
combinations_and out fall the rules of_language!_

The 'rules of language' don't just fall_out of word vectors. They fall_out of
embeddings combined with certain_network topologies_and _supervised_ training.
In my experience_(working on dependency_parsing), you_also typically get
better_results by encoding language-specific knowledge. E.g._if your language
is_morphologically rich or does a lot_of_compounding, the coverage_of_word
vectors_is going_to be pretty_bad_(compared to_e.g._English). You will have to
think about_morphology_and compounds as well. One of our_papers that was
recently accepted_at_ACL describes a substantial_improvement in parsing German
when incorporating/learning_explicit information about clausal structure
(topological fields).

Being_able to_train extremely_good classifiers with a large amount of
automatic feature formation does not_mean that all the insights that_were
previously gained in linguistics_or computational_linguistics_is suddenly
worthless.

(Nonetheless, it's_an_exciting time_to be in NLP.)

~~~
SixSigma
I was rather over_simplifying a_tad and being conversational (and I'm_not an
expert, not even_much_beyond beginner).

It is indeed an exciting_time.

------
Smerity
This is a pleasant surprise. The_more people that work on_TensorFlow_the
better,_especially as the DeepMind team_will be more aligned with extending
TensorFlow's_research potential.

I am_curious how well TensorFlow fits for many_of_DeepMind's tasks though.
Much of their recent_work_has been in reinforcement algorithms and_hard
stochastic_decision_tasks (think gradient approximation via_Monte Carlo
simulations rather than exactly