anchor.diff:  
   
   
  -Subsection: NLI and machine translation. 
  +
  +
  +
  +
  +=== NLI and machine translation. 
   Why is NLI important ? It's a key problem in natural language understanding. Many other NLU problems can be reduced to NLI: such as summarization 
   (given a piece of text and a suggested summary, does the former entail the latter), information extraction (does the text entail the extracted fact), 
   question answering (does the data source entail a given question and answer pair) as well as machine machine translation 
  
positive.code: So why hs DL been more successful in ASR and vision than NLP/NLU? That's a topic for another post! 
  
  
  
  [1] 
  [2] http://arxiv.org/pdf/1606.01933v1.pdf [A Decomposable Attention Model for Natural Language Inference] 
  [2]
  [3]
  [4]
  [5] Karpathy
  [6] Word2Vec
  [7] Glove
  [8] Google Translate
  [9] Smart Reply
  [10] http://www.foldl.me/2016/solving-language/
  
  
negative.code: = Natural language modeling: How deep is too deep?
  :hp-tags: Deep Learning, NLP
  
  *TL;DR* Is Deep Learning always the best tool for Natural Language Understanding tasks? Not necessarily!
  
  Recently, a paper from Facebook AI Research (FAIR) appeared on arXiv, under the intriguing title https://arxiv.org/pdf/1607.01759v2.pdf[_Bag of Tricks for Efficient Text Classification_][1], promptly catching the NLP community's attentuion. Even more intruguingly, FAIR soon followed up with an open source implementation a.k.a. https://github.com/facebookresearch/fastText[fastText][2]. In this work (essentially a simple modification of the unsupervised https://en.wikipedia.org/wiki/Word2vec[Word2Vec] algorithm to deal with supervised learning tasks), the authors made a convincing case for the frequent superiority of *shallow* 
  text representations - as opposed to deep ones - for common NLU tasks such as sentence classification, dispelling the myth that "deeper is always better".  
  Furthermore, this paper is but one example in the recent slew of results casting the "silver bullet" efficacy of complex neural architectures into question for text-related tasks (see some examples here[3], here[4] and here[5]). 
  
  Hold on, you might say, isn't Deep Learning a new disruptive force in AI, shown beyond doubt to be clearly superior to prior "shallow"
  learning approaches? Well, it depends who you ask. Ask a Computer Vision or a Speech Recognition expert, and you'll get and enthusiastic Yes!
  In computer vision, novel DL archirectures (such as VGG, GoogleNet, Inception, etc.) have delivered extremely impressive 
  results on ImageNet, CIFAR, even defying expectations of some computer vision DL champions[5]. In speech, commercial heavyweights such as 
  Google and Baidu have longs since switch to DL architectures. It should only be natural to expect, then, to see the 
  same trend in NLP/NLU, correct? 
  Well, not so fast. 
  
  The tide of enthusiasm in Deep Learning has of course spilled over to the NLU community, triggering a massive conversion of both 
  academics and industry practitioners to the newfound DL religion. Impressive results from other fields, 
  helped by the success of the seminal Word2Vec[6] (followed by Glove[7] and the like) was too much to resist. RNN and LSTM have since become mainstream techniques, offered by 
  popular DL libraries such as TensorFlow, Keras, DL4J, etc. Among other big companies, Google has been at the forefront of both open-sourcing DL techniques (with https://www.tensorflow.org/[TensorFlow]) and adopting DL architectures in production (see, e.g., SmartReply[9]). 
  
  So, you ask, isn't that enough? Well, dear friend: if you are reading this, chances are, you are not Google! And as a result, you 
  don't have the same massive amounts of training data, nor their virtually unlimited computational resources. To the rest of us, 
  it is important to understand the performance/computation tradeoffs that come with DL  -  which is what this post is really about. 
  
  So let us look at some key problems one by one. In this post, we will focus on text classification. 
  
  === Тext Classification
  Clearly, text classification is a common task with plenty of applications: search, user intent determination, sentiment analysis, topic modeling 
  (slightly different, but close), seqeuence labeling (related), etc. 
  For text classification, we typically have 3 options: 
  A) Logit or RF with bag of words or averaged word vectors.  
  B) Shallow neural networks such as FastText
  C) Deep networks such as RNNs/LSTMs or CNNs. 
  
  