anchor.diff:  '''
   
   ## Hard vs. Soft Targets
  +
  +https://arxiv.org/pdf/1503.02531.pdf[Hinton et al., 2015] introduced the concept of *"softmax temperature"*(Fig1). The probability p~i~ of class i is calculated from the logits z. T is the temperature parameter - as you can see, setting T = 1 gives us the standard softmax function. In the paper, they also found that if the ground truth labels are available, it improves the model if the loss to optimize includes training on ground truth labels in addition to the soft target outputs of the teacher model(as shown in the figure, student loss is added to distillation loss).
  +
  +'''
  +.Softmax Function with Temperature
  +image::https://github.com/anshu92/blog/raw/gh-pages/images/tempsoftmax.png[Model Distillation,align="center"]
   
   [quote, Hinton et al., 2015]
   ____
  
positive.code: 
  ## Can we mine the knowledge in a training set by collecting a pile of dirt before filtering it for gold nuggets, i.e, can we transfer knowledge learnt by an ensemble of models into a single efficient one?
  
  https://arxiv.org/pdf/1503.02531.pdf[Hinton et al., 2015] introduced the concept of *"softmax temperature"*(Fig1). The probability p~i~ of class i is calculated from the logits z. T is the temperature parameter - as you can see, setting T = 1 gives us the standard softmax function. In the paper, they also found that if the ground truth labels are available, it improves the model if the loss to optimize includes training on ground truth labels in addition to the soft target outputs of the teacher model(as shown in the figure, student loss is added to distillation loss).
  
  '''
  .Softmax Function with Temperature
  image::https://github.com/anshu92/blog/raw/gh-pages/images/tempsoftmax.png[Model Distillation,align="center"]
  
  '''
  
  On a philosophical line of thought, the paper presents an analogy of some insects in nature that optimize based on their stage-of-life requirement - larval forms to extract nutrition from environment and an adult form for travelling and reproduction; whereas cumbersome, large and often ensemble models remain the same during training and deployment. A conceptual block that may explain our hesitation of modifying trained models is our assumption that changing learned parameters may lead to loss of knowledge. Another assumption made is that the optimizing an objective function for the training set is close to optimizing the user's true objective - _generalizing well to unseen data._
  
  .Analogy. graphic by Anshuman Sahoo
  image::https://github.com/anshu92/blog/raw/gh-pages/images/distill1.png[Analogy]
  
  
negative.code: As you can see in Fig 3., whereas hard targets encode the label of the correct class, soft targets encode a lot more knowledge. For instance, the fact that a cow is 1000 times closer to a dog than a car is captured by the high temperature softmax output of the ensemble.
  
  
  In experiments involving MNIST, a smaller model of fully connected layers 784x800x800x10 produces 146 test errors. A larger model of fully connected layers 784x1200x1200x10 when trained using dropout, weight constraints and input jitter achieves 67 test errors. When the smaller model is trained on the soft output of the larger model, it achieves an error score of 74 errors. It achieves this without the need for the regularizing steps like dropout, jitter etc. This shows almost all the "dark" knowledge that helps the larger model generalize is successfully passed on to the smaller model. For instance, the larger model learns a similarity metric for the digit classes which helps the smaller model distinguish between them much easier.
  
  Another surprising result from MNIST is that if one class, say 3, is left out of the training set, and the smaller model is trained on the outputs of the larger. If we then adjust the bias of class 3 (this is needed because the smaller model has never seen 3, and assumes it to be improbable), we can actually get 98.6% of the test cases for 3 by the knowledge we learn about it from the other classes. In a similar way, if we only train on 7s and 8s, and lower their biases appropriately, the smaller model achieves 87% across all 10 classes!
  
  
  