anchor.diff:  
   '''
   
  -As shown above, the method of distillation relies on the idea of *_soft targets_* and *_softmax temperature_*.
  -
  -'''
  -
  -.Softmax Function with Temperature
  -image::https://github.com/anshu92/blog/raw/gh-pages/images/tempsoftmax.png[Model Distillation,align="center"]
  -
  -'''
  -
  -.hard vs soft targets
  -image::https://github.com/anshu92/blog/raw/gh-pages/images/distill3.png[Targets,align="center"]
  -
  -'''
  -
  -https://arxiv.org/pdf/1503.02531.pdf[Hinton et al., 2015] introduced the concept of *"softmax temperature"*. The probability p~i~ of class i is calculated from the logits z. T is the temperature parameter - as you can see, setting T = 1 gives us the standard softmax function. In the paper, they also found that if the ground truth labels are available, it improves the model if the loss to optimize includes training on ground truth labels in addition to the soft target outputs of the teacher model(as shown in the figure, student loss is added to distillation loss).
  -
  -On a philosophical line of thought, the paper presents an analogy of some insects in nature that optimize based on their stage-of-life requirement - larval forms to extract nutrition from environment and an adult form for travelling and reproduction; whereas cumbersome, large and often ensemble models remain the same during training and deployment. A conceptual block that may explain our hesitation of modifying trained models is our assumption that changing learned parameters may lead to loss of knowledge. Another assumption made is that the optimizing an objective function for the training set is close to optimizing the user's true objective - _generalizing well to unseen data._
  -
  -Ideal ways to generalize to the user objective requires knowledge that might not be available in the training data and is hard to model. Distillation can help. Using the output of the teacher model allows the student model to learn not just about the correct class(as it would in the case of hard labels) but also the relative incorrectness of the other classes. For instance, knowledge that a cat is closer to a tiger than a carrot is valuable for generalization of the model. Quoting from the paper - 
  +## Hard vs. Soft Targets
   
   [quote, Hinton et al., 2015]
   ____
  
positive.code: 
  Most machine learning competitions are won by ensemble of models that each understand the training set in a different way. They are often made as different(architectures, initializations, subsets of data) as possible to minimize correlations between their errors. It is also useful that each of these models should overfit on the data. Although useful when processing time is not a hindrance (like while producing an output for a non-real time competition), ensembles are impractical in most real world test/production times. They also encode very little knowledge per param which is not ideal for an efficient model.
  
  .Analogy. graphic by Anshuman Sahoo
  image::https://github.com/anshu92/blog/raw/gh-pages/images/distill1.png[Analogy]
  
  
negative.code: less data than the original cumbersome model and using a much higher learning rate.
  ____
  
  .hard vs soft targets
  image::https://github.com/anshu92/blog/raw/gh-pages/images/distill3.png[Targets,align="center"]
  
  '''
  
  As you can see in Fig 3., whereas hard targets encode the label of the correct class, soft targets encode a lot more knowledge. For instance, the fact that a cow is 1000 times closer to a dog than a car.
  
  
  