class_docstr: Distillation from a teacher to student network.
  
  First, a teacher is trained on a task; Second, a student is trained to perform
  the task while matching the teacher's softened outputs. For more details, see
  the paper below.
  
  In the hparams passed to this model include the desired
  {teacher/student}_model and {teacher/student}_hparams to be used. Also,
  specify the distillation temperature and task-distillation balance.
  
  Distilling the Knowledge in a Neural Network
  Hinton, Vinyals and Dean
  https://arxiv.org/abs/1503.02531
class_docstr_tok: ["Distillation","from","a","teacher","to","student","network",".","First",",","a","teacher","is","trained","on","a","task",";","Second",",","a","student","is","trained","to","perform","the","task","while","matching","the","teacher","'s","softened","outputs",".","For","more","details",",","see","the","paper","below",".","In","the","hparams","passed","to","this","model","include","the","desired","{","teacher\/student","}","_model","and","{","teacher\/student","}","_hparams","to","be","used",".","Also",",","specify","the","distillation","temperature","and","task-distillation","balance",".","Distilling","the","Knowledge","in","a","Neural","Network","Hinton",",","Vinyals","and","Dean","https",":","\/\/arxiv.org\/abs\/1503.02531"]
code: def body(self, features):
      hp = self.hparams
      is_distill = hp.distill_phase == "distill"
  
      targets = features["targets_raw"]
      targets = tf.squeeze(targets, [1, 2, 3])
      one_hot_targets = tf.one_hot(
          targets, hp.num_classes, dtype=tf.float32
      )
  
      # Teacher Network
      with tf.variable_scope("teacher"):
          teacher_outputs = self.teacher_model.body(features)
          tf.logging.info(
              "teacher output shape: %s" % teacher_outputs.get_shape()
          )
          teacher_outputs = tf.reduce_mean(teacher_outputs, axis=[1, 2])
          teacher_logits = tf.layers.dense(
              teacher_outputs, hp.num_classes
          )
  
          teacher_task_xent = (
              tf.nn.softmax_cross_entropy_with_logits_v2(
                  labels=one_hot_targets, logits=teacher_logits
              )
          )
          outputs = teacher_logits
  
      if is_distill:
          # Load teacher weights
          tf.train.init_from_checkpoint(
              hp.teacher_dir, {"teacher/": "teacher/"}
          )
          # Do not train the teacher
          trainable_vars = tf.get_collection_ref(
              tf.GraphKeys.TRAINABLE_VARIABLES
          )
          del trainable_vars[:]
  
      # Student Network
      if is_distill:
          with tf.variable_scope("student"):
              student_outputs = self.student_model.body(features)
              tf.logging.info(
                  "student output shape: %s"
                  % student_outputs.get_shape()
              )
              student_outputs = tf.reduce_mean(
                  student_outputs, axis=[1, 2]
              )
              student_logits = tf.layers.dense(
                  student_outputs, hp.num_classes
              )
  
              student_task_xent = (
                  tf.nn.softmax_cross_entropy_with_logits_v2(
                      labels=one_hot_targets, logits=student_logits
                  )
              )
              teacher_targets = tf.nn.softmax(
                  teacher_logits / hp.distill_temperature
              )
              student_distill_xent = (
                  tf.nn.softmax_cross_entropy_with_logits_v2(
                      labels=tf.stop_gradient(teacher_targets),
                      logits=student_logits / hp.distill_temperature,
                  )
              )
              # scale soft target obj. to match hard target obj. scale
              student_distill_xent *= hp.distill_temperature**2
  
              outputs = student_logits
  
              # Summaries
              tf.summary.scalar("distill_xent", student_distill_xent)
  
      if not is_distill:
          phase_loss = teacher_task_xent
      else:
          phase_loss = hp.task_balance * student_task_xent
          phase_loss += (1 - hp.task_balance) * student_distill_xent
  
      losses = {"training": phase_loss}
      outputs = tf.reshape(outputs, [-1, 1, 1, 1, outputs.shape[1]])
  
      return outputs, losses
  
file_path: tensor2tensor/tensor2tensor/models/distillation.py
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  from tensor2tensor.layers import common_hparams
  from tensor2tensor.utils import registry
  from tensor2tensor.utils import t2t_model
  import tensorflow.compat.v1
  from tensorflow.compat.v1 import estimator
label_desc: Use image node_id 2 for calling the Distillation obj's underlying member method code with example usage: obj.body(features) and returns: outputs,  losses