class_docstr: This Teacher is responsible for performing the task of Dialogue State Tracking.
  
  It can be used to evaluate LM on JGA (Joint Goal Accuracy) metric (as shown in
  [SimpleTOD](https://arxiv.org/abs/2005.00796) and
  [Soloist](https://arxiv.org/abs/2005.05298)).
class_docstr_tok: ["This","Teacher","is","responsible","for","performing","the","task","of","Dialogue","State","Tracking",".","It","can","be","used","to","evaluate","LM","on","JGA","(","Joint","Goal","Accuracy",")","metric","(","as","shown","in","[","SimpleTOD","]","(","https",":","\/\/arxiv.org\/abs\/2005.00796",")","and","[","Soloist","]","(","https",":","\/\/arxiv.org\/abs\/2005.05298",")",")","."]
docstr: for dialog state tracking, we compute the joint goal accuracy, which is the
  percentage of the turns where the model correctly and precisely predicts all
  slots(domain, slot_type, slot_value).
docstr_tok: ["for","dialog","state","tracking",",","we","compute","the","joint","goal","accuracy",",","which","is","the","percentage","of","the","turns","where","the","model","correctly","and","precisely","predicts","all","slots","(","domain",",","slot_type",",","slot_value",")","."]
code: def custom_evaluation(
      self, teacher_action: Message, labels, model_response: Message
  ):
      """
      for dialog state tracking, we compute the joint goal accuracy, which is the
      percentage of the turns where the model correctly and precisely predicts all
      slots(domain, slot_type, slot_value).
      """
      resp = model_response.get("text")
      if not resp:
          return
  
      # extract ground truth from labels
      (
          slots_truth,
          slots_truth_per_domain,
          slots_truth_named_entity,
      ) = self._extract_slot_from_string(labels[0])
  
      # extract generated slots from model_response
      (
          slots_pred,
          slots_pred_per_domain,
          slots_pred_named_entity,
      ) = self._extract_slot_from_string(resp)
  
      for gt_slot in slots_truth:
          self.metrics.add(
              "all/slot_r", AverageMetric(gt_slot in slots_pred)
          )
          curr_domain = gt_slot.split("--")[0]
          self.metrics.add(
              f"{curr_domain}/slot_r",
              AverageMetric(gt_slot in slots_pred),
          )
  
      for gt_slot in slots_pred_named_entity:
          self.metrics.add(
              "hallucination",
              AverageMetric(gt_slot not in slots_truth_named_entity),
          )
  
      for predicted_slot in slots_pred:
          self.metrics.add(
              "all/slot_p", AverageMetric(predicted_slot in slots_truth)
          )
          curr_domain = predicted_slot.split("--")[0]
          self.metrics.add(
              f"{curr_domain}/slot_p",
              AverageMetric(predicted_slot in slots_truth),
          )
  
      self.metrics.add(
          "jga", AverageMetric(set(slots_truth) == set(slots_pred))
      )
      self.metrics.add(
          "named_entities/jga",
          AverageMetric(
              set(slots_truth_named_entity)
              == set(slots_pred_named_entity)
          ),
      )
      for gt_slot in slots_truth_named_entity:
          self.metrics.add(
              "all_ne/slot_r", AverageMetric(gt_slot in slots_pred)
          )
          curr_domain = gt_slot.split("--")[0]
          self.metrics.add(
              f"{curr_domain}_ne/slot_r",
              AverageMetric(gt_slot in slots_pred),
          )
      for predicted_slot in slots_pred_named_entity:
          self.metrics.add(
              "all_ne/slot_p",
              AverageMetric(predicted_slot in slots_truth),
          )
          curr_domain = predicted_slot.split("--")[0]
          self.metrics.add(
              f"{curr_domain}_ne/slot_p",
              AverageMetric(predicted_slot in slots_truth),
          )
  
      for domain in slots_truth_per_domain:
          if domain in slots_pred_per_domain:
              self.metrics.add(
                  f"{domain}/jga",
                  AverageMetric(
                      slots_truth_per_domain[domain]
                      == slots_pred_per_domain[domain]
                  ),
              )
  
imports: import copy
  import json
  import os
  from typing import Optional
  import numpy
  import pandas
  import parlai.core.tod.tod_agents
  import parlai.core.tod.tod_core
  import parlai.tasks.multiwoz_v22.build
  from parlai.core.message import Message
  from parlai.core.metrics import AverageMetric
  from parlai.core.opt import Opt
  from parlai.core.params import ParlaiParser
  from parlai.utils.data import DatatypeHelper
  from parlai.utils.io import PathManager
  from parlai.utils.testing import is_this_circleci
label_desc: Use image node_id 5 for calling the MultiWOZv22DSTTeacher obj's underlying member method code with example usage: obj.custom_evaluation(teacher_action, labels, model_response) without return types