class_docstr: A ``WikiTablesErmSemanticParser`` is a :class:`WikiTablesSemanticParser` that learns to search
  for logical forms that yield the correct denotations.
  
  Parameters
  ----------
  vocab : ``Vocabulary``
  question_embedder : ``TextFieldEmbedder``
      Embedder for questions. Passed to super class.
  action_embedding_dim : ``int``
      Dimension to use for action embeddings. Passed to super class.
  encoder : ``Seq2SeqEncoder``
      The encoder to use for the input question. Passed to super class.
  entity_encoder : ``Seq2VecEncoder``
      The encoder to used for averaging the words of an entity. Passed to super class.
  attention : ``Attention``
      We compute an attention over the input question at each step of the decoder, using the
      decoder hidden state as the query.  Passed to WikiTablesDecoderStep.
  decoder_beam_size : ``int``
      Beam size to be used by the ExpectedRiskMinimization algorithm.
  decoder_num_finished_states : ``int``
      Number of finished states for which costs will be computed by the ExpectedRiskMinimization
      algorithm.
  max_decoding_steps : ``int``
      Maximum number of steps the decoder should take before giving up. Used both during training
      and evaluation. Passed to super class.
  normalize_beam_score_by_length : ``bool``, optional (default=False)
      Should we normalize the log-probabilities by length before renormalizing the beam? This was
      shown to work better for NML by Edunov et al., but that many not be the case for semantic
      parsing.
  checklist_cost_weight : ``float``, optional (default=0.6)
      Mixture weight (0-1) for combining coverage cost and denotation cost. As this increases, we
      weigh the coverage cost higher, with a value of 1.0 meaning that we do not care about
      denotation accuracy.
  use_neighbor_similarity_for_linking : ``bool``, optional (default=False)
      If ``True``, we will compute a max similarity between a question token and the `neighbors`
      of an entity as a component of the linking scores.  This is meant to capture the same kind
      of information as the ``related_column`` feature. Passed to super class.
  dropout : ``float``, optional (default=0)
      If greater than 0, we will apply dropout with this probability after all encoders (pytorch
      LSTMs do not apply dropout to their last layer). Passed to super class.
  num_linking_features : ``int``, optional (default=10)
      We need to construct a parameter vector for the linking features, so we need to know how
      many there are.  The default of 10 here matches the default in the ``KnowledgeGraphField``,
      which is to use all ten defined features. If this is 0, another term will be added to the
      linking score. This term contains the maximum similarity value from the entity's neighbors
      and the question. Passed to super class.
  rule_namespace : ``str``, optional (default=rule_labels)
      The vocabulary namespace to use for production rules.  The default corresponds to the
      default used in the dataset reader, so you likely don't need to modify this. Passed to super
      class.
  tables_directory : ``str``, optional (default=/wikitables/)
      The directory to find tables when evaluating logical forms.  We rely on a call to SEMPRE to
      evaluate logical forms, and SEMPRE needs to read the table from disk itself.  This tells
      SEMPRE where to find the tables. Passed to super class.
  mml_model_file : ``str``, optional (default=None)
      If you want to initialize this model using weights from another model trained using MML,
      pass the path to the ``model.tar.gz`` file of that model here.
class_docstr_tok: ["A","``","WikiTablesErmSemanticParser","``","is","a",":","class",":","`","WikiTablesSemanticParser","`","that","learns","to","search","for","logical","forms","that","yield","the","correct","denotations",".","Parameters","--","--","--","--","--","vocab",":","``","Vocabulary","``","question_embedder",":","``","TextFieldEmbedder","``","Embedder","for","questions",".","Passed","to","super","class",".","action_embedding_dim",":","``","int","``","Dimension","to","use","for","action","embeddings",".","Passed","to","super","class",".","encoder",":","``","Seq2SeqEncoder","``","The","encoder","to","use","for","the","input","question",".","Passed","to","super","class",".","entity_encoder",":","``","Seq2VecEncoder","``","The","encoder","to","used","for","averaging","the","words","of","an","entity",".","Passed","to","super","class",".","attention",":","``","Attention","``","We","compute","an","attention","over","the","input","question","at","each","step","of","the","decoder",",","using","the","decoder","hidden","state","as","the","query",".","Passed","to","WikiTablesDecoderStep",".","decoder_beam_size",":","``","int","``","Beam","size","to","be","used","by","the","ExpectedRiskMinimization","algorithm",".","decoder_num_finished_states",":","``","int","``","Number","of","finished","states","for","which","costs","will","be","computed","by","the","ExpectedRiskMinimization","algorithm",".","max_decoding_steps",":","``","int","``","Maximum","number","of","steps","the","decoder","should","take","before","giving","up",".","Used","both","during","training","and","evaluation",".","Passed","to","super","class",".","normalize_beam_score_by_length",":","``","bool","``",",","optional","(","default=False",")","Should","we","normalize","the","log-probabilities","by","length","before","renormalizing","the","beam","?","This","was","shown","to","work","better","for","NML","by","Edunov","et","al.",",","but","that","many","not","be","the","case","for","semantic","parsing",".","checklist_cost_weight",":","``","float","``",",","optional","(","default=0.6",")","Mixture","weight","(","0-1",")","for","combining","coverage","cost","and","denotation","cost",".","As","this","increases",",","we","weigh","the","coverage","cost","higher",",","with","a","value","of","1.0","meaning","that","we","do","not","care","about","denotation","accuracy",".","use_neighbor_similarity_for_linking",":","``","bool","``",",","optional","(","default=False",")","If","``","True","``",",","we","will","compute","a","max","similarity","between","a","question","token","and","the","`","neighbors","`","of","an","entity","as","a","component","of","the","linking","scores",".","This","is","meant","to","capture","the","same","kind","of","information","as","the","``","related_column","``","feature",".","Passed","to","super","class",".","dropout",":","``","float","``",",","optional","(","default=0",")","If","greater","than","0",",","we","will","apply","dropout","with","this","probability","after","all","encoders","(","pytorch","LSTMs","do","not","apply","dropout","to","their","last","layer",")",".","Passed","to","super","class",".","num_linking_features",":","``","int","``",",","optional","(","default=10",")","We","need","to","construct","a","parameter","vector","for","the","linking","features",",","so","we","need","to","know","how","many","there","are",".","The","default","of","10","here","matches","the","default","in","the","``","KnowledgeGraphField","``",",","which","is","to","use","all","ten","defined","features",".","If","this","is","0",",","another","term","will","be","added","to","the","linking","score",".","This","term","contains","the","maximum","similarity","value","from","the","entity","'s","neighbors","and","the","question",".","Passed","to","super","class",".","rule_namespace",":","``","str","``",",","optional","(","default=rule_labels",")","The","vocabulary","namespace","to","use","for","production","rules",".","The","default","corresponds","to","the","default","used","in","the","dataset","reader",",","so","you","likely","do","n't","need","to","modify","this",".","Passed","to","super","class",".","tables_directory",":","``","str","``",",","optional","(","default=\/wikitables\/",")","The","directory","to","find","tables","when","evaluating","logical","forms",".","We","rely","on","a","call","to","SEMPRE","to","evaluate","logical","forms",",","and","SEMPRE","needs","to","read","the","table","from","disk","itself",".","This","tells","SEMPRE","where","to","find","the","tables",".","Passed","to","super","class",".","mml_model_file",":","``","str","``",",","optional","(","default=None",")","If","you","want","to","initialize","this","model","using","weights","from","another","model","trained","using","MML",",","pass","the","path","to","the","``","model.tar.gz","``","file","of","that","model","here","."]
docstr: Parameters
  ----------
  question : Dict[str, torch.LongTensor]
     The output of ``TextField.as_array()`` applied on the question ``TextField``. This will
     be passed through a ``TextFieldEmbedder`` and then through an encoder.
  table : ``Dict[str, torch.LongTensor]``
      The output of ``KnowledgeGraphField.as_array()`` applied on the table
      ``KnowledgeGraphField``.  This output is similar to a ``TextField`` output, where each
      entity in the table is treated as a "token", and we will use a ``TextFieldEmbedder`` to
      get embeddings for each entity.
  world : ``List[WikiTablesWorld]``
      We use a ``MetadataField`` to get the ``World`` for each input instance.  Because of
      how ``MetadataField`` works, this gets passed to us as a ``List[WikiTablesWorld]``,
  actions : ``List[List[ProductionRuleArray]]``
      A list of all possible actions for each ``World`` in the batch, indexed into a
      ``ProductionRuleArray`` using a ``ProductionRuleField``.  We will embed all of these
      and use the embeddings to determine which action to take at each timestep in the
      decoder.
  example_lisp_string : ``List[str]``
      The example (lisp-formatted) string corresponding to the given input.  This comes
      directly from the ``.examples`` file provided with the dataset.  We pass this to SEMPRE
      when evaluating denotation accuracy; it is otherwise unused.
  metadata : ``List[Dict[str, Any]]``, optional, (default = None)
      Metadata containing the original tokenized question within a 'question_tokens' key.
docstr_tok: ["Parameters","--","--","--","--","--","question",":","Dict","[","str",",","torch.LongTensor","]","The","output","of","``","TextField.as_array","(",")","``","applied","on","the","question","``","TextField","``",".","This","will","be","passed","through","a","``","TextFieldEmbedder","``","and","then","through","an","encoder",".","table",":","``","Dict","[","str",",","torch.LongTensor","]","``","The","output","of","``","KnowledgeGraphField.as_array","(",")","``","applied","on","the","table","``","KnowledgeGraphField","``",".","This","output","is","similar","to","a","``","TextField","``","output",",","where","each","entity","in","the","table","is","treated","as","a","``","token","''",",","and","we","will","use","a","``","TextFieldEmbedder","``","to","get","embeddings","for","each","entity",".","world",":","``","List","[","WikiTablesWorld","]","``","We","use","a","``","MetadataField","``","to","get","the","``","World","``","for","each","input","instance",".","Because","of","how","``","MetadataField","``","works",",","this","gets","passed","to","us","as","a","``","List","[","WikiTablesWorld","]","``",",","actions",":","``","List","[","List","[","ProductionRuleArray","]","]","``","A","list","of","all","possible","actions","for","each","``","World","``","in","the","batch",",","indexed","into","a","``","ProductionRuleArray","``","using","a","``","ProductionRuleField","``",".","We","will","embed","all","of","these","and","use","the","embeddings","to","determine","which","action","to","take","at","each","timestep","in","the","decoder",".","example_lisp_string",":","``","List","[","str","]","``","The","example","(","lisp-formatted",")","string","corresponding","to","the","given","input",".","This","comes","directly","from","the","``",".examples","``","file","provided","with","the","dataset",".","We","pass","this","to","SEMPRE","when","evaluating","denotation","accuracy",";","it","is","otherwise","unused",".","metadata",":","``","List","[","Dict","[","str",",","Any","]","]","``",",","optional",",","(","default","=","None",")","Metadata","containing","the","original","tokenized","question","within","a","'question_tokens","'","key","."]
code: def forward(
      self,  # type: ignore
      question,
      table,
      world,
      actions,
      agenda,
      example_lisp_string,
      metadata=None,
  ):
      # pylint: disable=arguments-differ
      """
      Parameters
      ----------
      question : Dict[str, torch.LongTensor]
         The output of ``TextField.as_array()`` applied on the question ``TextField``. This will
         be passed through a ``TextFieldEmbedder`` and then through an encoder.
      table : ``Dict[str, torch.LongTensor]``
          The output of ``KnowledgeGraphField.as_array()`` applied on the table
          ``KnowledgeGraphField``.  This output is similar to a ``TextField`` output, where each
          entity in the table is treated as a "token", and we will use a ``TextFieldEmbedder`` to
          get embeddings for each entity.
      world : ``List[WikiTablesWorld]``
          We use a ``MetadataField`` to get the ``World`` for each input instance.  Because of
          how ``MetadataField`` works, this gets passed to us as a ``List[WikiTablesWorld]``,
      actions : ``List[List[ProductionRuleArray]]``
          A list of all possible actions for each ``World`` in the batch, indexed into a
          ``ProductionRuleArray`` using a ``ProductionRuleField``.  We will embed all of these
          and use the embeddings to determine which action to take at each timestep in the
          decoder.
      example_lisp_string : ``List[str]``
          The example (lisp-formatted) string corresponding to the given input.  This comes
          directly from the ``.examples`` file provided with the dataset.  We pass this to SEMPRE
          when evaluating denotation accuracy; it is otherwise unused.
      metadata : ``List[Dict[str, Any]]``, optional, (default = None)
          Metadata containing the original tokenized question within a 'question_tokens' key.
      """
      batch_size = list(question.values())[0].size(0)
      # Each instance's agenda is of size (agenda_size, 1)
      agenda_list = [agenda[i] for i in range(batch_size)]
      checklist_states = []
      all_terminal_productions = [
          set(instance_world.terminal_productions.values())
          for instance_world in world
      ]
      max_num_terminals = max(
          [len(terminals) for terminals in all_terminal_productions]
      )
      for (
          instance_actions,
          instance_agenda,
          terminal_productions,
      ) in izip(actions, agenda_list, all_terminal_productions):
          checklist_info = self._get_checklist_info(
              instance_agenda,
              instance_actions,
              terminal_productions,
              max_num_terminals,
          )
          (
              checklist_target,
              terminal_actions,
              checklist_mask,
          ) = checklist_info
          initial_checklist = checklist_target.new_zeros(
              checklist_target.size()
          )
          checklist_states.append(
              ChecklistState(
                  terminal_actions=terminal_actions,
                  checklist_target=checklist_target,
                  checklist_mask=checklist_mask,
                  checklist=initial_checklist,
              )
          )
      initial_info = self._get_initial_state_and_scores(
          question=question,
          table=table,
          world=world,
          actions=actions,
          example_lisp_string=example_lisp_string,
          add_world_to_initial_state=True,
          checklist_states=checklist_states,
      )
      initial_state = initial_info["initial_state"]
      # TODO(pradeep): Keep track of debug info. It's not straightforward currently because the
      # ERM's decode does not return the best states.
      outputs = self._decoder_trainer.decode(
          initial_state, self._decoder_step, self._get_state_cost
      )
      if not self.training:
          # TODO(pradeep): Can move most of this block to super class.
          linking_scores = initial_info["linking_scores"]
          feature_scores = initial_info["feature_scores"]
          similarity_scores = initial_info["similarity_scores"]
          batch_size = list(question.values())[0].size(0)
          action_mapping = {}
          for batch_index, batch_actions in enumerate(actions):
              for action_index, action in enumerate(batch_actions):
                  action_mapping[(batch_index, action_index)] = action[
                      0
                  ]
          outputs["action_mapping"] = action_mapping
          outputs["entities"] = []
          outputs["linking_scores"] = linking_scores
          if feature_scores is not None:
              outputs["feature_scores"] = feature_scores
          outputs["similarity_scores"] = similarity_scores
          outputs["logical_form"] = []
          best_action_sequences = outputs["best_action_sequences"]
          outputs["best_action_sequence"] = []
          outputs["debug_info"] = []
          agenda_indices = [
              actions_[:, 0].cpu().data for actions_ in agenda
          ]
          for i in range(batch_size):
              in_agenda_ratio = 0.0
              # Decoding may not have terminated with any completed logical forms, if `num_steps`
              # isn't long enough (or if the model is not trained enough and gets into an
              # infinite action loop).
              outputs["logical_form"].append([])
              if i in best_action_sequences:
                  for j, action_sequence in enumerate(
                      best_action_sequences[i]
                  ):
                      action_strings = [
                          action_mapping[(i, action_index)]
                          for action_index in action_sequence
                      ]
                      try:
                          logical_form = world[i].get_logical_form(
                              action_strings, add_var_function=False
                          )
                          outputs["logical_form"][-1].append(
                              logical_form
                          )
                      except ParsingError:
                          logical_form = "Error producing logical form"
                      if j == 0:
                          # Updating denotation accuracy and has_logical_form only based on the
                          # first logical form.
                          if logical_form.startswith("Error"):
                              self._has_logical_form(0.0)
                          else:
                              self._has_logical_form(1.0)
                          if example_lisp_string:
                              self._denotation_accuracy(
                                  logical_form, example_lisp_string[i]
                              )
                          outputs["best_action_sequence"].append(
                              action_strings
                          )
                  outputs["entities"].append(
                      world[i].table_graph.entities
                  )
                  instance_possible_actions = actions[i]
                  agenda_actions = []
                  for rule_id in agenda_indices[i]:
                      rule_id = int(rule_id)
                      if rule_id == -1:
                          continue
                      action_string = instance_possible_actions[
                          rule_id
                      ][0]
                      agenda_actions.append(action_string)
                  actions_in_agenda = [
                      action in action_strings
                      for action in agenda_actions
                  ]
                  if actions_in_agenda:
                      # Note: This means that when there are no actions on agenda, agenda coverage
                      # will be 0, not 1.
                      in_agenda_ratio = sum(actions_in_agenda) / len(
                          actions_in_agenda
                      )
              else:
                  outputs["best_action_sequence"].append([])
                  outputs["logical_form"][-1].append("")
                  self._has_logical_form(0.0)
                  if example_lisp_string:
                      self._denotation_accuracy(
                          None, example_lisp_string[i]
                      )
              self._agenda_coverage(in_agenda_ratio)
      if metadata is not None:
          outputs["question_tokens"] = [
              x["question_tokens"] for x in metadata
          ]
          outputs["original_table"] = [
              x["original_table"] for x in metadata
          ]
      return outputs
  
imports: from __future__ import division
  from __future__ import absolute_import
  import logging
  import os
  import torch
  from allennlp.data import Vocabulary
  from allennlp.data.fields.production_rule_field import ProductionRuleArray
  from allennlp.models.model import Model
  from allennlp.models.archival import load_archive, Archive
  from allennlp.models.semantic_parsing.wikitables.wikitables_decoder_state import WikiTablesDecoderState
  from allennlp.models.semantic_parsing.wikitables.wikitables_decoder_step import WikiTablesDecoderStep
  from allennlp.models.semantic_parsing.wikitables.wikitables_semantic_parser import WikiTablesSemanticParser
  from allennlp.modules import Attention, FeedForward, Seq2SeqEncoder, Seq2VecEncoder, TextFieldEmbedder
  from allennlp.nn.decoding import ChecklistState
  from allennlp.nn.decoding.decoder_trainers import ExpectedRiskMinimization
  from allennlp.semparse import ParsingError
  from allennlp.semparse.type_declarations import wikitables_type_declaration
  from allennlp.semparse.worlds import WikiTablesWorld
  from allennlp.training.metrics import Average
label_desc: Use image node_id 4 for calling the WikiTablesErmSemanticParser obj's underlying member method code with example usage: obj.forward(question, table, world, actions, agenda, example_lisp_string, metadata) and returns: outputs