class_docstr: This class loads in the Facebook bAbI goal oriented dialog dataset and
  vectorizes them into user utterances, bot utterances, and answers.
  
  As described in: "Learning End-to-End Goal Oriented Dialog".
  https://arxiv.org/abs/1605.07683.
  
  For a particular task, the class will read both train and test files
  and combine the vocabulary.
  
  Args:
      path (str): Directory to store the dataset
      task (str): a particular task to solve (all bAbI tasks are train
          and tested separately)
      oov (bool, optional): Load test set with out of vocabulary entity words
      use_match_type (bool, optional): Flag to use match-type features
      use_time (bool, optional): Add time words to each memory, encoding when the
          memory was formed
      use_speaker_tag (bool, optional): Add speaker words to each memory
          (<BOT> or <USER>) indicating who spoke each memory.
      cache_match_type (bool, optional): Flag to save match-type features after processing
      cache_vectorized (bool, optional): Flag to save all vectorized data after processing
  
  Attributes:
      data_dict (dict): Dictionary containing final vectorized train, val, and test datasets
      cands (np.array): Vectorized array of potential candidate answers, encoded
      as integers, as returned by BABI_Dialog class. Shape = [num_cands, max_cand_length]
      num_cands (int): Number of potential candidate answers.
      max_cand_len (int): Maximum length of a candidate answer sentence in number of words.
      memory_size (int): Maximum number of sentences to keep in memory at any given time.
      max_utt_len (int): Maximum length of any given sentence / user utterance
      vocab_size (int): Number of unique words in the vocabulary + 2 (0 is reserved for
          a padding symbol, and 1 is reserved for OOV)
      use_match_type (bool, optional): Flag to use match-type features
      kb_ents_to_type (dict, optional): For use with match-type features, dictionary of
          entities found in the dataset mapping to their associated match-type
      kb_ents_to_cand_idxs (dict, optional): For use with match-type features, dictionary
          mapping from each entity in the  knowledge base to the set of indicies in the
          candidate_answers array that contain that entity.
      match_type_idxs (dict, optional): For use with match-type features, dictionary
          mapping from match-type to the associated fixed index of the candidate vector
          which indicated this match type.
class_docstr_tok: ["This","class","loads","in","the","Facebook","bAbI","goal","oriented","dialog","dataset","and","vectorizes","them","into","user","utterances",",","bot","utterances",",","and","answers",".","As","described","in",":","``","Learning","End-to-End","Goal","Oriented","Dialog","''",".","https",":","\/\/arxiv.org\/abs\/1605.07683",".","For","a","particular","task",",","the","class","will","read","both","train","and","test","files","and","combine","the","vocabulary",".","Args",":","path","(","str",")",":","Directory","to","store","the","dataset","task","(","str",")",":","a","particular","task","to","solve","(","all","bAbI","tasks","are","train","and","tested","separately",")","oov","(","bool",",","optional",")",":","Load","test","set","with","out","of","vocabulary","entity","words","use_match_type","(","bool",",","optional",")",":","Flag","to","use","match-type","features","use_time","(","bool",",","optional",")",":","Add","time","words","to","each","memory",",","encoding","when","the","memory","was","formed","use_speaker_tag","(","bool",",","optional",")",":","Add","speaker","words","to","each","memory","(","<","BOT",">","or","<","USER",">",")","indicating","who","spoke","each","memory",".","cache_match_type","(","bool",",","optional",")",":","Flag","to","save","match-type","features","after","processing","cache_vectorized","(","bool",",","optional",")",":","Flag","to","save","all","vectorized","data","after","processing","Attributes",":","data_dict","(","dict",")",":","Dictionary","containing","final","vectorized","train",",","val",",","and","test","datasets","cands","(","np.array",")",":","Vectorized","array","of","potential","candidate","answers",",","encoded","as","integers",",","as","returned","by","BABI_Dialog","class",".","Shape","=","[","num_cands",",","max_cand_length","]","num_cands","(","int",")",":","Number","of","potential","candidate","answers",".","max_cand_len","(","int",")",":","Maximum","length","of","a","candidate","answer","sentence","in","number","of","words",".","memory_size","(","int",")",":","Maximum","number","of","sentences","to","keep","in","memory","at","any","given","time",".","max_utt_len","(","int",")",":","Maximum","length","of","any","given","sentence","\/","user","utterance","vocab_size","(","int",")",":","Number","of","unique","words","in","the","vocabulary","+","2","(","0","is","reserved","for","a","padding","symbol",",","and","1","is","reserved","for","OOV",")","use_match_type","(","bool",",","optional",")",":","Flag","to","use","match-type","features","kb_ents_to_type","(","dict",",","optional",")",":","For","use","with","match-type","features",",","dictionary","of","entities","found","in","the","dataset","mapping","to","their","associated","match-type","kb_ents_to_cand_idxs","(","dict",",","optional",")",":","For","use","with","match-type","features",",","dictionary","mapping","from","each","entity","in","the","knowledge","base","to","the","set","of","indicies","in","the","candidate_answers","array","that","contain","that","entity",".","match_type_idxs","(","dict",",","optional",")",":","For","use","with","match-type","features",",","dictionary","mapping","from","match-type","to","the","associated","fixed","index","of","the","candidate","vector","which","indicated","this","match","type","."]
code: def __init__(
      self,
      path=".",
      task=1,
      oov=False,
      use_match_type=False,
      use_time=True,
      use_speaker_tag=True,
      cache_match_type=False,
      cache_vectorized=False,
  ):
      self.url = "http://www.thespermwhale.com/jaseweston/babi"
      self.size = 3032766
      self.filename = "dialog-bAbI-tasks.tgz"
      self.path = path
      self.task = task - 1
      self.oov = oov
      self.use_match_type = use_match_type
      self.cache_vectorized = cache_vectorized
      self.match_type_vocab = None
      self.match_type_idxs = None
  
      self.tasks = [
          "dialog-babi-task1-API-calls-",
          "dialog-babi-task2-API-refine-",
          "dialog-babi-task3-options-",
          "dialog-babi-task4-phone-address-",
          "dialog-babi-task5-full-dialogs-",
          "dialog-babi-task6-dstc2-",
      ]
  
      print("Preparing bAbI-dialog dataset. Looking in ./%s" % path)
  
      assert task in range(
          1, 7
      ), "given task is not in the bAbI-dialog dataset"
  
      print("Task is %s" % (self.tasks[self.task]))
  
      (
          self.train_file,
          self.dev_file,
          self.test_file,
          self.cand_file,
          self.kb_file,
          self.vocab_file,
          self.vectorized_file,
      ) = self.load_data()
  
      # Parse files into sets of dialogue and user/bot utterance pairs
      self.train_dialog = self.parse_dialog(
          self.train_file, use_time, use_speaker_tag
      )
      self.dev_dialog = self.parse_dialog(
          self.dev_file, use_time, use_speaker_tag
      )
      self.test_dialog = self.parse_dialog(
          self.test_file, use_time, use_speaker_tag
      )
  
      self.candidate_answers_w = self.load_candidate_answers()
  
      if self.use_match_type:
          self.kb_ents_to_type = self.load_kb()
          self.kb_ents_to_cand_idxs = self.create_match_maps()
      else:
          self.kb_ents_to_type = None
          self.kb_ents_to_cand_idxs = None
  
      self.compute_statistics()
  
      if self.use_match_type:
          self.encode_match_feats()
  
      if self.cache_vectorized and os.path.exists(self.vectorized_file):
          print(
              "Loading cached vectorized data from: {}".format(
                  self.vectorized_file
              )
          )
          with open(self.vectorized_file, "rb") as f:
              (
                  self.train,
                  self.dev,
                  self.test,
                  self.cands,
              ) = pickle.load(f)
  
      else:
          self.train = self.vectorize_stories(self.train_dialog)
          self.dev = self.vectorize_stories(self.dev_dialog)
          self.test = self.vectorize_stories(self.test_dialog)
          self.cands = self.vectorize_cands(self.candidate_answers_w)
  
          if self.cache_vectorized:
              print(
                  "Caching vectorized data at {}".format(
                      self.vectorized_file
                  )
              )
              with open(self.vectorized_file, "wb") as f:
                  pickle.dump(
                      (self.train, self.dev, self.test, self.cands), f
                  )
  
      self.data_dict["train"] = {
          "memory": {
              "data": self.train[0],
              "axes": ("batch", "memory_axis", "sentence_axis"),
          },
          "memory_mask": {
              "data": self.train[1],
              "axes": ("batch", "memory_axis"),
          },
          "user_utt": {
              "data": self.train[2],
              "axes": ("batch", "sentence_axis"),
          },
          "answer": {
              "data": self.train[3],
              "axes": ("batch", "cand_axis"),
          },
      }
  
      self.data_dict["dev"] = {
          "memory": {
              "data": self.dev[0],
              "axes": ("batch", "memory_axis", "sentence_axis"),
          },
          "memory_mask": {
              "data": self.dev[1],
              "axes": ("batch", "memory_axis"),
          },
          "user_utt": {
              "data": self.dev[2],
              "axes": ("batch", "sentence_axis"),
          },
          "answer": {
              "data": self.dev[3],
              "axes": ("batch", "cand_axis"),
          },
      }
  
      self.data_dict["test"] = {
          "memory": {
              "data": self.test[0],
              "axes": ("batch", "memory_axis", "sentence_axis"),
          },
          "memory_mask": {
              "data": self.test[1],
              "axes": ("batch", "memory_axis"),
          },
          "user_utt": {
              "data": self.test[2],
              "axes": ("batch", "sentence_axis"),
          },
          "answer": {
              "data": self.test[3],
              "axes": ("batch", "cand_axis"),
          },
      }
  
      # Add question-specific candidate answers if we are using match-type
      if self.use_match_type:
          self.data_dict["train"]["cands_mat"] = {
              "data": self.create_cands_mat("train", cache_match_type),
              "axes": ("batch", "cand_axis", "REC"),
          }
          self.data_dict["dev"]["cands_mat"] = {
              "data": self.create_cands_mat("dev", cache_match_type),
              "axes": ("batch", "cand_axis", "REC"),
          }
          self.data_dict["test"]["cands_mat"] = {
              "data": self.create_cands_mat("test", cache_match_type),
              "axes": ("batch", "cand_axis", "REC"),
          }
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  from __future__ import unicode_literals
  import itertools
  import os
  import pickle
  import sys
  import tarfile
  import numpy
  from tqdm import tqdm
  from nlp_architect.utils.generic import license_prompt
  from nlp_architect.utils.io import download_unlicensed_file, valid_path_append
label_desc: Use image node_id 1 to create a new BABI_Dialog object from inherited base classes: object with example: obj = BABI_Dialog(path, task, oov, use_match_type, use_time, use_speaker_tag, cache_match_type, cache_vectorized)