class_docstr: This class loads in the Facebook bAbI goal oriented dialog dataset and
  vectorizes them into user utterances, bot utterances, and answers.
  
  As described in: "Learning End-to-End Goal Oriented Dialog".
  https://arxiv.org/abs/1605.07683.
  
  For a particular task, the class will read both train and test files
  and combine the vocabulary.
  
  Args:
      path (str): Directory to store the dataset
      task (str): a particular task to solve (all bAbI tasks are train
          and tested separately)
      oov (bool, optional): Load test set with out of vocabulary entity words
      use_match_type (bool, optional): Flag to use match-type features
      use_time (bool, optional): Add time words to each memory, encoding when the
          memory was formed
      use_speaker_tag (bool, optional): Add speaker words to each memory
          (<BOT> or <USER>) indicating who spoke each memory.
      cache_match_type (bool, optional): Flag to save match-type features after processing
      cache_vectorized (bool, optional): Flag to save all vectorized data after processing
  
  Attributes:
      data_dict (dict): Dictionary containing final vectorized train, val, and test datasets
      cands (np.array): Vectorized array of potential candidate answers, encoded
      as integers, as returned by BABI_Dialog class. Shape = [num_cands, max_cand_length]
      num_cands (int): Number of potential candidate answers.
      max_cand_len (int): Maximum length of a candidate answer sentence in number of words.
      memory_size (int): Maximum number of sentences to keep in memory at any given time.
      max_utt_len (int): Maximum length of any given sentence / user utterance
      vocab_size (int): Number of unique words in the vocabulary + 2 (0 is reserved for
          a padding symbol, and 1 is reserved for OOV)
      use_match_type (bool, optional): Flag to use match-type features
      kb_ents_to_type (dict, optional): For use with match-type features, dictionary of
          entities found in the dataset mapping to their associated match-type
      kb_ents_to_cand_idxs (dict, optional): For use with match-type features, dictionary
          mapping from each entity in the  knowledge base to the set of indicies in the
          candidate_answers array that contain that entity.
      match_type_idxs (dict, optional): For use with match-type features, dictionary
          mapping from match-type to the associated fixed index of the candidate vector
          which indicated this match type.
class_docstr_tok: ["This","class","loads","in","the","Facebook","bAbI","goal","oriented","dialog","dataset","and","vectorizes","them","into","user","utterances",",","bot","utterances",",","and","answers",".","As","described","in",":","``","Learning","End-to-End","Goal","Oriented","Dialog","''",".","https",":","\/\/arxiv.org\/abs\/1605.07683",".","For","a","particular","task",",","the","class","will","read","both","train","and","test","files","and","combine","the","vocabulary",".","Args",":","path","(","str",")",":","Directory","to","store","the","dataset","task","(","str",")",":","a","particular","task","to","solve","(","all","bAbI","tasks","are","train","and","tested","separately",")","oov","(","bool",",","optional",")",":","Load","test","set","with","out","of","vocabulary","entity","words","use_match_type","(","bool",",","optional",")",":","Flag","to","use","match-type","features","use_time","(","bool",",","optional",")",":","Add","time","words","to","each","memory",",","encoding","when","the","memory","was","formed","use_speaker_tag","(","bool",",","optional",")",":","Add","speaker","words","to","each","memory","(","<","BOT",">","or","<","USER",">",")","indicating","who","spoke","each","memory",".","cache_match_type","(","bool",",","optional",")",":","Flag","to","save","match-type","features","after","processing","cache_vectorized","(","bool",",","optional",")",":","Flag","to","save","all","vectorized","data","after","processing","Attributes",":","data_dict","(","dict",")",":","Dictionary","containing","final","vectorized","train",",","val",",","and","test","datasets","cands","(","np.array",")",":","Vectorized","array","of","potential","candidate","answers",",","encoded","as","integers",",","as","returned","by","BABI_Dialog","class",".","Shape","=","[","num_cands",",","max_cand_length","]","num_cands","(","int",")",":","Number","of","potential","candidate","answers",".","max_cand_len","(","int",")",":","Maximum","length","of","a","candidate","answer","sentence","in","number","of","words",".","memory_size","(","int",")",":","Maximum","number","of","sentences","to","keep","in","memory","at","any","given","time",".","max_utt_len","(","int",")",":","Maximum","length","of","any","given","sentence","\/","user","utterance","vocab_size","(","int",")",":","Number","of","unique","words","in","the","vocabulary","+","2","(","0","is","reserved","for","a","padding","symbol",",","and","1","is","reserved","for","OOV",")","use_match_type","(","bool",",","optional",")",":","Flag","to","use","match-type","features","kb_ents_to_type","(","dict",",","optional",")",":","For","use","with","match-type","features",",","dictionary","of","entities","found","in","the","dataset","mapping","to","their","associated","match-type","kb_ents_to_cand_idxs","(","dict",",","optional",")",":","For","use","with","match-type","features",",","dictionary","mapping","from","each","entity","in","the","knowledge","base","to","the","set","of","indicies","in","the","candidate_answers","array","that","contain","that","entity",".","match_type_idxs","(","dict",",","optional",")",":","For","use","with","match-type","features",",","dictionary","mapping","from","match-type","to","the","associated","fixed","index","of","the","candidate","vector","which","indicated","this","match","type","."]
docstr: Fetch and extract the Facebook bAbI-dialog dataset if not already downloaded.
  
  Returns:
      tuple: training and test filenames are returned
docstr_tok: ["Fetch","and","extract","the","Facebook","bAbI-dialog","dataset","if","not","already","downloaded",".","Returns",":","tuple",":","training","and","test","filenames","are","returned"]
code: def load_data(self):
      """
      Fetch and extract the Facebook bAbI-dialog dataset if not already downloaded.
  
      Returns:
          tuple: training and test filenames are returned
      """
      if self.task < 5:
          self.candidate_answer_filename = "dialog-babi-candidates.txt"
          self.kb_filename = "dialog-babi-kb-all.txt"
          self.cands_mat_filename = "babi-cands-with-matchtype_{}.npy"
          self.vocab_filename = "dialog-babi-vocab-task{}".format(
              self.task + 1
          ) + "_matchtype{}.pkl".format(self.use_match_type)
      else:
          self.candidate_answer_filename = (
              "dialog-babi-task6-dstc2-candidates.txt"
          )
          self.kb_filename = "dialog-babi-task6-dstc2-kb.txt"
          self.cands_mat_filename = "dstc2-cands-with-matchtype_{}.npy"
          self.vocab_filename = (
              "dstc2-vocab-task{}_matchtype{}.pkl".format(
                  self.task + 1, self.use_match_type
              )
          )
  
      self.vectorized_filename = "vectorized_task{}.pkl".format(
          self.task + 1
      )
  
      self.data_dict = {}
      self.vocab = None
      self.workdir, filepath = valid_path_append(
          self.path, "", self.filename
      )
      if not os.path.exists(filepath):
          if (
              license_prompt(
                  "bAbI-dialog",
                  "https://research.fb.com/downloads/babi/",
                  self.path,
              )
              is False
          ):
              sys.exit(0)
  
          download_unlicensed_file(
              self.url, self.filename, filepath, self.size
          )
  
      self.babi_dir_name = self.filename.split(".")[0]
  
      self.candidate_answer_filename = (
          self.babi_dir_name + "/" + self.candidate_answer_filename
      )
      self.kb_filename = self.babi_dir_name + "/" + self.kb_filename
      self.cands_mat_filename = os.path.join(
          self.workdir,
          self.babi_dir_name + "/" + self.cands_mat_filename,
      )
      self.vocab_filename = (
          self.babi_dir_name + "/" + self.vocab_filename
      )
      self.vectorized_filename = (
          self.babi_dir_name + "/" + self.vectorized_filename
      )
  
      task_name = (
          self.babi_dir_name + "/" + self.tasks[self.task] + "{}.txt"
      )
  
      train_file = os.path.join(self.workdir, task_name.format("trn"))
      dev_file = os.path.join(self.workdir, task_name.format("dev"))
      test_file_postfix = "tst-OOV" if self.oov else "tst"
      test_file = os.path.join(
          self.workdir, task_name.format(test_file_postfix)
      )
  
      cand_file = os.path.join(
          self.workdir, self.candidate_answer_filename
      )
      kb_file = os.path.join(self.workdir, self.kb_filename)
      vocab_file = os.path.join(self.workdir, self.vocab_filename)
      vectorized_file = os.path.join(
          self.workdir, self.vectorized_filename
      )
  
      if (
          os.path.exists(train_file) is False
          or os.path.exists(dev_file) is False
          or os.path.exists(test_file) is False
          or os.path.exists(cand_file) is False
      ):
          with tarfile.open(filepath, "r:gz") as f:
              f.extractall(self.workdir)
  
      return (
          train_file,
          dev_file,
          test_file,
          cand_file,
          kb_file,
          vocab_file,
          vectorized_file,
      )
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  from __future__ import unicode_literals
  import itertools
  import os
  import pickle
  import sys
  import tarfile
  import numpy
  from tqdm import tqdm
  from nlp_architect.utils.generic import license_prompt
  from nlp_architect.utils.io import download_unlicensed_file, valid_path_append
label_desc: Use image node_id 2 for calling the BABI_Dialog obj's underlying member method code with example usage: obj.load_data() and returns: train_file,  dev_file,  test_file,  cand_file,  kb_file,  vocab_file,  vectorized_file