class_docstr: This class loads in the Facebook bAbI goal oriented dialog dataset and
  vectorizes them into user utterances, bot utterances, and answers.
  
  As described in: "Learning End-to-End Goal Oriented Dialog".
  https://arxiv.org/abs/1605.07683.
  
  For a particular task, the class will read both train and test files
  and combine the vocabulary.
  
  Args:
      path (str): Directory to store the dataset
      task (str): a particular task to solve (all bAbI tasks are train
          and tested separately)
      oov (bool, optional): Load test set with out of vocabulary entity words
      use_match_type (bool, optional): Flag to use match-type features
      use_time (bool, optional): Add time words to each memory, encoding when the
          memory was formed
      use_speaker_tag (bool, optional): Add speaker words to each memory
          (<BOT> or <USER>) indicating who spoke each memory.
      cache_match_type (bool, optional): Flag to save match-type features after processing
      cache_vectorized (bool, optional): Flag to save all vectorized data after processing
  
  Attributes:
      data_dict (dict): Dictionary containing final vectorized train, val, and test datasets
      cands (np.array): Vectorized array of potential candidate answers, encoded
      as integers, as returned by BABI_Dialog class. Shape = [num_cands, max_cand_length]
      num_cands (int): Number of potential candidate answers.
      max_cand_len (int): Maximum length of a candidate answer sentence in number of words.
      memory_size (int): Maximum number of sentences to keep in memory at any given time.
      max_utt_len (int): Maximum length of any given sentence / user utterance
      vocab_size (int): Number of unique words in the vocabulary + 2 (0 is reserved for
          a padding symbol, and 1 is reserved for OOV)
      use_match_type (bool, optional): Flag to use match-type features
      kb_ents_to_type (dict, optional): For use with match-type features, dictionary of
          entities found in the dataset mapping to their associated match-type
      kb_ents_to_cand_idxs (dict, optional): For use with match-type features, dictionary
          mapping from each entity in the  knowledge base to the set of indicies in the
          candidate_answers array that contain that entity.
      match_type_idxs (dict, optional): For use with match-type features, dictionary
          mapping from match-type to the associated fixed index of the candidate vector
          which indicated this match type.
class_docstr_tok: ["This","class","loads","in","the","Facebook","bAbI","goal","oriented","dialog","dataset","and","vectorizes","them","into","user","utterances",",","bot","utterances",",","and","answers",".","As","described","in",":","``","Learning","End-to-End","Goal","Oriented","Dialog","''",".","https",":","\/\/arxiv.org\/abs\/1605.07683",".","For","a","particular","task",",","the","class","will","read","both","train","and","test","files","and","combine","the","vocabulary",".","Args",":","path","(","str",")",":","Directory","to","store","the","dataset","task","(","str",")",":","a","particular","task","to","solve","(","all","bAbI","tasks","are","train","and","tested","separately",")","oov","(","bool",",","optional",")",":","Load","test","set","with","out","of","vocabulary","entity","words","use_match_type","(","bool",",","optional",")",":","Flag","to","use","match-type","features","use_time","(","bool",",","optional",")",":","Add","time","words","to","each","memory",",","encoding","when","the","memory","was","formed","use_speaker_tag","(","bool",",","optional",")",":","Add","speaker","words","to","each","memory","(","<","BOT",">","or","<","USER",">",")","indicating","who","spoke","each","memory",".","cache_match_type","(","bool",",","optional",")",":","Flag","to","save","match-type","features","after","processing","cache_vectorized","(","bool",",","optional",")",":","Flag","to","save","all","vectorized","data","after","processing","Attributes",":","data_dict","(","dict",")",":","Dictionary","containing","final","vectorized","train",",","val",",","and","test","datasets","cands","(","np.array",")",":","Vectorized","array","of","potential","candidate","answers",",","encoded","as","integers",",","as","returned","by","BABI_Dialog","class",".","Shape","=","[","num_cands",",","max_cand_length","]","num_cands","(","int",")",":","Number","of","potential","candidate","answers",".","max_cand_len","(","int",")",":","Maximum","length","of","a","candidate","answer","sentence","in","number","of","words",".","memory_size","(","int",")",":","Maximum","number","of","sentences","to","keep","in","memory","at","any","given","time",".","max_utt_len","(","int",")",":","Maximum","length","of","any","given","sentence","\/","user","utterance","vocab_size","(","int",")",":","Number","of","unique","words","in","the","vocabulary","+","2","(","0","is","reserved","for","a","padding","symbol",",","and","1","is","reserved","for","OOV",")","use_match_type","(","bool",",","optional",")",":","Flag","to","use","match-type","features","kb_ents_to_type","(","dict",",","optional",")",":","For","use","with","match-type","features",",","dictionary","of","entities","found","in","the","dataset","mapping","to","their","associated","match-type","kb_ents_to_cand_idxs","(","dict",",","optional",")",":","For","use","with","match-type","features",",","dictionary","mapping","from","each","entity","in","the","knowledge","base","to","the","set","of","indicies","in","the","candidate_answers","array","that","contain","that","entity",".","match_type_idxs","(","dict",",","optional",")",":","For","use","with","match-type","features",",","dictionary","mapping","from","match-type","to","the","associated","fixed","index","of","the","candidate","vector","which","indicated","this","match","type","."]
docstr: Convert (memory, user_utt, answer) word data into vectors.
  
  If sentence length < max_utt_len it is padded with 0's
  If memory length < memory size, it is padded with empty memorys (max_utt_len 0's)
  
  Args:
      data (tuple) : Tuple of memories, user_utt, answer word data.
  
  Returns:
      tuple : Tuple of memories, memory_lengths, user_utt, answer vectors.
docstr_tok: ["Convert","(","memory",",","user_utt",",","answer",")","word","data","into","vectors",".","If","sentence","length","<","max_utt_len","it","is","padded","with","0's","If","memory","length","<","memory","size",",","it","is","padded","with","empty","memorys","(","max_utt_len","0","'s",")","Args",":","data","(","tuple",")",":","Tuple","of","memories",",","user_utt",",","answer","word","data",".","Returns",":","tuple",":","Tuple","of","memories",",","memory_lengths",",","user_utt",",","answer","vectors","."]
code: def vectorize_stories(self, data):
      """
      Convert (memory, user_utt, answer) word data into vectors.
  
      If sentence length < max_utt_len it is padded with 0's
      If memory length < memory size, it is padded with empty memorys (max_utt_len 0's)
  
      Args:
          data (tuple) : Tuple of memories, user_utt, answer word data.
  
      Returns:
          tuple : Tuple of memories, memory_lengths, user_utt, answer vectors.
      """
      m, ml, m_mask, u, a = [], [], [], [], []
      for mem, utt, answer in tqdm(data, desc="Vectorizing"):
          m.append([self.words_to_vector(sent) for sent in mem])
          ml.append(len(mem))
          mask_zero_len = self.memory_size - ml[-1]
          m_mask.append(
              [1.0 for _ in range(ml[-1])]
              + [0.0 for _ in range(mask_zero_len)]
          )
  
          u.append(self.words_to_vector(utt))
          a.append(self.one_hot_vector(answer))
  
      m = np.array(
          [pad_sentences(sents, self.max_utt_len) for sents in m]
      )
      m = pad_stories(m, self.max_utt_len, self.memory_size)
      m_mask = np.array(m_mask)
  
      u = pad_sentences(u, self.max_utt_len)
      a = np.array(a)
  
      return (m, m_mask, u, a)
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  from __future__ import unicode_literals
  import itertools
  import os
  import pickle
  import sys
  import tarfile
  import numpy
  from tqdm import tqdm
  from nlp_architect.utils.generic import license_prompt
  from nlp_architect.utils.io import download_unlicensed_file, valid_path_append
label_desc: Use image node_id 6 for calling the BABI_Dialog obj's underlying member method code with example usage: obj.vectorize_stories(data) and returns: m,  m_mask,  u,  a