class_docstr: Abstract class for dialog problems.
docstr: This function generates train and validation pairs in t2t-datagen style.
  
  The function assumes that if you have data at one level of the pipeline,
  you don't want to re-generate it, so for example if the 4 txt files exist,
  the function continues by generating the t2t-datagen format files.
  So if you want to re-download or re-generate data,
  you have to delete it first from the appropriate directories.
  
  Args:
    data_dir: string, Directory where the data will be generated. The raw
                      data has to be downloaded one directory level higher.
    tmp_dir: string, temp directory.
    data_split: string, which data split to generate samples for
  
  Yields:
    dict
docstr_tok: ["This","function","generates","train","and","validation","pairs","in","t2t-datagen","style",".","The","function","assumes","that","if","you","have","data","at","one","level","of","the","pipeline",",","you","do","n't","want","to","re-generate","it",",","so","for","example","if","the","4","txt","files","exist",",","the","function","continues","by","generating","the","t2t-datagen","format","files",".","So","if","you","want","to","re-download","or","re-generate","data",",","you","have","to","delete","it","first","from","the","appropriate","directories",".","Args",":","data_dir",":","string",",","Directory","where","the","data","will","be","generated",".","The","raw","data","has","to","be","downloaded","one","directory","level","higher",".","tmp_dir",":","string",",","temp","directory",".","data_split",":","string",",","which","data","split","to","generate","samples","for","Yields",":","dict"]
code: def generate_samples(self, data_dir, tmp_dir, data_split):
      """This function generates train and validation pairs in t2t-datagen style.
  
      The function assumes that if you have data at one level of the pipeline,
      you don't want to re-generate it, so for example if the 4 txt files exist,
      the function continues by generating the t2t-datagen format files.
      So if you want to re-download or re-generate data,
      you have to delete it first from the appropriate directories.
  
      Args:
        data_dir: string, Directory where the data will be generated. The raw
                          data has to be downloaded one directory level higher.
        tmp_dir: string, temp directory.
        data_split: string, which data split to generate samples for
  
      Yields:
        dict
      """
  
      self.data_dir = data_dir
      print(
          "problem_log: "
          + self.mode[data_split]
          + " data generation activated."
      )
  
      s_path = os.path.join(
          data_dir, self.mode[data_split] + "Source.txt"
      )
      t_path = os.path.join(
          data_dir, self.mode[data_split] + "Target.txt"
      )
  
      # Open the files and yield source-target lines.
      with tf.gfile.GFile(s_path, mode="r") as source_file:
          with tf.gfile.GFile(t_path, mode="r") as target_file:
              source, target = (
                  source_file.readline(),
                  target_file.readline(),
              )
              while source and target:
                  yield {
                      "inputs": source.strip(),
                      "targets": target.strip(),
                  }
                  source, target = (
                      source_file.readline(),
                      target_file.readline(),
                  )
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import os
  import re
  import tarfile
  import zipfile
  import requests
  from tensor2tensor.data_generators import generator_utils
  from tensor2tensor.data_generators import problem
  from tensor2tensor.data_generators import text_encoder
  from tensor2tensor.data_generators import text_problems
  from tensor2tensor.data_generators.text_problems import VocabType
  from tensor2tensor.layers import modalities
  from tensor2tensor.utils import metrics
  import tensorflow.compat.v1
label_desc: Use image node_id 31 for calling the DialogAbstract obj's underlying member method code with example usage: obj.generate_samples(data_dir, tmp_dir, data_split) without return types