class_docstr: Old segment -> revised segment.
  
  Data filtration heuristics:
    wiki_revision_max_page_size_exp:
      pages above this # of bytes are thrown out
  
    wiki_revision_revision_skip_factor:
      rate of logarithmic downsampling of revision history list
  
    wiki_revision_percent_identical_examples:
      how many identitcal examples to admit, as percent of total examples
  
    wiki_revision_introduce_errors:
      whether or not to introduce spelling-type errors on the source side
  
    wiki_revision_max_equal_to_diff_ratio:
      whether or not to introduce spelling-type errors on the source side
  
  
  Vocab size=32k
  Maximum input/target length = 1024 wordpiece tokens
class_docstr_tok: ["Old","segment","-",">","revised","segment",".","Data","filtration","heuristics",":","wiki_revision_max_page_size_exp",":","pages","above","this","#","of","bytes","are","thrown","out","wiki_revision_revision_skip_factor",":","rate","of","logarithmic","downsampling","of","revision","history","list","wiki_revision_percent_identical_examples",":","how","many","identitcal","examples","to","admit",",","as","percent","of","total","examples","wiki_revision_introduce_errors",":","whether","or","not","to","introduce","spelling-type","errors","on","the","source","side","wiki_revision_max_equal_to_diff_ratio",":","whether","or","not","to","introduce","spelling-type","errors","on","the","source","side","Vocab","size=32k","Maximum","input\/target","length","=","1024","wordpiece","tokens"]
docstr: Produce training examples based on a pair of snapshots.
  
  Aligns the snapshots, then chops at a random subset of the alignment points
  to create (old snippet -> new snippet) examples.
  
  Most negative examples (those with no changes) are discarded, but we
  keep some of them, maintaining a proportion in the final data
  determined by percent_identical_examples.
  
  Args:
    encoder: the subword text encoder
    old_snapshot: a list of ids
    new_snapshot: a list of ids
    max_length: an integer.  Maximum length of "inputs" and "targets".
    percent_identical_examples: a float
    max_length_distance: an integer. Max token edit dist for admitted examples
  
  Returns:
    a list of feature dictionaries.  The dictionaries have
    "inputs" and "targets" populated. text_encoder.EOS is appended to both.
docstr_tok: ["Produce","training","examples","based","on","a","pair","of","snapshots",".","Aligns","the","snapshots",",","then","chops","at","a","random","subset","of","the","alignment","points","to","create","(","old","snippet","-",">","new","snippet",")","examples",".","Most","negative","examples","(","those","with","no","changes",")","are","discarded",",","but","we","keep","some","of","them",",","maintaining","a","proportion","in","the","final","data","determined","by","percent_identical_examples",".","Args",":","encoder",":","the","subword","text","encoder","old_snapshot",":","a","list","of","ids","new_snapshot",":","a","list","of","ids","max_length",":","an","integer",".","Maximum","length","of","``","inputs","''","and","``","targets","''",".","percent_identical_examples",":","a","float","max_length_distance",":","an","integer",".","Max","token","edit","dist","for","admitted","examples","Returns",":","a","list","of","feature","dictionaries",".","The","dictionaries","have","``","inputs","''","and","``","targets","''","populated",".","text_encoder.EOS","is","appended","to","both","."]
code: def make_examples(
      self,
      encoder,
      old_snapshot,
      new_snapshot,
      max_length=1024,
      percent_identical_examples=0.01,
      max_length_distance=0,
  ):
      """Produce training examples based on a pair of snapshots.
  
      Aligns the snapshots, then chops at a random subset of the alignment points
      to create (old snippet -> new snippet) examples.
  
      Most negative examples (those with no changes) are discarded, but we
      keep some of them, maintaining a proportion in the final data
      determined by percent_identical_examples.
  
      Args:
        encoder: the subword text encoder
        old_snapshot: a list of ids
        new_snapshot: a list of ids
        max_length: an integer.  Maximum length of "inputs" and "targets".
        percent_identical_examples: a float
        max_length_distance: an integer. Max token edit dist for admitted examples
  
      Returns:
        a list of feature dictionaries.  The dictionaries have
        "inputs" and "targets" populated. text_encoder.EOS is appended to both.
      """
      ret = []
      eos_sequence = [text_encoder.EOS_ID]
      # Pick a per-token cut probability with a log-uniform distribution between
      # 1/4 and 1/(max_length / 2)
      bound1 = -math.log(4.0)
      bound2 = -math.log(max_length / 2.0)
      cut_prob = math.exp(random.random() * (bound2 - bound1) + bound1)
      opcodes = wiki_revision_utils.fast_match_sequences(
          old_snapshot, new_snapshot
      )
      cut_points = [(0, 0)]
      for tag, i1, i2, j1, j2 in opcodes:
          if tag == "equal":
              for i in range(i1, i2 + 1):
                  if random.random() < cut_prob:
                      cut_points.append((i, i + j1 - i1))
      cut_points.append((len(old_snapshot), len(new_snapshot)))
      src_tgt_pairs = []
      for cut_number in range(len(cut_points) - 1):
          i1, j1 = cut_points[cut_number]
          i2, j2 = cut_points[cut_number + 1]
          old_segment = old_snapshot[i1:i2]
          new_segment = new_snapshot[j1:j2]
          src_tgt_pairs.append((old_segment, new_segment))
  
      (
          src_tgt_pairs,
          thrown_edit_count,
      ) = wiki_revision_utils.edit_distance_filter(
          wiki_revision_utils.throw_empty_pairs(src_tgt_pairs),
          FLAGS.wiki_revision_max_equal_to_diff_ratio,
      )
  
      self.num_examples_thrown_out_edit_distance += thrown_edit_count
  
      for source, target in src_tgt_pairs:
          # Add EOS segment.
          old_segment = source + eos_sequence
          new_segment = target + eos_sequence
          if (
              len(old_segment) <= max_length
              and len(new_segment) <= max_length
          ):
              if max_length_distance and (
                  abs(len(old_segment) - len(new_segment))
                  > max_length_distance
              ):
                  self.num_examples_thrown_out_edit_distance += 1
                  continue
              if old_segment == new_segment:
                  # If current proportion of identity is below target
                  # percent_identical_examples, then roll for a 50% chance to add an
                  # identitical example. Random roll preserves nondeterminism.
                  # percent_identical_examples, then add identitical example.
                  # Random roll preserves nondeterminism in selecting identity examples.
                  if (
                      (
                          (self.num_identity_examples)
                          / (1 + self.num_total_examples)
                      )
                      > percent_identical_examples
                  ) or random.random() > 0.5:
                      self.num_examples_thrown_out_identity += 1
                      continue
                  else:
                      self.num_identity_examples += 1
              self.num_total_examples += 1
              self.num_source_tokens += len(old_segment) - 1
              self.num_target_tokens += len(new_segment) - 1
              ret.append(
                  {"inputs": old_segment, "targets": new_segment}
              )
          else:
              self.num_examples_thrown_out_too_long += 1
  
      return ret
  
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import math
  import random
  from absl import flags
  from tensor2tensor.data_generators import generator_utils
  from tensor2tensor.data_generators import text_encoder
  from tensor2tensor.data_generators import text_problems
  from tensor2tensor.data_generators import wiki_revision_utils
  from tensor2tensor.utils import metrics
  from tensor2tensor.utils import registry
  import tensorflow.compat.v1
label_desc: Use image node_id 10 for calling the WikiRevision obj's underlying member method code with example usage: obj.make_examples(encoder, old_snapshot, new_snapshot, max_length, percent_identical_examples, max_length_distance) and returns: ret