class_docstr: Align two similiar tokenizations.
  
  Args:
      source (Union[Iterable[str], str]): Source text tokens or string.
      target (Union[Iterable[str], str]): Target text tokens or string.
  
  Usage:
      ta = TokenAligner(source_tokens, target_tokens)
      target_span = ta.project_span(*source_span)
  
  Uses Levenshtein distance to align two similar tokenizations, and provide facilities to project
  indices and spans from the source tokenization to the target.
  
  Let source contain m tokens and M chars, and target contain n tokens and N chars. The token
  alignment is treated as a (sparse) m x n adjacency matrix T representing the bipartite graph
  between the source and target tokens.
  
  This is constructed by performing a character-level alignment using Levenshtein distance to
  obtain a (M x N) character adjacency matrix C. We then construct token-to-character matricies
  U (m x M) and V (n x N) and construct T as:
      T = (U C V')
  where V' denotes the transpose.
  
  Spans of non-aligned bytes are assumed to contain a many-to-many alignment of all chars in that
  range. This can lead to unwanted alignments if, for example, two consecutive tokens are mapped
  to escape sequences:
      source: ["'s", "["]
      target: ["&apos;", "s", "&#91;"]
  In the above case, "'s'" may be spuriously aligned to "&apos;" while "[" has no character match
  with "s" or "&#91;", and so is aligned to both. To make a correct alignment more likely, ensure
  that the characters in target correspond as closely as possible to those in source. For example,
  the following will align correctly:
      source: ["'s", "["]
      target: ["'", "s", "["]
class_docstr_tok: ["Align","two","similiar","tokenizations",".","Args",":","source","(","Union","[","Iterable","[","str","]",",","str","]",")",":","Source","text","tokens","or","string",".","target","(","Union","[","Iterable","[","str","]",",","str","]",")",":","Target","text","tokens","or","string",".","Usage",":","ta","=","TokenAligner","(","source_tokens",",","target_tokens",")","target_span","=","ta.project_span","(","*","source_span",")","Uses","Levenshtein","distance","to","align","two","similar","tokenizations",",","and","provide","facilities","to","project","indices","and","spans","from","the","source","tokenization","to","the","target",".","Let","source","contain","m","tokens","and","M","chars",",","and","target","contain","n","tokens","and","N","chars",".","The","token","alignment","is","treated","as","a","(","sparse",")","m","x","n","adjacency","matrix","T","representing","the","bipartite","graph","between","the","source","and","target","tokens",".","This","is","constructed","by","performing","a","character-level","alignment","using","Levenshtein","distance","to","obtain","a","(","M","x","N",")","character","adjacency","matrix","C.","We","then","construct","token-to-character","matricies","U","(","m","x","M",")","and","V","(","n","x","N",")","and","construct","T","as",":","T","=","(","U","C","V","'",")","where","V","'","denotes","the","transpose",".","Spans","of","non-aligned","bytes","are","assumed","to","contain","a","many-to-many","alignment","of","all","chars","in","that","range",".","This","can","lead","to","unwanted","alignments","if",",","for","example",",","two","consecutive","tokens","are","mapped","to","escape","sequences",":","source",":","[","``","'s","''",",","``","[","``","]","target",":","[","``","&","apos",";","''",",","``","s","''",",","``","&","#","91",";","''","]","In","the","above","case",",","``","'s","'","''","may","be","spuriously","aligned","to","``","&","apos",";","''","while","``","[","``","has","no","character","match","with","``","s","''","or","``","&","#","91",";","''",",","and","so","is","aligned","to","both",".","To","make","a","correct","alignment","more","likely",",","ensure","that","the","characters","in","target","correspond","as","closely","as","possible","to","those","in","source",".","For","example",",","the","following","will","align","correctly",":","source",":","[","``","'s","''",",","``","[","``","]","target",":","[","``","'","''",",","``","s","''",",","``","[","``","]"]
code: def _project_span(mat, start, end, inclusive):
      if inclusive:
          end = end + 1
      target_matches = mat[start:end].nonzero()[1].tolist()
      if len(target_matches) == 0:
          raise ValueError(
              f"Project {(start, end)} into empty span in target sequence"
          )
      output_start, output_end = min(target_matches), max(
          target_matches
      )
      if not inclusive:
          output_end = output_end + 1
      return (output_start, output_end)
  
file_path: jiant/jiant/utils/retokenize.py
imports: from typing import Iterable, Sequence, Tuple, Union
  from Levenshtein.StringMatcher import StringMatcher
  from nltk.tokenize.util import string_span_tokenize
  import numpy
label_desc: Use image node_id 3 for calling the TokenAligner obj's underlying member method code with example usage: obj._project_span(mat, start, end, inclusive) and returns: output_start,  output_end