class_docstr: Tokenizes a tensor of UTF-8 string into words according to logits.
docstr: Initializes a new instance.
  
  Args:
    force_split_at_break_character: a bool that indicates whether to force
      start a new word after an ICU-defined whitespace character.  Regardless
      of this parameter, we never include a whitespace into a token, and we
      always ignore the split/merge action for the whitespace character
      itself.  This parameter indicates what happens after a whitespace.
      * if force_split_at_break_character is true, create a new word starting
          at the first non-space character, regardless of the 0/1 label for
          that character, for instance:
  
          ```python
          s = [2.0, 1.0]  # sample pair of logits indicating a split action
          m = [1.0, 3.0]  # sample pair of logits indicating a merge action
  
          strings=["New York"]
          logits=[[s, m, m, s, m, m, m, m]]
          output tokens=[["New", "York"]]
  
          strings=["New York"]
          logits=[[s, m, m, m, m, m, m, m]]
          output tokens=[["New", "York"]]
  
          strings=["New York"],
          logits=[[s, m, m, m, s, m, m, m]]
          output tokens=[["New", "York"]]
          ```
      * otherwise, create a new word / continue the current one depending on
          the action for the first non-whitespace character.
  
          ```python
          s = [2.0, 1.0]  # sample pair of logits indicating a split action
          m = [1.0, 3.0]  # sample pair of logits indicating a merge action
  
          strings=["New York"],
          logits=[[s, m, m, s, m, m, m, m]]
          output tokens=[["NewYork"]]
  
          strings=["New York"],
          logits=[[s, m, m, m, m, m, m, m]]
          output tokens=[["NewYork"]]
  
          strings=["New York"],
          logits=[[s, m, m, m, s, m, m, m]]
          output tokens=[["New", "York"]]
          ```
docstr_tok: ["Initializes","a","new","instance",".","Args",":","force_split_at_break_character",":","a","bool","that","indicates","whether","to","force","start","a","new","word","after","an","ICU-defined","whitespace","character",".","Regardless","of","this","parameter",",","we","never","include","a","whitespace","into","a","token",",","and","we","always","ignore","the","split\/merge","action","for","the","whitespace","character","itself",".","This","parameter","indicates","what","happens","after","a","whitespace",".","*","if","force_split_at_break_character","is","true",",","create","a","new","word","starting","at","the","first","non-space","character",",","regardless","of","the","0\/1","label","for","that","character",",","for","instance",":","``","`","python","s","=","[","2.0",",","1.0","]","#","sample","pair","of","logits","indicating","a","split","action","m","=","[","1.0",",","3.0","]","#","sample","pair","of","logits","indicating","a","merge","action","strings=","[","``","New","York","''","]","logits=","[","[","s",",","m",",","m",",","s",",","m",",","m",",","m",",","m","]","]","output","tokens=","[","[","``","New","''",",","``","York","''","]","]","strings=","[","``","New","York","''","]","logits=","[","[","s",",","m",",","m",",","m",",","m",",","m",",","m",",","m","]","]","output","tokens=","[","[","``","New","''",",","``","York","''","]","]","strings=","[","``","New","York","''","]",",","logits=","[","[","s",",","m",",","m",",","m",",","s",",","m",",","m",",","m","]","]","output","tokens=","[","[","``","New","''",",","``","York","''","]","]","``","`","*","otherwise",",","create","a","new","word","\/","continue","the","current","one","depending","on","the","action","for","the","first","non-whitespace","character",".","``","`","python","s","=","[","2.0",",","1.0","]","#","sample","pair","of","logits","indicating","a","split","action","m","=","[","1.0",",","3.0","]","#","sample","pair","of","logits","indicating","a","merge","action","strings=","[","``","New","York","''","]",",","logits=","[","[","s",",","m",",","m",",","s",",","m",",","m",",","m",",","m","]","]","output","tokens=","[","[","``","NewYork","''","]","]","strings=","[","``","New","York","''","]",",","logits=","[","[","s",",","m",",","m",",","m",",","m",",","m",",","m",",","m","]","]","output","tokens=","[","[","``","NewYork","''","]","]","strings=","[","``","New","York","''","]",",","logits=","[","[","s",",","m",",","m",",","m",",","s",",","m",",","m",",","m","]","]","output","tokens=","[","[","``","New","''",",","``","York","''","]","]","``","`"]
code: def __init__(self, force_split_at_break_character=True):
      """Initializes a new instance.
  
      Args:
        force_split_at_break_character: a bool that indicates whether to force
          start a new word after an ICU-defined whitespace character.  Regardless
          of this parameter, we never include a whitespace into a token, and we
          always ignore the split/merge action for the whitespace character
          itself.  This parameter indicates what happens after a whitespace.
          * if force_split_at_break_character is true, create a new word starting
              at the first non-space character, regardless of the 0/1 label for
              that character, for instance:
  
              ```python
              s = [2.0, 1.0]  # sample pair of logits indicating a split action
              m = [1.0, 3.0]  # sample pair of logits indicating a merge action
  
              strings=["New York"]
              logits=[[s, m, m, s, m, m, m, m]]
              output tokens=[["New", "York"]]
  
              strings=["New York"]
              logits=[[s, m, m, m, m, m, m, m]]
              output tokens=[["New", "York"]]
  
              strings=["New York"],
              logits=[[s, m, m, m, s, m, m, m]]
              output tokens=[["New", "York"]]
              ```
          * otherwise, create a new word / continue the current one depending on
              the action for the first non-whitespace character.
  
              ```python
              s = [2.0, 1.0]  # sample pair of logits indicating a split action
              m = [1.0, 3.0]  # sample pair of logits indicating a merge action
  
              strings=["New York"],
              logits=[[s, m, m, s, m, m, m, m]]
              output tokens=[["NewYork"]]
  
              strings=["New York"],
              logits=[[s, m, m, m, m, m, m, m]]
              output tokens=[["NewYork"]]
  
              strings=["New York"],
              logits=[[s, m, m, m, s, m, m, m]]
              output tokens=[["New", "York"]]
              ```
      """
      super(SplitMergeFromLogitsTokenizer, self).__init__()
      self._force_split_at_break_character = (
          force_split_at_break_character
      )
      counter = (
          _tf_text_split_merge_from_logits_tokenizer_op_create_counter
      )
      counter.get_cell().increase_by(1)
  
file_path: text/tensorflow_text/python/ops/split_merge_from_logits_tokenizer.py
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  from tensorflow.python.eager import monitoring
  from tensorflow.python.framework import ops
  from tensorflow.python.ops.ragged.ragged_tensor import RaggedTensor
  from tensorflow_text.python.ops.tokenization import TokenizerWithOffsets
  from tensorflow.python.framework import load_library
  from tensorflow.python.platform import resource_loader
label_desc: Use image node_id 1 to create a new SplitMergeFromLogitsTokenizer object from inherited base classes: TokenizerWithOffsets with example: obj = SplitMergeFromLogitsTokenizer(force_split_at_break_character)