third_party/tensorflow-text/src/tensorflow_text/python/ops/whitespace_tokenizer.py
def _whitespace_tokenize_codepoints_with_offsets(self, codepoints_tensor):
    """Tokenizes a tensor of codepoints with rank of 1.

    Args:
      codepoints_tensor: Single-dimension Tensor of codepoints to tokenize.

    Returns:
      Tuple of tokenized codepoints with offsets relative to the codepoints have
      a shape of [num_strings, (num_tokens or num_offsets)].
    """
    (output_values, output_values_inner_splits, output_offset_starts,
     output_offset_ends, output_outer_splits) = (
         gen_whitespace_tokenizer.whitespace_tokenize_with_offsets(
             input_values=codepoints_tensor.flat_values,
             input_splits=codepoints_tensor.row_splits))
    codepoint_tokens = RaggedTensor.from_nested_row_splits(
        flat_values=output_values,
        nested_row_splits=[output_outer_splits, output_values_inner_splits])
    codepoint_offset_starts = RaggedTensor.from_nested_row_splits(
        flat_values=output_offset_starts,
        nested_row_splits=[output_outer_splits])
    codepoint_offset_ends = RaggedTensor.from_nested_row_splits(
        flat_values=output_offset_ends,
        nested_row_splits=[output_outer_splits])
    return (codepoint_tokens, codepoint_offset_starts, codepoint_offset_ends)
Tokenizes a tensor of codepoints with rank of 1.

    Args:
      codepoints_tensor: Single-dimension Tensor of codepoints to tokenize.

    Returns:
      Tuple of tokenized codepoints with offsets relative to the codepoints have
      a shape of [num_strings, (num_tokens or num_offsets)].
    
Tokenizes a tensor of codepoints with rank of 1.
Tokenizes a tensor of codepoints with rank of 1 .
def _whitespace_tokenize_codepoints_with_offsets(self, codepoints_tensor):
    (output_values, output_values_inner_splits, output_offset_starts,
     output_offset_ends, output_outer_splits) = (
         gen_whitespace_tokenizer.whitespace_tokenize_with_offsets(
             input_values=codepoints_tensor.flat_values,
             input_splits=codepoints_tensor.row_splits))
    codepoint_tokens = RaggedTensor.from_nested_row_splits(
        flat_values=output_values,
        nested_row_splits=[output_outer_splits, output_values_inner_splits])
    codepoint_offset_starts = RaggedTensor.from_nested_row_splits(
        flat_values=output_offset_starts,
        nested_row_splits=[output_outer_splits])
    codepoint_offset_ends = RaggedTensor.from_nested_row_splits(
        flat_values=output_offset_ends,
        nested_row_splits=[output_outer_splits])
    return (codepoint_tokens, codepoint_offset_starts, codepoint_offset_ends)
def _whitespace_tokenize_codepoints_with_offsets ( self , codepoints_tensor ) : ( output_values , output_values_inner_splits , output_offset_starts , output_offset_ends , output_outer_splits ) = ( gen_whitespace_tokenizer . whitespace_tokenize_with_offsets ( input_values = codepoints_tensor . flat_values , input_splits = codepoints_tensor . row_splits ) ) codepoint_tokens = RaggedTensor . from_nested_row_splits ( flat_values = output_values , nested_row_splits = [ output_outer_splits , output_values_inner_splits ] ) codepoint_offset_starts = RaggedTensor . from_nested_row_splits ( flat_values = output_offset_starts , nested_row_splits = [ output_outer_splits ] ) codepoint_offset_ends = RaggedTensor . from_nested_row_splits ( flat_values = output_offset_ends , nested_row_splits = [ output_outer_splits ] ) return ( codepoint_tokens , codepoint_offset_starts , codepoint_offset_ends )
Tokenizes a tensor of codepoints with rank of 1.
Tokenizes a tensor of codepoints with rank of 1 .
"""Tokenizes a tensor of codepoints with rank of 1.

    Args:
      codepoints_tensor: Single-dimension Tensor of codepoints to tokenize.

    Returns:
      Tuple of tokenized codepoints with offsets relative to the codepoints have
      a shape of [num_strings, (num_tokens or num_offsets)].
    """
Tuple of tokenized codepoints with offsets relative to the codepoints have
a shape of [num_strings, (num_tokens or num_offsets)].
Tuple of tokenized codepoints with offsets relative to the codepoints have a shape of [ num_strings ( num_tokens or num_offsets ) ] .
self
codepoints_tensor
Single-dimension Tensor of codepoints to tokenize.
Single - dimension Tensor of codepoints to tokenize .