class_docstr: __init__(features=None, excluded_features=None,
      to_lower=False, delimiters=["\\r", "\\v", "\\n", "\\f", "\\t", " "],
      output_column_prefix=None)
  
  Tokenizing is a method of breaking natural language text into its smallest
  standalone and meaningful components (in English, usually space-delimited
  words, but not always).
  
  By default, Tokenizer tokenizes strings by space characters. The user may
  specify a customized list of delimiters, or use Penn treebank-style
  tokenization.
  
  .. warning::
      The default tokenization setting is now different from that of
      Turi Create v1.6. The old default was Penn treebank-style
      tokenization. (This is still available by setting `delimiters=None`.)
      The current default is to tokenize by space characters.
  
  Parameters
  ----------
  features : list[str] | str | None, optional
      Name(s) of feature column(s) to be transformed. If set to None, then all
      feature columns are used.
  
  excluded_features : list[str] | str | None, optional
      Name(s) of feature columns in the input dataset to be ignored. Either
      `excluded_features` or `features` can be passed, but not both.
  
  to_lower : bool, optional
      Indicates whether to map the input strings to lower case before counting.
  
  delimiters: list[string], optional
      A list of delimiter characters for tokenization. By default, the list
      is defined to be the list of space characters. The user can define
      any custom list of single-character delimiters. Alternatively, setting
      `delimiters=None` will use a Penn treebank-style tokenization that
      separates individual punctuation marks and detects positive and negative
      real numbers, phone numbers with no spaces, urls, and emails. The
      Penn treebank-style tokenization also attempts to separate contractions
      and possessives. For instance, "don't" would be tokenized as
      ["do", "n't"].
  
  output_column_prefix : str, optional
      The prefix to use for the column name of each transformed column.
      When provided, the transformation will add columns to the input data,
      where the new name is "`output_column_prefix`.original_column_name".
      If `output_column_prefix=None` (default), then the output column name
      is the same as the original feature column name.
  
  Returns
  -------
  out : Tokenizer
      A Tokenizer object which is initialized with the defined parameters.
  
  Notes
  -----
  This implementation of Tokenizer applies regular expressions to the natural
  language text to capture a high-recall set of valid text patterns.
  
  If the SFrame to be transformed already contains a column with the
  designated output column name, then that column will be replaced with the
  new output. In particular, this means that `output_column_prefix=None` will
  overwrite the original feature columns.
  
  References
  ----------
  - `Penn treebank tokenization <https://web.archive.org/web/19970614072242/http://www.cis.upenn.edu:80/~treebank/tokenization.html>`_
  
  See Also
  --------
  turicreate.toolkits.text_analytics.tokenize,
  turicreate.toolkits.feature_engineering._word_counter.WordCounter,
  turicreate.toolkits.feature_engineering._ngram_counter.NGramCounter,
  turicreate.toolkits.feature_engineering._tfidf.TFIDF,
  turicreate.toolkits.feature_engineering.create
  
  Examples
  --------
  
  .. sourcecode:: python
  
      >>> import turicreate
      >>> from turicreate.feature_engineering import *
  
      # Create a sample dataset
      >>> sf = turicreate.SFrame({
      ...    'docs': ["This is a document!", "This one's also a document."]})
  
      # Construct a tokenizer with default options.
      >>> tokenizer = Tokenizer()
  
      # Transform the data using the tokenizer.
      >>> tokenized_sf = tokenizer.fit_transform(sf)
      >>> tokenized_sf
      Columns:
          docs    list
  
      Rows: 2
  
      Data:
      +-----------------------------------+
      |                docs               |
      +-----------------------------------+
      |      [This, is, a, document!]     |
      | [This, one's, also, a, document.] |
      +-----------------------------------+
      [2 rows x 1 columns]
  
      # Convert to lower case and use Penn treebank-style tokenization.
      >>> ptb_tokenizer = Tokenizer(to_lower=True, delimiters=None)
      >>> tokenized_sf = ptb_tokenizer.fit_transform(sf)
      >>> tokenized_sf
      Columns:
          docs    list
  
      Rows: 2
  
      Data:
      +---------------------------------------+
      |                  docs                 |
      +---------------------------------------+
      |       [this, is, a, document, !]      |
      | [this, one, 's, also, a, document, .] |
      +---------------------------------------+
      [2 rows x 1 columns]
  
      # Tokenize only a single column 'docs'.
      >>> tokenizer = Tokenizer(features = ['docs'])
      >>> tokenizer['features']
      ['docs']
  
      # Tokenize all columns except 'docs'.
      >>> tokenizer = Tokenizer(excluded_features = ['docs'])
      >>> tokenizer['features']  # `features` are set to `None`
code: def __repr__(self):
      (sections, section_titles) = self._get_summary_struct()
      return _toolkit_repr_print(
          self, sections, section_titles, width=30
      )
  
file_path: turicreate/src/python/turicreate/toolkits/_feature_engineering/_tokenizer.py
imports: from __future__ import print_function
  from __future__ import division
  from __future__ import absolute_import
  import turicreate
  from ._feature_engineering import Transformer
  from turicreate.toolkits._internal_utils import _toolkit_repr_print
  from turicreate.toolkits._internal_utils import _precomputed_field
  from turicreate.util import _raise_error_if_not_of_type
  from .None import _internal_utils
label_desc: Use image node_id 3 for calling the Tokenizer obj's underlying member method code with example usage: obj.__repr__() and returns: _toolkit_repr_print