class_docstr: Provides a consistent `Transform` interface to tokenizers operating on `DataFrame`s and folders
class_docstr_tok: ["Provides","a","consistent","`","Transform","`","interface","to","tokenizers","operating","on","`","DataFrame","`","s","and","folders"]
code: def __init__(
      self,
      tok,
      rules=None,
      counter=None,
      lengths=None,
      mode=None,
      sep=" ",
  ):
      if isinstance(tok, type):
          tok = tok()
      store_attr("tok,counter,lengths,mode,sep")
      self.rules = defaults.text_proc_rules if rules is None else rules
  
imports: from __future__ import annotations
  from ..torch_basics import *
  from ..data.all import *
  import html
label_desc: Use image node_id 1 to create a new Tokenizer object from inherited base classes: Transform with example: obj = Tokenizer(tok, rules, counter, lengths, mode, sep)