class_docstr: Text tokenization utility class.
  
  This class allows to vectorize a text corpus, by turning each
  text into either a sequence of integers (each integer being the index
  of a token in a dictionary) or into a vector where the coefficient
  for each token could be binary, based on word count, based on tf-idf...
  
  # Arguments
      num_words: the maximum number of words to keep, based
          on word frequency. Only the most common `num_words-1` words will
          be kept.
      filters: a string where each element is a character that will be
          filtered from the texts. The default is all punctuation, plus
          tabs and line breaks, minus the `'` character.
      lower: boolean. Whether to convert the texts to lowercase.
      split: str. Separator for word splitting.
      char_level: if True, every character will be treated as a token.
      oov_token: if given, it will be added to word_index and used to
          replace out-of-vocabulary words during text_to_sequence calls
      analyzer: function. Custom analyzer to split the text.
          The default analyzer is text_to_word_sequence
  
  By default, all punctuation is removed, turning the texts into
  space-separated sequences of words
  (words maybe include the `'` character). These sequences are then
  split into lists of tokens. They will then be indexed or vectorized.
  
  `0` is a reserved index that won't be assigned to any word.
class_docstr_tok: ["Text","tokenization","utility","class",".","This","class","allows","to","vectorize","a","text","corpus",",","by","turning","each","text","into","either","a","sequence","of","integers","(","each","integer","being","the","index","of","a","token","in","a","dictionary",")","or","into","a","vector","where","the","coefficient","for","each","token","could","be","binary",",","based","on","word","count",",","based","on","tf-idf","...","#","Arguments","num_words",":","the","maximum","number","of","words","to","keep",",","based","on","word","frequency",".","Only","the","most","common","`","num_words-1","`","words","will","be","kept",".","filters",":","a","string","where","each","element","is","a","character","that","will","be","filtered","from","the","texts",".","The","default","is","all","punctuation",",","plus","tabs","and","line","breaks",",","minus","the","`","'","`","character",".","lower",":","boolean",".","Whether","to","convert","the","texts","to","lowercase",".","split",":","str",".","Separator","for","word","splitting",".","char_level",":","if","True",",","every","character","will","be","treated","as","a","token",".","oov_token",":","if","given",",","it","will","be","added","to","word_index","and","used","to","replace","out-of-vocabulary","words","during","text_to_sequence","calls","analyzer",":","function",".","Custom","analyzer","to","split","the","text",".","The","default","analyzer","is","text_to_word_sequence","By","default",",","all","punctuation","is","removed",",","turning","the","texts","into","space-separated","sequences","of","words","(","words","maybe","include","the","`","'","`","character",")",".","These","sequences","are","then","split","into","lists","of","tokens",".","They","will","then","be","indexed","or","vectorized",".","`","0","`","is","a","reserved","index","that","wo","n't","be","assigned","to","any","word","."]
docstr: Updates internal vocabulary based on a list of texts.
  
  In the case where texts contains lists,
  we assume each entry of the lists to be a token.
  
  Required before using `texts_to_sequences` or `texts_to_matrix`.
  
  # Arguments
      texts: can be a list of strings,
          a generator of strings (for memory-efficiency),
          or a list of list of strings.
docstr_tok: ["Updates","internal","vocabulary","based","on","a","list","of","texts",".","In","the","case","where","texts","contains","lists",",","we","assume","each","entry","of","the","lists","to","be","a","token",".","Required","before","using","`","texts_to_sequences","`","or","`","texts_to_matrix","`",".","#","Arguments","texts",":","can","be","a","list","of","strings",",","a","generator","of","strings","(","for","memory-efficiency",")",",","or","a","list","of","list","of","strings","."]
code: def fit_on_texts(self, texts):
      """Updates internal vocabulary based on a list of texts.
  
      In the case where texts contains lists,
      we assume each entry of the lists to be a token.
  
      Required before using `texts_to_sequences` or `texts_to_matrix`.
  
      # Arguments
          texts: can be a list of strings,
              a generator of strings (for memory-efficiency),
              or a list of list of strings.
      """
      for text in texts:
          self.document_count += 1
          if self.char_level or isinstance(text, list):
              if self.lower:
                  if isinstance(text, list):
                      text = [text_elem.lower() for text_elem in text]
                  else:
                      text = text.lower()
              seq = text
          else:
              if self.analyzer is None:
                  seq = text_to_word_sequence(
                      text,
                      filters=self.filters,
                      lower=self.lower,
                      split=self.split,
                  )
              else:
                  seq = self.analyzer(text)
          for w in seq:
              if w in self.word_counts:
                  self.word_counts[w] += 1
              else:
                  self.word_counts[w] = 1
          for w in set(seq):
              # In how many documents each word occurs
              self.word_docs[w] += 1
  
      wcounts = list(self.word_counts.items())
      wcounts.sort(key=lambda x: x[1], reverse=True)
      # forcing the oov_token to index 1 if it exists
      if self.oov_token is None:
          sorted_voc = []
      else:
          sorted_voc = [self.oov_token]
      sorted_voc.extend(wc[0] for wc in wcounts)
  
      # note that index 0 is reserved, never assigned to an existing word
      self.word_index = dict(
          zip(sorted_voc, list(range(1, len(sorted_voc) + 1)))
      )
  
      self.index_word = {c: w for w, c in self.word_index.items()}
  
      for w, c in list(self.word_docs.items()):
          self.index_docs[self.word_index[w]] = c
  
imports: import json
  import warnings
  from collections import OrderedDict, defaultdict
  from hashlib import md5
  import numpy
label_desc: Use image node_id 2 for calling the Tokenizer obj's underlying member method code with example usage: obj.fit_on_texts(texts) without return types