title: Getting the number of words from tf.Tokenizer after fitting
body: <p>I initially tried making an RNN that can predict Shakespeare text, and I did it successfully using character level-encoding. But when I switched to word level encoding, I ran into a multitude of issues. Specifically, I am having a hard time getting the total
  number of characters (I was told it was just <strong>dataset_size = tokenizer.document_count</strong> but this just returns <em>1</em> )  so that I can set <strong>steps_per_epoch = dataset_size // batch_size</strong> when fitting my model (Now, both char and word level encoding return 1). I tried setting <strong>dataset_size = sum(tokenizer.word_counts.values())</strong> but when I fit the model, I get this error right before the first epoch ends:</p>
  <blockquote>
  <p>WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 32 batches). You may need to use the repeat() function when building your dataset.</p>
  </blockquote>
  <p>So I assume that my code believes that I have slightly more training sets available than I actually do. Or it may be the fact that I am programming on the new M1 chip which doesn't have a production version of TF? So really, I'm just not sure how to get the exact number of words in this text.</p>
  <p>Here's the code:</p>
  <pre><code>import tensorflow as tf
  from tensorflow import keras
  import numpy as np
  import re 
  
  shakespeare_url = &quot;https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt&quot;
  filepath = keras.utils.get_file(&quot;shakespeare.txt&quot;, shakespeare_url)
  
  with open(filepath) as f:
      shakespeare_text = f.read()
  
  tokenizer = keras.preprocessing.text.Tokenizer(char_level=False) #Set to word-level encoding
  tokenizer.fit_on_texts([shakespeare_text])
  
  max_id = len(tokenizer.word_index) # number of distinct characters
  #dataset_size = sum(tokenizer.word_counts.values()) #Returns 204089
  dataset_size = tokenizer.document_count # Returns 1
  </code></pre>
  <p>Thanks:)</p>