title: Difference between split() and tokenize()
body: <p>I am counting the number of unique words in a loaded text file. However I have realized that when I use <code>split()</code> and tensorflow_datasets <code>tokenize()</code> I get different results yet I thought they achieve the same thing. Here is my code. Can somebody help me know the difference between the two.</p>
  
  <pre><code>import tensorflow as tf
  import tensorflow_datasets as tfds
  
  tf.enable_eager_execution()
  
  BUFFER_SIZE = 50000
  TAKE_SIZE = 5000
  BATCH_SIZE = 64
  
  tokenizer = tfds.features.text.Tokenizer()
  data = open("news.2011.en.shuffled","r").read()
  vocab = list(set(data.split()))  # gives more count
  print(len(vocab))
  
  tokenized_data = tokenizer.tokenize(data)
  print(len(set(tokenized_data)))  # gives less count
  </code></pre>