How to preprocess tensorflow imdb_review dataset I am using tensorflow imdb_review dataset, and I want to preprocess it using Tokenizer and pad_sequences
When I am using the Tokenizer instance and using the following code:
tokenizer=Tokenizer(num_words=100)
tokenizer.fit_on_texts(df['text'])
word_index = tokenizer.word_index
sequences=tokenizer.texts_to_sequences(df['text'])

print(word_index)
print(sequences)

I am getting the error
TypeError: a bytes-like object is required, not 'dict'

What I've tried
store dataset as dataframe and then iterate over the text column, and store it in a list, and then tokenize it.
df = tfds.as_dataframe(ds.take(4), info)
# list to store corpus
corpus = []
for sentences in df['text'].iteritems():
  corpus.append(sentences)

tokenizer=Tokenizer(num_words=100)
tokenizer.fit_on_texts(corpus)
word_index=tokenizer.word_index
print(word_index)

But i'm getting the error
AttributeError: 'tuple' object has no attribute 'lower'

How can I use the text column and preprocess it to feed it to my neural network?
For TypeErrors it is always good to check the exact type of the variable causing issues, especially when troubleshooting TensorFlow and Keras:
print(type(df['text']))

fit_on_texts expects a list of string or similar, but you are providing a dictionary, so you'll want to convert accordingly. Example:
from typing import List, Dict

foo_dict: Dict[str, str] = {&quot;one&quot;: &quot;first sentence&quot;, &quot;two&quot;: &quot;second sentence&quot;}
foo_list: List[str] = list(foo_dict.values())

print(foo_dict)
print(type(foo_dict))

print(foo_list)
print(type(foo_list))

Output:
{'one': 'first sentence', 'two': 'second sentence'}
&lt;class 'dict'&gt;
['first sentence', 'second sentence']
&lt;class 'list'&gt;

API docs will help here:
https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer#fit_on_texts
I'm using type annotations in the example above which I find helps troubleshooting and avoiding bugs.