title: How do I go from Pandas DataFrame to Tensorflow BatchDataset for NLP?
body: <p>I'm honestly trying to figure out how to convert a dataset (format: pandas <code>DataFrame</code> or numpy array) to a form that a simple text-classification tensorflow model can train on for sentiment analysis. The dataset I'm using is similar to IMDB (containing both text and labels (positive or negative)). Every tutorial I've looked at has either prepared data differently, or didn't bother with data preparation and left it to your imagination. (For instance, all the IMDB tutorials import a preprocessed Tensorflow <code>BatchDataset</code> from <code>tensorflow_datasets</code>, which isn't helpful when I'm using my own set of data). My own attempts to convert a Pandas <code>DataFrame</code> to Tensorflow's <code>Dataset</code> types have resulted in ValueErrors or a negative loss during training. Any help would be appreciated.</p>
  
  <p>I had originally prepared my data as follows, where <code>training</code> and <code>validation</code> are already shuffled Pandas <code>DataFrame</code>s containing <code>text</code> and <code>label</code> columns:</p>
  
  <pre><code># IMPORT STUFF
  
  from __future__ import absolute_import, division, print_function, unicode_literals
  import tensorflow as tf # (I'm using tensorflow 2.0)
  from tensorflow import keras
  from tensorflow.keras import layers
  from tensorflow.keras.preprocessing.text import Tokenizer
  import pandas as pd
  import numpy as np
  # ... [code for importing and preparing the pandas dataframe omitted]
  
  # TOKENIZE
  
  train_text = training['text'].to_numpy()
  tok = Tokenizer(oov_token='&lt;unk&gt;')
  tok.fit_on_texts(train_text)
  tok.word_index['&lt;pad&gt;'] = 0
  tok.index_word[0] = '&lt;pad&gt;'
  
  train_seqs = tok.texts_to_sequences(train_text)
  train_seqs = tf.keras.preprocessing.sequence.pad_sequences(train_seqs, padding='post')
  
  train_labels = training['label'].to_numpy().flatten()
  
  valid_text = validation['text'].to_numpy()
  valid_seqs = tok.texts_to_sequences(valid_text)
  valid_seqs = tf.keras.preprocessing.sequence.pad_sequences(valid_seqs, padding='post')
  
  valid_labels = validation['label'].to_numpy().flatten()
  
  # CONVERT TO TF DATASETS
  
  train_ds = tf.data.Dataset.from_tensor_slices((train_seqs,train_labels))
  valid_ds = tf.data.Dataset.from_tensor_slices((valid_seqs,valid_labels))
  
  train_ds = train_ds.shuffle(BUFFER_SIZE).batch(BATCH_SIZE)
  valid_ds = valid_ds.batch(BATCH_SIZE)
  
  # PREFETCH
  
  train_ds = train_ds.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
  valid_ds = valid_ds.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
  </code></pre>
  
  <p>This resulted train_ds and valid_ds being tokenized and of type <code>PrefetchDataset</code> or <code>&lt;PrefetchDataset shapes: ((None, None, None, 118), (None, None, None)), types: (tf.int32, tf.int64)&gt;</code>.</p>
  
  <p>I then trained as follows, but got a <em>large negative loss and an accuracy of 0</em>.</p>
  
  <pre><code>model = keras.Sequential([
      layers.Embedding(vocab_size, embedding_dim),
      layers.GlobalAveragePooling1D(),
      layers.Dense(1, activation='sigmoid') # also tried activation='softmax'
  ])
  
  model.compile(optimizer='adam',
                loss='binary_crossentropy', # binary_crossentropy
                metrics=['accuracy'])
  
  history = model.fit(
      train_ds,
      epochs=1,
      validation_data=valid_ds, validation_steps=1, steps_per_epoch=BUFFER_SIZE)
  </code></pre>
  
  <p>If I don't do the fancy prefetch stuff, <code>train_ds</code> would be of type <code>BatchDataset</code> or <code>&lt;BatchDataset shapes: ((None, 118), (None,)), types: (tf.int32, tf.int64)&gt;</code>, but that also is getting me a negative loss and an accuracy of 0.</p>
  
  <p>And if I just do the following: </p>
  
  <pre><code>x, y = training['text'].to_numpy(), training['label'].to_numpy()
  x, y = tf.convert_to_tensor(x),tf.convert_to_tensor(y)
  </code></pre>
  
  <p>then <code>x</code> and <code>y</code> are of type <code>EagerTensor</code>, but I can't seem to figure out how to Batch an <code>EagerTensor</code>.</p>
  
  <p><strong>What types and shapes do I really need for <code>train_ds</code>? What am I missing or doing wrong?</strong> </p>
  
  <p>The <a href="https://www.tensorflow.org/tutorials/keras/text_classification_with_hub" rel="noreferrer">text_classification_with_hub tutorial</a> trains an already prepared imdb dataset as shown:</p>
  
  <pre><code>model = tf.keras.Sequential()
  model.add(hub_layer)
  model.add(tf.keras.layers.Dense(16, activation='relu'))
  model.add(tf.keras.layers.Dense(1, activation='sigmoid'))
  
  model.compile(optimizer='adam',
                loss='binary_crossentropy',
                metrics=['accuracy'])
  
  history = model.fit(train_data.shuffle(10000).batch(512),
                      epochs=20,
                      validation_data=validation_data.batch(512),
                      verbose=1)
  </code></pre>
  
  <p>In this example, <code>train_data</code> is of form <code>tensorflow.python.data.ops.dataset_ops._OptionsDataset</code>, and <code>train_data.shuffle(1000).batch(512)</code> is <code>tensorflow.python.data.ops.dataset_ops.BatchDataset</code> (or <code>&lt;BatchDataset shapes: ((None,), (None,)), types: (tf.string, tf.int64)&gt;</code>). </p>
  
  <p>They apparently didn't bother with tokenization with this dataset, but I doubt tokenization is my issue. Why does their <code>train_data.shuffle(10000).batch(512)</code> work but my <code>train_ds</code> not work? </p>
  
  <p>It's possible the issue is with the model setup, the <code>Embedding</code> layer, or with tokenization, but I'm not so sure that's the case. I've already looked at the following tutorials for inspiration:</p>
  
  <p><a href="https://www.tensorflow.org/tutorials/keras/text_classification_with_hub" rel="noreferrer">https://www.tensorflow.org/tutorials/keras/text_classification_with_hub</a></p>
  
  <p><a href="https://www.kaggle.com/drscarlat/imdb-sentiment-analysis-keras-and-tensorflow" rel="noreferrer">https://www.kaggle.com/drscarlat/imdb-sentiment-analysis-keras-and-tensorflow</a></p>
  
  <p><a href="https://www.tensorflow.org/tutorials/text/image_captioning" rel="noreferrer">https://www.tensorflow.org/tutorials/text/image_captioning</a></p>
  
  <p><a href="https://www.tensorflow.org/tutorials/text/word_embeddings#learning_embeddings_from_scratch" rel="noreferrer">https://www.tensorflow.org/tutorials/text/word_embeddings#learning_embeddings_from_scratch</a></p>
  
  <p><a href="https://thedatafrog.com/word-embedding-sentiment-analysis/" rel="noreferrer">https://thedatafrog.com/word-embedding-sentiment-analysis/</a></p>