title: Tensorflow 1.9.0: A question about training my model with generators
body: <h1>Situation</h1>
  <p>I am trying to build my own CNN with images I compiled.
  I have two folders with training and validation data. This is my code so far:</p>
  <pre class="lang-py prettyprint-override"><code>import os
  import sys
  import shutil
  import numpy as np
  import tensorflow as tf
  
  SCRIPT_DIR = os.getcwd()
  
  TRAIN_GRAPH = 'training_graph.pb'
  CHKPT_FILE = 'float_model.ckpt'
  
  CHKPT_DIR = os.path.join(SCRIPT_DIR, 'chkpts')
  TB_LOG_DIR = os.path.join(SCRIPT_DIR, 'tb_logs')
  CHKPT_PATH = os.path.join(CHKPT_DIR, CHKPT_FILE)
  CNN_DIR = os.path.join(SCRIPT_DIR, 'cnn_dir')
  
  # create a directory for the dataset if it doesn't already exist
  if not (os.path.exists(CNN_DIR)):
      os.makedirs(CNN_DIR)
      print(&quot;Directory &quot; , CNN_DIR ,  &quot;created &quot;) 
  
  # create a directory for the TensorBoard data if it doesn't already exist
  # delete it and recreate if it already exists
  if (os.path.exists(TB_LOG_DIR)):
      shutil.rmtree(TB_LOG_DIR)
  os.makedirs(TB_LOG_DIR)
  print(&quot;Directory &quot; , TB_LOG_DIR ,  &quot;created &quot;) 
  
  # create a directory for the checkpoint if it doesn't already exist
  # delete it and recreate if it already exists
  if (os.path.exists(CHKPT_DIR)):
      shutil.rmtree(CHKPT_DIR)
  os.makedirs(CHKPT_DIR)
  print(&quot;Directory &quot; , CHKPT_DIR ,  &quot;created &quot;)
  
  LEARN_RATE = 0.0001
  BATCHSIZE = 64
  EPOCHS = 10
  IMG_SIZE = 224
  
  # ===== IMPORTANT PART STARTS HERE ================================
  
  # SETTING UP DATA ---------------
  datagen_training = tf.keras.preprocessing.image.ImageDataGenerator(
      rescale = 1./255,
      shear_range = 0.2,
      zoom_range = 0.2,
      horizontal_flip = True)
  
  dataset_training = datagen_training.flow_from_directory(
          &quot;./dataset/prepared/training&quot;,
          target_size = (IMG_SIZE, IMG_SIZE),
          batch_size = BATCHSIZE,
          class_mode = &quot;binary&quot;)
  
  datagen_test = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1./255)
  
  dataset_test = datagen_training.flow_from_directory(
          &quot;./dataset/prepared/validation&quot;,
          target_size=(IMG_SIZE, IMG_SIZE),
          batch_size=BATCHSIZE,
          class_mode=&quot;binary&quot;)
  # SETTING UP CNN ---------------
  
  cnn = tf.keras.models.Sequential()
  
  # First convolutional layer
  cnn.add(tf.keras.layers.Conv2D(
      filters = 16, 
      kernel_size = 3, 
      activation = tf.nn.relu, 
      input_shape = [IMG_SIZE, IMG_SIZE, 3]))
  
  # Maxpooling layer
  cnn.add(tf.keras.layers.MaxPool2D(
      pool_size = 2, 
      strides = 2))
  
  # Second convolutional layer
  cnn.add(tf.keras.layers.Conv2D(
      filters = 32, 
      kernel_size = 3, 
      activation = tf.nn.relu))
    
  # Flattening the vector
  cnn.add(tf.keras.layers.Flatten())
          
  # Fully connected layer #1
  # Hidden layer with 256 neurons
  cnn.add(tf.keras.layers.Dense(
      units = 256,
      activation = tf.keras.activations.softmax))
  
  # Fully connected layer #2
  # Densing down to 8 variables, output layer
  cnn.add(tf.keras.layers.Dense(
      units = 8,
      activation = None))
  
  cnn.compile(optimizer = &quot;adam&quot;, loss = &quot;binary_crossentropy&quot;, metrics = [&quot;accuracy&quot;])
  
  # PROBLEM LIES HERE ------------------------
  cnn.fit(x = dataset_training, 
      validation_data = dataset_test,
      epochs = 10)
  </code></pre>
  <p>When I execute this, I get this error:</p>
  <blockquote>
  <p>--------------------------------------------------------------------------- AttributeError                            Traceback (most recent call
  last)  in 
  1 cnn.fit(x = dataset_training,
  2     validation_data = dataset_test,
  ----&gt; 3     epochs = 10)</p>
  <p>~/anaconda3/envs/decent/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py
  in fit(self, x, y, batch_size, epochs, verbose, callbacks,
  validation_split, validation_data, shuffle, class_weight,
  sample_weight, initial_epoch, steps_per_epoch, validation_steps,
  **kwargs)    1261         steps_name='steps_per_epoch',    1262         steps=steps_per_epoch,
  -&gt; 1263         validation_split=validation_split)    1264     1265     # Prepare validation data.</p>
  <p>~/anaconda3/envs/decent/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py
  in _standardize_user_data(self, x, y, sample_weight, class_weight,
  batch_size, check_steps, steps_name, steps, validation_split)
  866         feed_input_shapes,
  867         check_batch_axis=False,  # Don't enforce the batch size.
  --&gt; 868         exception_prefix='input')
  869
  870     if y is not None:</p>
  <p>~/anaconda3/envs/decent/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_utils.py
  in standardize_input_data(data, names, shapes, check_batch_axis,
  exception_prefix)
  141     data = data.values if data.<strong>class</strong>.<strong>name</strong> == 'DataFrame' else data
  142     data = [data]
  --&gt; 143   data = [standardize_single_array(x) for x in data]
  144
  145   if len(data) != len(names):</p>
  <p>~/anaconda3/envs/decent/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_utils.py
  in (.0)
  141     data = data.values if data.<strong>class</strong>.<strong>name</strong> == 'DataFrame' else data
  142     data = [data]
  --&gt; 143   data = [standardize_single_array(x) for x in data]
  144
  145   if len(data) != len(names):</p>
  <p>~/anaconda3/envs/decent/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_utils.py
  in standardize_single_array(x)
  79   elif tensor_util.is_tensor(x):
  80     return x
  ---&gt; 81   elif x.ndim == 1:
  82     x = np.expand_dims(x, 1)
  83   return x</p>
  <p>AttributeError: 'DirectoryIterator' object has no attribute 'ndim'</p>
  </blockquote>
  <p>The tutorial I am following probably uses a more recent version of Tensorflow. However, due to restrictions in my system I need to use the version I am using. Now <a href="https://stackoverflow.com/questions/52779602/error-attributeerror-directoryiterator-object-has-no-attribute-ndim-in-auto/52780095">according to this</a> SO post, I need to use <code>fit_generator(...)</code>. Looking at <a href="https://www.geeksforgeeks.org/keras-fit-and-keras-fit_generator/" rel="nofollow noreferrer">this post</a> the syntax of <code>fit_generator(...)</code> is:</p>
  <pre class="lang-py prettyprint-override"><code>fit_generator(object, generator, steps_per_epoch, epochs = 1,
    verbose = getOption(&quot;keras.fit_verbose&quot;, default = 1),
    callbacks = NULL, view_metrics = getOption(&quot;keras.view_metrics&quot;,
    default = &quot;auto&quot;), validation_data = NULL, validation_steps = NULL,
    class_weight = NULL, max_queue_size = 10, workers = 1,
    initial_epoch = 0)
  </code></pre>
  <p>Which means, I need to give the function a generator. However, the example given in the post looks like this:</p>
  <pre class="lang-py prettyprint-override"><code># performing data argumentation by training image generator
  dataAugmentaion = ImageDataGenerator(rotation_range = 30, zoom_range = 0.20, 
  fill_mode = &quot;nearest&quot;, shear_range = 0.20, horizontal_flip = True, 
  width_shift_range = 0.1, height_shift_range = 0.1)
  
  # training the model
  model.fit_generator(dataAugmentaion.flow(trainX, trainY, batch_size = 32),
   validation_data = (testX, testY), steps_per_epoch = len(trainX) // 32,
   epochs = 10)
  </code></pre>
  <p>Which is weird for me. Where does the <code>trainX</code> and <code>trainY</code> come from? I don't have variables like that, because I (don't seem to) have some kind of vector of input and output data, at least it's not obvious for me where those arrays/vectors/tuples come from.
  I also noticed in another example from Avnet, that they also get such kind of data:</p>
  <pre><code>mnist_dataset = tf.keras.datasets.mnist.load_data('mnist_data')
  (x_train, y_train), (x_test, y_test) = mnist_dataset
  </code></pre>
  <p>I don't get what is in those arrays, how that data is formatted, and what to use them for. I understand that I need some kind of input vector/image, which, in my code, should be provided by the <code>ImageDataGenerator</code>, right? There seems to be a big thing I am missing to understand how this works. The tutorial I am following doesn't elaborate further on this, since they can just plug in the generators into <code>cnn.fit(...)</code>.</p>
  <hr />
  <h1>Questions</h1>
  <p><strong>So I guess my questions are the following:</strong></p>
  <ol>
  <li>What are those arrays like <code>trainX</code> and <code>trainY</code> or <code>(x_train, y_train), (x_test, y_test)</code>?</li>
  <li>What do I need them for?</li>
  <li>How are they formatted?</li>
  <li>How can I extract the data in that format from my <code>ImageDataGenerator</code> or rather from the <code>DirectoryIterator</code> which I use to &quot;load&quot; my data with <code>flow_from_directory(...)</code> (here: <code>dataset_training</code> and <code>dataset_test</code>)</li>
  <li>Any other recommendations or suggestions for my code/methodology?</li>
  </ol>
  <p><strong>Tensorflow Version: 1.9.0</strong></p>
  <p><strong>Keras Version: 2.1.6-tf</strong></p>
  <hr />
  <h1>Update</h1>
  <p>According to <a href="https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator" rel="nofollow noreferrer">this post</a>, the function <code>fit_generator(...)</code> should look a bit like this:</p>
  <pre><code># ...
  model.fit(
          train_generator,
          steps_per_epoch=2000,
          epochs=50,
          validation_data=validation_generator,
          validation_steps=800)
  # ...
  </code></pre>
  <p>I changed the <code>fit</code> into <code>fit_generator</code>. Which basically results in this:</p>
  <pre class="lang-py prettyprint-override"><code>cnn.fit_generator(
          generator=datagen_training,
          steps_per_epoch=50000/32,
          epochs=EPOCHS,
          validation_data=datagen_test,
          validation_steps=800
  )
  </code></pre>
  <p>Unfortunately, that throws this error:</p>
  <blockquote>
  <p>--------------------------------------------------------------------------- TypeError                                 Traceback (most recent call
  last)  in 
  9         epochs=EPOCHS,
  10         validation_data=datagen_test,
  ---&gt; 11         validation_steps=800
  12 )</p>
  <p>~/anaconda3/envs/decent/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py
  in fit_generator(self, generator, steps_per_epoch, epochs, verbose,
  callbacks, validation_data, validation_steps, class_weight,
  max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
  1759         use_multiprocessing=use_multiprocessing,    1760<br />
  shuffle=shuffle,
  -&gt; 1761         initial_epoch=initial_epoch)    1762     1763   def evaluate_generator(self,</p>
  <p>~/anaconda3/envs/decent/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose,
  callbacks, validation_data, validation_steps, class_weight,
  max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
  112     if do_validation and not val_gen:
  113       # Prepare data for validation
  --&gt; 114       if len(validation_data) == 2:
  115         val_x, val_y = validation_data  # pylint: disable=unpacking-non-sequence
  116         val_sample_weight = None</p>
  <p>TypeError: object of type 'ImageDataGenerator' has no len()</p>
  </blockquote>