title: Generator for `model.fit()` Input rather than Collection? (training data too large for memory)
body: <p>tf-learn models' fit function can be passed training and test data like so:</p>
  
  <pre><code>model = tflearn.DNN(nn)
  
  model.fit({'input': X_train},
            {'targets': Y_train},
            n_epoch=10,
            validation_set=(
                {'input': X_test},
                {'targets': Y_test}
            ))
  </code></pre>
  
  <p>where <code>nn</code> is the definition of a model. However, what if the collections such as <code>X_train</code> are too large for memory?</p>
  
  <p>In my case I have compressed sparse binary vectors (cells either being 1 or 0) in the form of a list of indices and an integer coding for the number of dimensions of the vector, which allow me to reconstruct the original vectors.</p>
  
  <p>That collections of compressed vectors do fit into memory, not the collections containing the full vectors though. Therefore I tried to instead pass a generator over <code>X_train</code> and the other collections (now containing the compressed vectors) that reconstructs the full vectors on the fly, but <code>model.fit</code> required a <code>len()</code> function. So I defined a custom feeder class like so:</p>
  
  <pre><code>class Feeder:
      def __init__(self, data, convert):
          self.data = data
          self.convert = convert
  
      def __len__(self):
          return len(self.data)
  
      def __iter__(self):
          return self
  
      def __next__(self):
          for item in self.data:
              yield self.convert(item)
  </code></pre>
  
  <p>which I call like this:</p>
  
  <pre><code>def reconstruct_vector(non_zero_indices, dimensionality):
      """
      returns a vector of zeros and ones reconstructed from a sparse vector
      and a dimensions value
      """
      vec = np.zeros(dimensionality)
      for i in non_zero_intraining data too large for memorydices:
          vec[i] = 1
      return vec
  
  item_to_input_vector = lambda item : reconstruct_vector(item[0], item[1])
  item_to_target_vector = lambda item : np.array([1,0]) if item else np.array([0,1])
  
  model.fit({'input': Feeder(X_train, item_to_input_vector)},
            {'targets': Feeder(Y_train, item_to_target_vector)},
            n_epoch=10,
            validation_set=(
                {'input': Feeder(X_test, item_to_input_vector)},
                {'targets': Feeder(Y_test, item_to_target_vector)}
            ))
  </code></pre>
  
  <p>But that also does not work, as I get some cryptic error:</p>
  
  <pre><code>Exception in thread Thread-3:
  Traceback (most recent call last):
    File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
      self.run()
    File "/usr/lib64/python3.6/threading.py", line 864, in run
      self._target(*self._args, **self._kwargs)
    File "/usr/lib/python3.6/site-packages/tflearn/data_flow.py", line 187, in fill_feed_dict_queue
      data = self.retrieve_data(batch_ids)
    File "/usr/lib/python3.6/site-packages/tflearn/data_flow.py", line 222, in retrieve_data
      utils.slice_array(self.feed_dict[key], batch_ids)
    File "/usr/lib/python3.6/site-packages/tflearn/utils.py", line 187, in slice_array
      return X[start]
  TypeError: only integer scalar arrays can be converted to a scalar index
  </code></pre>
  
  <p>So, what would be the correct way to go about this?</p>