title: TensorFlow/Keras multi-threaded model fitting
body: <p>I'm attempting to train multiple <code>keras</code> models with different parameter values using multiple threads (and the <code>tensorflow</code> backend). I've seen a few examples of using the same model within multiple threads, but in this particular case, I run into various errors regarding conflicting graphs, etc. Here's a simple example of what I'd like to be able to do:</p>
  
  <pre class="lang-py prettyprint-override"><code>from concurrent.futures import ThreadPoolExecutor
  import numpy as np
  import tensorflow as tf
  from keras import backend as K
  from keras.layers import Dense
  from keras.models import Sequential
  
  
  sess = tf.Session()
  
  
  def example_model(size):
      model = Sequential()
      model.add(Dense(size, input_shape=(5,)))
      model.add(Dense(1))
      model.compile(optimizer='sgd', loss='mse')
      return model
  
  
  if __name__ == '__main__':
      K.set_session(sess)
      X = np.random.random((10, 5))
      y = np.random.random((10, 1))
      models = [example_model(i) for i in range(5, 10)]
  
      e = ThreadPoolExecutor(4)
      res_list = [e.submit(model.fit, X, y) for model in models]
  
      for res in res_list:
          print(res.result())
  </code></pre>
  
  <p>The resulting error is <code>ValueError: Tensor("Variable:0", shape=(5, 5), dtype=float32_ref) must be from the same graph as Tensor("Variable_2/read:0", shape=(), dtype=float32).</code>. I've also tried initializing the models within the threads which gives a similar failure. </p>
  
  <p>Any thoughts on the best way to go about this? I'm not at all attached to this exact structure, but I'd prefer to be able to use multiple threads rather than processes so all the models are trained within the same GPU memory allocation.</p>