title: Keras tf backend predict speed slow for batch size of 1
body: <p>I am combining a Monte-Carlo Tree Search with a convolutional neural network as the rollout policy. I've identified the Keras model.predict function as being very slow. After experimentation, I found that surprisingly model parameter size and prediction sample size don't affect the speed significantly. For reference:</p>
  
  <ul>
  <li>0.00135549 s for 3 samples with batch_size = 3</li>
  <li>0.00303991 s for 3 samples with batch_size = 1</li>
  <li>0.00115528 s for 1 sample with batch_size = 1</li>
  <li>0.00136132 s for 10 samples with batch_size = 10</li>
  </ul>
  
  <p>as you can see I can predict 10 samples at about the same speed as 1 sample. The change is also very minimal though noticeable if I decrease parameter size by 100X but I'd rather not change parameter size by that much anyway. In addition, the predict function is very slow the first time run through (~0.2s) though I don't think that's the problem here since the same model is predicting multiple times. </p>
  
  <p>I wonder if there is some workaround because clearly the 10 samples can be evaluated very quickly, all I want to be able to do is predict the samples at different times and not all at once since I need to update the Tree Search before making a new prediction. Perhaps should I work with tensorflow instead?</p>