title: image_dim_ordering - what am I missing here?
body: <p>EDIT: Could not reproduce this issue using cuda 8.0 and using titan X (Pascal)</p>
  
  <p>Using tensorflow backend for keras I have issues that are related to image_dim_ordering.
  When I use image_dim_ordering='th' in the keras config file, everything works well But when I use 'tf', training simply doesn't really improve much from 0.5 accuracy.</p>
  
  <p>The motivation is that currently my live augmentations are very costly, and I'd love to remove the unneeded reshape from theano dim order convention to tensorflow.</p>
  
  <p>I tried recreating the issue with simple  code to allow reproduction by other people which may assist me to  understand what am I doing wrong here. I'm well aware of the channel,height,width different conventions, and at least I think that I handle that.</p>
  
  <p>While I didn't fully reproduce my problem in the compact example (maybe because it's a trivial task), the training results are repeatedly different, and much worse for the 'tf' case, even when I try different seed values.
  Note - in this reproducing code, all that the network needs to do is to tell apart full patches of -1.0 from full patches of 1.0</p>
  
  <p>This is my '~/.keras/keras.json'</p>
  
  <pre><code>{
      "floatx": "float32",
      "epsilon": 1e-07,
      "backend": "tensorflow",
      "image_dim_ordering": "th"  
  }
  </code></pre>
  
  <p>my tensorflow version is ''0.11.0rc0'' (it happened on 0,10 as well)
  my keras is latest git pull of today. </p>
  
  <p>Using 'th' for the image_dim_ordering I get accuracy >=0.99 at epoch 4 for three different seeds.
  Using 'tf' for the dim order I get accuracy >= 0.9 much latest as you can see below in the log, only at around epoch 24</p>
  
  <p><strong>The following is a standalone code that should reproduce the problem:</strong></p>
  
  <pre><code>from keras import backend as K
  import keras.optimizers
  from keras.layers import Convolution2D, MaxPooling2D
  from keras.layers import Activation, Dropout, Flatten, Dense, Input
  from keras.models import Model
  import numpy as np
  
  def make_model(input_dim_size):
      if K.image_dim_ordering() == 'tf':
          input_shape = (input_dim_size, input_dim_size,1)
      else:
          input_shape = (1, input_dim_size, input_dim_size)
      img_input = Input(shape=input_shape)
  
      x = Convolution2D(64,5,5,border_mode='same')(img_input)
      x = Activation('relu')(x)
      x = MaxPooling2D((2,2),strides=(2,2))(x)
  
      x = Convolution2D(64, 5, 5, border_mode='same')(x)
      x = Activation('relu')(x)
      x = MaxPooling2D((2, 2), strides=(2, 2))(x)
  
      x = Convolution2D(64, 5, 5, border_mode='same')(x)
      x = Activation('relu')(x)
      x = MaxPooling2D((2, 2), strides=(2, 2))(x)
  
      x = Convolution2D(128, 5, 5, border_mode='same')(x)
      x = Activation('relu')(x)
      x = MaxPooling2D((2, 2), strides=(2, 2))(x)
  
      x = Convolution2D(128, 5, 5, border_mode='same')(x)
      x = Activation('relu')(x)
      x = MaxPooling2D((2, 2), strides=(2, 2))(x)
  
      x = Flatten()(x)
      x = Dense(1024*2)(x)
      x = Activation('relu')(x)
      x = Dropout(0.5)(x)
  
      x = Dense(1024 * 2)(x)
      x = Activation('relu')(x)
      x = Dropout(0.75)(x)
  
      x = Dense(200)(x)
      x = Activation('relu')(x)
      x = Dropout(0.75)(x)
  
      x = Dense(1,activation='sigmoid')(x)
  
      model = Model(img_input, x)
  
      learning_rate = 0.01
  
      sgd = keras.optimizers.sgd(lr=learning_rate, momentum=0.9, nesterov=True)
  
      model.summary()
  
      model.compile(loss='binary_crossentropy',
                    optimizer=sgd,
                    metrics=['accuracy']
                    )
      return model
  
  np.random.seed(456)
  
  def dummy_generator(mini_batch_size=64, block_size=100):
      if K.image_dim_ordering() == 'tf':
          tensor_X_shape = (mini_batch_size,block_size, block_size,1)
      else:
          tensor_X_shape = (mini_batch_size, 1, block_size, block_size)
  
      X = np.zeros(tensor_X_shape, dtype=np.float32)
      y = np.zeros((mini_batch_size, 1))
  
      while True:
          for b in range(mini_batch_size):
              X[b, :, :, :] = (float(b % 2) * 2.0) - 1.0
              y[b, :] = float(b % 2)
          yield X,y
  
  with K.tf.device('/gpu:2'):
      K.set_session(K.tf.Session(config=K.tf.ConfigProto(allow_soft_placement=True, log_device_placement=False)))
      MINI_BATCH_SIZE = 64
      PATCH_SIZE = 100
      model = make_model(PATCH_SIZE)
      gen = dummy_generator(mini_batch_size=MINI_BATCH_SIZE,block_size=PATCH_SIZE)
      model.fit_generator(gen, MINI_BATCH_SIZE*10,
                          100, verbose=1,
                          callbacks=[],
                          validation_data=None,
                          nb_val_samples=None,
                          max_q_size=1,
                          nb_worker=1, pickle_safe=False)
  </code></pre>
  
  <p>For the 'tf' case this is the training log: (and looks very similar on different seeds):</p>
  
  <pre><code>Epoch 1/100
  640/640 [==============================] - 1s - loss: 0.6932 - acc: 0.4781     
  Epoch 2/100
  640/640 [==============================] - 0s - loss: 0.6932 - acc: 0.4938     
  Epoch 3/100
  640/640 [==============================] - 0s - loss: 0.6921 - acc: 0.5203     
  Epoch 4/100
  640/640 [==============================] - 0s - loss: 0.6920 - acc: 0.5469     
  Epoch 5/100
  640/640 [==============================] - 0s - loss: 0.6935 - acc: 0.4875     
  Epoch 6/100
  640/640 [==============================] - 0s - loss: 0.6941 - acc: 0.4969     
  Epoch 7/100
  640/640 [==============================] - 0s - loss: 0.6937 - acc: 0.5047     
  Epoch 8/100
  640/640 [==============================] - 0s - loss: 0.6931 - acc: 0.5312     
  Epoch 9/100
  640/640 [==============================] - 0s - loss: 0.6923 - acc: 0.5250     
  Epoch 10/100
  640/640 [==============================] - 0s - loss: 0.6929 - acc: 0.5281     
  Epoch 11/100
  640/640 [==============================] - 0s - loss: 0.6934 - acc: 0.4953     
  Epoch 12/100
  640/640 [==============================] - 0s - loss: 0.6918 - acc: 0.5234     
  Epoch 13/100
  640/640 [==============================] - 0s - loss: 0.6930 - acc: 0.5125     
  Epoch 14/100
  640/640 [==============================] - 0s - loss: 0.6939 - acc: 0.4797     
  Epoch 15/100
  640/640 [==============================] - 0s - loss: 0.6936 - acc: 0.5047     
  Epoch 16/100
  640/640 [==============================] - 0s - loss: 0.6917 - acc: 0.4922     
  Epoch 17/100
  640/640 [==============================] - 0s - loss: 0.6945 - acc: 0.4891     
  Epoch 18/100
  640/640 [==============================] - 0s - loss: 0.6948 - acc: 0.5000     
  Epoch 19/100
  640/640 [==============================] - 0s - loss: 0.6968 - acc: 0.4594     
  Epoch 20/100
  640/640 [==============================] - 0s - loss: 0.6919 - acc: 0.5391     
  Epoch 21/100
  640/640 [==============================] - 0s - loss: 0.6904 - acc: 0.5172     
  Epoch 22/100
  640/640 [==============================] - 0s - loss: 0.6881 - acc: 0.5906     
  Epoch 23/100
  640/640 [==============================] - 0s - loss: 0.6804 - acc: 0.6359     
  Epoch 24/100
  640/640 [==============================] - 0s - loss: 0.6470 - acc: 0.8219     
  Epoch 25/100
  640/640 [==============================] - 0s - loss: 0.4134 - acc: 0.9625     
  Epoch 26/100
  640/640 [==============================] - 0s - loss: 0.2347 - acc: 0.9953     
  Epoch 27/100
  640/640 [==============================] - 0s - loss: 0.1231 - acc: 1.0000 
  </code></pre>
  
  <p>And for the 'th' case the training log is (and looks very similar on different seeds):</p>
  
  <pre><code>Epoch 1/100
  640/640 [==============================] - 3s - loss: 0.6891 - acc: 0.5594     
  Epoch 2/100
  640/640 [==============================] - 2s - loss: 0.6079 - acc: 0.7328     
  Epoch 3/100
  640/640 [==============================] - 2s - loss: 0.3166 - acc: 0.9422     
  Epoch 4/100
  640/640 [==============================] - 2s - loss: 0.1767 - acc: 0.9969  
  </code></pre>
  
  <p>I find it suspicious that it's so fast in the tensorflow case, (0s), but after adding debug printing to the generator it does seem to get called.
  I thought that maybe it's related to keras not needing to reshape anything, but 2-3 seconds sounds too much time for this amount of reshapes </p>
  
  <p>If anyone can try to reproduce the results that I see and help me understand what the heck am I missing here, I'd be grateful :)</p>