Why is training take so long on my GPU? Details:
GPU: GTX 1080
Training: ~1.1 Million images belonging to 10 classes
Validation: ~150 Thousand images belonging to 10 classes
Time per Epoch: ~10 hours
I've setup CUDA, cuDNN and Tensorflow( Tensorflow GPU as well).
I don't think my model is that complicated that is takes 10 hours per epoch. I even checked if my GPU was the problem but it wasn't.
Is the training time due to the Fully connected layers?
My model:
model = Sequential()
model.add()
model.add(Conv2D(64, (3, 3), padding=&quot;same&quot;, strides=2))
model.add(Activation('relu'))
model.add(Dropout(0.25))
model.add(Conv2D(64, (3, 3), padding=&quot;same&quot;, strides=2))
model.add(Activation('relu'))
model.add(Dropout(0.25))
model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3, 3), strides=2))
model.add(Flatten())
model.add(Dense(256))
model.add(Activation('relu'))
model.add(Dense(4096))
model.add(Activation('relu'))
model.add(Dense(10))
model.add(Activation('softmax'))

model.summary()

opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6)

model.compile(loss='categorical_crossentropy',
          optimizer=opt,
          metrics=['accuracy']
          )

Because there is a lot of data I used the ImageDataGenerator.
gen = ImageDataGenerator(
 horizontal_flip=True
)

train_gen = gen.flow_from_directory(
        'train/',
        target_size=(512, 512),
        batch_size=5,
        class_mode=&quot;categorical&quot;
)

valid_gen = gen.flow_from_directory(
        'validation/',
        target_size=(512, 512),
        batch_size=5,
        class_mode=&quot;categorical&quot;
)
That's about expected. If you divide the number of seconds by the number of images you processed, you get 33 milliseconds per image, which seems about right for such a small network. Larger networks usually take in the ballpark of 50 to 200 milliseconds per image. 

Yes, a large dense layer is likely to hurt your performance, since that's a huge matrix (256 by 4096) and a large matrix multiplication to go along with it every time you run the network.