Need a little help Understanding how to build model's in Keras I am trying to make a CNN in Keras, and to test the validity of my model i am trying to get it to train on MNIST dataset, so i am sure that everything is working fine, but unfortunately model is barely training and i suspect that nothing updating.
My model is :

model=Sequential()

#conv1_1
model.add(Conv2D(128,kernel_size=3, strides=1,
                 padding='SAME', use_bias=False, 
                 activation='relu',name='conv1_1',input_shape=(28,28,1)))
#conv1_2
model.add(Conv2D(128, kernel_size=3, strides=1,
                 padding='SAME', use_bias=False, 
                 activation='relu',name='conv1_2'))
model.add(MaxPooling2D(pool_size=2,strides=2))

#conv2_1
model.add(Conv2D(64, kernel_size=3, strides=1,
                 padding='SAME', use_bias=False, 
                 activation='relu',name="conv2_1"))

#conv2_2
model.add(Conv2D(64, kernel_size=3, strides=1,
                 padding='SAME', use_bias=False, 
                 activation='relu',name='conv2_2'))
model.add(MaxPooling2D(pool_size=2,strides=2))

model.add(Flatten())
model.add(Dense(1024, activation='relu',name='Dense1'))
model.add(Dropout(0.2))
model.add(Dense(512, activation='relu',name='Dense2'))
model.add(Dense(10, activation='softmax',name='output'))


Compiled with:

model.compile(loss='categorical_crossentropy', optimizer='adadelta', metrics=['accuracy'])

model.fit(X_train,y_train,batch_size=10,validation_split=0.2,epochs=10)


My X_train and y_train look like:

plt.imshow(X_train[0].reshape(28,28))
plt.show()




y_train[0]
array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.])


Here are the Results of first 3 epochs:

Epoch 1/10
48000/48000 [==============================] - 45s 927us/step - loss: 14.2813 - acc: 0.1140 - val_loss: 14.4096 - val_acc: 0.1060
Epoch 2/10
48000/48000 [==============================] - 44s 915us/step - loss: 14.2813 - acc: 0.1140 - val_loss: 14.4096 - val_acc: 0.1060
Epoch 3/10
48000/48000 [==============================] - 44s 924us/step - loss: 14.2813 - acc: 0.1140 - val_loss: 14.4096 - val_acc: 0.1060
Epoch 4/10
48000/48000 [==============================] - 45s 930us/step - loss: 14.2813 - acc: 0.1140 - val_loss: 14.4096 - val_acc: 0.1060


This is my first Keras Model, and i think i am missing something important here.
I have implemented your model to the astonishment, there is a very minute error that is hard to notice.

The way, I was able to get better accuracy is by changing the optimizer to "SGD" or "ADAM".

As you have used "ADADELTA" which is an extension of "ADAGRAD" optimizer. In "ADAGRAD" has good performs on sparse data &amp; while training a large scale neural network. Its monotonic learning rate usually proves too aggressive, stops learning too early.

Refer to this link for understanding on optimizers