CNN not learning properly Before marking my question as duplicate, I would like to say that I have tried all the possible solutions mentioned in similar questions, but that doesn't seem to work. 

I am currently working on blood cells classification problem where we basically have to classify blood images (4 classes). The dataset consists of 9957 images, nearly equal number of images of all the 4 classes. The accuracy always hovers around 25-27%  even after trying different optimizers and learning rates. I even tried training upto 100 epochs. Image augmentation doesn't help. Also, it is not that it is predicting same class for all images although for 1 particular batch of images, it predicts the same class. It again predicts some other class for all images in the next batch. So, I would just like to know, what am I possibly doing wrong? Is the dataset not sufficient, or the architecture should have more hidden layers, or am I not implementing optimizer or loss function correctly or is there any silly mistake I am overlooking in my code ?

My CNN architecture: (fs means filter_size, nf means number of filters, s is no. strides)

Input(80,80,1)-&gt;Conv(fs = 3, nf = 80, s = [1,1,1,1])
Activation(LeakyReLU)-&gt;Conv(fs = 3,nf=64,s=[1,1,1,1])
Activation(LReLU)-&gt;Pool(ps = [1,2,2,1],s=[1,2,2,1]
Conv(fs = 3,nf = 64,s=[1,1,1,1])-&gt;Activation(LReLU)
Dropout(prob = 0.75)-&gt;Flatten
FullyConnected(output_features = 128)-&gt;Dropout(prob = 0.5)
FullyConnected(output_features = 4)


loss_value = tf.reduce_mean(loss_fn)
optimizer = tf.train.AdamOptimizer()

loss_min_fn = optimizer.minimize(loss = loss_value)
check_prediction = tf.equal(tf.argmax(y,axis=1),y_pred)
model_accuracy = tf.reduce_mean(tf.cast(check_prediction, tf.float32)

sess.run(loss_min_fn, feed_dict = {x:X_train_batch, y:y_train_batch})         
train_accuracy = train_accuracy + sess.run(model_accuracy, feed_dict={x : X_train_batch,y:y_train_batch})
train_loss = train_loss + sess.run(loss_value, feed_dict={x : X_train_batch,y:y_train_batch})


The images kind of look like this
I'll make a few observations that will hopefully help.


I would remove the dropout layer until you have evidence of overfitting on the data.  A dropout layer is generally used to make a model more generalized which is not your problem in this case and it may be hurting your model.
The images may not contain enough information to distinguish between the classes.  Can an expert assign a class label by looking at the images? If not, it may be that it is just not possible from the images.
Your accuracy scores imply that the model is just guessing since a random chance classifier would be expected to perform as well. However, it is curious that you seem to get reasonable accuracy on one class.  You may find that you need multiple models configured as one-vs-the-rest binary classifiers. Each model can be uniquely configured and trained to specialize on one class.  After the training is complete, which could be time consuming, prediction should be efficient.
Lastly, do you think that the model has stopped learning by the time you stop training?  Intuitively, I would anticipate the need for tens of thousands of epochs, and potentially millions, before seeing a really refined model.  100 epochs seems like it would be just starting to learn. Because of this cost, I would recommend you configure your model so you can continue training from a stopping point.


And by all means, keep trying different configurations of your layers.  One thing to remember though is that each time you use a convolutional layer you are losing information.  You need to consider how much information you are losing with each convolutional layer.