Loss for CNN decreases and settles but training accuracy does not improve
I guess I was able to figure out the answer to this when I looked carefully at the feature space and printed out the outputs from different convolutional layers. It appears that this issue appears when you have a lot of redundant features in your feature space. Since, this was a binary classification problem with FALSE and TRUE, each image by default had a lot of features that would correspond to a FALSE and the TRUEs only had a small percentage of the feature space being different. The trainer easily learnt the large percentage of the FALSE feature space and hence the loss easily went down while the small TRUE feature space was missed. This is the reason why the loss was going down but accuracy not improving. The trick to avoiding this issue was to downsample the initial image in the beginning before going to the convolutional layer. One could do it with some pyramidal pooling if on Tensorflow, or simply with imresize in matlab.