Why doesn't my image classification network get better with training? I am attempting to train a network to do something I thought would be a relatively simple case to learn with: identify whether the back of a scanned vintage postcard has one of 'no postage stamp', a '1 cent stamp', or a '2 cent stamp.' The images are 250px by about 150px, RGB color, and there are about two thousand of them. Ballpark 75% of them are no-stamp, 20% 1-cent, and 10% 2-cent.

When I attempt to train the network it seems like it is starting at 70 +/- 1 % accurate and hovers in that range for 50 epochs, never improving. I'm not sure I'm reading the metrics correctly, though, as this doesn't seem quite right.

I set this up by following the tutorial on the Keras blog: https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html 

I haven't implemented the latter part of the tutorial, where a pre-trained network is used, because I haven't found one that seems like it would be a similar problem.

My training and validation sets are here: https://drive.google.com/open?id=1-TxEKVVvP7RuFC7kFgH7Wt5A8z8QGTR3 

And my Google Colab Jupyter notebook is here: https://colab.research.google.com/drive/1UuKDF1wDwYlXszB2ahIrygRnfcs2D_sD
The answer (as detailed quite thoroughly here and here) is that specifying

model.compile(optimizer = 'rmsprop', loss = 'binary_crossentropy', metrics=['accuracy'])


causes keras to guess, incorrectly, that because I am using binary_crossentropy for the loss function, that I would want to use binary_accuracy as the way of reporting the accuracy metrics. Apparently, one should specify that one wants the categorical_accuracy metrics if one, as I do, has more than two classes.