Improving Validation Loss and Accuracy for CNN
Your data set is very small, so you definitely should try your luck at transfer learning, if it is an option.

Other than that, you probably should have a dropout layer after the dense-128 layer. Also, it is probably a good idea to remove dropouts after pooling layers. Instead, you can try using SpatialDropout after convolutional layers.