How do I deal with the fact that I have images which are not consistent with the class they belong in an image classification problem with CNN?
There are different ways of doing this, but the final idea in regard to this is that you need to clean your dataset.

You could go through it manually and then separate the images. This is extremely slow if you're dealing with a large dataset.

A faster less robust way would be to do a Principle component analysis

step1 :  Do PCA to reduce the dimensionality to a 2D or 3D space
step2 : Plot and see if there are any clusters. Usually, images which are different fall into different clusters
step3 : Cluster it via a convex clustering algorithm like K-means
step4 : Store the images belonging to a cluster into a folder.
step5 : Go through the folders and make the necessary cleaning.