Make the CNN to say "I don't know"
You could put a one-class classification model before your CNN.

This would mean that you treat both your classes as one and then frame it as an anomaly detection problem. There are some different ways of achieving this. One way could be to do dimensionality reduction on the images and then use that image encoding to train an outlier detector like an one-class SVM. You can get some ideas here:

Learning Deep Features for One-ClassClassification

Anomaly Detection using One-Class Neural Networks

You can also look at cbeleites answer to this related questions. This was actually where I learned about one-class classification on images and it goes into a bit more detail and explains some caveats with using it.