CNN for unsupervised anomaly detection
I'm not really sure if if P(cat) and P(dog) ~0.5 then it's an anomaly would be sound. 

But what you are referring to is not a strange idea: A CNN forms a conceptual description of inputs based on patterns in previously seen data. New data that is less like the train data doens't compress and reconstruct well. The approach usually used for this is called autoencoding. 

Roughly: If you would build an autoencoder by reconstructing images of cats and dogs, the in and out images would eventually (after enough data) approach each other (the distance you'd define would decrease). A new image from another domain would reconstruct less well (The distance between reconstruction and input would be larger than average).