CNN for unsupervised anomaly detection I'm wondering if the following strategy has been already used and could work

Let's says you have a CNN which work well to classify image data, dog and cat. You only have cat and dog image as training data.
Is there any way to use it to detect image of horse as anomaly?

For example, with a ruled based system we could says 

if P(cat) and P(dog) ~0.5 then it's an anomaly



another way could be to take feature vector at last fully connected layer and compare vector very different from other could be considerated as anomaly

Do you have any related paper? is it a totaly dumb idea?
I'm not really sure if if P(cat) and P(dog) ~0.5 then it's an anomaly would be sound. 

But what you are referring to is not a strange idea: A CNN forms a conceptual description of inputs based on patterns in previously seen data. New data that is less like the train data doens't compress and reconstruct well. The approach usually used for this is called autoencoding. 

Roughly: If you would build an autoencoder by reconstructing images of cats and dogs, the in and out images would eventually (after enough data) approach each other (the distance you'd define would decrease). A new image from another domain would reconstruct less well (The distance between reconstruction and input would be larger than average).