How to deal with a binary classification problem, where the instances in the negative class are very similar?
The CNN in this case does not learn what is a cat but rather what differentiate an image with a cat from one without a cat.
If all of your &quot;no-cat&quot; images contain spider(s), the CNN could also converge only by detecting the spider(s) : images with spider(s) belong to &quot;no-cat&quot; and others belong to &quot;cat&quot;. That explains why you have such a good confusion matrix on testing data and such poor performance in real-world.
You have some options to avoid such a situation:

add other images to your &quot;no-cat&quot; label as you said (using public datasets for instance)
try one-class classification approaches. If you are familiar with neural networks, you could dig into reconstruction error of autoencoders (see this post: How to use a dataset with only one category of data or this sklearn guide)

Hope it helps