Overfitting CNN models
Correct me if I'm wrong, but it seems you have a binary classification problem - the image either contains a cat or a dog. 

If that's the case, then:

output = Dense(1, activation='sigmoid')(flatten)


and

xception_model.compile(Adam(lr=.0001, decay=1e-6), loss='binary_crossentropy', metrics=['binary_accuracy'])


This thread is a great resource on the difference between the two questions.

Let me know if it works.

EDIT: Further explanation

Although we normally talk about “binary classification”, the way the outcome is usually modeled is as a Bernoulli random variable, conditioned on the input data. So:

$$P(y = 1|\mathbf{x}) = p, \ 0\leq p\leq1$$

A Bernoulli random variable takes on values between 0 and 1. So that’s what our network should produce. One idea might be to just clip all values of $wth+b$ outside that interval. But if we do this, the gradient in these regions will be 0: The network cannot learn.

A better way is to squish the complete incoming interval into the range (0,1), using the sigmoid function:

$$\sigma(x) = \frac{1}{1 + e^{(-x)}}$$

But the sigmoid function saturates when its input gets very large, or very small. Is this problematic? It depends. In the end, what we care about is if the cost function saturates. If we follow the general principle of maximum likelihood/cross entropy, the loss will be:

$$ - log P (y|\mathbf{x}) $$

where the $log$ undoes the $exp$ in the sigmoid.

In Keras, the corresponding loss function is binary_crossentropy. For a single item, the loss will be:


$−log(p)$ when the ground truth is 1
$−log(1−p)$ when the ground truth is 0


Here, you can see that when for an individual example, the network predicts the wrong class and is highly confident about it, this example will contribute very strongly to the loss. Therefore, you only need to have one output node, because binary classification is "symmetrical". By that, I mean that if we say that an item is of class 0 with probability p, we are also referring to the fact that this same item is of class 1 with probability 1 - p.