Loss function for optimising precision & recall / sensitivity & specificity? I've been using precision and recall as my metrics, as per keras-team/keras/pull/9393/files

Sensitivity &amp; specificity is what I want to optimise for. Every epoch I output it:


class SensitivitySpecificityCallback(Callback):
    def on_epoch_end(self, epoch, logs=None):
        if epoch:
            x_test = self.validation_data[0]
            y_test = self.validation_data[1]
            predictions = self.model.predict(x_test)
            output_sensitivity_specificity(epoch, predictions, y_test)


def output_sensitivity_specificity(epoch, predictions, y_test):
    y_test = np.argmax(y_test, axis=-1)
    predictions = np.argmax(predictions, axis=-1)
    c = confusion_matrix(y_test, predictions)
    print('Confusion matrix:\n', c)
    print('[{:03d}] sensitivity'.format(epoch), c[0, 0] / (c[0, 1] + c[0, 0]))
    print('[{:03d}] specificity'.format(epoch), c[1, 1] / (c[1, 1] + c[1, 0]))


On my 203rd epoch I get a great result, then it's all downhill—in the wrong direction!—from there.

How do I optimise for sensitivity and specificity? - Thinking to update the weights, and/or develop a custom loss function…
The comments raise two important points:

you can't directly optimize for sensitivity or specificity
you should save the model at the best epoch and use that one

Below I explain something that is not directly related, but might be interesting for you:
You can add the weights to the loss of some examples. For example, you can increase the weight of positive examples which means that the model will focus more on correctly classifying positive examples at the potential expense of negative examples. This is especially useful if your data is suffering from class imbalance.