How to define a custom performance metric in Keras?
You have to use Keras backend functions. Unfortunately they do not support the &amp;-operator, so that you have to build a workaround: We generate matrices of the dimension batch_size x 3, where (e.g. for true positive) the first column is the ground truth vector, the second the actual prediction and the third is kind of a label-helper column, that contains in the case of true positive only ones. Then we check which instances are positive instances, are predicted as positive and the label-helper is also positive. Those are the true positives.
We can make this analog with false positives, false negatives and true negatives with some reverse-calculations of the labels.
Your f1-metric may look as follows:
def f1_score(y_true, y_pred):
    &quot;&quot;&quot;
    f1 score

    :param y_true:
    :param y_pred:
    :return:
    &quot;&quot;&quot;
    tp_3d = K.concatenate(
        [
            K.cast(y_true, 'bool'),
            K.cast(K.round(y_pred), 'bool'),
            K.cast(K.ones_like(y_pred), 'bool')
        ], axis=1
    )

    fp_3d = K.concatenate(
        [
            K.cast(K.abs(y_true - K.ones_like(y_true)), 'bool'),
            K.cast(K.round(y_pred), 'bool'),
            K.cast(K.ones_like(y_pred), 'bool')
        ], axis=1
    )

    fn_3d = K.concatenate(
        [
            K.cast(y_true, 'bool'),
            K.cast(K.abs(K.round(y_pred) - K.ones_like(y_pred)), 'bool'),
            K.cast(K.ones_like(y_pred), 'bool')
        ], axis=1
    )

    tp = K.sum(K.cast(K.all(tp_3d, axis=1), 'int32'))
    fp = K.sum(K.cast(K.all(fp_3d, axis=1), 'int32'))
    fn = K.sum(K.cast(K.all(fn_3d, axis=1), 'int32'))

    precision = tp / (tp + fp)
    recall = tp / (tp + fn)
    return 2 * ((precision * recall) / (precision + recall))

Since the Keras-backend calculator returns nan for division by zero, we do not need the if-else-statement for the return statement.
Edit:
I have found a pretty good idea for a exact implementation. The problem with our first approach is, that it is only &quot;approximated&quot;, since it is computed batchwise and subsequently averaged. One could also calculate this after each epoch with the keras.callbacks. Please find the idea here.
An example implementation would be:
import keras
import numpy as np
import sklearn.metrics as sklm


class Metrics(keras.callbacks.Callback):
    def on_train_begin(self, logs={}):
        self.confusion = []
        self.precision = []
        self.recall = []
        self.f1s = []
        self.kappa = []
        self.auc = []

    def on_epoch_end(self, epoch, logs={}):
        score = np.asarray(self.model.predict(self.validation_data[0]))
        predict = np.round(np.asarray(self.model.predict(self.validation_data[0])))
        targ = self.validation_data[1]

        self.auc.append(sklm.roc_auc_score(targ, score))
        self.confusion.append(sklm.confusion_matrix(targ, predict))
        self.precision.append(sklm.precision_score(targ, predict))
        self.recall.append(sklm.recall_score(targ, predict))
        self.f1s.append(sklm.f1_score(targ, predict))
        self.kappa.append(sklm.cohen_kappa_score(targ, predict))

        return

To make the network to call this function you simply add it to you callbacks like
metrics = Metrics()
model.fit(
    train_instances.x,
    train_instances.y,
    batch_size,
    epochs,
    verbose=2,
    callbacks=[metrics],
    validation_data=(valid_instances.x, valid_instances.y),
)

Then you can simply access the members of the metrics variable.