How to write a custom loss function in Keras/Tensorflow that uses loops/iterations with reference numpy code
I saw this question: Implementing custom loss function in keras with condition And I need to do the same thing but with code that seems to need loops.

I have a custom numpy function which calculates the mean Euclid distance from the mean vector. I wrote this based on the paper https://arxiv.org/pdf/1801.05365.pdf:



import numpy as np

def mean_euclid_distance_from_mean_vector(n_vectors):

    dists = []

    for (i, v) in enumerate(n_vectors):
        n_vectors_rest = n_vectors[np.arange(len(n_vectors)) != i]

        print("rest of vectors: ")
        print(n_vectors_rest)

        # calculate mean vector
        mean_rest = n_vectors_rest.mean(axis=0)

        print("mean rest vector")
        print(mean_rest)

        dist = v - mean_rest

        print("dist vector")
        print(dist)
        dists.append(dist)

    # dists is now a matrix of distance vectors (distance from the mean vector)
    dists = np.array(dists)

    print("distance vector matrix")
    print(dists)

    # here we matmult each vector
    # sum them up
    # and divide by the total number of elements
    result = np.sum([np.matmul(d, d) for d in dists]) / dists.size

    return result


features = np.array([
    [1,2,3,4],
    [4,3,2,1]
])

c = mean_euclid_distance_from_mean_vector(features)

print(c)


I need this function however to work inside tensorflow with Keras. So a custom lambda https://www.tensorflow.org/api_docs/python/tf/keras/layers/Lambda

However, I'm not sure how to implement the above in Keras/Tensorflow since it has loops, and the way the paper talked about calculating the m_i seems to require loops like the way I implemented the above.

For reference, the PyTorch version of this code is here: https://github.com/PramuPerera/DeepOneClass

Given a feature map like:

features = np.array([
    [1, 2, 3, 4],
    [2, 4, 4, 3],
    [3, 2, 1, 4],
], dtype=np.float64)


reflecting a batch_size of

batch_size = features.shape[0]


and 

k = features.shape[1]


One has that implementing the above Formulas in Tensorflow could be expressed (prototyped) by:

dim = (batch_size, features.shape[1])
def zero(i):
    arr = np.ones(dim)
    arr[i] = 0
    return arr


mapper = [zero(i) for i in range(batch_size)]
elems = (features, mapper)
m = (1 / (batch_size - 1)) * tf.map_fn(lambda x: tf.math.reduce_sum(x[0] * x[1], axis=0), elems, dtype=tf.float64)
pairs = tf.map_fn(lambda x: tf.concat(x, axis=0) , tf.stack([features, m], 1), dtype=tf.float64)
compactness_loss = (1 / (batch_size * k)) * tf.map_fn(lambda x: tf.math.reduce_euclidean_norm(x), pairs, dtype=tf.float64)

with tf.Session() as sess:
    print("loss value output is: ", compactness_loss.eval())


Which yields:

loss value output is:  [0.64549722 0.79056942 0.64549722]


However a single measure is required for the batch, therefore it is necessary to reduce it; by the summation of all values.

The wanted Compactness Loss function à la Tensorflow is:

def compactness_loss(actual, features):
    features = Flatten()(features)
    k = 7 * 7 * 512
    dim = (batch_size, k)

    def zero(i):
        z = tf.zeros((1, dim[1]), dtype=tf.dtypes.float32)
        o = tf.ones((1, dim[1]), dtype=tf.dtypes.float32)
        arr = []
        for k in range(dim[0]):
            arr.append(o if k != i else z)
        res = tf.concat(arr, axis=0)
        return res

    masks = [zero(i) for i in range(batch_size)]
    m = (1 / (batch_size - 1)) * tf.map_fn(
        # row-wise summation
        lambda mask: tf.math.reduce_sum(features * mask, axis=0),
        masks,
        dtype=tf.float32,
    )
    dists = features - m
    sqrd_dists = tf.pow(dists, 2)
    red_dists = tf.math.reduce_sum(sqrd_dists, axis=1)
    compact_loss = (1 / (batch_size * k)) * tf.math.reduce_sum(red_dists)
    return compact_loss


Of course the Flatten() could be moved back into the model for convenience and the k could be derived directly from the feature map; this answers your question. You may just have some trouble finding out the the expected values for the model are - feature maps from the VGG16 (or any other architechture) trained against the imagenet for instance?

The paper says:


  In  our  formulation  (shown  in  Figure  2  (e)),  starting  froma  pre-trained  deep  model,  we  freeze  initial  features  (gs)  and learn  (gl)  and  (hc).  Based  on  the  output  of  the  classification sub-network (hc),  two  losses compactness  loss and descriptiveness  loss are  evaluated.  These  two  losses,  introduced  in the subsequent sections, are used to assess the quality of the learned  deep  feature.  We  use  the  provided  one-class  dataset to  calculate  the compactness  loss.  An  external  multi-class reference  dataset  is  used  to  evaluate  the descriptiveness  loss.As  shown  in  Figure  3,  weights  of gl and hc are  learned in  the  proposed  method  through  back-propagation  from  the composite loss. Once training is converged, system shown in setup  in  Figure  2(d)  is  used  to  perform  classification  where the resulting model is used as the pre-trained model.


then looking at the "Framework" backbone here plus:


  AlexNet   Binary   and   VGG16   Binary   (Baseline). A  binary  CNN  is  trained  by  having  ImageNet  samples  and  one-class  image  samples  as  the  two  classes  using  AlexNet  andVGG16 architectures, respectively. Testing is performed using k-nearest neighbor, One-class SVM [43], Isolation Forest [3]and Gaussian Mixture Model [3] classifiers.


Makes me wonder whether it would not be reasonable to add suggested the dense layers to both the Secondary and the Reference Networks to a single class output (Sigmoid) or even and binary class output (using Softmax) and using the mean_squared_error as the so called Compactness Loss and binary_cross_entropy as the Descriptveness Loss.
