rfho/utils.py
def cross_entropy_loss(labels, logits, linear_input=True, eps=1.e-5, name='cross_entropy_loss'):
    """
    Clipped standard-version cross entropy loss. Implemented because  the standard function
    tf.nn.softmax_cross_entropy_with_logits has wrong (?) Hessian.
    Clipped because it easily brings to nan otherwise, especially when calculating the Hessian.

    Maybe the code could be optimized since ln(softmax(z_j)) = z_j - prod z_i . Should benchmark it.

    :param labels:
    :param logits: softmax or linear output of the model
    :param linear_input: True (default) if y is linear in which case tf.nn.softmax will be applied to y
    :param eps: (optional, default 1.e-5) clipping value for log.
    :param name: (optional, default cross_entropy_loss) name scope for the defined operations.
    :return: tensor for the cross_entropy_loss (WITHOUT MEAN ON THE EXAMPLES)
    """
    with tf.name_scope(name):
        softmax_out = tf.nn.softmax(logits) if linear_input else logits
        return -tf.reduce_sum(
            labels * tf.log(tf.clip_by_value(softmax_out, eps, 1. - eps)), reduction_indices=[1]
        )

    Clipped standard-version cross entropy loss. Implemented because  the standard function
    tf.nn.softmax_cross_entropy_with_logits has wrong (?) Hessian.
    Clipped because it easily brings to nan otherwise, especially when calculating the Hessian.

    Maybe the code could be optimized since ln(softmax(z_j)) = z_j - prod z_i . Should benchmark it.

    :param labels:
    :param logits: softmax or linear output of the model
    :param linear_input: True (default) if y is linear in which case tf.nn.softmax will be applied to y
    :param eps: (optional, default 1.e-5) clipping value for log.
    :param name: (optional, default cross_entropy_loss) name scope for the defined operations.
    :return: tensor for the cross_entropy_loss (WITHOUT MEAN ON THE EXAMPLES)
    
Clipped standard-version cross entropy loss. Implemented because  the standard function
tf.nn.softmax_cross_entropy_with_logits has wrong (?) Hessian.
Clipped because it easily brings to nan otherwise, especially when calculating the Hessian.

Maybe the code could be optimized since ln(softmax(z_j)) = z_j - prod z_i . Should benchmark it.
Clipped standard - version cross entropy loss . Implemented because the standard function tf . nn . softmax_cross_entropy_with_logits has wrong ( ? ) Hessian . Clipped because it easily brings to nan otherwise especially when calculating the Hessian . Maybe the code could be optimized since ln ( softmax ( z_j )) = z_j - prod z_i . Should benchmark it .
def cross_entropy_loss(labels, logits, linear_input=True, eps=1.e-5, name='cross_entropy_loss'):
    with tf.name_scope(name):
        softmax_out = tf.nn.softmax(logits) if linear_input else logits
        return -tf.reduce_sum(
            labels * tf.log(tf.clip_by_value(softmax_out, eps, 1. - eps)), reduction_indices=[1]
        )
def cross_entropy_loss ( labels , logits , linear_input = True , eps = 1.e-5 , name = 'cross_entropy_loss' ) : with tf . name_scope ( name ) : softmax_out = tf . nn . softmax ( logits ) if linear_input else logits return - tf . reduce_sum ( labels * tf . log ( tf . clip_by_value ( softmax_out , eps , 1. - eps ) ) , reduction_indices = [ 1 ] )
Clipped standard-version cross entropy loss.
Clipped standard - version cross entropy loss .
"""
    Clipped standard-version cross entropy loss. Implemented because  the standard function
    tf.nn.softmax_cross_entropy_with_logits has wrong (?) Hessian.
    Clipped because it easily brings to nan otherwise, especially when calculating the Hessian.

    Maybe the code could be optimized since ln(softmax(z_j)) = z_j - prod z_i . Should benchmark it.

    :param labels:
    :param logits: softmax or linear output of the model
    :param linear_input: True (default) if y is linear in which case tf.nn.softmax will be applied to y
    :param eps: (optional, default 1.e-5) clipping value for log.
    :param name: (optional, default cross_entropy_loss) name scope for the defined operations.
    :return: tensor for the cross_entropy_loss (WITHOUT MEAN ON THE EXAMPLES)
    """
tensor for the cross_entropy_loss (WITHOUT MEAN ON THE EXAMPLES)
tensor for the cross_entropy_loss ( WITHOUT MEAN ON THE EXAMPLES )
labels
None
logits
softmax or linear output of the model
softmax or linear output of the model
linear_input
True (default) if y is linear in which case tf.nn.softmax will be applied to y
True ( default ) if y is linear in which case tf . nn . softmax will be applied to y
eps
(optional, default 1.e-5) clipping value for log.
( optional default 1 . e - 5 ) clipping value for log .
name
(optional, default cross_entropy_loss) name scope for the defined operations.
( optional default cross_entropy_loss ) name scope for the defined operations .