<h2 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature</h2>

<p dir="auto">label smoothing addition to standard cross entropy?</p>
<h2 dir="auto">Motivation</h2>

<h2 dir="auto">Pitch</h2>

<h2 dir="auto">Alternatives</h2>

<h2 dir="auto">Additional context</h2>
<p dir="auto">Currently our cross entropy loss (i.e., <a href="https://pytorch.org/docs/stable/nn.html#crossentropyloss" rel="nofollow">nn.CrossEntropyLoss</a>) only supports a hard target class, i.e., wanting to maximize the output (log) probability of a particular class. But in many times training w.r.t. a soft target distribution (i.e., wanting the output to match a particular distribution) is quite useful too, e.g., preventing overfitting.</p>
<h2 dir="auto">Math</h2>
<p dir="auto">Cross entropy loss operates on logits after softmax.</p>
<p dir="auto">Denote the input vector as <code class="notranslate">x</code>. Log softmax computes a vector <code class="notranslate">y</code> of same length as <code class="notranslate">x</code>, where <code class="notranslate">y_i = x_i - log( \sum_j exp(x_j) )</code>, representing the log likelihood of each class.</p>
<ul dir="auto">
<li>
<p dir="auto">In the hard target case, if the target clss is <code class="notranslate">c</code>, the loss is simply negative log likelihood loss <code class="notranslate">-y_c</code>.</p>
</li>
<li>
<p dir="auto">In the soft target case, let the target distribution vector be <code class="notranslate">p</code> (i.e., <code class="notranslate">p_i</code> is the target probability for predicting class <code class="notranslate">i</code>). The loss is the KL divergence</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="D( softmax(x) || p) = \sum_i p_i (log p_i  / softmax(x)_i) = -\sum_i p_i y_i + constant"><pre class="notranslate"><code class="notranslate">D( softmax(x) || p) = \sum_i p_i (log p_i  / softmax(x)_i) = -\sum_i p_i y_i + constant
</code></pre></div>
<p dir="auto">The constant is independent of <code class="notranslate">x</code> and thus discarded. Our loss formula is just <code class="notranslate">-\sum_i p_i y_i</code>.</p>
<p dir="auto">When <code class="notranslate">p_c = 1</code> for some class <code class="notranslate">c</code>, this simplifies to the hard target class.</p>
<p dir="auto">The formula for gradient computation can be easily derived from this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="
d l / d y_i = -p_i

d y_i / d x_i = 1 - exp(x_i) / \sum_j exp(x_j) = 1 - exp(y_i)

# suppose k != i
d y_k / d_x_i = -exp(x_i) / \sum_j exp(x_j) = - exp(y_i)

# so
d l / d x_i = exp(y_i) (\sum p) - p_i = exp (y_i) - p_i (= softmax(x) - p_i).
"><pre class="notranslate"><code class="notranslate">
d l / d y_i = -p_i

d y_i / d x_i = 1 - exp(x_i) / \sum_j exp(x_j) = 1 - exp(y_i)

# suppose k != i
d y_k / d_x_i = -exp(x_i) / \sum_j exp(x_j) = - exp(y_i)

# so
d l / d x_i = exp(y_i) (\sum p) - p_i = exp (y_i) - p_i (= softmax(x) - p_i).

</code></pre></div>
</li>
</ul>
<h2 dir="auto">Possible Implementation</h2>
<p dir="auto">Currently our cross entropy loss implementation takes in batched <code class="notranslate">x</code> of shape <code class="notranslate">(N, C)</code> and floating point dtype (<code class="notranslate">N</code> is the batch size and <code class="notranslate">C</code> is the number of classes), and a batched target class indices vector <code class="notranslate">target</code> of shape <code class="notranslate">(N)</code>, where <code class="notranslate">target[i]</code> is the index of the desired output class, and dtype <code class="notranslate">long</code> (an integral type).</p>
<p dir="auto">Since we want it to also take in soft target distribution as target, we can allow it to also take in <code class="notranslate">target</code> as a target batched distribution of shape <code class="notranslate">(N, C)</code>, and detect whether we want soft target or hard target basing on shape and dtype.</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a></p>