title: What's the difference between tf.nn.ctc_loss with pytorch.nn.CTCLoss
body: <p>For the same input and label:</p>
  
  <ul>
  <li>the output of <code>pytorch.nn.CTCLoss</code> is 5.74, </li>
  <li>the output of <code>tf.nn.ctc_loss</code> is 129.69, </li>
  <li>but the output of <code>math.log(tf ctc loss)</code> is 4.86</li>
  </ul>
  
  <p>So what's the difference between <code>pytorch.nn.CTCLoss</code> with <code>tf.nn.ctc_loss</code>?</p>
  
  <pre><code>tf: 1.13.1
  pytorch: 1.1.0
  </code></pre>
  
  <p>I had try to these:</p>
  
  <ol>
  <li><p><code>log_softmax</code> the input, and then send it to <code>pytorch.nn.CTCLoss</code>,</p></li>
  <li><p><code>tf.nn.log_softmax</code> the input, and then send it to <code>tf.nn.ctc_loss</code></p></li>
  <li><p>directly send the input to <code>tf.nn.ctc_loss</code></p></li>
  <li><p>directly send the input to <code>tf.nn.ctc_loss</code>, and then <code>math.log(output of tf.nn.ctc_loss)</code></p></li>
  </ol>
  
  <p>In the case 2, case 3, and case 4, the result of calculation is difference from <code>pytorch.nn.CTCLoss</code></p>
  
  <pre class="lang-py prettyprint-override"><code>from torch import nn
  import torch
  import tensorflow as tf
  import math
  
  time_step = 50  # Input sequence length
  vocab_size = 20  # Number of classes
  batch_size = 16  # Batch size
  target_sequence_length = 30  # Target sequence length
  
  
  def dense_to_sparse(dense_tensor, sequence_length):
      indices = tf.where(tf.sequence_mask(sequence_length))
      values = tf.gather_nd(dense_tensor, indices)
      shape = tf.shape(dense_tensor, out_type=tf.int64)
      return tf.SparseTensor(indices, values, shape)
  
  
  def compute_loss(x, y, x_len):
      ctclosses = tf.nn.ctc_loss(
          y,
          tf.cast(x, dtype=tf.float32),
          x_len,
          preprocess_collapse_repeated=False,
          ctc_merge_repeated=False,
          ignore_longer_outputs_than_inputs=False
      )
      ctclosses = tf.reduce_mean(ctclosses)
  
      with tf.Session() as sess:
          ctclosses = sess.run(ctclosses)
          print(f"tf ctc loss: {ctclosses}")
          print(f"tf log(ctc loss): {math.log(ctclosses)}")
  
  
  minimum_target_length = 10
  
  ctc_loss = nn.CTCLoss(blank=vocab_size - 1)
  x = torch.randn(time_step, batch_size, vocab_size)  # [size] = T,N,C
  y = torch.randint(0, vocab_size - 2, (batch_size, target_sequence_length), dtype=torch.long)  # low, high, [size]
  
  x_lengths = torch.full((batch_size,), time_step, dtype=torch.long)  # Length of inputs
  y_lengths = torch.randint(minimum_target_length, target_sequence_length, (batch_size,),
                            dtype=torch.long)  # Length of targets can be variable (even if target sequences are constant length)
  
  loss = ctc_loss(x.log_softmax(2).detach(), y, x_lengths, y_lengths)
  print(f"torch ctc loss: {loss}")
  
  x = x.numpy()
  y = y.numpy()
  x_lengths = x_lengths.numpy()
  y_lengths = y_lengths.numpy()
  x = tf.cast(x, dtype=tf.float32)
  y = tf.cast(dense_to_sparse(y, y_lengths), dtype=tf.int32)
  compute_loss(x, y, x_lengths)
  
  </code></pre>
  
  <p>I expect the output of <code>tf.nn.ctc_loss</code> is same with the output of <code>pytorch.nn.CTCLoss</code>, but actually they are not, but how can i make them same?</p>