title: How to use model input in loss function?
body: <p>I am trying to use a custom loss-function which depends on some arguments that the model does not have.</p>
  <p>The model has two inputs (<code>mel_specs</code> and <code>pred_inp</code>) and expects a <code>labels</code> tensor for training:</p>
  <pre class="lang-py prettyprint-override"><code>def to_keras_example(example):
      # Preparing inputs
      return (mel_specs, pred_inp), labels
  
  # Is a tf.train.Dataset for model.fit(train_data, ...)
  train_data = load_dataset(fp, 'train).map(to_keras_example).repeat()
  </code></pre>
  <p>In my loss function I need to calculate the lengths of <code>mel_specs</code> and <code>pred_inp</code>. This means my loss looks like this:</p>
  <pre class="lang-py prettyprint-override"><code>def rnnt_loss_wrapper(y_true, y_pred, mel_specs_inputs_):
      input_lengths = get_padded_length(mel_specs_inputs_[:, :, 0])
      label_lengths = get_padded_length(y_true)
      return rnnt_loss(
          acts=y_pred,
          labels=tf.cast(y_true, dtype=tf.int32),
          input_lengths=input_lengths,
          label_lengths=label_lengths
      )
  </code></pre>
  <p>However, no matter which approach I choose, I am facing some issue.</p>
  <hr />
  <h2>Option 1) Setting the loss-function in model.compile()</h2>
  <p>If I actually wrap the loss function s.t. it returns a function which takes <code>y_true</code>  and <code>y_pred</code> like this:</p>
  <pre class="lang-py prettyprint-override"><code>def rnnt_loss_wrapper(mel_specs_inputs_):
      def inner_(y_true, y_pred):
          input_lengths = get_padded_length(mel_specs_inputs_[:, :, 0])
          label_lengths = get_padded_length(y_true)
          return rnnt_loss(
              acts=y_pred,
              labels=tf.cast(y_true, dtype=tf.int32),
              input_lengths=input_lengths,
              label_lengths=label_lengths
          )
      return inner_
  
  model = create_model(hparams)
  model.compile(
      optimizer=optimizer,
      loss=rnnt_loss_wrapper(model.inputs[0]
  )
  </code></pre>
  <p>Here I get a <code>_SymbolicException</code> after calling <code>model.fit()</code>:</p>
  <pre class="lang-none prettyprint-override"><code>tensorflow.python.eager.core._SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors, but found [...]
  </code></pre>
  <hr />
  <h2>Option 2) Using model.add_loss()</h2>
  <p>The documentation of <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer#add_loss" rel="nofollow noreferrer"><code>add_loss()</code></a> states:</p>
  <blockquote>
  <pre class="lang-none prettyprint-override"><code>[Adds a..] loss tensor(s), potentially dependent on layer inputs.
  ..
  Arguments:
    losses: Loss tensor, or list/tuple of tensors. Rather than tensors, losses
      may also be zero-argument callables which create a loss tensor.
    inputs: Ignored when executing eagerly. If anything ...
  </code></pre>
  </blockquote>
  <p>So I tried to do the following:</p>
  <pre class="lang-py prettyprint-override"><code>def rnnt_loss_wrapper(y_true, y_pred, mel_specs_inputs_):
      input_lengths = get_padded_length(mel_specs_inputs_[:, :, 0])
      label_lengths = get_padded_length(y_true)
      return rnnt_loss(
          acts=y_pred,
          labels=tf.cast(y_true, dtype=tf.int32),
          input_lengths=input_lengths,
          label_lengths=label_lengths
      )
  
  model = create_model(hparams)
  model.add_loss(
      rnnt_loss_wrapper(
          y_true=model.inputs[2],
          y_pred=model.outputs[0],
          mel_specs_inputs_=model.inputs[0],
      ),
      inputs=True
  )
  model.compile(
      optimizer=optimizer
  )
  </code></pre>
  <p>However, calling <code>model.fit()</code> throws a <code>ValueError</code>:</p>
  <pre class="lang-none prettyprint-override"><code>ValueError: No gradients provided for any variable: [...]
  </code></pre>
  <hr />
  <p>Is any of the above options supposed to work?</p>