title: Combine Time-series with time-invariant data in RNN/LSTM using Keras Functional API
body: <p><em>Update: As <a href="https://stackoverflow.com/users/5140223/rvinas">rvinas</a> pointed out, I had forgotten to add inputs_aux as second <code>input</code> in <code>Model</code>. Fixed now, and it works. So ConditionalRNN can readily be used to do what I want.</em></p>
  
  <p>I'd like to treat time-series together with non-time-series characteristics in extended LSTM cells (a requirement also discussed <a href="https://datascience.stackexchange.com/q/17099">here</a>). <a href="https://github.com/philipperemy/cond_rnn" rel="nofollow noreferrer">ConditionalRNN (cond-rnn)</a> for Tensorflow in Python seems to allow this.</p>
  
  <p>Can it be used in Keras Functional API (without eager execution)?
  That is, does anyone have a clue how to fix my failed approach below, or a different example where ConditionalRNN (or alternatives) are used to readily combine TS and non-TS data in LSTM-style cells or any equivalent?</p>
  
  <p>I've seen the eager execution-bare tf example on <a href="https://github.com/philipperemy/cond_rnn" rel="nofollow noreferrer">Pilippe Remy's ConditionalRNN github page</a>, but I did not manage to extend it to a readily fittable version in Keras Functional API.</p>
  
  <p>My code looks as follows; it works if, instead of the ConditionalRNN, I use a standard LSTM cell (and adjust the model 'x' input correspondingly). With ConditionalRNN, I did not get it to execute; I receive either the <code>must feed a value for placeholder tensor 'in_aux'</code> error (cf. below), or instead some different type of input size complaints when I change the code, despite trying to be careful about data dimensions compatibility.</p>
  
  <p>(Using Python 3.6, Tensorflow 2.1, cond-rnn 2.1, on Ubuntu 16.04)</p>
  
  <pre><code>import numpy as np
  
  from tensorflow.keras.models import Model
  from tensorflow.keras.layers import LSTM, Dense, Input
  from cond_rnn import ConditionalRNN
  
  inputs = Input(name='in',shape=(5,5)) # Each observation has 5 dimensions à 5 time-steps each
  x = Dense(64)(inputs)
  
  inputs_aux = Input(name='in_aux', shape=[5]) # For each of the 5 dimensions, a non-time-series observation too
  x = ConditionalRNN(7, cell='LSTM')([x,inputs_aux]) # Updated Syntax for cond_rnn v2.1
  # x = ConditionalRNN(7, cell='LSTM', cond=inputs_aux)(x) # Syntax for cond_rnn in some version before v2.1
  
  predictions = Dense(1)(x)
  model = Model(inputs=[inputs, inputs_aux], outputs=predictions) # With this fix, [inputs, inputs_aux], it now works, solving the issue
  #model = Model(inputs=inputs, outputs=predictions)
  model.compile(optimizer='rmsprop', loss='mean_squared_error', metrics=['mse'])
  data = np.random.standard_normal([100,5,5]) # Sample of 100 observations with 5 dimensions à 5 time-steps each
  data_aux = np.random.standard_normal([100,5]) # Sample of 100 observations with 5 dimensions à only 1 non-time-series value each
  labels = np.random.standard_normal(size=[100]) # For each of the 100 obs., a corresponding (single) outcome variable
  
  model.fit([data,data_aux], labels)
  </code></pre>
  
  <p>The error I get is </p>
  
  <pre><code>tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'in_aux' with dtype float and shape [?,5]
       [[{{node in_aux}}]]
  </code></pre>
  
  <p>and the traceback is</p>
  
  <pre><code>Traceback (most recent call last):
    File "/home/florian/temp_nonclear/playground/test/est1ls_bare.py", line 20, in &lt;module&gt;
      model.fit({'in': data, 'in_aux': data_aux}, labels) #model.fit([data,data_aux], labels) # Also crashes when using model.fit({'in': data, 'in_aux': data_aux}, labels)
    File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/keras/engine/training.py", line 643, in fit
      use_multiprocessing=use_multiprocessing)
    File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 664, in fit
      steps_name='steps_per_epoch')
    File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 383, in model_iteration
      batch_outs = f(ins_batch)
    File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/keras/backend.py", line 3353, in __call__
      run_metadata=self.run_metadata)
    File "/home/florian/BB/tsgenerator/ts_wgan/venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1458, in __call__
      run_metadata_ptr)
  </code></pre>