How to build a recurrent neural net in Keras where each input goes through a layer first?
What should the input look like?


You are right to think a 2D tensor, but usually we add one more dimension for the batch. You can indeed have a variable length number_of_x_inputs, but to train during batch processing all inputs in a single batch will need to have the same shape. (Setting batch size to 1 will get around this.) During inference, you can have whatever length you want. See below code sample.


How do I pass each input vector through the same transformation before feeding it to the RNN layer?


Use TimeDistributed. The example below passes all vectors $x_i$ through the same feed forward network (Dense(5, ...)), but you should be able to swap that out for whatever f you were thinking.

from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed

x_dimension = 16
num_classes = 2

model = Sequential()

model.add(TimeDistributed(Dense(5, activation='relu'),
          input_shape=(None, x_dimension)))
model.add(LSTM(32, return_sequences=True))
model.add(LSTM(8))
model.add(Dense(num_classes, activation='softmax'))

print(model.summary(90))


This prints the following model:

Layer (type)                            Output Shape                        Param #
==========================================================================================
time_distributed_1 (TimeDistributed)    (None, None, 5)                     85
__________________________________________________________________________________________
lstm_1 (LSTM)                           (None, None, 32)                    4864
__________________________________________________________________________________________
lstm_2 (LSTM)                           (None, 8)                           1312
__________________________________________________________________________________________
dense_2 (Dense)                         (None, 2)                           18
==========================================================================================
Total params: 6,279
Trainable params: 6,279
Non-trainable params: 0
__________________________________________________________________________________________