title: passing keras TimeDistributed wrapper multiple inputs, only one with time dimension
body: <p>I'm brand new to Keras...  been reading the documentation...  referencing the last example ("Video Question Answering Model") in the Intro to Keras Functional API here:  <a href="https://keras.io/getting-started/functional-api-guide/" rel="nofollow noreferrer">https://keras.io/getting-started/functional-api-guide/</a></p>
  
  <p>In a nutshell, this example takes a natural language question about a video and does a softmax on possible answers.  Essentially a vision_model is developed to encode one frame of video.  It's wrapped with a TimeDistributed() and applied to a video sequence...  that's passed to an LSTM to get one vector for the whole video sequence...  and that's concatenated with an encoded question.</p>
  
  <p>original code here:</p>
  
  <pre><code>video_input = Input(shape=(100, 3, 224, 224))
  # This is our video encoded via the previously trained vision_model (weights are reused)
  encoded_frame_sequence = TimeDistributed(vision_model)(video_input)  # the output will be a sequence of vectors
  encoded_video = LSTM(256)(encoded_frame_sequence)  # the output will be a vector
  
  # This is a model-level representation of the question encoder, reusing the same weights as before:
  question_encoder = Model(inputs=question_input, outputs=encoded_question)
  
  # Let's use it to encode the question:
  video_question_input = Input(shape=(100,), dtype='int32')
  encoded_video_question = question_encoder(video_question_input)
  
  # And this is our video question answering model:
  merged = keras.layers.concatenate([encoded_video, encoded_video_question])
  output = Dense(1000, activation='softmax')(merged)
  video_qa_model = Model(inputs=[video_input, video_question_input], outputs=output)
  </code></pre>
  
  <p>What if I wanted to throw out the LSTM on the video sequence and ask the same question about every frame, and do a softmax on potential outputs for all 100 frames?  I can conceptually think of two approaches, but I don't really know how to implement either, or if there's an entirely better approach for what seems like a common use case.</p>
  
  <p><strong>Approach 1</strong></p>
  
  <p>I suspect this is the way to go...  concatenate the encoded question with the output of vision_model, which is designed to encode one image...  this was done in the previous example at the same link ("Visual question answering model") and is called vqa_model.  </p>
  
  <pre><code>vqa_model = Model(inputs=[image_input, question_input], outputs=output)
  </code></pre>
  
  <p>Now can I pass this to a TimeDistributed wrapper??</p>
  
  <pre><code>approach_1 = TimeDistributed(vqa_model)([video_input, question_input])
  </code></pre>
  
  <p>This yields an assertion error, for (I think) obvious reasons.  </p>
  
  <pre><code>AssertionError
  ----&gt; 1 approach_1 = TimeDistributed(vqa_model)([video_input, question_input])
  
  Path\Continuum\Anaconda3\lib\site-packages\keras\engine\topology.py in __call__(self, inputs, **kwargs)
      558                     self.build(input_shapes[0])
      559                 else:
  --&gt; 560                     self.build(input_shapes)
      561                 self.built = True
      562 
  
  Path\Continuum\Anaconda3\lib\site-packages\keras\layers\wrappers.py in build(self, input_shape)
      139 
      140     def build(self, input_shape):
  --&gt; 141         assert len(input_shape) &gt;= 3
      142         self.input_spec = InputSpec(shape=input_shape)
      143         child_input_shape = (input_shape[0],) + input_shape[2:]
  
  AssertionError: 
  </code></pre>
  
  <p>I need to pass it two inputs, only one of which (video_input) actually has the time dimension (question_input does not).  Is there a way to make something like this work...  or do I need to broadcast the embedded question 100x to match the time dimension of video input?  That seems awfully inefficient.</p>
  
  <p><strong>Approach 2</strong></p>
  
  <p>This idea seems super clunky...  but if I made vqa_model a shared layer, and manually called it 100x on the same question, but the next frame of video.</p>
  
  <pre><code>vqa_1 = vqa_model([frame_1, question_input])
  vqa_2 = vqa_model([frame_2, question_input])
  ...
  vqa_100 = vqa_model([frame_100, question_input])
  </code></pre>
  
  <p>or...  more likely...  am I going about this all wrong?</p>
  
  <p>Thanks in advance.</p>