title: Keras how to add an attention layer for a weighted sum
body: <p>I have the following a network architecture (only the relevant part of the network is shown below) </p>
  
  <pre><code>vocab_dimension = 1500
  embed_dimension = 10
  
  
  x = [Input(shape=(None, ), name='input', dtype='int32'),
       Input(shape=(None, ), name='weights'),
       Input(shape=(None, ), name='neg_examples', dtype='int32')]
  
  
  embedding_layer = Embedding(input_dim=vocab_dimension, output_dim=embed_dimension)
  
  
  def _weighted_sum(x):
      return K.sum(x[0] * K.expand_dims(x[1], -1), axis=1, keepdims=True)
  
  
  weighted_sum = Lambda(_weighted_sum, name='weighted_sum')
  
  item_vecs = embedding_layer(x[2])
  user_vecs = weighted_sum([embedding_layer(x[0]), x[1]])
  </code></pre>
  
  <p>The problem here is that I would like to not pass the weights as input, but I would like to 'learn' them, like in an attention layer.</p>
  
  <p>I know that attention layers could be created this way</p>
  
  <pre><code>attention_probs = Dense(h, activation='softmax', name='attention_probs')(x[0])
  weighted_sum = Lambda(_weighted_sum)([x[0], attention_probs])
  </code></pre>
  
  <p><code>h</code> is equal to the dimension of the length of the input, which I set to 5. However, if I do the above I get the following error
  <code>TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'</code></p>
  
  <p>I think this has to do with the dimensions of the input parameters, but I'm not sure how to fix this.</p>