title: How to add a tensor of trainable weights to a Keras tensor?
body: <p>Following the "Temporal Encoding" section on page 5 of <a href="https://arxiv.org/pdf/1503.08895.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1503.08895.pdf</a> (an excellent paper by the way), I have say N many embedded vectors of dimension M. So my Keras tensor is (batch size, N, M) and I want to add an N by M matrix of weights to each of the batch-size-many samples. To that end I've created my own Keras layer:</p>
  
  <pre><code>from constants import BATCH_SIZE
  class Added_Weights(Layer):
  
    def __init__(self, input_dim, output_dim, **kwargs):
        self.output_dim = output_dim
        self.input_dim = input_dim
  
        super(Added_Weights, self).__init__(**kwargs)
  
    def build(self, input_shape):
        # Create a trainable weight variable for this layer.
        self.kernel = self.add_weight(name='kernel',
                                    shape=(BATCH_SIZE, self.input_dim[0], self.input_dim[1]),
                                    initializer=RandomNormal(mean=0., stddev=0.05, seed=None),
                                    trainable=True)
  
        print("kernel has shape "+self.kernel.shape + " or "+K.int_shape(self.kernel))
        super(Added_Weights, self).build(input_shape)  
  
    def call(self, x, **kwargs):
        return Add()([x, self.kernel])
  
    def compute_output_shape(self, input_shape):
        return (BATCH_SIZE, self.input_dim[0], self.input_dim[1])
  </code></pre>
  
  <p>And this WORKS, but the problem is that each of the BATCH_SIZE many matrices has DifferenT weights. I need to be adding the same weights to each of the samples in the batch.</p>
  
  <p>So I've tried a couple things. Keras has a built in RepeatVector layer, so I tried giving the kernel shape (N, M) and doing RepeatVector (BATCH_SIZE)(kernel), but for some reason that ends up with shape (N, BATCH_SIZE, M). I'd like to use a Reshape there, but Reshape() treats the first dimension as the batch_size and won't allow me to modify it. Permute() has the same problem.</p>
  
  <p>Another thought was to make the initial shape as it is in the code, and then loop over the tensor to set slices 1 through BATCH_SIZE-1 equal to slice 0, so they're all holding the same weights, but I'm not allowed to assign values to Keras tensors that way.</p>
  
  <p>The only other thought I had was to just try it with shape (N, M) and hope Keras is smart enough to add it to each slice of the input, but after the Add() is applied to my (?, N, M) and the (N, M) kernel, somehow I end up with an (N, N, M) tensor, at which point we're dead.</p>