title: Weighted summation of embeddings in pytorch
body: <p>I have a sequence of 12 words which I represent using a 12x256 matrix (using word embeddings). Let us refer to these as <img src="https://chart.googleapis.com/chart?cht=tx&amp;chl=e_1%2C%5Cdots%2Ce_%7B12%7D" alt="">. I wish to take this as input and output a 1x256 vector. However I don't want to use a (12x256) x 256 dense layer. Instead I want to create the output embedding using a weighted summation of the 12 embeddings</p>
  
  <p><img src="https://chart.googleapis.com/chart?cht=tx&amp;chl=w_1%20e_1%20%2B%20%5Cdots%20%2B%20w_%7B12%7D%20e_%7B12%7D" alt=""></p>
  
  <p>where the wi s are scalars (thus there is weight sharing).</p>
  
  <p>How can I create trainable wi s in pytorch? I am new and only familiar with the standard modules like nn.Linear.</p>