title: One-hot encoding in pytorch/torchtext
body: <p>I have a <code>Bucketiterator</code> from <code>torchtext</code> that I feed to a model in <code>pytorch</code>. An example of how the iterator is constructed:</p>
  
  <pre><code>train_iter, val_iter = BucketIterator.splits((train,val),
                                               batch_size=batch_size,
                                               sort_within_batch = True, 
                                               device = device, 
                                               shuffle=True, 
                                               sort_key=lambda x: (len(x.src), len(x.trg)))
  </code></pre>
  
  <p>The data is then fed to a model like this, where I use the <code>nn.Embedding</code> layer. </p>
  
  <pre><code>class encoder(nn.Module):
      def __init__(self, input_dim, emb_dim, hid_dim, n_layers, dropout):
          super().__init__()
  
          self.input_dim = input_dim
          self.emb_dim = emb_dim
          self.hid_dim = hid_dim
          self.n_layers = n_layers
          self.dropout = dropout
  
          self.embedding = nn.Embedding(input_dim, emb_dim)
  
          self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, dropout = dropout)
  
          self.dropout = nn.Dropout(dropout)
  
      def forward(self, src):
  
          #src = [src sent len, batch size]
  
          embedded = self.dropout(self.embedding(src))
  
          #embedded = [src sent len, batch size, emb dim]
          hidden_enc = []
          outputs, hidden = self.rnn(embedded[0,:,:].unsqueeze(0))
          for i in range(1,len(embedded[:,1,1])):
              outputs, hidden = self.rnn(embedded[i,:,:].unsqueeze(0),hidden)
              hidden_cpu = []
              for k in range(len(hidden)):
                  hidden_cpu.append(hidden[k])
                  hidden_cpu[k] = hidden[k].cpu()
              hidden_enc.append(tuple(hidden_cpu))
  
  
  
          #outputs, hidden = self.rnn(embedded)
  
          #outputs = [src sent len, batch size, hid dim * n directions]
          #hidden = [n layers * n directions, batch size, hid dim]
          #cell = [n layers * n directions, batch size, hid dim]
          None
          #outputs are always from the top hidden layer
  
          return hidden, hidden_enc
  </code></pre>
  
  <p>But what if I wanted the embedding to be one-hot encoded? I work on formal languages and it would be nice to preserve orthogonality between tokens. It doesn't seem like <code>pytorch</code> or <code>torchtext</code> has any functionality for doing this. </p>