embedding layer outputs nan
I am trying to learn a seq2seq model.
An embedding layer is located in the encoder and it sometimes outputs nan value after some iterations.
I cannot identify the reason.
How can I solve this??
The problem is the first emb_layer in the forward function in the code below.


class TransformerEncoder(nn.Module):
    def __init__(self, vocab_size, hidden_size=1024, num_layers=6, dropout=0.2, input_pad=1, batch_first=False, embedder=None, init_weight=0.1):
        super(TransformerEncoder, self).__init__()
        self.input_pad = input_pad
        self.vocab_size = vocab_size
        self.num_layers = num_layers
        self.embedder = embedder

        if embedder is not None:
            self.emb_layer = embedder
        else:
            self.emb_layer = nn.Embedding(vocab_size, hidden_size, padding_idx=1)

        self.positional_encoder = PositionalEncoder()
        self.transformer_layers = nn.ModuleList()
        for _ in range(num_layers):
            self.transformer_layers.append(
                    TransformerEncoderBlock(num_heads=8, embedding_dim=1024, dropout=dropout))

    def set_mask(self, inputs):
        self.input_mask = (inputs == self.input_pad).unsqueeze(1)

    def forward(self, inputs):
        x = self.emb_layer(inputs)
        x = self.positional_encoder(x)


It is usually the inputs more than the weights which tend to become nan (either goes too high or too low). Maybe these are incorrect to start out with and worsen after some gradients. You can identify these inputs by running the tensor or np.array thru' a simple condition check like:
print(&quot;Inp value too high&quot;) if len(bert_embeddings[bert_embeddings&gt;1000]) &gt; 1 else None

A common mistake for a beginner is to use a torch.empty instead of torch.zeros. This invariably leads to Nan over time.
If all your inputs are good, then it is the vanishing or exploding gradients issue. See if the problem worsens after a few iterations. Explore different activations or clipping gradients which usually fix these type of issues. If you are using latest optimizers you usually need not worry about adjusting the learning rate.
