Word-embedding does not provide expected relations between words
I am trying to train a word embedding to a list of repeated sentences where only the subject changes. I expected that the generated vectors corresponding the subjects provide a strong correlation after training as it is expected from a word embedding. However, the angle between the vectors of subjects is not always larger than the angle between subjects and a random word.
Man   is going to write a very long novel that no one can read.
Woman is going to write a very long novel that no one can read.
Boy   is going to write a very long novel that no one can read.

The code is based on pytorch tutorial:
import torch
from torch import nn
import torch.nn.functional as F
import numpy as np

class EmbedTrainer(nn.Module):
    def __init__(self, d_vocab, d_embed, d_context):
        super(EmbedTrainer, self).__init__()
        self.embed = nn.Embedding(d_vocab, d_embed)
        self.fc_1 = nn.Linear(d_embed * d_context, 128)
        self.fc_2 = nn.Linear(128, d_vocab)

    def forward(self, x):
        x = self.embed(x).view((1, -1)) # flatten after embedding
        x = self.fc_2(F.relu(self.fc_1(x)))
        x = F.log_softmax(x, dim=1)
        return x

text = &quot; &quot;.join([&quot;{} is going to write a very long novel that no one can read.&quot;.format(x) for x in [&quot;Man&quot;, &quot;Woman&quot;, &quot;Boy&quot;]])
text_split = text.split()
trigrams = [([text_split[i], text_split[i+1]], text_split[i+2]) for i in range(len(text_split)-2)]
dic = list(set(text.split()))
tok_to_ids = {w:i for i, w in enumerate(dic)}
tokens_text = text.split(&quot; &quot;)
d_vocab, d_embed, d_context = len(dic), 10, 2

&quot;&quot;&quot; Train &quot;&quot;&quot;
loss_func = nn.NLLLoss()
model = EmbedTrainer(d_vocab, d_embed, d_context)
print(model)
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)

losses = []
epochs = 10
for epoch in range(epochs):
    total_loss = 0
    for input, target in trigrams:
        tok_ids = torch.tensor([tok_to_ids[tok] for tok in input], dtype=torch.long)
        target_id = torch.tensor([tok_to_ids[target]], dtype=torch.long)
        model.zero_grad()
        log_prob = model(tok_ids)
        #if total_loss == 0: print(&quot;train &quot;, log_prob, target_id)
        loss = loss_func(log_prob, target_id)
        total_loss += loss.item()
        loss.backward()
        optimizer.step()
    print(total_loss)
    losses.append(total_loss)

embed_map = {}
for word in [&quot;Man&quot;, &quot;Woman&quot;, &quot;Boy&quot;, &quot;novel&quot;]:
    embed_map[word] = model.embed.weight[tok_to_ids[word]]
    print(word, embed_map[word])

def angle(a, b):
    from numpy.linalg import norm
    a, b = a.detach().numpy(), b.detach().numpy()
    return np.dot(a, b) / norm(a) / norm(b)

print(&quot;man.woman&quot;, angle(embed_map[&quot;Man&quot;], embed_map[&quot;Woman&quot;]))
print(&quot;man.novel&quot;, angle(embed_map[&quot;Man&quot;], embed_map[&quot;novel&quot;]))


It's most probably the training size. Training a 128d embedding is definitely overkill. Rule of thumb from the the google developers blog:

Why is the embedding vector size 3 in our example? Well, the following &quot;formula&quot; provides a general rule of thumb about the number of embedding dimensions: 
embedding_dimensions =  number_of_categories**0.25 
That is, the embedding vector dimension should be the 4th root of the number of categories. Since our vocabulary size in this example is 81, the recommended number of dimensions is 3: 
3 =  81**0.25

