Possible reasons for word2vec learning context words as most similar rather than words in similar contexts I am observing my word2vec model learning context words as most similar rather than words in similar contexts. I don't understand why it (word2vec in general, not my model in particular) can behave like that and would like to know why.

I have implemented the original word2vec in keras. I chose the variant with the dot product layer rather than the hierarchical softmax and trained the model on a Wikipedia dump that I split into 5-grams. For each word I construct 8 pairs with a binary target label as training items. I use the 4 context words with the label True and choose 4 random words that are not one of the context words with the label 0.

Intuitively this model should learns similar representations for words in similar context, because it modifies the representation of these words in a similar manner, as it optimizes them independently with similar context words.
So these similar words are not directly made similar but rather indirectly, as they are subject to similar nudges due to their similar contexts.

The model is this:

input_target = Input((1,))
input_context = Input((1,))

embedding = Embedding(vocab_size, vector_dim, input_length=1, name='embedding')

target = embedding(input_target)
target = Reshape((vector_dim, 1))(target)
context = embedding(input_context)
context = Reshape((vector_dim, 1))(context)

dot_product = Dot(axes=1)([target, context])
dot_product = Reshape((1,))(dot_product)

output = Dense(1, activation='sigmoid')(dot_product)

model = Model(inputs=[input_target, input_context], outputs=output)
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])


However, what I observe from training this model is, that when I rank the most similar words given a certain word, that I get words that appear in the context of that word, rather than words that appear in similar context, as the most similar words.

For example:

words most similar to "plant":

rank |   word

0    |   amount
1    |   surface
2    |   electron
3    |   mass
4    |   plant # also: Why is plant not most similar to plant? How can that happen?
5    |   fluid
6    |   air
7    |   metal
8    |   molecule
9    |   cell
10   |   electric
11   |   per
12   |   oxygen
13   |   demonstrate
14   |   smooth


To me that looks a lot more like words that appear in the context of plant rather than words that appear in similar context as it.

The function to compute these is:

def get_most_similar(word_vector, embeddings, n=15):
    """
    find the `n` words that are most similar to `word_vector` in `embeddings`
    measured by their cosine similarity
    """ 
    v = word_vector
    m = embeddings
    cosines = (np.dot(v, m.T))/(np.linalg.norm(m.T, axis=0)*np.linalg.norm(v))
    ranked_by_similarity = np.argpartition(cosines, -n)[-n:]
    return reversed(ranked_by_similarity)


Is there a simple reason for that? 

I have the following other parameters:

word vector size: 300
batch size: 128
vocabulary size: 169161 (distinct lemmas)
training sample count: 27793586 (5-grams, overlapping within sentences)


I trained the model for 1 epoch and did only observe marginal further convergence into the second epoch.
Neural networks are different from other Machine Learning techniques in that they have to learn by repetition.  In Neural Network parlance, the repetition is called an epoch. During an epoch, each instance, in your case each word, is evaluated and the error is applied backwards through the layers and the weights are adjusted.  The changes to the weights can happen more or less aggressively based on the hyperparameters, but they tend to only move a little at a time.

You have trained your word2vec model for two epochs.  I didn't see how you initialized the weights for your model, but typically they are set to random values or set to a zeros. Given your results look a little random, I am sure it is because the model has barely begun to learn.

In order to get a well trained word embedding model, or any other Neural Network, you may need to train for hundreds, thousands, or even millions of epochs.  This is why the pre-trained word embedding models are so valuable even though they are freely available.  If you just want a well trained embedding model, download it. If you need to incorporate some domain specific usage, you can further train the downloaded embeddings. 

HTH