Word2Vec: Why do some dimensions of an embedding have an interpretation, and why does addition/subtraction of embedding vectors work?
The dimensions for an embedding space are only be accidentally interpretable.
However, vectors through the space can be interpretable. That is why word analogies are possible in an embedding space . The addition / subtraction of word vectors describe another vector through the embedding space. For example, &quot;king&quot; - &quot;man&quot; + &quot;woman&quot; approximate the &quot;queen&quot; vector.
Words are consistently used in relationship to other words. Word embeddings can model these consistent relationship by finding which words co-occur together and projection the words into lower dimension space that retains the most common occurrences.