Which matrix represents the similarity between words when using SVD?
You can find some material here and here but the idea (at least in this case) is the following: consider the full SVD decomposition of the symmetric matrix $A = W \Delta W^T$. We want to calculate the cosine similarity between the $i$-th column (aka word) $a_i$ and the $j$-th column $a_j$ of $A$. Then $a_k = A e_k$, where $e_k$ is the $k$-th vector of the canonical basis of $\mathbb{R}$. Let's call $\cos(a_i,a_j)$ the cosine between $a_i,a_j$. Then
$$\cos(a_i,a_j) = \cos(Ae_i,Ae_j) = \cos(W \Delta W^T e_i,W \Delta W^T e_j) = \cos(\Delta W^T e_i,\Delta W^T e_j)$$
where the last equality holds because $W$ is an orthogonal matrix (and so $W$ is conformal, i.e. it preserves angles). So you can calculate the cosine similarity between the columns of $\Delta W^T$. A $k$-truncated SVD gives a well-enough approximation. In general, columns of $W \Delta$ and rows of $W$ have different meanings!