clustering 2-dimensional euclidean vectors - appropriate dissimilarity measure
I hope I followed your question correctly. If you have those data points on as 2D vectors, that would mean, that you have, say $N=50,000$ data points each represented as $\left(x_i,y_i\right)$ , right? If angle and length are what you are concerned about, then your formulation seems right. You can convert each data point $\left(x_i,y_i\right)$ into $\left(\theta_i, d_i\right)$, which are the angle and length of the vector. These two parameters $\left(\theta_i, d_i\right)$ act as two features in each of your samples and you can run k-means on this. You have to be careful about using a consistent measure on your angle (always anti clockwise or clockwise). Your dissimilarity measure seems quite correct as far as I can tell. I'm sure you are aware of this python package but just for the sake of completion you can use  this or in MATLAB you can use this. I'm not sure about R, so pardon me if that was the essential part of your question.