Why does an imbalanced data set badly effect distance measures like Mahalanobis? I'm relatively new to data science and I am struggling to understand why the Mahalanobis distance (or any other distance measure) applied to an imbalanced data-set becomes inaccurate. I have a data set that consists of three classes A, B and C. There are 100 observations for class A, 60 for class B, and 20 for class C. When I calculate the Mahalanobis distance between each class, the results do not appear consistent with my PCA (principal component analysis) plot. In the PCA plot, class C is the most separate class; however, the Mahalanobis distance does not reflect this.
For balanced data sets, i.e., where classes A, B and C have the same number of observations, this has never been an issue. The Mahalanobis distance has always quite accurately reflected the results of PCA for balanced data.
I have read some similar questions and answers on here about why imbalanced data must be handled carefully for classification algorithms, but is this the same for distance measures? From what I can tell the Mahalanobis distance doesn't explicitly depend on sample size.
Therefore, I ask why does this measure lose reliability for imbalanced data?
Mahalanobis distance is defined as a distance between a point and a distribution. The key is how you define the distribution and I would say the imbalance of classses is not the problem in itself here. What could be a problem is mahalanobis is sensitive to initialization and the sample sizes of your classes are not that big. You could check the covariance matrix if it is reasonable for you task.
About more general question -- the distance between two points obviously does not depend on size of the classes. If we talk about distance between a point and a set, then it could affect the result: i.e. you defined the distance as the distance to the closest point in set, then obviously the more points, the bigger chance to get the closer point.