Otsu Thresholding : Why 'minimum within class variance' gives the optimum threshold?
Otsu threshold assumes a bi-modal (e.g. two-class) histogram. Nobuyuki Otsu shows that in such a case, minimizing the within-class variance is the same as maximizing between-class variance. To separate the bi-modal histogram, that is actually what you need to do. It would correspond to maximizing the distance between the clusters, which would give you the optimum point of separation.

Please check the paper for further details:

https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&amp;arnumber=4310076

Nobuyuki Otsu (1979). "A threshold selection method from gray-level histograms". IEEE Trans. Sys., Man., Cyber. 9 (1): 62–66