How can I compute the ideal variance threshold value for my data?
Variance thresholding is used to select those features with a variance above the suggested threshold. Ideally you would want to take in all features with a non-zero variance but I'm not sure of the data youre handling, it would be better to calculate the variance of the individual feature, arrange them in the increasing order of variance and then select that value where the variance sharply increases.
Or you could do PCA and find the order of the importance of features and then set the threshold.