About optimal distance (or dissimilarity) measure in proportion data My data is proportional data, but variables are correlated with each other. So I am wondering what kinds of distance or dissimilarity we should use to take correlation into account.
If we would like to use data with proportion (e.g.,total sum scaling or compositional data), what kinds of distance or dissimilarity can we use?
Bray-Curtis is one of them, but does this make sense to use arcisin or logit transformation and use Pearson correlation?
In continuous variables, we can use Pearson and Spearman distance. In our case, what kinds of distance can we use? Pearson distance cannot be applied to data with 0-1 scale. Logit transformation -&gt; Pearson?
I would suggest reading a little on the topic; my understanding is that distance measures can be pretty robust to proportion/compositional data. Here is a relevant textbook chapter:

Other distance measures assume that the data are proportions ranging from zero to one, inclusive.

Among these are indeed Bray-Curtis as well as Jaccard distance (and others). I think that you will want to use one of the more traditional distances rather than a Pearson/Spearman, though they are getting at the same problem. A transformation might help, I suppose, but I would prefer to choose a simple but predictable distance measure rather than a correction on a possibly unsuitable distance that might have unpredictable behavior.
It's a little hard to tell from your question, but it seems that you are wanting to in some way account for indirect correlations using a distance metric. I would not suggest trying to make your distance measure do all this work. Instead, I would use a simple but clear distance method, and then use a multidimensional analysis such as principal coordinate analysis (or similar) to understand the correlation structure of your compositional variables. This is a fairly standard workflow in metagenomics, for instance, which relies heavily on high-dimensional compositional data. I would recommend looking more into metagenomics statistical applications for inspiration if needed.
The choice does also depend to a certain extent on the specific application. In some fields certain methods are familiar and trusted even if there is not a strong empirical/theoretical rule.
You've asked a lot of questions about Pearson/Spearman correlations. Much has been written on the use of correlation analysis on compositional data, most of it cautioning against doing this kind of analysis without special treatment. This leads to difficulties in building meaningful association models for different variables. One method that I'm familiar with in metagenomics is sparse correlation, e.g. SPARCC. But there are newer methods that may be better, for example here and here.