about the normalization of RNAseq data for calculating distance for unsupervised learning I have been working on clustering using RNAseq data. To compute distance, what kinds of normalization is optimal?

Can we use normalization using relative log normalization? I understand this should be used for differential gene expression analysis, not for unsupervised learning in which we do not use information of disease condition.

If you can show good papers, please share.
Typically some sort of variance stabilizing transform is used before clustering. Popular options are regularized log transformation or a vst transform, which are available in DESeq2. Note that these are NOT used for performing differential expression, just things like clustering and PCA. For differential expression one would use TMM, RLE or similar.