How does it make sense to speak of heritability of qualitative traits?
The concept of heritability is a concept coming from quantitative genetics. It only applies to quantitative traits. This does not mean that the trait must be continuous. Discrete traits (such as the number of eyes for example) are quantitative traits.

For boolean traits, it is common to set one outcome to 0 and the other to 1. Which outcome is set to which value does not matter as it won't affect the variance. To take your example of homosexuality, one would classify all individuals as homosexual or heterosexual (with not in between or other variants) and set homosexual to 0 and heterosexual to 1 (or vice-versa) and compute the heritability from there.

As far as I know, for non-boolean nominal traits, the concept of heritability is undefined.