What is the conversion between r and FST?
I am very unsure but here are my thoughts...

We can work out from 

$$R_{XY} = F_{XY}/F_{XX}$$

I do not really understand the concept of coefficient of kinship and the concept of coefficient of kinship with oneself but you say


  According to the paper, kinship with himself is 0.5 in a sexual population


So, let's assume

$$R_{XY} = 2F_{XY}$$

Let's assume that the individual Y is drawn from anywhere in the total population. Again, I don't really understand the concept of coefficient of kinship but let's assume that $F_{XY}$ is the same as the probability of identity by descent between these two individuals, then in Nei (1973) terms, $F_{XY} = J_T = 1 - H_T$. Hence, $R_{XY} = 2\left(1 - H_T\right)$ or $H_T = 1-\frac{R_{XY}}{2}$

Given that 

$$F_{ST} = 1 - \frac{H_S}{H_T}$$

it results that 

$$H_T = \frac{H_S}{1-F_{ST}}$$

and therefore,

$$1-\frac{R_{XY}}{2} = \frac{H_S}{1-F_{ST}}$$

solving into

$$R_{XY} = \frac{ 1 - \frac{H_S}{1-F_{ST}}  }{2} = \frac{1-F_{ST} - H_T}{2(1-F_{ST})}$$

I would recommend you making some more reading on the coefficient of kinship to ensure that my interpretation here is correct.