Can I use computational diffie hellman problem in the following scheme?
Well, the problem "given $X, Y$, recover $R$" is impossible, even to a computationally unbounded adversary.  The reason is that any value of the form $e(X, Y)^k$ is possible; for any set of nonzero values for $X, Y, n, m, k$, there exists values for $r, P, Q$ that is consistent with the observed $X, Y$ values, and which $R=e(X,Y)^k$

However, that is unlikely to be the only way to attack this system; as stated, your system is informationally secure (in that the attacker literally doesn't have enough information to attack the system), and that is quite rare in cryptography.  Most likely there is some additional information that is available to the attacker; what this information is is vitally important in deciding on the security of the system.