How to prove a train of sinc pulses in digital communicaton system are orthogonal to each other?
The idea of solving the integral in the frequency domain is good, but you made a mistake rewriting the integral. Note that
$$\int_{-\infty}^{\infty}\phi_n(t)\phi_k(t)dt\tag{1}$$
equals the Fourier transform of $\phi_n(t)\phi_k(t)$ evaluated at $f=0$. As you know, that Fourier transform is given by the convolution of the two individual Fourier transforms of $\phi_n(t)$ and $\phi_k(t)$, respectively:
$$\mathcal{F}\big\{\phi_n(t)\phi_k(t)\big\}=\int_{-\infty}^{\infty}\Phi_n(\xi)\Phi_k(f-\xi)d\xi\tag{2}$$
Evaluating $(2)$ at $f=0$ gives
$$\int_{-\infty}^{\infty}\phi_n(t)\phi_k(t)dt=\int_{-\infty}^{\infty}\Phi_n(\xi)\Phi_k(-\xi)d\xi=\int_{-\infty}^{\infty}\Phi_n(\xi)\Phi_k^*(\xi)d\xi\tag{3}$$
where the last equality in $(3)$ is true because $\phi_k(t)$ is real-valued. Eq. $(3)$ is just Parseval's theorem.
I'm sure you can continue from here and show that the right-hand side of $(3)$ equals zero for $n\neq k$.

Note that the integral you tried to compute equals the inverse Fourier transform of $(\Phi_n\star\Phi_k)(f)$ evaluated at $t=0$, i.e., it equals $\phi_n(0)\phi_k(0)$ which also satisfies
$$\phi_n(0)\phi_k(0)=\delta[n-k]\tag{4}$$