How to prove a train of sinc pulses in digital communicaton system are orthogonal to each other? Consider a train of sinc pulses:
$$\phi_n(t)= \frac{\sin(\omega_M(t-nT_s))}{\omega_M(t-nT_s)}\quad; n=0,\pm1,\pm2,\dots$$
$\quad$where,$\quad T_s=\frac{\pi}{\omega_M}$
Now ,in order to show sinc pulses are orthogonal we need to prove:
$$\int_{-\infty}^{\infty}\phi_n(t)\phi_k(t)dt=T_s \delta_{nk} \quad \dots(1)$$
where, $\delta_{nk}$ is kronecker's delta.
So, i began doing it as follows:
$$T_s=\frac{\pi}{\omega_M}=\frac{\pi}{2\pi f_M}=\frac{1}{2f_M}=\frac{1}{f_N} \quad \dots(2)$$
where, $f_N$ is the Nyquist frequency
$$\phi_0(t)=\frac{\sin(\omega_Mt)}{\omega_Mt}=\frac{\sin(2\pi f_Mt)}{2\pi f_Mt}=sinc(2f_Mt)=sinc(f_Nt) \quad \dots(3)$$
Now, $\mathscr{F}\{ sinc(f_Nt) \}=\frac{1}{f_N} rect(\frac{f}{f_N})$ ,where $rect$ is a rectangular function centred at origin and having width= $f_N$
$$\implies \mathscr{F}\{ sinc(f_N(t-nT_s)) \}=\frac{1}{f_N} \exp(-i2\pi f n T_s) rect(\frac{f}{f_N}) \quad \dots(4)$$
Now we can write:
$$\int_{-\infty}^{\infty}\phi_n(t)\phi_k(t)dt=\int_{-\infty}^{\infty} \{ \Phi_n(f) \circledast \Phi_k(f) \} df$$
$$=\int_{-\infty}^{\infty} \int_{-\infty}^{\infty}\Phi_n(\tau) \Phi_k(f-\tau) d\tau df$$
$$=\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \exp(-i2\pi \tau n T_s) 
\frac{1}{f_N} rect(\frac{\tau}{f_N}) \exp(-i2\pi (f-\tau) k T_s) \frac{1}{f_N} rect(\frac{f-\tau}{f_N}) d\tau df$$
$$=\int_{-\infty}^{\infty} \exp(-i2\pi \tau n T_s) \frac{1}{f_N} rect(\frac{\tau}{f_N}) \exp(i2\pi \tau k T_s) \{ \int_{-\infty}^{\infty} \exp(-i2\pi f k T_s) \frac{1}{f_N} rect(\frac{f-\tau}{f_N}) df \} d\tau \quad \dots(5)$$
The inner integral of $(5)$ can be simplified as:
$$\int_{\tau -\frac{f_N}{2}}^{\tau +\frac{f_N}{2}} \frac{1}{f_N} \exp(-i2\pi f k T_s) df$$
$$=\frac{\exp(-i2\pi \tau k T_s) \sin(\pi k)}{\pi k} \quad \dots(6)$$
So, $(5)$ can be rewritten as:
$$\int_{-\infty}^{\infty} \exp(-i2\pi \tau n T_s) \frac{1}{f_N} rect(\frac{\tau}{f_N}) \exp(i2\pi \tau k T_s) \frac{\exp(-i2\pi \tau k T_s) \sin(\pi k)}{\pi k} d\tau$$
$$=\frac{\sin(\pi k)}{\pi k} \int_{-\infty}^{\infty} \exp(-i2\pi \tau n T_s) \frac{1}{f_N} rect(\frac{\tau}{f_N})d\tau $$
$$=\frac{\sin(\pi k)}{\pi k} \frac{\sin(\pi n)}{\pi n} \quad \dots(7)$$
Now, $(7)$ is even equal to $0$ when $k=2$ and $n=2$
So,where i missed ? any help or suggestions please...
The idea of solving the integral in the frequency domain is good, but you made a mistake rewriting the integral. Note that
$$\int_{-\infty}^{\infty}\phi_n(t)\phi_k(t)dt\tag{1}$$
equals the Fourier transform of $\phi_n(t)\phi_k(t)$ evaluated at $f=0$. As you know, that Fourier transform is given by the convolution of the two individual Fourier transforms of $\phi_n(t)$ and $\phi_k(t)$, respectively:
$$\mathcal{F}\big\{\phi_n(t)\phi_k(t)\big\}=\int_{-\infty}^{\infty}\Phi_n(\xi)\Phi_k(f-\xi)d\xi\tag{2}$$
Evaluating $(2)$ at $f=0$ gives
$$\int_{-\infty}^{\infty}\phi_n(t)\phi_k(t)dt=\int_{-\infty}^{\infty}\Phi_n(\xi)\Phi_k(-\xi)d\xi=\int_{-\infty}^{\infty}\Phi_n(\xi)\Phi_k^*(\xi)d\xi\tag{3}$$
where the last equality in $(3)$ is true because $\phi_k(t)$ is real-valued. Eq. $(3)$ is just Parseval's theorem.
I'm sure you can continue from here and show that the right-hand side of $(3)$ equals zero for $n\neq k$.

Note that the integral you tried to compute equals the inverse Fourier transform of $(\Phi_n\star\Phi_k)(f)$ evaluated at $t=0$, i.e., it equals $\phi_n(0)\phi_k(0)$ which also satisfies
$$\phi_n(0)\phi_k(0)=\delta[n-k]\tag{4}$$