Calculating penalty to SNR for non-matched filter
Let's consider a received signal

$$Y(t)=Ap(t)+N(t)\tag{1}$$

where $A$ is the information symbol (modeled as a random variable), $p(t)$ is the transmit pulse, and $N(t)$ is additive white Gaussian noise (AWGN, modeled as a random process). For the sake of simplicity let's assume that all signals are real-valued (baseband case).

Filtering with an LTI system with impulse response $h(t)$ and sampling at $t=0$ gives

$$Y_0=\int_{-\infty}^{\infty}Y(t-\tau)h(\tau)d\tau\Big |_{t=0}=\int_{-\infty}^{\infty}Y(-\tau)h(\tau)d\tau\tag{2}$$

With $(1)$ you get

$$Y_0=A\int_{-\infty}^{\infty}p(-\tau)h(\tau)d\tau+\int_{-\infty}^{\infty}N(-\tau)h(\tau)d\tau\tag{3}$$

The power of the desired signal (the first term in $(3)$) is

$$P_S=E[A^2]\left(\int_{-\infty}^{\infty}p(-\tau)h(\tau)d\tau\right)^2\tag{4}$$

where $E[\cdot]$ is the expected value, and $E[A^2]$ is the power of the information symbol $A$. From the second term in $(3)$, the noise power is

$$P_N=\frac{N_0}{2}\int_{-\infty}^{\infty}h^2(\tau)d\tau\tag{5}$$

where $N_0$ is the one-sided power spectral density of the noise. From $(4)$ and $(5)$ the SNR is

$$SNR=\frac{P_S}{P_N}=\frac{E[A^2]\displaystyle\left(\int_{-\infty}^{\infty}p(-\tau)h(\tau)d\tau\right)^2}{\displaystyle\frac{N_0}{2}\int_{-\infty}^{\infty}h^2(\tau)d\tau}\tag{6}$$

We know that $(6)$ can be maximized by a matched filter:

$$h_{opt}(t)=Cp(-t)\tag{7}$$

with some constant $C$. From $(6)$ and $(7)$, the optimal (maximum) SNR is given by

$$SNR_{opt}=\frac{E[A^2]}{\frac{N_0}{2}}\tag{8}$$

The actual SNR for an arbitrary receive filter impulse response $h(t)$ can be written in terms of $SNR_{opt}$:

$$SNR=SNR_{opt}\underbrace{\frac{\displaystyle\left(\int_{-\infty}^{\infty}p(-\tau)h(\tau)d\tau\right)^2}{\displaystyle\int_{-\infty}^{\infty}h^2(\tau)d\tau}}_{\le 1}\tag{9}$$

where the last term in $(9)$ (or its inverse) could be called the "SNR penalty".