vestigial sideband modulation
Practical VSB transmitters generally haven't proper antisymmetrical output signal filtering around the carrier frequency as Harry Nyquist in his VSB modulation theory proved they should have. One sideband is transmitted nearly as is, the carrier is attenuated and the the other sideband is band limited radically. The total filtering in the transmitter is a non-ideal bandpass filtering which is presented by transfer function Hi in your formula.

The non-ideality of the VSB filtering in the transmitter can be compensated in the receiver, if we have there after the detector a correction filter which has transfer function

$$      H_0(f)=\frac{1}{H_i(f+f_c)+H_i(f-f_c)}                       $$

The formula can be used to determine the frequency response of the correction filtering if the filtering in the transmitter is known. 

VSB was common in the era of analog TV. These calculations for analog television design were unnecessary because the needed filterings were determined in TV signal norms. I believe (I haven't proven it) the formula isn't valid if the detector is a diode, synchronous detection by mixing down to 0Hz is needed to avoid unwanted new frequency components in detection. In analog TV receivers the correction filtering was in intermediate frequency stages before the detector to avoid the need of complex synchronous detection.

Warning: Your sourcebook, where you have taken this formula from, seems to assume the reader is well familiar with usual university level communication engineering math, where  transfer functions are complex and negative frequencies are as meaningful as positive ones. 

If you happen NOT to have any sourcebook, only the given formula titled with "VSB" and you are wondering what in the hell it exactly and precisely presents in relative to VSB, you can find more complete theoretical discussion from here: http://faculty.kfupm.edu.sa/EE/wajih/files/EE%20370/EE%20370,%20Lecture%2013.pdf 

Be sure you need math knowledge worth several months of full time studies to understand the calculations.