What type of correlation is this equation?
I understand your confusion because the equation is barely understandable from the information given in the book. It becomes more clear from the original paper by Classen and Meyr [1] from which it has been taken. They propose a two stage frequency offset estimation that consists of an acquistion stage and a tracking stage. The equation you've cited represents the acquisition algorithm. In fact, it is a searching algorithm that "tries out" various values of frequency offset $\epsilon$ and chooses the one value $\hat\epsilon_\mathrm{acq}$ that yields the maximum correlation sum. It can be described as follows



For every $\epsilon$


Compensate the received time domain signal for the supposed frequency offset $\epsilon$
Calculate the DFT of the frequency offset compensated RX signal. This yields $Y_{l}[p[j],\epsilon]$
Calculate the correlation sum $R_\epsilon$. (The term inside $|\cdots|$)


Then, find the maximum $\hat R_\epsilon$ of all $R_\epsilon$ and calculate the estimated frequency offset by 
$$
\hat \epsilon_{acq} = \dfrac{\hat R_\epsilon}{2\pi \cdot T_{sub}}
$$



Of course it's not feasible to evaluate the correlation sum for every $\epsilon$ because the number of possbile values is infinite. Instead, you have to restrict the range of values to a reasonable frequency offset that is determined by the hardware you're using. Additionaly, this "search range" has to be partitioned into discrete values. The authors of [1] note


  In practice we found that it is sufficient to space the trial parameters $0.1/T_{sub}$ apart from each other.


Where $T_\mathrm{sub}$ is the length of one OFDM symbol including guard interval.



[1] Classen, F. and Myer, H. (June 1994) Frequency synchronization algorithm for OFDM systems suitable for
communication over frequency selective fading channels. IEEE VTC’94, pp. 1655 1659.