Gradient descent with infinite gradient value
Is the method replacing infinitely large values to some large but
  finite values correct?


Yes. For example, the same problem happens for the logarithm in cross-entropy loss function, i.e. $p_i \text{log}(p'_i)$ when $p'_i \rightarrow 0$. This is avoided by replacing $\text{log}(x)$ with $\hat{\text{log}}(x) = \text{log}(x+\epsilon)$ for some small $\epsilon$. 

Similarly, you are changing $f(x)$ in the denominator to $\hat{f}(x) = max(\epsilon, f(x))$. 

However, I would suggest $\hat{f}(x) = f(x) + \epsilon$ instead of a cut-off threshold. This way, the difference in $f(x_1) &lt; f(x_2) &lt; \epsilon$ would not be ignored unlike the max cut-off.