Gradient descent in a noisy environment
If I understand correctly, your question is whether if you cannot compute the gradient of the function exactly, but only within some errors, then gradient descent method still works in that it gives you convergence to a minimum point?

The answer to this is Yes, if you use Backtracking gradient descent. (And not the standard version of gradient descent, that version works only if you have some strong assumptions on the cost function, such as being convex or having Lipschitz continuous gradients, together with some other assumptions such as the function has compact sub level sets. The point is that if you consider a regularisation technique, as mentioned in the answer by Victor, then even your original function has Lipschitz continuous gradients and has compact sub level sets, after you do regularisation, the new function won't satisfy these assumptions any more.) You can find more details in the paper I mention in my answer in the following link: 

Does gradient descent always converge to an optimum?