How do you generalize from stochastic to batch learning using gradient descent?
Do you average them or something ?


Yes! The gradient of the loss for the minibatch is just a simple average of the gradients of the individual examples in the minibatch. See Section 8.1.3 of the Deep Learning Book:


  Hence, we can obtain an unbiased estimator of the exact gradient of the generalization error by sampling a minibatch of examples $\{x(1),...x(m)\}$
  with corresponding targets $y(i)$ from the data-generating
  distribution $p_{data}$, then computing the gradient of the loss with respect
  to the parameters for that minibatch: 
  $$\hat{g} = \frac{1}{m} ∇_\theta \sum_{i} L(f(x^{(i)}; \theta), y^{(i)}).$$