best way to regularize gradient boosting regressor?
The hyper parameters that you could tune in any boosting technique are:


Depth of each tree: As you rightly pointed out this is very important because each tree in boosting technique learns from the errors of the previous trees. Hence underfitting the initial trees ensure that the later trees learn actual patterns and not noise.
Number of trees: this is kind of intuitive from previous point as the number of trees increase the learnable signal decreases and hence the ideal number of trees is more than underfitting trees and less than overfitted trees.
Learning rate: this parameter gives weights to previous trees according to a value between 0 and 1. Lower learning rates give lesser importance to previous trees. Higher weights lead to faster steps towards optimization.  Lower weights typically lead to global optimum. But lower learning rates need more trees to learn the function.


4.Sub sample: if the value is less than 1 a subset of variables is used to build the tree making it robust and learn signal from more variables. This variable reduces overfitting by not fitting only 1 variable but a group of variables.

These  variables if tuned correctly are sufficient to reduce overfitting.