Why do we use gradients instead of residuals in Gradient Boosting?
Hmmm, I am little perplexed by your question.

In gradient boosting, we do use the residuals. The residuals are the gradients.

You can check my simple implementation of gradient boosting. This is where the magic happens:

def fit(self, X, y):
    self.fs = [self.first_estimator]
    # step 0
    f = self.first_estimator.fit(X, y)
    # step 1
    for m in range(self.M):
        f = self.predict(X)
        # step 2
        U = self.loss(y, f)
        # step 3
        g = clone(self.base_estimator).fit(X, U)
        # step 4
        self.fs.append(g)


You start by a dummy model $f$. Then you create a new model $g$ based on the errors of the existing ensemble $L(f(x),y)$. The code should be pretty straight-forward. Check the algorithm in the wikipedia page for a more formal presentation.



Why am I using residuals and gradients as interchangeable words?

You have discrete mathematics and continuous mathematics.

In neural networks, we use the definition of gradient from continuous mathematics. You have a loss function that is $C_1$ continuous (meaning that the loss must be differentiable at least once).

$$f'(x) = \lim_{h\to0} \frac{f(x+h)-f(x)}{h}$$

In gradient boosting, we use the definition of gradient from discrete mathematics. It's usually called fininite differences, instead of derivative or gradient.

$$f'(x) = f(x+1)-f(x)$$

Personally, I think gradient is a misnomer. But it's marketing. Gradient boosting sounds more mathematical and sophisticated than "differences boosting" or "residuals boosting".

By the way, the term boosting already existed when gradient boosting was invented. It was used for AdaBoost, which changes the weights of the observations based on the loss, rather than training in the residuals. AdaBoost only works for weak models (each model must commit errors), while gradient boosting can do resample and work for strong models.



Important point on gradient boosting

One important difference between gradient boosting (discrete optimization) and neural networks (continuous optimization) is that gradient boosting allows you to work with functions whose derivative is constant.

In gradient boosting, you can use "weird" functions like MAE or the Pinball function. In my code, you can see that you can choose between MSE, MAE and quantile (which is the pinball loss). These latter two do not work very well for neural networks.

The pinball loss is used for quantile predictions, like wind forecasts in energy systems. This is the reason why gradient boosting is much used in these systems.

Small point: many people think that the reason why MAE and Pinball functions work bad for neural networks is because they are not continuous. While it is true they are not continuous when $y=\hat y$, it's very easy to work-around that. This is not the reason why results are so poor with those losses. The reason why they don't work well is because the optimization steps use the gradient (in gradient descent), and the gradient is constant for the entire function $L'(y,\hat y)=1$ or $L'(y,\hat y)=-1$, so the magnitude of the gradient is always 1. The optimization will depend solely on the learning rate heuristic that you are using, which usually will not work out very well.