Conflicting directions of weights gradients in gradient descent?
That is, maybe Err reduces when w1 goes towards ∂Err/∂w1 alone, but it might well be the case that Err actually increases when w1 is updated along with w2, that is when we are taking steps together in direction of all these weights, we might actually not go down the Err.
Isn't this the case?

Not exactly. Using back propagation, the weight gradients may be calculated precisely (for the given training data). It doesn't matter that there are many of them updating at the same time. The gradients don't &quot;conflict&quot; as such, they are 100% compatible. But they are only valid locally to the current values of $w_i$
An update to the all the weights at once in the opposite direction to the gradient is guaranteed to reduce the error value for that training data, with an important caveat that it is only fully guaranteed to make an infinitesimal small improvement, when the step size is also infinitesimal.
If you make an update step that is too large (for some value of &quot;too large&quot; which varies a lot depending on context), then the curve may not remain consistent over the step and your error value could increase.
This problem is not directly related to updating multiple weights at once. It can occur even when you have a single weight. However, when there is a more complex function, with more weights all changing at once, there can be more places where the function does this.
In practice, infinitesimal updates would take too long to train, so you need to find a larger value - but not so large it causes problems - as a compromise.
In addition:

The training data you have will usually allow you at best to create a rough approximation to the &quot;true&quot; function you are searching for. So you don't really want to find an actual global minimum in the error function for your training data set, as it would overfit.

Typically, using mini-batches, the gradient is also only a rough approximation, so updates only go roughly in the right direction (this can sometimes be good as it can help escape from local minima and saddle points)