How is direction of weight change determined by Gradient Descent algorithm
First thing is that what does gradient descent do? Gradient Descent is a tool of calculus which we use to determine the parameters (here weights) used in a Machine Learning algorithm or a Neural Network, by running the Gradient Descent algorithm iteratively. 

What does the vector obtained from one iteration of gradient descent tell us? It tells us the direction of weight change (when weights are treated as a vector) for the maximum reduction in the value outputted by a loss function. The intuition behind why Gradient Descent gives such a direction can be found here:


Why the gradient is the direction of steepest ascent - Khan Academy
Why is gradient the direction of steepest ascent?
Why does gradient descent work?


In general cases the cost/loss function is an n-dimensional paraboloid (we design function in such a way such that its convex).



A 2-D paraboloid with x and y as independent variables.

Now why do we reduce the weights in the direction of negative gradient only? Why not some other direction?

Since we want to reduce the cost in Linear Regression for better predictions we choose only the negative Gradient Descent direction, as it is the direction of steepest descent even though we could have chosen some other possible vector direction to reduce cost, but the negative gradient descent direction ensures that:


Cost is always decreased (if the step size is correct and in general only in cases of convex functions e.g paraboloid).
Cost is decreased by the maximal amount if we move in that direction, so we don't have to worry whether cost will decrease or not if we move in that direction.


Also we use learning rate alpha to scale by how much amount we want to decrease the weights.

EDIT: As pointed by @pasaba the error function may not be a paraboloid but in general a good cost function looks like a paraboloid with skewed axis.