Interpreting Gradients and Partial Derivatives when training Neural Networks
My question is, when we calculate partial derivative with respect to one parameter (e.g. weight between input x1 and 1st hidden layer neuron) then we are treating all other weights and biases as constants and we are evaluating how will cost function change if we were to take a step in the direction that is represented by that particular weight. Is this correct?


Yes.

Long answer: that is exactly the meaning of partial derivative. It's the impact of a specific variable, while keeping all the others constant. It's an all things equal condition (or ceteris paribus, if you like Latin).

You need it in order to understand how much that specific parameter, at that current value, is contributing to the final Loss. You have to keep everything else equal in order to understand what is that parameter's responsibility of the final model's error.




  Also, what is the role of Jacobian matrix in NN training?


More generally, the Jacobian matrix of a Neural Network is a matrix of the partial derivatives of the y. It's size is:

( Number of observations, Number of parameters )

It's a way to store/represent your gradient information. The application of the chain rule to execute backprop is executed on those in practice.