Neural networks, optimization math intuition When I look into the following partial derivative, I see it as being the key element of any optimization algorithm out there. Correct me if I'm wrong, but this gets us the slope of the loss function, so we can go opposite to that slope, therefore minimizing the loss.

$$\frac{\partial \theta}{\partial \mathcal{L}}$$

where: $\theta$ is the weights, and the $\mathcal{L}$ is the loss;



Does that make sense? Is there any other calculation step that is arguably more fundamental to the optimization of neural networks other than this derivative?

This topic is specially important for me right now, because I was thinking of tattoing this derivative, as a cool A.I. tattoo, and I want it to be fundamental and simple.
Note that $\frac{\partial L}{\partial \theta}$ is different from $\frac{\partial \theta}{\partial L}$. What you tried to describe seems to be $\frac{\partial L}{\partial \theta}$ where $\theta$ is a variable.  If $\theta$ is high dimensional, sometimes we just use the $\nabla$ notation. 

Gradient descent is 

$$\theta_{n+1}=\theta_n-\gamma \nabla L(\theta_n)$$


Not everything is differentiable and gradient might not be well defined for some optimization problem. 
In the event that there are constraints, $L$ might need to take the role of Langragian rather than the objective function. 
Gradient descent is just a means to find the parameters for a model. Gradient based approach seems to be the norm for now but things can change.


What you proposed to tattoo is just "gradient" or "slope". Not objection but just want to let you know what you are doing.