How can the derivative of a neural network be calculated, given no mathematical expression?
I think what you mean to ask is how can differentiation occur when there's no obvious neural network function to differentiate?

Don't worry - lots of people get confused about this, because it seems like an obvious hole in the puzzle. As mentioned by @AtillaOzgur, neural networks use partial differentiation through backpropagation. 

First, take the output of all the neurons (except the one you're about to differentiate by) as a function:



The above diagram represents the output of one neuron. Do this for every neuron in your network until you have a set. Let's call this set function NN. The output of NN (given all your neuron outputs) is what you'd normally plug into your RL policy.

You then differentiate NN by a single neuron (n) as shown:

$$\frac{\partial NN}{\partial n} = \lim_{h\to0} \left(\frac{NN(\text{all other neuron outputs}, n + h) - NN(\text{all other neuron outputs}, n)}{h} \right)$$

In reality however, it's the partial derivative of the activation function (A) with respect to the output of a single neuron (n):

$$\frac{\partial A}{\partial n}$$

So, depending on your activation function, you just plug in your neuron output to a certain expression and you've found the value by which to update your neural network.

I hope this helps. Deep learning is definitely a field with a learning curve, but places like StackExchange are great resources to help you out.