PReLU Activation Function update rule
Is it updated before weight update or after weight update?


It doesn't usually matter, provided your NN framework has separate gradient calculation stage from parameter update stage.

The gradients should be collected with all weights and parameters at their current values (during back-propagation). Then the parameters are updated, using those gradients. There should be no interaction between the parameters during the update step.

However, if you have implemented the training code yourself from scratch, and have interleaved the gradient calculations and weight updates, then this only works in a pure online situation with weights updated as the gradients are calculated. In that scenario, you should probably update the weights for a layer before updating the PReLU parameter for the same layer, to prevent changes in the activation function altering the gradients during your calculations (this is not a concern with usual weight updates). 

Alternatively, separate your gradient calculations from the weight update steps. This is more flexible and allows you to use other more advanced optimisations and layer designs.