Can you learn parameters in nonlinear function?
In general, you can learn any parameter of the network, provided you can find the partial derivative of the loss function with respect to the desired parameter. Given that $\rho$ is assumed to be differentiable (as the authors state in the paper), you can take the partial derivative of the loss function with respect to the parameter $\alpha$.

In this paper, $\rho$ is a non-linear function (that is, a function that is not linear, e.g. the sigmoid function) that applies element-wise to its input. So, if you pass a vector to this $\rho$, you will get a vector of the same shape out of it. The authors do not explicitly call it an "activation function", but $\rho$ does an analogous job of an activation function, that is, it introduces non-linearity. Furthermore, in this architecture, $\rho$ is also followed by a matrix. In general, this is not forbidden, even though it is not common. 

In general, in each layer of a neural network, you can have several different learnable parameters or weights. A parameter is learnable if you can differentiate the loss function with respect to it. You can have more than one weight matrix. For example, recurrent neural networks have usually more than one weight matrix associated with each layer: one matrix is associated with the feed-forward connections and the other matrix is associated with the recurrent connections.