How to enforce covariance-matrix output as part of the last layer of a Policy Network? I have a continuous state space, and a continuous action space. The way I understand it, I can build a policy network which takes as input a continuous state vector and outputs both mean vector and covariance matrix of the action-distribution. To get a valid action I then sample from that distribution.

However, when trying to implement such a network, I get the error message that the parts of my output layer which I want to be the covariance matrix are singular/not positive-semi-definite. How can I fix this? I tried different activation-functions and initializations for the last layer, but once in a while I run into the same problem again. 

How can I enforce that my network outputs a valid covariance matrix?
Usually it is assumed that there is no correlation between different actions, so the covariance matrix will be zero everywhere except on the main diagonal. Diagonal will represent variances of actions. Diagonal covariance matrix will be positive semidefinite if all values on diagonal are $\geq$ 0 so you need to insure that output of final layer is $\geq$ 0, which can be done with ReLU activation for example.