How to properly optimize shared network between actor and critic?
Just use one class inheriting from nn.Module called e.g. ActorCriticModel. 

Then, have two members called self.actor and self.critic and define them to have the desired architecture.Then, in the forward() method return two values, one for the actor output (which is a vector) and one for the critic value (which is a scalar). 

This way you can use only one optimizer.