Is there a rule of thumb when designing neural network in deep reinforcement learning?
The process of reinforcement learning already implies that you have a base model to work from, that's what you're reinforcing. So, presumably, that underlying model is already good, otherwise you wouldn't be using it, right? 

The whole point of reinforcement learning is to introduce your (functioning) model to new information and/or changing conditions. Reinforcement learning isn't going to take a bad model and turn it into a good one. If your base model is not performing to your satisfaction, then you should go back to a model design/selection phase; reinforcement learning is not going to help you in that scenario.