How to use mirror but affect different sides differently?
It will be difficult in this way, you may make it simple in following ways;

First work for the model in all aspect in which both sides are same - using mirror 

When you satisfied with your model apply the mirror.

Secondly, Now you are able to select any side as per your need and change any side individually