Do we need to use off-policy methods for policy shaping?
You can use Policy Gradients for this which is on-policy. One of the most straightforward ways to do that is to provide the teacher/expert's probabilities over actions at each time-step. Then you can use the usual policy gradient loss plus the cross entropy between expert's actions distribution and agent's action distribution:

$$\mathcal{L(\boldsymbol{\theta})} =(R_t-V_{\boldsymbol{\theta}}(\boldsymbol{s}_t)) \log \pi_{\boldsymbol{\theta}}(a_t|\boldsymbol{s}_t) + \lambda H(\pi_{expert},\pi_{\boldsymbol{\theta}})$$

You could get a look at this paper for basic ways to affect on-policy learning in PG methods.

On the why this works, I will give you a descriptive explanation: Cross entropy is related to KL divergence which is a "measure" of the distance between two distributions. With the extra term in the Loss function we are trying to minimize the distance between the expert's distribution and agent's distribution (i.e. making the agent's one more similar to the expert's one).