Initialising DQN with weights from imitation learning rather than policy gradient network
My understanding is that you are first training a policy network using imitation learning. Then you are adjusting that trained network in some way to be a value network for DQN. The most obvious change would be to remove softmax activation whilst keeping the network layer sizes identical. This would then present Q estimates for all actions from any given state.
The initial estimates would not be trained Q values though, they would be the &quot;preferences&quot; or the logits for probabilities to support a near optimal action choice. The main thing that will be likely in the new network is that for the one near optimal action choice, the network would predict the highest action value. As you derive the target policy by taking the maximising action, initially this looks good. However, the problem is that the Q values that this network predicts can have little to no relation to the real expected returns experienced by the agent under the target policy.

Initial rewards from running the policy are high but start to decrease (for a while) as the DQN trains and later increases again.

I think what is happening is that initially the greedy policy derived from your Q network is very similar to the policy learned during imitation learning. However, the value estimates are very wrong. This leads to large error values, large corrections needed, and radical changes to network weights throughout in order to change the network from an approximate policy function to an approximate action value function. The loss of performance occurs because there is not a smooth transition between the two very different functions that also maintains correct maximising actions.
I don't think this can be completely fixed. However you might get some insight into potential work-arounds by considering that you are not just doing imitation learning here. Instead you are performing both imitation learning (to copy a near optimal policy) and transfer learning (to re-use network weights on a related task).
Approaches that help with transfer learning may also help here. For instance, you could freeze the layers closer to input features, or reduce the learning rate for those layers. You do either of these things on the assumption that the low-level derived features (in the hidden layers) that the first network has learned are still useful for the new task.