Can we combine Off-Policy with On-Policy Algorithms?
In the DRL nanodegree in Udacity, the instructor says it is possible to combine on- and off-policy learning and suggests the following paper where this has been done: Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (ICLR 2017). 
Citing the paper:


  The core idea is to use the first-order Taylor expansion of the critic
  as a control variate, resulting in an analytical gradient term through
  the critic and a Monte Carlo policy gradient term consisting of the
  residuals in advantage approximations. The method helps unify policy
  gradient and actor-critic methods: it can be seen as using the
  off-policy critic to reduce variance in policy gradient or using
  on-policy Monte Carlo returns to correct for bias in the critic
  gradient.


The authors provide an open source implementation of it in https://github.com/shaneshixiang/rllabplusplus

There is a follow-up paper by the same authors also addressing this problem: Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (NIPS 2017).

The Related Work section in both papers might be also worth looking at.