Policy gradient on data only, without emulators
I think your best approach is to use Imitation Learning. Many techniques in imitation learning use Supervised Learning so you do not need to use emulator. Check DAGGER which is used in continuous action scenarios or the recent AggreVated algorithm (just ignore the theoretical parts of the paper).

As a start you can use Supervised Learning just for experimentation and then use the above algorithms. I would suggest though to use even a poor simulator just to have an idea of how your implementations behave after the. Bare in mind that RL tries to solve an optimization problem (maximize an utility/cost function) whereas the Supervised Learning methods try to optimize the difference between model's prediction and ground truth. Just be cautious of the algorithm's behavior before you go live.