LSTM in reinforcement learning
You can use LSTM in reinforcement learning, of course. You don't give actions to the agent, it doesn't work like that.

The agent give actions to your MDP and you must return proper reward in order to teach the agent. For example if you implement trading bot, the policy(policy=the agent, which is your LSTM network) will say that at step T it is going to have action 34, which means something to your MDP and you return reward for example -0.03 or +0.05 or whatever depending what that actions is doing at the moment T. 

So I get the question like you want to do a supervised learning on a reinforcement learning environment.

You can mimic supervised learning as well, but the idea of reinforcement learning is not that.

Here is how to mimic:

Scenario: you are at step T, lets say you have 3 possible actions -1,0,+1;

In a supervised learning you must give the desired action to the learning process.
In reinforcement learning you must give reward based on if you are happy or not from the agent's action.

So you must have predefined that for -1 you are not happy and you give reward 0.0, for action 0 you are not happy and you give reward 0.0 and for action +1 you are happy and you give reward +100;

I hope this makes things clear.