How to use the LSTM layer in PPO architecture?
Not sure if you're using TensorFlow-Agents for this, but if you are, there's some nice functionality for this built out using the ActorDistributionRnnNetwork and ValueDistributionRnnNetwork that I've found to be very helpful.  The LSTM architecture is used in both the actor and critic networks.  See this page for more information - hope this helps!