What are the strategies for computationally heavy environments or long-time waiting environments?
You could use Ray RLlib. It has support for parallel environments, even over multiple GPUs and compute nodes.