Can A3C update the policy / critic on a local machine without needing to copy?
You might want to check out this paper relating to Phizaz's comment: Asynchronous Methods for Deep Reinforcement Learning (specifically search for Hogwild).