Can A3C update the policy / critic on a local machine without needing to copy? To make A2C into A3C you make it asynchronous. From what I understand the 'correct' way to do that is to thread off workers with a copy of the policy and critic, and then return the state/action/reward tuples to the main thread, which then performs the gradients updates on the main policy and critic, and then repeat the process.

I understand why the copying would be necessary in a distributed environment, but if I were to always run it locally then could I just perform the updates on a global variable of the policy and critic, i.e. avoid the need for copying? Provided the concurrency of the updates was handled correctly, would that be fine?
You might want to check out this paper relating to Phizaz's comment: Asynchronous Methods for Deep Reinforcement Learning (specifically search for Hogwild).