What could be the cause of the drop in the reward in A3C?
I'll share my understanding so far.
This kind of behavior is actually normal when using on-policy algorithms with a sparse final reward.
Issue stems from the fact that once you get stuck in a behavior policy which does nothing (uses a "do nothing" action, for instance, until timeout), it's quite hard to get out of it, because you keep getting experiences that teach you nothing (no reward signal at all) and keeps you in the same policy.
Possible mitigations:


Encourage more exploration (in A3C, make the entropy loss coefficient bigger) in order to recover more quickly from this type of stationary behavior.
Use an off-policy algorithm with a big enough replay buffer, so that even if you start behaving this way you still use experiences from a "healthy" older policy. 


If you stick to a totally on-policy algorithm, making the batch_size bigger might help a little.