Does Eraserhead have to be continuosly watching someone to erase their quirks?
Yes. As far as I can recall after reading the manga and as mentioned here, 


  Erasure requires a continuous focus on a specific target for its effects to be maintained... Erasure is naturally impractical against groups of surrounding enemies, because it's impossible for Shota to keep watching every potential target within a 360º radius. 


This has a drawback as eventually, Aizawa would need to blink every once in a while so the effects are not permanent.

If you can cite which episode you are referring to about Overhaul, please do because I haven't watched the 4th season yet, I only read the manga.