Can it hurt me to accept a brief "Visiting Lecturer" position?
The title of lecturer does not necessarily imply that you would teach full time. Sure, it usually implies more teaching than research, but that does not mean it's strictly teaching only. Especially for a visiting lecturer position, one could imagine there be arrangements possible.

So: have you checked the exact term under which you would be employed?

Also: how long would be this “brief” position? And what is your current seniority in your local research system? Those factors will play heavily into the decision.

Overall, I expect it's a net positive for you if you manage to keep some research activity going on, especially if you can travel relatively often to visit your non-US lab. Otherwise, if it's purely teaching with no opportunity to maintain some research activity, and you are not far from getting a job where you are, accepting might be risky.

“Long absent, soon forgotten”