Will swishing or moving a teabag around affect the steeping time in any way?
You are not hurting the flavor.
In theory, it will shorten the steeping time a miniscule (probably imperceptable) amount, but the effect is more psychologial perception than reality.
I pour boiling hot water over my teabag and start my timer for five minutes and don't touch or swirl the tea bag. If the bag sits in the hot water 20% longer (6 minutes instead of 5), I would be lying if I tried to pretend that I could preceive a taste difference from the previous cuppa I'd sipped. I cannot.
Costs you nothing - so try it and see if you can taste any difference.