Exactly. In a real fight, if you were holding a towel like that someone would probably just swipe it from you.

That said, fabric can be super helpful in fights. Jackie Chan has some great tips on using a scarf to disarm a knife with minimal telegraphing.