If I remove all my upvotes from a user's posts which I ever upvoted, will it be considered as serial voting fraud?

Can I be blocked for doing so?

Note: Just a question. I have no intention of doing it.