Is there a TSQL command or another way to rename AOAG (Always On Availability Group)?
I do not believe that there is a way to rename it (safely). You will need to rebuild the AG. 

If you do it during an outage window where you can eliminate or minimize data changes then you should be able to do it without having to perform new backups / restores.