Is there a way to remove everything in my Android phone Contacts and then transfer everything in Gmail contacts into the phone contacts?
If you only want a particular contact group to appear in the contacts app on your phone, then you can achieve it pretty easily by using the Contacts to display option in your Contacts app.


Open Contacts app. 
Tap on the 3 dots at top right, choose Settings. 
Select Contacts to display > customize. 
Expand your Google account and uncheck all the other groups and keep only My Contacts.




Do note that when performing a search within contacts, it will still search within all the available contacts. But under normal display it will only show your 97 contacts stored under My Contacts group.