How can I setup ACL for user to modify only custom fields?
Your ACL allows to view and edit all custom fields sets. I guess you previously had CiviCRM: view all contacts to be able to see the contacts at all. 

You still need to grant a view permission to access the contact or an edit permission to allow any modification to your desired set of contacts: 

I'm not aware that you can grant permission to edit custom fields without permission to edit the contact while using the UI.