How to prevent data loss during intra-site failover
The crux of your issue is:


  Given two physical sites, each site has two SQL servers with requirement that at any given time application is able to access one of its local SQL server instance to write/read data. Then the data need to be synchronized cross site and all four servers will contain synchronized data. Which SQL server’s HA solution or replication solution or combination of both should we pick to achieve such result?


The "one size fits all" answer is, the current offerings that SQL Server has won't fit into this. SQL Server doesn't work well in a multi-master zero data loss game. Other database solutions have similar issues where multi-master has the potential (like SQL Server) for data loss. I don't believe you're going to get a single bullet one size fits all solution here from the database layer itself.

The only way, with SQL Server, to get what you're asking for it to create the application or middle tier around this philosophy and put the onus on the application/middle tier to make sure all databases are in sync and let the application/middle tier determine how to handle what happens when they aren't (for example, a write fails on one database but 3 the other databases successfully committed - do you proceed?).