Why does the secondary server not failover?
Wouldn't the server in site2 think that it was the last remaining server and failover to become the primary?

No, because it is part of a group with only two votes, which is not a majority.  For all the DR side knows there are three healthy nodes on the primary side which have quorum.
When you gracefully shut down the primary side, Dynamic Quorum reconfigured the votes.
Quorum is explained here: https://docs.microsoft.com/en-us/windows-server/failover-clustering/manage-cluster-quorum