How to prevent Azure Traffic Manager from routing over high latency connections in a fail-over scenario?
Disclaimer: I have not setup a working example of this

I believe you are looking for Nested Traffic Managers

If you setup a main traffic manager that uses performance to select between North America and Europe child traffic managers.  As long as one of the services in East or West North America is functioning the child traffic manager will report its status as available and the main traffic manager will continue to send traffic to the child based on performance

Docs:
https://docs.microsoft.com/en-us/azure/traffic-manager/traffic-manager-nested-profiles