AlwaysOn read only replica slower then prime
To fix our problem we had to enable Trace Flag 3459. 

We are currently having this issue on SQL Server 2017 CU6 (14.0.3025.34). Not much documentation on this Flag, but from what I gather, enabling this flag will slow down our failover process. I'm going to test some more, and update if i run into anything show stoppers.