Read MS Sql query high latency w.r.t. Remote Query
Create a copy of the table into a #temporary table on the server you are running the query on, then run the query based off the temp table. 

I am not an expert on why certain queries run slower over linked servers and others do not. However, if you move that data temporarily over to the server you are running the query on, it often makes it run faster.  Someone more experienced can answer why this is the case.