Linked server to SQL Azure very slow
Try setting the "Collation Compatible" to True on the Linked Server definition.

If you set it to False, it will pull the entire table down and do the compare on the receiving end. Consider the scenario where the local server is case insensitive, and the remote server is case sensitive, the results will differ based on where the compare is done.