Performance issue with larger resultsets SQL Server
Looking at the execution plan XML, there are a few things sticking out that may be the cause of your issue.


You are selecting everything back to the client. There are inherent wait times with transferring data over a network. The larger query is transferring almost 8x more data.
The larger query has a spill to TempDB. This means that parts of the query are using disk instead of memory. This will clearly slow it down much more than the smaller query.
You are running the under the compatibility level for SQL Server 7.0, so the fact that your arithabort setting is different for each execution can be causing issues. See here - http://www.sommarskog.se/query-plan-mysteries.html.


Regarding #3, you are either not running these the same way, or you are changing session level settings in between runs. You need to run these exactly the same way to get execution plans that can be compared accurately.