Entity Framework Skip and Take too slow
I would first suggest logging the database operation to see what SQL is executed.  EF may not be running the SQL you think it is.

The red flag I see is that you call ToList before the Skip and Take.  Generally in LINQ statements, ToList forces execution immediately.

This suggests your EF code is pulling every record from the table up front, building a list, and then applying the Skip/Take to the list.  If that's the case, the logged SQL should show a select statement without any filtering.

I suspect what you really want to do is run the ToList after the Skip and Take.  This should apply the filtering inside the executed SQL, but the logging will show you for sure.