Reduce query time for higher offset in sql server
Answer based on comments originally left by Dan Guzman:

OFFSET isn't magic; time will be progressively slower with higher offsets. Also, you should have a separate index for each column to be ordered but SQL Server can read each either forward or backward so you don't need additional permutations for descending order.

Instead of row-number pagination, you could use key pagination instead, passing the last retrieved values of the ordered column and primary key. The query could then specify `SELECT TOP(n) WHERE '. That would allow users to scroll forward (and backwards with similar logic). If you must do row number pagination, do that in a caching layer.

See Optimising Server-Side Paging - Part I and T-SQL Querying: TOP and OFFSET-FETCH (sample chapter) by Itzik Ben-Gan.

An anchor filter is required to avoid a progressively larger scan, which is costly to return a page far into the result with a large paginated result like yours. I've used that method with multi-billion row tables with sub-second response time, albeit that was back in the SQL 2000 days and I don't think users ever scrolled to the end.