SQL Server - Order by case statement very slow
Any calculation or function, when applied to an index key column, will prevent SQL Server from using that index. 

In your case, ORDER BY Id DESC can make use of an ordered scan of the Clustered Index on your table, as Id is the key column on that index. That means, only 20 rows have to be read. 

Once you apply the CASE statement, SQL Server cannot use that index anymore and instead has to scan the entire row set and execute (very expensive) sorting on all rows to find the 20 rows you are interested in.

One of the solutions is to use dynamic SQL to include the appropriate ORDER BY without any calculations (as Lamak suggested in the comments above) or to just write the query 4 times and use an IF statement to decide which one to execute.