PostgreSQL: Performance: ORDER BY an indexed column is slowing down the query performance
It is normal that retrieving a sorted result is more effort.
If you want the total count, all rows have to be scanned, even if you only return the first 50.
You could try order by name || '' to see if a sort would be faster.
My recommendation is not to retrieve the total count together with the query.
The optimizer's estimates are good in this case, so you should use an approximate result count by running
EXPLAIN (FORMAT JSON) SELECT ...

and retrieving the Plan Rows.