ORDER BY ASC is 300x faster than ORDER BY DESC
I think the problem is two-fold (or 3-fold):


missing composite index on both columns filtered: (x, inserted_at)
uneven distribution
old statistics



   Filter: (a.x = 'some_x_val'::uuid)
   Rows Removed by Filter: 2745418



The unusual distribution means that the single index scan has to do a lot of work to remove the rows that don't match the 2nd filter (by the other (x) column). The index scan for ORDER BY ASC is just lucky and finds the 20 rows a bit more quickly, only by luck. It could be the other way around with other values in the parameters of inserted_at and x

So the best way to solve this is to add this composite index. You could follow the advice on the linked question and answer by Erwin, to update statistics on the table but this may help with the issue or not. It's not certain.