Select distinct combinations of rows
If I get it right you would like the last row for each column combination (regardless of column order). Easiest is probably to enumerate the rows (using least, GREATEST for column order) in desc order and choose the first one

SELECT ... FROM (
    SELECT...
           , row_number() over (partition by least(author_id, recipient_id)
                                           , greatest(author_id, recipient_id) 
                                order by updated_at desc) as rn
    FROM private_message
    WHERE 665 IN (author_id, recipient_id) 
) as t
WHERE rn = 1
LIMIT 20