Can I tell postgres to use a different "collation" for an integer column?
You can use a CASE expression for the ORDER BY:

order by case the_column 
            when 0 then 0 
            when 3 then 1
            when 1 then 2
            when 4 then 3
            when 2 then 4
            else 5 --- everything else
         end