Combining array_agg and unnest
You need to do the unnest in a sub-query:

select d."key", array_agg(distinct x.v) 
from data d
  cross join lateral unnest(d."values") as x(v)
group by d."key";


Set returning functions (like unnest()) should in general be used in the from part of a query. But to be able to reference a column from the table you need a lateral join. 

from data cross join lateral unnest(...) is an explicit way of writing from data, unnest(...) which also generates a cross join. But I prefer the explicit cross join operator to document that I indeed intended to write a cross join, rather than accidentally.

This will however not preserve the order of the elements.

Online example: https://rextester.com/TVIDB57711