Postgresql - Distinct array aggregation
You could create your own aggregate to append arrays:
CREATE AGGREGATE array_append_agg(anyarray) (
   SFUNC = pg_catalog.array_cat,
   STYPE = pg_catalog.anyarray
);

Then use that aggregate function in your outer query.