Adding LIMIT changes PostgreSQL plan from good to a bad one
Tell PostgreSQL not to use the index by using

ORDER BY contacts__aggregated_1.value + 0 ASC


The problem is that PostgreSQL thinks it will be fastest if it scans contacts__aggregated in the ORDER BY order using the index, uses a nested loop to add rows from the other tables and filters out values that don't satisfy the conditions.

This is often a good strategy if not many values get filtered out.

It produces a bad plan, however, if


there are fewer rows that satisfy the filter condition than PostgreSQL estimates
the rows that satisfy the filter condition all have high value


One of these things must have happend. Look for high Rows removed by filter in the EXPLAIN (ANALYZE, BUFFERS) output.