Why so many loops in pkey index scan?
The issue is here

Index Scan using entrytag_tag_id_row_idx on entrytag
    (cost=0.44..402.17 rows=99 width=4)
    (actual time=0.093..535.664 rows=989105 loops=1)

Your statistics are off for 

WHERE "entrytag"."tag_id" = 323456


Your planner thinks there are far fewer tag_id=323456 then there are. You may want to try ANALYZE entrytag;, and trying again. Or upping statistics. 

ALTER TABLE entrytag
  ALTER COLUMN tag_id
  SET STATISTICS 1000;


Then try ANALYZE entrytag; and trying again. Sounds like a typical case of bad statistics.

You may want to improve your schema, or denormallize. You're joining, selecting, and ordering million rows. It's not going to be instant.