Filter index slow initial use
Very selective partial indexes tend to get bloated quickly, because the table autovacuuming is triggered off the percent turn over in the table.  But 20% turn over in a table can mean many thousands of percent turn over in the partial index.  Nightly reindexing can also solve the bloat, but it won't stay solved for long if turn over is high.  You can crank up autovacuum so it happens multiple times throughout the day.
The first time you run this query in a while, it is probably marking a lot of obsolete index entries as killed.  That is probably one reason there are so many dirtied buffers, and also one reason it will be faster the next time, as it will no longer have to visit all the obsolete entries just to discover they are obsolete.  Of course the buffers it does still need to read are likely to be cached as well, so that would be another reason for it to be faster.
Note that this index-entry killing doesn't work in bitmap index scans, only on regular index scans.  Also, it doesn't work on hot standbys. Both of which can be a real problem for some setups.