How to optimize count(*) using only foreign key index on PostgreSQL
PostgreSQL can only use an index to count the number of rows if the table has been vacuumed recently so that most table blocks are marked "all visible" in the visibility map. Otherwise it has to inspect the table to check ifbthe row is visible or not, and then an index scan is more efficient.

I think that the solution is not to count the rows. For a cheap estimate, use pg_class.reltuples. For a more in-depth treatment of the topic, read my blog.