Why turning off condition_fanout_filter speed up query
you have one of the cases where the optimizer searching for a good solution costs much more time than the query it self, but that can change with increasing numbers.
You should read about what this optimization switch does.
if you need more speed you can try if one of https://dev.mysql.com/doc/refman/8.0/en/switchable-optimizations.html work better. additinal to different indexes