Order of reactivity of hydrogen halides
The halide ion is involved, indirectly.  The hydrogen halide has to protonate the alcohol first, and this requires dissociation of the proton from  the halide ion.  Chloride ion is more strongly basic than iodide ion, an effect we do not see in water (leveling effect) but can see it in the reaction with the alcohol.  So we can protonate more of the alcohol at any time, with hydrogen iodide, making more opportunities to then form the carbocation, than with hydrogen chloride.

Hydrogen fluoride would be way off the scale here, in the direction of less reactivity.  It is only a weak acid in water, let alone in this reaction mechanism.  It's one manifestation of fluorination being more difficult than other halogenations.