Why doesn't fluorine harm the ozone layer the way chlorine and bromine do?
TL;DR: The mechanism is different for fluorine and chlorine atoms. Fluorine atoms react and forms stable HF molecules while chlorine atoms turns into a radical by the action of UV which helps in the destruction of ozone.
Long answer:
Ozone depleting substance(ODS) are gases that take part in ozone depletion process. Most of the ODS are primarily chlorofluorocarbons(CFC) and halons that contain chlorine and bromine atoms respectively which reach the stratosphere and reacts with ozone and lead to ozone depletion. In this context, the chemical component responsible are chlorine radical (Cl·) and bromine radical (Br·). The ODS travel to the stratosphere without being destroyed in the troposphere due to their low reactivity and once in the stratosphere, the Cl and Br atoms are released from the parent compounds by the action of ultraviolet light, e.g.
$$\ce{CFCl3 -&gt;[UV] Cl· + ·CFCl2}$$
Ozone is a highly reactive molecule that easily reduces to the more stable oxygen form by the catalytic action from the halogen radicals. Cl and Br atoms helps in destruction of ozone molecules through a variety of complex cycles. A simple cycle is as follows:
$$\ce{Cl· + O3 → ClO + O2}$$
$$\ce{ClO + O3 → Cl· + 2 O2}$$
Chlorine radical is released and the process is continued and the overall effect result in ozone depletion. Halon gases contains bromine and the process is similar to that of what CFCs does but the action is more potent and is more efficient in this ozone depletion process.

What about fluorine and iodine?

CFCs also contains fluorine atoms which gets released along with chlorine and bromine on photodissociation in the stratosphere. But this fluorine by itself does not contribute to ozone depletion. Fluorine atoms released are quickly sequestered into carbonyl compounds and subsequently into hydrogen fluoride by reacting with water and methane molecules which is very stable in the stratosphere. As such, CFCs are now being replaced with Hydrofluorocarbons (HFCs) according to Montreal protocol (but they do contribute to global warming).
Iodine containing organic molecules react so rapidly in the lower atmosphere that they do not reach the stratosphere in significant quantities and does not play role in ozone depletion.
References

https://en.wikipedia.org/wiki/Ozone_depletion
https://csl.noaa.gov/assessments/ozone/2010/twentyquestions/Q7.pdf
https://hal.archives-ouvertes.fr/hal-00256296/document