LED downlights working in conjunction with bathroom fan
I think you may have a wiring problem. From your description you circuit is acting like you have hooked your fan up in series with the lights. In a series circuit the lights will all stay the same brightness if they are all the same wattage and type. Once the fan comes on part of the voltage is dropping through the motor and the rest is distributed through the lights.

Take a close look on how the fan and the lights are wired up and if you have a meter verify that all equipment is wired in parallel and check for any crossover wiring.

I am only using the information you have given us. If the problem persists, you might edit your question and give us some photos or diagrams.

Good luck.