Bathroom fan leaks during heavy rains
I would suspect your flashing around the  wall cap on the outside wall.  Seal it with caulking and then see if it leaks next hard rain.  There has to be a breach of some sort at the wall cap as that is the only portion of your system exposed to the elements.

When it does leak, is it just a hard rain, or one accompanied with high winds too?   Could be the wind is forcing the rain into the wall cap

If it still leaks after caulking and you dont think high wind is a culprit,  you could try replacing the the wall cap.