How to wire switch to ceiling fan
Alright so we know that whatever you have in place is not following a normal wiring pattern and right know it's a real mess (example brown is never the ground). But if you are able to identify the conductors at the switch box with the conductors at the fan box. Here is generic fan diagram that might help.


This should get you started in the right direction.

Good luck.