Installed a dimmer and now it's controlling everything
How about we start by marking some wires with colored tape.


Wires that are supposed to be hot all the time get marked black. 
Wires associated with the fan hot (that we switch) get red. 
Wires associated with the lamp hot (that is dimmed) get blue. 


Mark the appropriate hot wires amongst those five cables.  

On the dimmer, you mark the supply wire black (is probably already that way) and mark the load wire blue (since it goes to the lamp) (it might supply as red, but we mustn't connect it to the fan!) 

On the plain fan switch, you mark one terminal as black and the other as red.  

Now, hook like colors together. 

I don't understand how to hook colors together if the switch doesn't provide an on-board splice point for that many wires.

Then you use "Wire Nuts" generally, or you can use Wago style "lever connectors".   You probably want a red wire nut to connect as many blacks as you'll have to connect. 

But one of the blacks is not a wire.  It's a switch terminal screw.

You need to use a technique called a pigtail. For this you need a few scraps of wire.  Buy 1 foot of 12/3 Romex, and cut it into two 6" sections and then peel the sheath off of it.  That'll give you a supply of pigtails in a variety of colors. 

I use pigtails pretty much by default.  If I know my switch needs to go between black and red, I'll stick a black and red pigtail on the switch. This often makes for some redundant connections, but I don't care, and they work fine.