Do we ever bypass the noisy component rather than the control ICs?
I'll use this circuit as an example:



First, consider the current loops:


FET ON: Supply - Motor - Q1 - Ground - Supply
FET OFF: Motor - Diode


This will draw a square wave current from the supply, with relatively high di/dt edges depending on switching speed. 

di/dt combines with supply and ground inductance to create noise.

Placing a decoupling capacitor very close to this, between the MOSFET source (at ground) and the load's supply (cathode of FD1) provides a low inductance path for the high di/dt currents. These currents will naturally flow in the low inductance path, ie in the local decoupling cap, thus the supply will be less affected by HF noise. This is an example of treating the noise at the source, it is effective.

I mentioned current loops because you want the area of the loop which carries fast di/dt currents to be small, so it doesn't act as a loop antenna. Placing the cap close also helps with this.

Switching as fast as necessary, but no faster, is always a good idea.

As for the cap, in this example its ripple current is the full motor current, so it should be low-ESR. One or several ceramics (MLCCs) are ideal for this. Since ESR goes down with increasing capacitance, don't skimp on capacitance.