How hot should wires get?
How hot should wires get?


How how wires should get depends on application, but there are benefits to keeping just about any device other than a heater cool.  The question ends up being more "How much weight and expense is it worth adding to decrease wasted power?" in the end.


  I am using 10 AWG wire to power a BLDC motor, and these wires can get quite hot.


If wires are getting "quite hot" you're wasting a significant amount of electricity.  Specifically, wires get hot because of \$I^2R\$ losses, which is to say the power the wire uses to produce heat, \$P\$, is equal to current squared(\$I^2\$) times resistance (\$R\$).  If you can measure or calculate current, you can choose a wire size with appropriately low losses.  You can also measure the existing wire and calculate its resistance if you don't have a meter.  As a general rule, going down 3 AWG sizes doubles wire size and halves resistance.   AWG sizes step mostly in increments of 2, but that's beside the point.  The calculation of resistance based on cross sectional area measured or from a chart is easy as well.  In AC systems, inductive loads, such as motors produce current out of phase with the voltage (reactive current) and in some cases (probably not yours) it is worth using power factor correction to reduce reactive current on transmission lines.  


  When an IC is intended for high power uses, it is common practice to put a heat sink on it, but what about for wires? 


In almost every case short of superconductors it is more practical to use a larger wire and waste less energy as heat than add weight in the form of a cooling system.  When space is an object, though, such as in a motor or electromagnet, microprocessor, it becomes more likely that a cooling system might be used, even if what's producing the heat are conductors.


  Should I simply use a thicker gauge wire?


Yeppers.  If you calculate the current, take into account if they are in free air or a closed space and look at the appropriate DC wire ampacity chart, you can round the current up to the nearest size, apply a derating factor if necessary (a multiplier to decrease the acceptable current for a wire based on factors like ambient temperature) and choose the recommended size of wire for your current.  If you have a substantial length of wire you may wish to take voltage drop into account as well.  

If you don't have means to measure or calculate the current, you can at the very least massively improve your setup by embiggening or doubling up the wire until it no longer becomes warm.