Why is this switching transistor heating up?
This does not work as a switch. Instead, forms a current source or a linear voltage regulator (You can explain how by yourself, but detailed explanation is below). According to the notes on the schematic, the voltage across D and S of Q2 will be 12-3.0 = 9V at worst. If you multiply this with load (fan) current then you'll find the power dissipated (\$P_D\$) by Q2. Multiply \$P_D\$ with \$R_{th j-a}\$ of AO3401 which is given as min. 100 in the datasheet and you'll find temperature rise. This may explain the excessive heat. You can verify this by applying maximum control input voltage (1.2V) and seeing that Q2 does not heat up.

Now let me explain how this works as a linear regulator (According to the schematic in your question):

1) At the time of energizing the circuit, (assuming control input is 0) the output of comparator will be 0 due to pull-down resistor (R2). So, comparator output is low --> Q1 is off --> Q2 is off --> No load current/voltage --> Voltage across R2 is zero --> Output remains low.

2) When the control voltage is applied, comparator will attempt increasing its output to 12V. When this output voltage approaches/reaches Vbe threshold of Q1 (neglecting 100R tied to emitter), CE resistance of Q1 starts to decrease. Thus, G-S voltage (so, DS resistance) of Q2 starts to decrease, leading load current (so, load voltage) to increase. 

3) This load voltage is divided by 1+(R8+R3)/R2=1+90k/10k=10 and fed back to negative input of comparator (\$V_{in-} = V_L/10\$). When this FB voltage (i.e. voltage across R2) reaches and exceeds the voltage on the positive input terminal (i.e. control voltage), the comparator attempts decreasing its output to 0.

4) Comparator output starts to decrease, so Vbe of Q1 starts to decrease, leading CE resistance to increase and forces Q2 to increase its DS resistance. This results in decreasing load current (and so, load voltage). This voltage is divided by 10 and fed back to negative input of comparator (\$V_{in-} = V_L/10\$). 

5) Now the voltage at negative input is lower than positive input, so the comparator will attempt increasing its output to 12V. Output start to increase and the cycle begins afresh from (2).

Consequently, the voltage across the load will be 10 times control voltage: \$V_L = V_{ctrl} \cdot 10\$ and the voltage across MOSFET is \$V_{DS} = 12V - V_L\$. We don't have any information about your load, so it's quite hard to guess how much the load current is. Anyway, the power dissipated by MOSFET will be \$P_D = V_{DS} \cdot I_L\$. 

I made a simulation on Proteus 7. You can download from here and here's a screen shot: 



(I used LMV393 because LM393 is not defined in Proteus, but LMV393 is the low-voltage version of LM393).

Let's assume your fan current \$I_L=50mA\$ @ \$V_L=5V\$. So MOSFET's power dissipation will be \$P_D = 7V \cdot 0.05 = 0.35W\$. Multiplying this with \$R_{th(j-a)}=100\$ will give a temperature rise of \$\Delta T = 100°C/W \cdot 0.35W = 35°C\$. Assuming ambient temperature is 24°C, so MOSFET's final temperature will be 24+35=59°C.

Hope this explanation is enough and useful for you.