When does the NMOS form the channel? VGS > VTH or VGB > VTH?
When does the NMOS form the channel? 


An n-channel mosfet forms channel when 
$$V_{GS} &gt; V_{TS}\tag1$$ or when $$V_{GB} &gt; V_{TS} + V_{SB}\tag2$$
(Both conditions are actually same)

Where \$V_{TS}\$ is the threshold voltage with source as reference terminal and is given by:
$$V_{TS} = V_{T0} + \gamma(\sqrt{\phi_t+V_{SB}}-\sqrt{\phi_t})$$

\$\gamma\$ and \$\phi_t\$ are device parameters (see body effect). \$V_{T0}\$ is the threshold voltage with source and bulk shorted (\$V_{SB}=0\$). The threshold voltage given in datasheet is \$V_{T0}\$ . 

The point I want to make is that you can not say that a mosfet will form channel if either \$V_{GS}\$ is greater than a particular voltage or \$V_{GB}\$ is greater than a particular voltage without the knowing the voltage between source and bulk, \$V_{SB}\$. 

For example, for an n-MOS of threshold voltage 3V, \$V_{GB} = 4V\$ does not ensure channel formation if source terminal is at a higher potential than bulk (or body) terminal.


  With respect to which reference terminal threshold voltage is defined?


One can actually defined two threshold voltages- \$V_{TS}\$ with respect to source and \$V_{TB} (= V_{TS} + V_{SB})\$ with respect to bulk. Text books and articles, especially circuit related, usually defines it with respect to source terminal.