Using mesh analysis to get the transfer function of electrical system
But as I remember, the both currents I1 and I2 are in the same directions, so it should + instead of - ?


If you look at the total current through the inductor, then you see that \$I_1(s)\$ goes from top to bottom, while \$I_2(s)\$ goes from bottom to top. They go in opposite directions, hence the minus sign.


  And at the end he said that the transfer function is G(s) = I2/V(s). And as I remembered the transfer function should be Vc(s)/V(s).


Transfer functions can be from either voltage or current, to a voltage or current. Any combination is possible, and it just depends on what you want to treat as the input and output. In this case, the author is calculating the transconductance, ie. the output current \$I_2(s)\$ for a given input voltage \$V(S)\$.


  And when we should use the Nodal analysis instead of Mesh analysis ? When there is a lots of calculations?


Both Nodal analysis and Mesh analysis can be used to solve any circuit. But you usually can save yourself some work if you choose the method appropriately.

Generally:


Mesh analysis uses the KVL equations, ie. it works with voltages. It is easier to use when there are ideal voltage sources in the circuit.
Nodal analysis uses the KCL equations, meaning currents. If you have ideal current sources in your circuit, then this is probably easier.
If you have both ideal voltage and current sources, then you will need to apply Thevenin/Norton equivalents, or you will need to use source-shifting. Or you can also use Modified Nodal analysis which is the method used in simulators.
If you have many components/nodes, but only few loops, mesh analysis will lead to fewer equations, which is easier to solve.
If you have a highly branching circuit (ie. lots of loops and not as many nodes), then Nodal analysis is preferred as it will lead to less equations.




Appendix

The Thevenin and Norton equivalents are shown in this figure:



They are exactly equivalent if the following relationship holds:

$$V_{Th} = R_N\cdot I_{N}$$

$$R_N = R_{Th}$$

Just make sure you can draw a circle around the structure, and that there are only two connections leaving/entering that circle. If that is not the case, then you can't replace them by one another.

So you can use this to turn voltage sources into current sources and vice-versa.

The equivalents can be used more generally, but this is how you'd use it for Nodal analysis or Mesh current analysis.