Understanding the negative feedback and stablizing of a current limiter
To determine the loopgain properly you would need to make a small-signal equivalent of the circuit. The method of doing that will drag you straight into EE Analog circuit analysis territory. It's too complex to explain that all here so have a look here.

Since I know how to do this already I can tell you that in 1st order (omitting some 2nd order effects) the open loopgain is roughly:

$$G= gm * R1$$

Where

$$gm= 40 * I_{c,q1}$$

So we only need to know Ic,q1.

First determine the voltage across R1: 5V - Vgs,M1 - Vbe,q1 = 5 - 3.8 - 0.47 = 0.73 V 

The current flowing through R1 is 0.73/120k ohm = 6.1 uA

The same current is flowing through Q1 so gm = 40 * 6.1 uA = 240 uA/V

So G = 240 uA/V * 120 kohm = 29

Maybe you wonder why the gain of NMOS M1 is not in this equation, it is because M1 is configured as a common drain or source follower. This means it has a voltage gain of about 1 so I neglected it. Only if M1 was a very small NMOS like an on-chip NMOS would it have such a small W/L that the gain would be significantly smaller than 1. But this is a switching NMOS and these have very large W/L to get the small Rds,on value needed for switching MOSFETs.

Also R2 does not come into the equation as the voltage across it is (in 1st order) set by M1, if R2 had a different value M1 would simply provide more or less current. Again this is a 1st order approach, if you'd calculate in more detail you would find R2 influencing the loopgain. In the 1st order approach, the value of R2 only determines the DC biasing current.

As an (on-chip) circuit designer I would not do this calculation for this type of circuit as it is a well-known "local feedback" construction with a clear dominant pole (at the gate of M1) and therefore it will always be stable and will simply work as long as you dimension it properly such that all the DC voltages in the circuit remain "sensible".