How is the formula of the cutoff frequency of a 2nd order Butterworth low-pass filter derived?
First you need to find (calculate or look up) the transfer function of that circuit. It has the form

$$H(s)=\frac{a}{s^2+bs+c}\tag{1}$$

where the constants $a$, $b$, and $c$ depend on the values of the resistors and capacitors.

The $3$ dB cut-off frequency $\omega_c$ can be found by solving

$$\big|H(j\omega_c)\big|^2=\frac12\big|H(0)\big|^2=\frac12\left(\frac{a}{c}\right)^2\tag{2}$$

(because $H(0)=\frac{a}{c}$).

Solving $(2)$ involves solving a quadratic equation in $\omega_c^2$.