what is the theory behind matlab butter function?
The MATLAB function that you refer to uses this knowledge to calculate the coefficients

Briefly, the function you refer to goes something like this:


The transfer functions for analog Butterworth filters of any order are already known analytically. It selects the normalized transfer function polynomial for the order you select.
The normalized polynomial is scaled appropriately according to the desired cutoff frequency, giving an $s$-domain transfer function that has the response that you want.
The $s$-domain prototype filter is then mapped to a discrete-time approximation using the bilinear transform. This is the most commonly-used method of mapping a continuous-time filter to a discrete-time filter.
The result of the bilinear transform is a transfer function in the $z$-domain, from which the filter coefficients are easily found by inspection.