Butterworth filter's gain formula does not agree with R's `signal` package
The function butter() computes the coefficients of a discrete-time (&quot;digital&quot;) Butterworth filter, whereas the gain formula you used is valid for a continuous-time (&quot;analog&quot;) Butterworth filter. According to the R documentation you can use butter() to compute an analog filter using the plane argument.
The gain of a discrete-time Butterworth filter is obtained by the bilinear transform, which substitutes the &quot;analog&quot; frequency variable by $\tan(\omega/2)$, where $\omega$ is the frequency in radians, normalized by the sampling frequency. For a discrete-time unit gain lowpass filter of order $n$, the squared gain is given by
$$G^2(\omega)=\frac{1}{1+\left(\frac{\tan(\omega/2)}{\tan(\omega_c/2)}\right)^{2n}},\qquad |\omega|\le\pi\tag{1}$$