Why is a circular mask appropriate for Fourier filtering rectangular images?
For simplicity, let's not do any shifting and only consider non-negative frequencies.
Let's assume that the horizontal and vertical image dimensions are even integers $W$ and $H$. Looking at the output of a $H\times W$ 2-d DFT of the image, the $u$th column with $u\le W/2$ represents a horizontal frequency of $u/W$ times the horizontal sampling frequency and the $v$th row with $v\le H/2$ represents a vertical frequency of $v/H$ times the vertical sampling frequency. For a square image grid the horizontal and vertical sampling frequencies are equal and in the following denoted by a single variable $f_s$. The frequency-magnitude of a 2-d frequency at bin $u, v$ will then be $\sqrt{(u/W)^2 + (v/H)^2}f_s$.
For a cut-off frequency $f_c$ your mask would select frequencies:
$$\sqrt{\left(\frac{u}{W}\right)^2 + \left(\frac{v}{H}\right)^2}f_s &lt; f_c\tag{1}$$
$$\Rightarrow\frac{f_s^2}{W^2f_c^2}u^2 + \frac{f_s^2}{H^2f_c^2}v^2 &lt; 1.\tag{2}$$
That indeed defines an ellipse in coordinates $u, v$.
However, if you consider the actual frequencies $\frac{u}{W}f_s$, $\frac{v}{H}f_s$ as coordinates, then what you have is a circle:
$$\text{Eq. 1}$$
$$\Rightarrow\frac{1}{f_c^2}\left(\frac{u}{W}f_s\right)^2 + \frac{1}{f_c^2}\left(\frac{v}{H}f_s\right)^2 &lt; 1.\tag{3}$$
To summarize, it depends on how you express your frequencies.