FFT on non-rectangular part of image
Instead of having a hard edge between the image data of interest and the background, you could use a two-dimensional tapered window function, as is often done in spectral analysis. You might start by trying a Gaussian window, which for a two-dimensional case would look something like:

$$
w[x,y] = e^{-\frac{\left(x-\frac{N_x-1}{2}\right)^2}{2\left(\sigma_x \frac{N_x-1}{2}\right)^2}} e^{-\frac{\left(y-\frac{N_y-1}{2}\right)^2}{2\left(\sigma_y \frac{N_y-1}{2}\right)^2}}
$$

$N_x$ and $N_y$ are the dimensions of the desired transform in the $x$ and $y$ directions, respectively, and $\sigma_x$ and $\sigma_y$ are parameters that allow you to control the shape of the window; for small $\sigma$ values, most of the energy in the window function will be concentrated toward the center, with that effect decreasing as you increase $\sigma$.