Fourier transform possible on non-rectangular part of an image
Even when the image part is rectangular, the FFT result will be contaminated by boundary effects. The FFT of the part is the Fourier transform of an infinite tiling of the rectangular part. (An elliptical or circular part cannot be tiled. A hexagonal tiling would get pretty close though.) If there is a discontinuity going from one tile to another, this will create high-frequency components that are not descriptive of the insides of the part. Randomizing the phases will spatially translate these components so that they appear also inside the part, which is probably not what you want. Mirroring at the boundaries does not eliminate discontinuities of the derivative, and requires doubling of the FFT dimensions.

This is where a smoothly tapering 2-d window function can help. For a circular part you might like the Tukey window with the distance from the part center as its argument.



For an ellipse, stretch the window function accordingly. Multiply the data by the window function to select the area you want to analyze, and FFT the result. Then do the phase randomization. Mask the noise by multiplying it with an ellipitical disc-like mask that has value 0 outside the disc and value 1 inside the disc. You can also use the window function as the mask to get a fuzzy boundary. I did not find yet a perfect way to normalize the noise, but matching the mean and the standard deviation both weighted by the mask works quite OK:



Multiply the input image by the complementary mask: 1 - mask, and sum the two together. Octave source:

pkg load signal;
r = 120; # Circle radius
n = 512; # Image width and height
tukeyr = 0.5; # Window transition relative size
lena = double(imread("lena.gif"))/255;
tukeyn = round(r/(1-tukeyr));
x = repmat([1:n], n, 1);
y = repmat([1:n]', 1, n);
dist = sqrt((x-n/2).^2 + (y-n/2).^2);
tukey = tukeywin(tukeyn*2, tukeyr)(tukeyn+1:tukeyn*2);
win = reshape(interp1(1:tukeyn, tukey, reshape(dist, n*n, 1), 0), n, n);
randphase = 2*pi*rand(n, n);
noisefft = abs(fft2(lena.*win)).*exp(j*randphase);
noise = real(ifft2(noisefft));

#Circular mask
weight = r-dist;
weight = (weight &gt;= 1)*1 + (weight &gt; 0 &amp; weight &lt; 1).*weight;
#Or use window as mask
#weight = win;

weightsum = sum(sum(weight));
noiseweightedmean = sum(sum(noise.*weight))/weightsum;
noiseweightedstdev = sqrt(sum(sum((noise-noiseweightedmean).^2.*weight))/weightsum);
lenaweightedmean = sum(sum(lena.*weight))/weightsum;
lenaweightedstdev = sqrt(sum(sum((lena-lenaweightedmean).^2.*weight))/weightsum);
normalizednoise = ((noise - noiseweightedmean) / noiseweightedstdev * lenaweightedstdev) + lenaweightedmean;
imshow(lena.*(1-weight).+normalizednoise.*weight);


P.S. I found that highpass filtering the copy of the Lena image used for noise generation, by setting some of its lowest-frequency bins to zero, will reduce the error in the magnitudes of the fft of the final image compared to fft of the source image. I think the reason is that low-frequency amplitude fluctuations will be reduced. Within the circle (ellipse) those appear in practice as zero frequency, and in their absence less normalization is needed. So that's one thing to investigate.