Result of inverse FFT is sometimes shifted in real space
If you have a filter in the time/space domain, the location that is associated with zero linear phase in the frequency domain is the top left pixel of the filter. If the filter is centered in time/space, then you need to ifftshift prior to taking the FFT in order to remove the linear phase that centers it in the time domain. The following example Octave code (sorry, I don't use python) illustrates this:

x = zeros(10,10);   % Make an impulse in time, so that its 
x(6,6) = 1.0;       % magnitude in frequency is all ones.

% Display the angle of the FFT both with
% and without the time-domain quad-swapping.
figure;
imagesc( angle( fft2(x) ) );
title( 'Phase Without Quad-Swapping In Time' );

figure;
imagesc( angle( fft2( ifftshift(x) ) ) );
title( 'Phase With Quad-Swapping In Time' );


The same thing is true in the frequency domain, which is what is happening with your filter. Specifically, you have defined the magnitude of the filter in the frequency domain, but you have not given it the phase necessary to center it in the time/space domain.

I tried to answer a related question here.