Sub pixel shift causes weird results
The solution lies in the use of the x's. You need to convert the x's you use to lower frequencies, which can be either positive or negative. So only use x's between -N/2 and N/2 and the problem is solved.

def convertfreqs(x):
    return np.mod((x + N/2),N) - N/2


This works because it is the higher frequencies which cause the signal to change heavily inbetween pixels. Higher frequencies means higher x's.